What criteria would use to rate each supplier?

As a new Logistics/Supply Chain Manager for a national Heating, Ventilation, and Air Conditioning (HVAC) distributor. One of your first tasks is to review the list of suppliers and develop a Supplier Evaluation for the purpose of classifying vendors based on the selected criteria that you feel impacts your business. You start with a focus on steel piping and fittings purchased by five (5) vendors. (Use Vendor #1 - Vendor #5 as vendor names.). These vendors are the only suppliers of steel piping and fittings used by this distributor.
REQUIREMENT 1
1) What criteria would use to rate each supplier? (i.e one may be Cost.)
2) How would you evaluate each supplier (i.e letter grade, number)? What would each mean in terms of Excellent through Poor?
3) How would you differentiate an "A" supplier from a "B" supplier or "C" supplier?
4) Suppose one of your current suppliers is habitually late with deliveries, but it very good with regard to the other criteria you require. How would you handle this vendor?
5) Would you recommend terminating a supplier/vendor due to the criteria you selected? Is not, what would you do to bring them up to your expectations?
REQUIREMENT 2
Prepare a preliminary review for the Director of Logistics/Supply Chain Operations with a comparison of each of your 5 vendors and your recommendation regarding action you planned to take with each vendor.