Valuable evaluation metrics

by Dr. Robert A. Kemp

This article identifies and examines some of the metrics available to enhance MRO supply operations. Our responsibilities as supply pros are to search for, identify and develop methods that improve our performance from several critical perspectives. Regardless of the model we use for the evaluation, the metrics are critical perspectives in the supplier evaluation process.

From literature and experience, we can develop long lists of factors or criteria to support our supplier evaluation processes. I think that we must recognize that we shouldn’t use the same list of metrics for a “C” supplier as we do for the more important “A.”  Indeed, you probably shouldn’t use the same evaluation model. Found on Page 29 is a list of usable factors or criteria. It’s safe to say that every list I’ve ever used or seen included quality, delivery and price. We now call price “Total all in cost.”  After those three, the lists usually diverge significantly, and this is acceptable because the factors should fit our individual needs.

Broadly speaking, we have two kinds of metrics: qualitative and quantitative. Qualitative metrics are just words — acceptable or unacceptable, yes or no, plus or minus, or similar “either/or” situations. Quantitative metrics are numbers. We define the numbers by their ability to support analytical statistical tools. We generally name three categories of tests: nominal, interval and ratio. The names define their elegance and power.

If you recall from articles 2 and 3 of this series, we identified four concepts for models:
1) the Categorical Model,
2) the Weighted-Point Model,
3) the Cost Ratio Model, and
4) Web-Based Models.

Nominal data are just counts of factors or things. The Categorical model for supplier evaluation uses nominal data collected by qualitative processes. Our analyses are limited to counts of the words. Supplier C is better because it had a better score. We cannot speak to range or the significance of being “better.”

Ordinal or interval data show us a rank order for the factor evaluated. For example, a 1 to 9 scale, with 1 being “not important” and 9 being “very important.”  With this data, we can run sophisticated statistical analyses and make conclusions comparing categories of factors and suppliers. Typically, we use the Likert Scale to collect this type of data. Here is an example of this scale designed to measure importance or degree of feeling. It shows nine points for differentiation.   

    |          |          |          |         |          |          |          |          |
    -------------------------------------------------------------------------------------
    1         2         3         4        5         6         7         8         9
not important              average importance             very important

The Likert Scale can take many forms (for example, 1 to 5, or 1 to 7). It’s important, though, that we always see the intervals as equally spaced, finite, and clearly that 5 is five times greater than 1.

The Weighted Point and Cost Ratio models use both nominal and interval data and can generate ratio data for sound analyses. Web-Based models can use all data types. Correct use of the proper metric and correct data analysis processes ensure our ability to draw sound conclusions concerning comparisons between suppliers, calculating and using benchmarks, and searching for alternative solutions.

Robert Kemp is a consultant, speaker and the former president of the Institute for Supply Management. He can be reached at kempr@mchsi.com.

This article appeared in the Oct./Nov. 2002 issue of MRO Today magazine. Copyright 2002.

Back to top                                Back to MRO Coach archives