Data Quality Part 4: The Qualitative Components: Representativeness and Comparability

Posted: July 6th, 2023

Authors: Gene Y. Aditya S.

We started this journey with the following thought: Defining data quality and implementing a data quality program furthers the goal that the data collected serve the intended purpose, i.e., informed decision making. Up to this point, we’ve defined data quality, listed the components, and discussed and defined precision, accuracy, and completeness. In this article, we’ll talk about representativeness and comparability. These concepts reflect another facet of the data set. For these two components, it’s not about how “good” the data are, it’s about how and when the data were collected, and how the data can and cannot (should and should not) be used for decision-making.

I thought these would be easy, until I realized that U.S. EPA defined representativeness using the word “represent” and comparability using the word “compare.” But we shall soldier on.

Representativeness is a measure or description that the measurement reflects the issue that is being investigated. I’m sorry, what? More analogies to help explain:

I have three cars in my garage, and I go out and I measure the air pressure on all 15 tires (each car has 4 tires and a spare). But the question is: Can I safely drive my old pickup truck? (Yeah, I know that air pressure is not the only factor relative to safely driving that 1997 Nissan.) Back to our data set, the only data that is germane to my decision-making is the pressure in the four tires mounted on the truck. These data directly address the question being investigated. The other data, while they speak to the shape of my fleet, or to my general maintenance activities, do not address my question of interest. The pressure in the four tires mounted on the truck are representative of the physical state I care about (safe driving of my pickup).

Let’s explore the bathroom scale analogy. Why am I weighing myself? Maybe I’m trying to lose (or gain) weight. Maybe I’m using it to monitor my health. In either case, I’m developing a data set, and I want to look at the whole data set. I want each number to reflect the same physical status (e.g., unclothed, first thing in the morning). I can’t just willy-nilly jump on the scale any time I want, wearing anything I want, and expect that the data will tell an understandable story. Each weighing result must represent the same physical state, as a function of time.

Comparability is a measure or description of whether two “representative” pieces of data can be used for decision-making. Again…huh? Let’s try the analogies again.

I still have three cars in my garage, and after measuring the pressure on all 15 tires (4 tires and 1 spare per car), I find out that two of those tires are low. I fire up the ol’ compressor (and put on my safety glasses and earplugs) and fill the tires. The next day, I go out and check the pressure again. I grab a pressure gauge and check those two and they’re fine. Good to go! You’re thinking, but Gene, maybe you haven’t waited long enough. You’re right, so I do it every day for the next week. Now I find that the pressure is all over the map. What happened? I have a bunch of pressure gauges in my garage. And I’m not sure when or where I got them, for sure none of them are calibrated or documented or certified or what have you. The data from this series of undocumented measuring tools is not comparable. Of course, if we are looking for a correct (accurate) measure of pressure, we better know that the tool we are using is capable of the accuracy we are looking for. If we’re looking for a trend (does the tire hold air?), the absolute value is less critical, but repeatability (precision) is more critical. (We’ll get back to bringing this all together in a later article.)

For the bathroom scale analogy, it’s much the same. Have you ever noticed that you weigh a different amount at the doctor’s office than at home? Even if you correct for time of day and amount of clothing? It’s the comparability issue. The two scales give different results, and the data are not comparable. The equipment is different, the quality specifications for the equipment are different. There are likely different calibration procedures, calibration standards, and even maintenance practices.

Hopefully you’ve noticed by this point that I don’t have a “representativeness” test or a “comparability sample.” As noted in the title of this article, these data quality components are “qualitative.” They require all (or maybe even more) of the following procedural and planning tools:

Thoughtful experimental design. We need to define the question being asked, and the physical state of the item being measured. This may be operational specifications, or time of day.
Structured methodology. We need defined measurement procedures and of course, those procedures must be followed. There needs to be some kind of documentation and training to improve the likelihood that the appropriate steps are followed each time.
Again, we need procedures and defined appropriate standards. We also need independent standards to check the calibration.

Next Time: Measurement Sensitivity.

Until then, feel free to contact either of us: