In brief

Good measurement is the foundation of good science, but making measurements can be difficult. This might be for practical or technical reasons, but it can also be hard to know that we are measuring the right things, in the right way. Good measurements are underpinned by strong theories, are valid, minimise error and avoid bias.

A measurement is not the same as an observation. If we observe someone weeping then we might infer that they are sad; but an observation of weeping cannot itself a measure of how sad a person is. This is because emotions (like other psychological notions, such as motivation, or personality) are ‘hypothetical constructs’: We don’t see them directly — only via the many effects they have on the world. To convert observations of weeping to measurements of sadness we need to have a theory about how sadness relates to observable events.

This theory isn’t tested by the measurement, and reasonable people may have different theories about how constructs relate to observable events. I might claim, for example, that sadness primarily causes ice cream consumption rather than weeping. This can lead to arguments about the construct validity of a measure. That is to say, do our observations really measure what the things we wish them to? A common strategy to improve construct validity is to introduce diversity. If we make many different but related measurements we can capture different aspects of the same construct.

Because any single observation can have multiple causes — weeping can be caused by onions as well as sadness — using observations to make measurements creates measurement error. To reduce this kind of error, the main strategy is repetition. Repeated observations help by “averaging out” other causes, giving us a clearer picture of the construct of interest. For example, if we observe someone once—perhaps while they are chopping onions— then we might mistakenly think they are sad. If we observe them many times over the day then we are less likely to be misled in this way. Diversity can help here too because different observations may not share the ‘other’ causes, and so each can provide an independent signal.

Biases are a special kind of measurement error that can reduce validity (and where repetition can actually make things worse). For example, when answering questionnaires people can give misleading answers, perhaps to please the experimenter or make themselves look better. It’s not possible to avoid all bias, but it’s important to consider biases when interpreting our measurements.

How relaxed are you?

When we measure things like height, weight or speed the measurement itself seems easy — we use a tape measure or scales.

But (perhaps especially) in psychology, just making measurements can be a big part of the challenge. The difficulties can be both practical and conceptual, and we need to be mindful to maintain validity and minimise error and bias.

Measurement model

Self report scales

Cartoon: XKCD

Self report scales have many flaws but are often the least-worst option! They are cheap, simple to implement, and may be the only way to access some psychological constructs.

Good questionnaire items take time to construct, but should be simple, use familiar language, have face validity for the construct measured, be ‘unbiased’ and not be ‘leading’. Using a common structure is often advisable.

Taken at face value, self-report scales are deceptively simple: We ask participants questions and assume that their responses are ‘caused’ by the variable of interest.

There are lots of possible pitfalls with self report measures, not all of which can be avoided.

Research has shown that participants can give biased answers for many different reasons, and may lack any true insight on which to base their answer. For a quick overview see this helpful article but also Nisbett and Wilson (1977) for a classic illustration.

But psychologists persist with self report because it is cheap, simple, and because it may be hard to measure the constructs we are interested in any other way. Pain is a good example of this; see Chapman et al. (1985), and Katz and Melzack (1999) for a discussion.

‘Think aloud’

Before using new self-report measures it’s important to pilot your questions with volunteers to check that they can be understood. Some questions can be confusing to participants. Examples of bad questions would include double-negatives, or double-barrelled questions (“How enthusiastic and intelligent is your lecturer?”, for example; see Hinkin 1998 for other examples).

TARA-Led Workshop Tasks

Other important tasks to complete

See the Discourse post here and follow the instructions closely.

How does this help with the assessment? Learning how to collect data is an important component of the module; without some data you will find it hard to complete the next few workshop tasks. We will select examples of different groups’ questionnaires and use them in the subsequent teaching sessions.