Realist Evaluation
[Updated 2018 10 07] A core concept in Realist Evaluation is the CMO configuration. CMO stands for Context, Mechanism, Outcome. [The original text worth reading is Pawson and Tilley (1997) Realist Evaluation]
Context describes what are thought to be important features of the setting in which things may happen. This may include a description of an intervention taking place in that setting.
Mechanism is a causal process happening within individual people or within organisations (depending on the scale of analysis) that is triggered by the particular set of context conditions. It leads to an…
Outcome, which is the result of the interaction between Context and Mechanism
As in QCA there can be many different CMO configurations that can be at work in a given programme, policy or project. Each of these, as initially imagined, is a theory that needs to be tested.
One of my interests is how to do this. Especially if there are a lot of potential CMOs to consider, and not much time (if you are an evaluation team)
My conjecture is that you need a multi-stage process:
- Find out what happened.
- What outcomes occurred. Prior theory should help
- In what different kinds of contexts did they occur and not occur. Prior theory should help
- Find out what conjectured Context and Outcome events were actually co-occurring in reality. Use QCA or EvalC3 software to find this out. This is cross-case analysis.
- Focus in only on those CMOs where there is an association. In this order of priority:
- Where the context conditions are necessary and sufficient for the outcome
- Where the context conditions are necessary or sufficient for the outcome
- Then invest resources in within-case analyses to find out if the conjectured mechanism is at work, or if some other mechanism is in operation, or if there is none at all.
This conjecture relates to a discussion of the sequencing of realist evaluation and related methods, on a recent ITAD blog posting
Which brings us to Process Tracing…
Process Tracing
This is a method of within-case analysis to identify causal mechanisms at work within individual cases. [Look here for some references, there is a lot written on the subject] It is one way of investigating the Mechanism element of a CMO, and how it causally connects Context with Outcome. Key items of evidence, in a plausible story of how things happened, fall into four categories:
- Hoop tests: A person guilty of a murder will not have an alibi. If they do the theory that they committed the crime fails. Not having an alibi is a necessary part of the causal process, and the basis for prosecution. But it is not sufficient.
- Smoking gun test: Being in possession of a gun at the scene of a murder is sufficient to enable the murder, but it is not necessary.
- Doubly decisive test: Being in possession with a gun at the scene of the murder and the dead person’s body containing a bullet of the same character (bore marks) as those in the gun, is both necessary and sufficient
- Straw-in-the-Wind test: This is evidence that is relevant but neither necessary nor sufficient
How does this relate to EvalC3? The EvalC3 workflow ends with case selection and within-case analysis. One important form of within-case analysis is to examine True Positive cases, comparing modal and outlier cases. With each of these types of cases, it would be useful to do some form of process tracing, exploring the expected causal connection between the model attributes and the outcome. This is where the above four tests could be used.