…tools for exploring and evaluating complex causal configurations
EvalC3 is an Excel application designed for use in the monitoring and evaluation of the achievements of development aid projects (….but has wider applicability).
EvalC3 is a means of doing comparative configurational analysis using search algorithms and manual hypothesis testing
Its purpose is to enable users: (a) to identify sets of attributes that describe a project intervention & its context, and which are good predictors of the achievement of an outcome of interest, (b) to compare and evaluate the performance of these predictive models, and (c) to identify relevant cases for follow-up within-case investigations to uncover any causal mechanisms at work.
Examples of four different kinds of uses are described in Example Uses
These predictions are based on the screening of a data set that (ideally) describes the attributes of a set of those projects, their context and their outcomes. While it involves systematic quantitative cross-case comparisons, its use should be informed by within-case knowledge at both the pre-analysis planning and post-analysis interpretation stages.
The overall approach is based on the view that “association is a necessary but insufficient basis for a strong claim about causation”, which is a more useful perspective than simply saying “correlation does not equal causation”.
Influences: The design of this Excel package has been influenced by exposure to: (a) Qualitative Comparative Analysis (courtesy Barbara Befani), (b) RapidMiner open source predictive analytics software, (c) Goertz and Mahoney’s (2012) A Tale of Two Cultures” . Look here for relevant references
Four main tools are available to develop these predictions:
- Manual hypothesis-led inquiry, used to explore the predictive power of specific attributes of prior interest. Suitable for data sets of any size. Ideally the first step in the process of analysis using EvalC3
- Algorithm based searches
- Exhaustive searches of two types: (a) single attributes search – which is quick, and (b) multiple (combinations of) attribute search – which can be much slower if many attributes or cases are involved. Preferably the second step to take, to explore “outside the box” of our existing expectations
- Decision Tree searches. This is a type of reiterated and cumulative single attribute search, for a set of configurations best able to predict all outcomes
- Evolutionary search of combinations of attributes using a genetic algorithm. Especially ( but not only) suitable for larger data sets. Also helping us to explore “outside of the box, but on a larger scale.
The results are generated instantaneously in the case of manual hypothesis testing, quickly with evolutionary and Decision Tree searches and some times much longer with exhaustive searches for combinations of attributes.
A range of performance measures: The results of each search is a predictive model, which describes a sub-set of attributes that is consistently associated with a specific kind of outcome. The number of the cases identified (and missed) by a predictive models is summarised in the form of a truth table, commonly known as a Confusion Matrix. This table is then used to generate a range of measures of the performance of a given model, which are suitable for use in different contexts.
There is also a model store, where results of any previously model can be accessed: (a) to compare against the design and performance of the current model, and (b) reloaded for further exploration.
Supporting tools: The EvalC3 application also two supporting tools:
- Post cross-case analysis: A measure of project similarity which enables identification of cases most suitable for subsequent within-case investigation in order to identify the nature of any common causal mechanism underling the project attributes that have been found to be good predictors of outcomes
- Pre cross-case analysis: Two measures describing the whole data set.
- Diversity: The percentage of all possible configurations of the current set of attributes that are present in the data set. The higher the percentage the less likely a current model will be contradicted by new data
- Consistency: The proportion of all the configurations that have consistent outcomes e.g. all present or all absent. Higher levels of consistency will mean models that are found are less likely to have False Positive cases that will require additional attributes to explain their existence.
Analysis of “effects of a cause”: The default setting for EvalC3 is to analyse “causes of an effect” where multiple project attributes may be contributing to an outcome of interest.
However EvalC3 can also analyse “effects of a cause”, where a particular project intervention (described by a specific attribute in a data set) may be contributing to multiple outcomes.
Triangulation: Data that has been analysed using Qualitative Comparative Analysis(QCA) or Decision Tree algorithms can also be imported and analysed using EvalC3 tools. See the Data Sets page for examples that can be experimented with.
Predictive models first developed by EvalC3 can also be triangulated by later re-analysis using Qualitative Comparative Analysis(QCA) or Decision Tree algorithm
Origins: The original Excel application was designed in 2015 by Rick Davies, who is now working with Aptivate to develop the current more user-friendly and robust Excel version. This is being done with two purposes in mind: (a) To widen the range of tools available to identify and analyse complex causal configurations, (b) To widen the use of such tools, among the global community of evaluators.
A pdf version of this page is available here: EvalC3 _ tools for exploring and evaluating complex causal configurations
EvalC3 by Rick Davies is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Based on a work at https://ec3site.wordpress.com/.