What is special about EvalC3
- EvalC3 combines both manual hypothesis testing and algorithmic search
- EvalC3 provides a range of search algorithms
- EvalC3 provides a range of model performance measures
- EvalC3 provides case selection tools not available elsewhere
- EvalC3 is tolerant of missing data
- EvalC3 is available in user-friendly Excel
EvalC3 has similarities and differences with two other methods of analysis:
- Qualitative Comparative Analysis (QCA)
- Decision Tree learning, as available in software packages like Rapid Miner
The similarities
- All three can work with binary data, which is more widely available than numerical data […because all numeric data can be converted into binary data]
- All three analyse relationships between cases in terms of how their sets of attributes overlap, or not
- The results of all three analyses can be compared using some common performance measures
- Their findings can all be interpreted within the same view of causality, known as multiple conjunctural causation
The differences
- Missing data:
- EvalC3 is tolerant of missing data points, more so than QCA.
- Decision Tree software in packages like Rapid Miner also seem to be more tolerant than QCA, and it is currently more transparent than EvalC3 in the way it deals with missing data.
- Minimisation / Search: (i.e. finding a configuration that fits many cases)
- Exhaustive search is used by EvalC3, and I think to some degree by QCA software. When used to find important single attributes this is both quick and effective. When used to find combinations of multiple attributes this can be very slow, though still very effective.
- The Quine-McCluskey algorithm is used by QCA to reduce many configurations to the smallest possible number that have the same core elements. It is not helpful in data sets with limited diversity of configurations e.g. where many of the configurations are significantly different from each other (i.e. more than one attribute different).
- Decision Tree models can be generated using EvalC3 and Rapid Miner software to produce very readable results. However, Decision Tree algorithms can be prone to over-fitting i.e. prioritizing accuracy over generalisability.
- A genetic algorithm is also built into EvalC3. This is efficient when dealing with large data sets and many attributes, but it may require more than one application to find the best solution.
- In summary, all algorithms have their strengths and weaknesses. Ideally, we should test the results found by using one algorithm by also using an alternative algorithm.
- Performance measures:
- EvalC3 uses provides multiple measures of model performance, which will be suitable in different contexts.
- QCA focuses on two (consistency and coverage) and of these, it then seems to privilege consistency.
- Rapid Miner has a similar range to EvalC3 but fails to use QCA measures like necessity and sufficiency, which EvalC3 does use
- Visualisation of results:
- Decision Trees are the most user-friendly visualisation.
- Venn diagrams as sometimes used by QCA require more familiarisation/explanation.
- EvalC3 uses three methods: Decision Tree diagrams, plus a combination of a Design menu and a Confusion Matrix (truth table) to display the results.
- Manipulation of results:
- EvalC3 includes the capacity to manually configure prediction models, and to tweak models developed by its two algorithms.
- These options are not available in Rapid Miner and QCA software
- Sensitivity / contribution analysis:
- Prediction models developed by EvalC3 can be manually adjusted to identify which attribute in the model contributes most to its overall performance.
- QCA can do a similar form of analysis (known as INUS analysis) but not with the same degree of precision.
- Case selection:
- EvalC3 has a systematic process for identifying individual cases most suitable to follow up within-case analysis.
- This option is not available in Rapid Miner and QCA software