When a good prediction model has been found it may consist of multiple project attributes (required to be present and/or absent). A question may then asked as to how important each of these attributes are, within the model as a whole.
This question can be answered by systematically reversing each attribute from the model, one at a time, and on each occasion observing how the overall performance of the model changes. Reversal here means changing an attribute value from 1 to 0 or 0 to 1, not simply to n/a. The reversal of attributes which are more important will be associated with a bigger deterioration in the performance of a model. The question then is which attribute reversal has been associated with the biggest deterioration in model performance.
An example using the Krook data set
As a result of using an evolutionary search it was found that the presence of “quotas” for women in parliament and the country being in a “post-conflict” situation was sufficient but not necessary for high levels of women’s participation in parliament. This model accounted for 6 of the 9 cases where there were high levels of women in parliament.
When the “presence of quotas” was reversed to the “absence of quotas” in the model, the model performance fell from 83% to 41%, when using Averaged Accuracy as the performance measure. The number of TPs fell from 6 to 1, with FPs increasing from 0 to 5. In other words, the revised model was now a better predictor of the absence of the outcome
When the presence of a post-conflict situation was removed from the model, the model performance fell to 49%, with the number of FPs(4) being greater than the number of TPs(2).
It appears that the presence/absence of quotas was the attribute of the model that made the biggest difference to its performance
This type of analysis can be seen as a particular form of contribution analysis.
The following is a quote from the TOrs of an evaluation: “Even though it is well recognized that multiple factors can affect the livelihoods of individuals or the capacities of institutions, it is important for policy makers as well as stakeholders to know what the added value of the xxxx program is”
What they are looking for, I suggest, is an INUS condition, an attribute that is Insufficient but Necessary part of a configuration that is Sufficient but Unnecessary for an outcome to occur.
Here are two possible configurations:
A configuration that is sufficient for the outcome: 0010101 = 1 (outcome present)
And another, almost the same, which is not: 0000101 = 0 (outcome absent)
Here “–1—-” is the INUS condition. Its presence is associated with the presence of the outcome, and its absence is associated with the absence of the outcome.
INUS attributes can identified using EvalC3. The first step is to develop a good predictive model for an outcome. Typically this will consist of a number of attributes. Then by selectively changing the status of each attribute in a configuration and then observing its effects on the model’s performance we can identify the extent to which an attribute is an Insufficient but Necessary part of a configuration that is Sufficient but Unnecessary.
A model is sufficient for an outcome if there are multiple TP cases but no FPs. Where its attributes are present, the outcome is present, though the outcome may also be present under other conditions (as shown by the FN cases). If, for example, the value of an attribute in a model was changed from 1 to 0, ans some FPs then appeared, the model would no longer be sufficient for the outcome. A more demanding INUS requirement might be that the number of FPs cases now outnumber the TPs.