When a good prediction model has been found it may consist of multiple project attributes (required to be present and/or absent). A question may then asked as to how important each of these attributes are, within the model as a whole.
This question can be answered by systematically reversing each attribute from the model, one at a time, and on each occasion observing how the overall performance of the model changes. Reversal here means changing an attribute value from 1 to 0 or 0 to 1, not simply to n/a. The reversal of attributes which are more important will be associated with a bigger deterioration in the performance of a model. The question then is which attribute reversal has been associated with the biggest deterioration in model performance.
Example using the Krook data set
As a result of using an evolutionary search it was found that the presence of “quotas” for women in parliament and the country being in a “post-conflict” situation was sufficient but not necessary for high levels of women’s participation in parliament. This model accounted for 6 of the 9 cases where there were high levels of women in parliament.
When the “presence of quotas” was reversed to the “absence of quotas” in the model, the model performance fell from 83% to 41%, when using Averaged Accuracy as the performance measure. The number of TPs fell from 6 to 1, with FPs increasing from 0 to 5. In other words, the revised model was now a better predictor of the absence of the outcome
When the presence of a post-conflict situation was removed from the model, the model performance fell to 49%, with the number of FPs(4) being greater than the number of TPs(2).
It appears that the presence/absence of quotas was the attribute of the model that made the biggest difference to its performance
This type of analysis can be seen as a particular form of contribution analysis.
A QCA perspective: This type of analysis can also be seen as a form of INUS analysis as used in QCA. By selectively changing each attribute in a configuration and then observing its effects we are identifying the extent to which an attribute is an Insufficient but Necessary part of a configuration that is Sufficient but Unnecessary. If this reduces the performance of a predictive model from above 50% accuracy to below 50% (or more FPs than TPs) we could describe the attribute as INUS. In the above example changing the value of the “quotas” attribute had a noticeable effect on model accuracy, reducing it well below 50%. So, if we regard INUS as a categorical status (i.e. it either is or is not) then in this analysis “post-conflict situation was not quite an INUS attribute. If the number of TPs was reduced to zero then it would be.
A further example. Here are two possible configurations:
A configuration that is sufficient for the outcome: 0010101 = 1 (outcome present)
And another, almost the same, which is not: 0000101 = 0 (outcome absent)
– – 1- – – –
Here 1 is the INUS condition. Its presence is associated with the presence of the outcome, and vice versa.