Sensitivity Analysis

When a good prediction model has been found it may consist of multiple project attributes (required to be present and/or absent). A question may then asked as to how important each of these attributes are, within the model as a whole.

This question can be answered by systematically removing each attribute from the model, one at a time, and on each occasion observing how the overall performance of the model changes. The removal of attributes which are more important will be associated with a bigger deterioration in the performance of a model. The question then is which attribute removal has been associated with the biggest deterioration in model performance.

Example using the Krook data set

As a result of using an evolutionary search it was found that the presence of “quotas” for women in parliament and the country being in a “post-conflict” situation was sufficient but not necessary for high levels of women’s participation in parliament. This model accounted for 6 of the 9 cases where there were high levels of women in parliament.

When the presence of quotas was removed from the model, the model performance fell from 83% to 74%, when using Averaged Accuracy as the performance measure

When the presence of a post-conflict situation was removed from the model, the model performance remained at the same level of 83%. Therefore presence of quotas can be seen as the most important component of the model.

When the changes in the Confusion Matrix were examined it could be seen that the main contribution of the presence of a “post-conflict situation” within the model was to reduce the number of False Positives, from 4 to zero.

This type of analysis can be seen as a particular form of contribution analysis.

A QCA perspective: This type of analysis can also be seen as a form of INUS analysis as used in QCA. By selectively removing each attribute in a configuration and then  observing its effects we are identifying the extent to which an attribute is an Insufficient but Necessary part of a configuration that is Sufficient but Unnecessary.If this reduces the performance of a predictive model from above 50% accuracy to below 50% we could describe the attribute as INUS. In the above example the “post-conflict situation” attribute had a noticeable effect on model accuracy, but the model accuracy was still above 50%. So, if we regard INUS as a categorical status (i.e. it either is or is not) then in this analysis “post-conflict situation was  not an INUS attribute.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s