“Boring” versus “interesting” models

How to deal with each type

In an analysis of data on 65 projects funded by the Civil Society Challenge Fund data some of the prediction rules confirmed existing expectations, they held no surprise and ran the risk of being quickly dismissed as “boring”.

But there is a useful step that can then be taken with such “boring” cases. That is to examine the False Positives, where the model predicted the outcome to be present, but where the data showed the outcome to be absent. It is these kinds of cases that are important to examine in detail, to find out why, despite the presence of the model conditions, the outcome was not present.

Understanding these cases will help us define the boundaries of our confidence in the prediction model we have taken for granted. It may help prevent us from being excessively confident in the model, if the causes of the False Positives are beyond our control. Or it may help us widen the applicability of the model, if the causes of the False Positives are within our control.

On the other hand, where a prediction rule contradicts existing expectations it is the True Positives that are most in need of investigation in detail, in order to find out if and how the attributes of the model interacted  to cause the predicted and observed outcome.

So, it is worth asking clients of an analysis which of the results they expected versus which were surprises to them. Or, better still, before sharing the results, ask them to predict the results. That may give a more direct answer.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s