How to deal with each type
In an analysis of data on 65 projects funded by the Civil Society Challenge Fund data some of the prediction rules confirmed existing expectations, they held no surprise and ran the risk of being quickly dismissed as “boring”.
But there is a useful step that can then be taken with such “boring” cases. That is to examine the False Positives, where the model predicted the outcome to be present, but where the data showed the outcome to be absent. It is these kinds of cases that are important to examine in detail, to find out why, despite the presence of the model conditions, the outcome was not present.
Understanding these cases will help us define the boundaries of our confidence in the prediction model we have taken for granted. It may help prevent us from being excessively confident in the model, if the causes of the False Positives are beyond our control. Or it may help us widen the applicability of the model, if the causes of the False Positives are within our control.
On the other hand, where a prediction rule contradicts existing expectations it is the True Positives that are most in need of investigation in detail, in order to find out if and how the attributes of the model interacted to cause the predicted and observed outcome.
So, it is worth asking clients of an analysis which of the results they expected versus which were surprises to them. Or, better still, before sharing the results, ask them to predict the results. That may give a more direct answer.