Warning: This page is not about any of the core functions of EvalC3. It’s simply an exploration of related ideas…
The concept of a “fitness landscape” has its origins in evolutionary biology and was first proposed by Sewell Wright in 1932. Here is a recent book on the history of the idea: “The adaptive landscape in evolutionary biology“. See also Colin Reeve’s book chapter on Fitness Landscapes.
It was proposed as a metaphor, with height as the measure of fitness, and hills of different height might be spread across a landscape in different ways, as isolated peaks, ranges, and plateaus. An organism, or a species, would be located somewhere on this landscape and exploring it through genetic and behavioral variation. It might reach the top of a hill, but this might only be a “local optima” i.e not the highest hill in the whole landscape. one of the difficulties with the metaphor, that various people have struggled with is what are the horizontal (N/S, E/W) axes in this landscape? What aspect of evolutionary biology do they represent?
It occurred to me that a version of this idea could be implemented using data on prediction model fitness, as it is summarised in a Confusion Matrix. The N-S and E-W axes could be the two measures Coverage (/ True positive rate / Sensitivity / Recall) and Consistency (/ Positive Predictive Value / Precision ). The vertical dimension could be the number of True Positives, found with any given combination of Coverage and Consistency values.
The surface of the landscape could be mapped by generating every possible model with a given set of attributes, and recording and plotting the combination of Confusion Matrix values for each model.
Here is a very simple example, where performance measures from 9 different models, (among the 2^5 =32 possible models) were plotted on to a landscape surface diagram generated by Excel. In this landscape, the bottom right corner is where Necessary and Sufficient models would be found because there FP=0 & FN=0. But none were found in the example data set I was working with. But there were plenty of Sufficient models, represented by the mountains on the bottom left side, where FP = 0. Altitude represents numbers of True Positives that a model has.
What would this landscape meta-model then tell us? I had hoped some measure of landscape smoothness/ruggedness would tell us how risky or safe incremental innovation in model design might be, in a given dataset. Some landscapes might be rugged in the sense that a small change in model design would lead to a big change in fitness, and vice versa. But with the above version of a fitness landscape the number of changes in model attributes is not visible.
To be continued….