[Last updated 2018 10 06] Warning: This page is not about any of the core functions of EvalC3. It’s simply an exploration of related ideas…
The concept of a “fitness landscape” has its origins in evolutionary biology and was first proposed by Sewell Wright in 1932. Here is a recent book on the history of the idea: “The adaptive landscape in evolutionary biology“. See also Colin Reeve’s book chapter on Fitness Landscapes.
It was proposed as a metaphor, with altitude as the measure of fitness. Fitness, in the form of higher altitudes, might be spread across a landscape in different forms, as isolated peaks, ranges, and plateaus. The four compass directions represent a two-dimensional version of what in reality is a multi-dimensional space of attributes that would fully describe a species’ attributes. A species would be located somewhere on this 2D landscape and would be exploring it through genetic and behavioral variation. A species might reach the top of a hill, but this might only be a “local optimum” i.e not the highest hill in the whole landscape.
It occurred to me that a version of this idea could be implemented using data on prediction model fitness, as it is summarised in a Confusion Matrix. In place of species, we have prediction models, each of which has a potential location on a landscape. The N-S and E-W axes of the landscape could represent the number of False Positives and False Negatives and the vertical dimension could represent the number of True Positives. In this landscape, each prediction models will have a distinct location, defined by their number of True Positives, False Positives and False Negatives.
The shape of the whole surface of the landscape could be identified by locating every possible model for a given set of data i.e.2^number of attributes in that data set.
Here is a very simple and incomplete example, where performance measures from 9 different models, (among the 2^5 =32 possible models in the Krook dataset) were plotted on to a landscape surface diagram generated by Excel. In this landscape, the bottom right corner is where Necessary and Sufficient models would be found because it is there where FP=0 & FN=0. But no models of this type were found in the Krook dataset I was working with. But there were plenty of Sufficient models, represented by the mountains on the bottom left side, where FP = 0 but there some FNs. Higher altitudes represent models with greater numbers of True Positives. PS: I think this visualisation could be improved by using a better 3D graphing app. Also, a contour map might be better
What would this landscape shaped meta-model then tell us? I had hoped some measure of landscape smoothness/ruggedness would tell us how risky or safe incremental innovation in model design might be, in a given dataset. Some landscapes might be rugged in the sense that a small change in model design would lead to a big change in fitness and vice versa. However, I don’t think this type of landscape example let us see this – the distance between two locations in this landscape does not represent the degree of similarity in two models – rather it represents how different the performance of two models is, in terms of numbers of FPs and FNs. To show landscape ruggedness/smoothness some other way needs to be found to represent similarity of model attributes on the horizontal axes.
To be continued….