If a data set has information on 10 different attributes of projects this means that there could be 210 different combinations of these that might be the best predictor of an outcome of interest i.e. 1,024 possibilities. EvalC3 provides a number of ways of searching through these possibilities to find the most accurate predictor:
- Hypothesis-led manual selection of attributes, based on theory derived from past experience and/or research elsewhere. The advantage of this approach is that where the hypothesis is correct there may already be a good foundation knowledge, from prior research, on why it works. . In EvalC3 a prediction model can be developed manually by inserting relevant values into the model design (under the Design), and then observing its performance. Normally this should be the first step in an analysis process using EvalC3. However it is possible that there are other solutions with an even better fit with the data, which lay out of sight outside our current understanding,
- Exhaustive search, where every possibility is examined. This should be the second step in an analysis process using EvalC3, which should help us explore “outside the box” of our existing view of how things work. Because it is exhaustive the results will be conclusive. Exhaustive search can be very time consuming and will usually only be suitable when working with small data sets.
- Additional attribute search. This is a simplified and quicker version of an exhaustive search. There are two main ways of using it
- Where there is already an existing model the attributes of this model are treated as search constraints. An exhaustive search is then be made of x+1 attributes, where x is the number of attributes in the current model.
- Where there is no existing model, or where an existing model has been erased then using the “additional attribute search” will search for the best performing single attribute model.
- If this search is re-iterated it will treat the result of the first search as a constraint that has to be met. Unless the result of the first search is removed.
- Used without re-iteration this type of search is most useful when searching for single attributes that are necessary or sufficient fore the outcome.
- Latest update: It is now possible to generate Decision Tree models, using an automatic re-iteration of the additional attribute search. This will generate a set of models that will account for all outcomes, both positive and negative. Look here for detailed guidance on how this work and why it can be so useful
- Evolutionary search. When data sets are large (deep and/or wide) an exhaustive search described above can be too slow to implement. Evolutionary searches are a very efficient means of searching for complex models within much larger combinatorial spaces. EvalC3 makes use of an existing Excel add-in known as Solver, to carry out evolutionary searches.
- However evolutionary searches are not necessarily as conclusive in their findings as exhaustive searches, because they sample different combinations of attributes, rather than test all of them . For this reason the value of the results generated by an evolutionary search should be tested by repeating the search a number of times
The results of all the above types of searches can be manually tweaked (i.e.remove and replace one attribute at a time) to explore the consequences of incremental variations in model design, such as changing the status of an attribute from present to absent, or to N/A. This can help identify the relative contribution of different model attributes to the overall performance of a given model. This knowledge can be useful when carrying out within-case investigations, in search of likely causal mechanisms at work