3. Design model

There are two different ways of building a predictive model:

• Manually, based on a prior theory of any kind (e.g. your own or someone else’s, a formal or an informal theory)
• Automatically, using one of the search algorithms built into EvalC3

1. Manual design

1. Once the Design and Evaluate view is open look at “Design” on the left side (circled in red in the screenshot below). Here you choose what values to place next to each of the attributes that are automatically listed here. The drop-down menu in the Status column provides three options: N/A meaning ignore this attribute; 1 = this attribute is present, 0 = this attribute will be absent.
2. The default status for each attribute when this view is first opened is N/A.
3. You also need to choose whether the Outcome is expected to be present or absent when these attributes are as described above, using the same kind of drop down menu in the Status column
4. This combination of attribute values and the selected outcome then constitute a predictive model
5. The performance of this model can then be seen immediately in the Confusion Matrix under the heading “Evaluate”, which is explained more below
6. Click on the Save button (above left) to save details of this model and its performance. You will need to save the model with a name you will recognize later.
7. If you want to remove all the attributes of a model in one go i.e re-set them all to n/a click on the round “Stop”sign to the right of attribute “Status”

2. Search algorithm

1. Chose which type of model you want to find (circled in red in the screenshot above). There are four options, each represented by a button you can click on:
   1. Necessary & Sufficient: This kind of model will consist of a single attribute, or set of attributes, which are both necessary and sufficient for the outcome. In the Confusion matrix, there will be no False Positives and no False Negatives. In reality, this kind of model is rare.
   2. Necessary but Insufficient: This kind of model will consist of a single attribute, or set of attributes, which are necessary but insufficient for the outcome. In the Confusion matrix, there will be no False Negatives but there will be some False Positives.
   3. Sufficient but Unnecessary: This kind of model will consist of a single attribute, or set of attributes, which are sufficient but unnecessary for the outcome. In the Confusion matrix, there will be no False Positives but there will be some False Negatives.
   4. Most predictive – of any kind: This kind of model will consist of a single attribute, or set of attributes, which are likely to be in sufficient and unnecessary for the outcome, but still are good predictors, as measured by Accuracy, for example. In the Confusion matrix, there will be some False Positives and there will be some False Negatives. But it may also be the case that this search finds one of the above three models.
2. When any of the above buttons are clicked this will take you to a Find New Models pop-up menu. This presents a choice of four search algorithms. See Search Options on this website for more detailed information about these choices
3. Choose the performance indicator: the measure that should be maximised by the best models that can be found. There are three groups of these: Overall, Specific and Relative. For more information on these see Evaluate Model. Clue: Start by using the most widely used measure: Accuracy
4. Set constraints. These can be of three types, which canna be used by themselves or in combination:
   1. Particular attributes in the Design view whose values need to remain fixed. For example, as being present or absent
   2. Specific performance measures other than the one selected as the objective. For example that Lift =>100%
   3. Specific values for one or more cells in the Confusion Matrix
      1. Try setting False Positive = 0, to find Sufficient but Unnecessary attributes (or configurations of attributes)
      2. Try setting False Negative = 0, to find Necessary but Insufficient attributes (or configurations of attributes)
      3. Try setting False Negative = 0 and False Positive = 0 to find Necessary and Sufficient attributes
      4. Postscript: There are now three radio button options that can be used to set these constraints with one click
5. Implement the search by clicking Okay
   1. If using exhaustive search, watch the process bar in order to assess if the results will be ready within the time available. If not, cancel.

View the results of the search, given the settings above. 

The attributes that have been found as the best predictors of the outcome (known as “the model”) will appear in the Design area, replacing any previous selection. This found model will automatically be saved and the saved name will be visible to the right of the “Save Model” button

The raw results of the prediction model will be shown in the Confusion Matrix in the Evaluate area. See Evaluate Model for more information on how to read the Confusion Matrix.

The performance measures derived from the Confusion Matrix can be seen listed further below the matrix. These are used to summarise the performance of the current model in predicting the outcome of interest.

Revise the results

Within the Design & Evaluate worksheet you can tweak the values of the attributes in the new model in order to:

1. Incrementally improve performance of the model
2. Identify what attributes in the model contribute most/least to its overall performance. For more on this option see Sensitivity Analysis
3. Postscript: There is now a Sensitivity button on the right, which if clicked will then highlight the attribute in the current model which contributes the most to its good performance. This is measured by comparing the % point reduction in model performance when each attribute is selectively removed from the model

Save the results

Save the results of each version of the model that you find to be of value. This will be done automatically, with a unique name, if exhaustive of evolutionary searches have been carried out. But if there has been any manual tweaking the resulting model will then need to be saved manually