Internal validity
The design of the EvalC3 workflow, which progresses from cross-case analysis to within-case analysis (albeit with some recycling between the states sometimes), is orientated towards establishing a form of internal validity. In cross-case analysis an association is (hopefully) found between particular attributes of cases and an outcome of interest. Then, through within-case analyses, we might find evidence of causal mechanisms at work underlying that association. Ideally, we can then anchor the description of an abstract association with the verifiable particulars of individual known cases.
External validity
We can also give some attention to the prospect of external validity, the prospect that the association that has been found might also be found in other settings. Evidence of this possibility can be seen by looking at the diversity, or the lack of it, of the attributes of the cases within the dataset. The most extreme possibility in a set of cases would be where each had a unique set of attributes, there were no duplicates. We could also compare such set of cases in terms of the number of attributes that were documented. No duplication amongst a set of cases with many attributes per case would be indicative of greater diversity compared to no duplication amongst a set of cases with only a few attributes per case.
The presence of diversity within a set of cases is encouraging. The existence of each type of configuration tells us what is possible (outcome -wise) when that type of configuration occurs. But where there are gaps in the range of possible configurations in a dataset this does point to potential risks for external validity. It suggests that any model that has been found work within the current set of cases may not work when applied in other settings, where there are cases with configurations not found present in a dataset that was used to develop the model. Within EvalC3, on the Select Data worksheet there is a measure of diversity present in the dataset currently being used. This tells us what proportion of all the possible configurations of attributes that could exist amongst the cases are not actually present. So it can be seen as a kind of measure of risk, the risk of a predictive model not being applicable in the wider world.
Sampling and external validity
Random sampling of a population of cases is one way of ensuring that the results of an analysis can be generalised. But they only give confidence about generalisation from the sample to the population at large where the sample came from. Not necessarily beyond that population. However, if the diversity of the attributes of the cases within the sample is high, rather than low, we might have a bit more confidence about the ability of the model to work outside the sampled population. Where we are dealing with small populations we may not have to resort to sampling, but the degree of diversity of cases within the population will still have significance in terms of potential for generalisation of findings to other populations.