Scalability: The difficulty of problems should scale up, as comparing absolute performances may be less significant than comparing how performances scale up with problems of increasing difficulty.

Valid vs. not-valid balance: In a good test set, valid and not-valid problems should be more or less equal both in number and in difficulty. Moreover, the *maximum uncertainty* regarding the solution of the problems is desirable.

Reproducibility: A good test set should allow for easily reproducing the results.

The following criteria derive from or are significant sub-cases of the main criteria above.

Parameterization: Parameterized inputs with sufficient parameters and degrees of freedom allow the inputs to range over a large portion of the input space.

Control: In particular, it is very useful to have parameters that control *monotonically* the key features of the input test set, like the average difficulty and the "valid vs. non-valid" rate.

Modal vs. propositional balance: Reasoning in modal logics involves alternating between two orthogonal search efforts: pure modal reasoning and pure propositional reasoning. A good test set should be challenging from both viewpoints.

Data organization: The data should be summarizable —so as to make a comparison possible with a limited effort— and plottable —so as to enable the qualitative behavior of the system(s) to be highlighted.

Finally, particular care must be taken to avoid the following problems.

Redundancy: Empirical test sets must be carefully chosen so as not to include inadvertent redundancy. They should also be chosen so as not to include small sub-inputs that dictate the result of the entire input.

Triviality: A good test set should be flawless, that is, it should not contain significant subsets of inadvertent trivial problems.

Artificiality: A good empirical test set should correspond closely to inputs from applications.

Over-size: The single problems should not be too big w.r.t. their difficulty, so that the resources required for parsing and data managing do not seriously influence total performance.

These criteria, which are described and motivated in detail by Horrocks et al. (2000), have been proposed after a five-year debate on empirical testing in modal logics (Giunchiglia & Sebastiani, 1996; Heuerding & Schwendimann, 1996; Hustadt & Schmidt, 1999; Giunchiglia et al., 2000; Horrocks & Patel-Schneider, 2002). (Notice that some of these criteria are identical or similar to those suggested by Heuerding & Schwendimann, 1996.)
The above criteria are general, and in some cases they require some interpretation. First, some of them have to be implicitly interpreted as "unless the user deliberately wants the contrary for some reason". For instance, it might be the case that one wants to *deliberately* generate easy problems, e.g., to be sure that the tested procedure does not take too much time to solve them, or redundant