rapidly for the datasets with large number of features, moreover the estimates of importance returned by ACE and Boruta were very similar, with better results achieved by Boruta. Thus only Boruta heuristic algorithm was used for tests in this phase.

## 3.1 Synthetic Data

The first experiment was performed on a serious of synthetic geometrical problems analogous to the Madalon problem [10], constructed with the help of mlbench.xor function from the R package *mlbench* [19].

Each of sets used consisted of two attributes used to generate the decision, eight random linear combinations of them and as many irrelevant attributes made of an uniform noise to top up to the given size - this way in all cases there was 10 attributes important by design. All possible variants of such a data set containing 125, 250, 500, 1000 and 2000 objects described with 125, 250, 500, 1000, 2000 and 4000 attributes were generated. Finally in each case the set of relevant attributes was calculated with all three wrapper algorithms described above; as a reference also 10 top important attributes from a single random forest run were saved. One should note that given a feature importance, the essence of the *all relevant* feature selection is finding either an importance cutoff or correspondingly the number of top-scoring features that separates relevant and irrelevant attributes. Therefore an algorithm which deems important top N
attributes from the random forest importance ranking, where N is the known a priori number of relevant attributes, is a reference that shows the limit of information contained in the importance measure. The entire procedure was repeated fifteen times.

Each result was described by the following quantities, averaged over fifteen iterations: number of attributes correctly recognised as important (true positives, TP), attributes incorrectly recognised as important (false positives, FP) and attributes which where incorrectly not recognised as important (false negatives, FN ). The overall performance was assessed as an average F-score, F =
2Prec×Rec Prec+Rec , where *Prec* =TP
TP+FP and Rec =TP
TP+FN .

## 3.2 Semi-Synthetic Data

The second experiment used four series of semi-synthetic data sets. Each set was constructed from a real-world data set by extending it with an artificial attributes made of an uniform random noise to obtain respectively 125, 250, 500, 1000 and 2000 total attributes, similarly to the construction of the synthetic test sets. Thus one knows that the added attributes are irrelevant by design and correspondingly that the relevant attributes may be only found among original attributes. The Boruta algorithm was used to find the relevant attributes in all cases; the whole procedure was repeated 15 times and the results were averaged.

As a base real-world sets the following four well-known data sets were used:
German Credit [7], House Votes [19, 7], Ozone [19, 3] and Vehicle [19, 7, 28].