[ENH] classification test scenario with three classes

[fkiraly]

This adds a classification test scenario with three classes.

Currently, only two classes were tested.

[cedricdonie]

Would there be any harm in going to more than three classes? I understand that we are interested in multi-class prediction rather than three-class prediction, so e.g., five classes might catch more errors. Or perhaps this test can be made parametric with anywhere from three to e.g., 15 classes to increase coverage further (at the cost of more runtime)?

[fkiraly]

Yes, you are right, of course - it is runtime that worries me as you say, and the minimum sample size requirement.

Some classifiers have grid search internally, 5-fold as default. So you'd want to see at least 3 instances of each in the training set, which gets you to n_instances = n_classes * 4 (or better 5, 6).

Many classifiers have between second and third power scaling on the number of instances, and there are a large number of classifiers, on each of which the scenario is run. So, going from 3 to about 4 doubles the runtime caused by this scenario, I would guess, which is about 1/3 or 1/4 of the total classifier runtime already.

We can of course check how much it is really, empirically, if you would like to, I do not mind - though I wonder if 5 classes gives that much more coverage than 3. My gut feeling is, if sth breaks with 5, it already breaks with 3, e.g., the one-hot encoder example.