Panel to Primitive Transformation with Sparse Output

[fspinna]

I am currently working on a Panel to Primitive transformer, where the primitive is essentially a 2D matrix that contains numerous features but is predominantly sparse. As per my current understanding and attempts, it appears that generating a scipy sparse output (such as a csc array) directly from the transform method is not supported. Or am I missing something? Thank you in advance for any help!

[fkiraly]

Yes/no - you can turn off input and output checks in transformations via set_config, see documentation for which config fields to use, these should give granular control of checks on/off, or which output type you want.

This allows you to use the sktime framework without any internal controls on what goes in or out, but of course you pay for this flexibility by lack of inbuilt checks and tests (plus, conversions, broadcasting, etc).

(btw, if you are using this, feedback and contributed tests for usage with checks turned off would be appreciated! it's currently a "silent" feature due to not-so-thorough testing)

More in the middle of this trade-off, you could add sparse output as an mtype if you wanted. The data container system is not as conveniently extensible yet as the estimator framework, but this is on the current roadmap, as users seem to want customizability here, and estimator packages also increasingly deviate from the numpy/pandas canon.

Below is a first draft at the rework; to add your own, third party type in the current code base, you would currently have to modify the datatypes module directly; in the future, we hope to support allowing to have custom data types in a third party location and most things working out of the box.

See some current thinking here, feedback appreciated:
#6033

[fspinna]

Thank you so much for the help! I'll go with set_config(**{"output_conversion": "off"}) for now.
By the way (and off topic), this also solved a problem I had for concatenating primitive outputs using FeatureUnion, which seems to support only Panels or Series transformer outputs.

[fkiraly]

this also solved a problem I had for concatenating primitive outputs using FeatureUnion, which seems to support only Panels or Series transformer outputs.

Interesting - sounds like a bug, could you kindly report this with an example?

[fkiraly]

Addendum: there already is one sparse data mtype for primitives (Table scitype) - it's the list_of_dict specification. It's not quite the same as scipy sparse, but it's sparse.

See sktime.datatypes._table._registry.