arff deserialiser

[gf712]

Simple ARFF deserialiser. See the spec here.

@karlnapf shall I add a ARFF file as an example and a unittest/meta example?

TODO:

• datetime format
• string
• handle missing data

[gf712]

@vigsterkr does stuff like this work independent of platform?

[gf712]

@vigsterkr and this?

[gf712]

@karlnapf how does shogun handle missing data values? It is possible to have NaN values ("?" in arff files) ?

[karlnapf]

@karlnapf shall I add a ARFF file as an example and a unittest/meta example?

Yes you can add it under the toy data in data/toy for examples.
For unit test, I would rather not base it on a file ... although there seems to be no other sensible way or?

[karlnapf]

@karlnapf how does shogun handle missing data values? It is possible to have NaN values ("?" in arff files) ?

idk to be honest :D we might have to think/work a bit for that

[karlnapf]

Nice!

[gf712]

For unit test, I would rather not base it on a file ... although there seems to be no other sensible way or?

For unittest I could try and use strings? I can then feed the stream directly, rather than a file. Would just have to refactor the class a bit though by adding a constructor for streams rather than file names

[karlnapf]

I think that’d be a nice feature anyways or?

[gf712]

Yes, I need it for openml stuff!

[gf712]

So datetime somewhat works, but might have to add strptime definition for the windows build? (see https://stackoverflow.com/a/321877). The conversion to the c++ convention of datetime format needs to be tested more too. I convert dates to unix timestamps btw, because it is the only representation that makes sense to me that can be stored in a numeric matrix

[gf712]

@karlnapf I am not sure how to do the parsing of strings though, for that we would almost need a SGMatrix<variant>

[karlnapf]

So datetime somewhat works, but might have to add strptime definition for the windows build? (see https://stackoverflow.com/a/321877). The conversion to the c++ convention of datetime format needs to be tested more too. I convert dates to unix timestamps btw, because it is the only representation that makes sense to me that can be stored in a numeric matrix

Great. Ill check it out

[karlnapf]

@karlnapf I am not sure how to do the parsing of strings though, for that we would almost need a SGMatrix<variant>

Uh that will be tricky with parameter framework. Also SG* types are meant as mere (continuous!) memory wrappers around basic types, not the fancy stuff they are abused to do.

So one thing might be to think about a matrix-like structure to handle non-primitives. I mean std::vector is fine with the param framework

We have started using variant in the pipeline...although that causes all sort of headaches (needs an override clone/equals, but no idea how to make put/get work atm), but that might be fixed in any.h. Maybe a good point to start thinking about how to handle variant .... @vigsterkr ?

[gf712]

@karlnapf @vigsterkr so the return type is now CCombinedFeatures, and I use std::vector<variant> internally.
Current features:

1. handle numeric data with CDenseFeatures
2. handle dates (excluding timezone name parsing because it requires libtz from the date library to be compiled into a library)
3. handle strings with StringList and then StringFeatures

The missing data can be ignored for now I think, and just be skipped by the parser, and then one day we can add imputers.
The relational attributes I still need to look into.
And then I'll add a meta example too and that's it I think. @karlnapf can you think of an interesting dataset that shogun could handle?
At some point I can add a ARFFWriter, but it isn't really needed right now, and can be done once the new serialisation stuff is ready.

[gf712]

also the error seems to be due to AveragedPerceptron? I am guessing there's some UB going on

[gf712]

@karlnapf I had some time so I wrote this (pretty bad) serialiser.. Would it make sense to add a visitor pattern to SGObjects? Or at least for features?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.