[cleaner] option to skip cleaner on scrubbing some files

[pmoravec]

As the cleaner run time can be in hours and esp. until #3097 (parallel cleaner) is implemented, does it make sense to have an option "skip cleaning these files in the archive"?

When a user knows some collected data are without any sensitive information but they still needs to clean the rest, sos clean shall have an option to skip cleaning them. Like e.g. sos clean --skip-cleaning=var/log/chrony,sos_commands/kernel,sos_commands/process/ps_* - comma-separated list of paths (or prefixes of paths) such that cleaner will skip processing any file in the archive that starts with either path.

The option or its argument is just an example and I am happy to hear other proposal (e.g. does glob's * make sense at all?), my point is if that feature makes sense at all.

I think it can be a nice speed up of the cleaning process, while driven by users knowing on their own what is safe to skip.

[TurboTurtle]

I don't think I see anything wrong with that kind of feature. We already have a bunch of messaging around "we're really trying here, but nothing's perfect" and this would be an explicit "I know what I want" from users.

Globs make sense since we're talking about files. Would we want to assume a leading * for any entries? So that users can simply specify --skip-cleaning='ps_*' instead of --skip-cleaning='sos_commands/process/ps_*' (which could of course lead to unintended skipping), or take the "tell us exactly what you want done" mindset?

[dnegreira]

Globs make sense since we're talking about files. Would we want to assume a leading * for any entries? So that users can simply specify --skip-cleaning='ps_' instead of --skip-cleaning='sos_commands/process/ps_' (which could of course lead to unintended skipping), or take the "tell us exactly what you want done" mindset?

I tend to err on the side of doing more cleaning than wanted rather than less cleaning.
I would prefer, for the specific case of cleaner, that we go for the later example of specifically defining what the user wants.

[pmoravec]

Same here. If a user wants to skip cleaning something, they know well what in particular they want to skip. That means the second option. The first one can too easily end up in unintended skipping to clean / unitended private data left in final tarball.

[NikhilKakade-1]

If the user has specified the --clean option for collecting and cleaning packages, will it be logical to initiate the cleaning of logs simultaneously during the collection process itself ?

[TurboTurtle]

That would require a significant rewrite of both report and clean to be able to work in tandem like that in a way that is still performant. I'm not saying it can't be done, just that it's a much larger effort than it appears to be at first glance.

[pmoravec]

#3520 proposed here. It was bit tricky to come up with an option name since --skip-files is already an option for report.

Also, adding a new option to so many places in report+clean+collector is error-prone - we duplicate code and sometimes it is not clear why or where to add an option. This confusion will grow if/when we will introduce more subcommands of sos

Ideally we should have (in sosnode?) one place where we will keep all options (and arg_defaults?), such that adding an option means changing one place only. But such code refactorization can be challenging.

[pmoravec]

Closing discussion as resolved by the merged #3520.