FR: Time based root scope destructors

[scudette]

Some operations attach destructors to the root scope - this ensure these run when the query completes but remain available for the life of the query . For example certain temporary files are attached to the root scope because it is difficult to know at which scope level they will be used.

This approach fails in event queries which never really complete (unless the event table is refreshed). This causes things like temp files to not be cleaned up in a reasonable time (because they are still in scope so may be used).

For example these constructs leave temp file behind:

--- This materializes into a file based object
LET X <= SELECT _value AS V FROM range(end=10000)

-- This materializes into a tempfile as well
SELECT { SELECT _value AS V FROM range(end=10000) } AS X FROM scope()

In both the above cases it is hard to know if the expand will be needed further in the query. Since an event query never terminates this can leave temp files behind.

There is really no good solution here:

1. We can define a timeout where we call destructors anyway to remove the rempfile. This will break any access of the materialized object after the timeout. For events that occur rarely (e.g. ETW events) this is a real possibility and will break this construct. A possible work around here is to use materialize() instead of a simple LET <= - this is probably the right thing to do anyway.

2. I am not sure that materializing in the second example above is a valid operation - we should maybe exclude that from file based materialize operations completely - I am ok with just breaking this operation because it is totally a misuse of VQL columns. There is no sane case where it is valid to expand a single column of 1000 items in a list.

Hopefully if we offer a way to tune this it will be possible to cater for specific edge cases but the common case will behave well.