experimental spike for SO66453263 - zero padding

[mgravell]

See https://stackoverflow.com/q/66453263/23354 for context

(existing data with zero-padding at the end)

[bmueller84]

As mentioned in my comment on stackoverflow, this solution works perfectly for my needs. Good job! So what are the next steps? Will that code be part of an official release?

[mgravell]

Fundamentally, the problem here at least with the stream API is that it is very easy to consume data from a stream that you can't then put back; it would be become very tempting for users to think "oh, great, I can just use a zero sentinel between records", and then when they try: it works in their tests, and then they lose real data in production, because data was swallowed and dropped. We might be able to fix this partially for "pipelines", but: I think this is just too dangerous as written

[bmueller84]

I get your point. So, what alternative would you recommend for my zero-padded legacy data? It does work in older versions of protobuf, after all. Any help would be highly appreciated.

[mgravell]

Question: in your case it is just trailing padding data; not a separator - is that right? Maybe we can resurrect this but treble-doen on the warnings on the API, meaning "this will leave your stream in an undefined and unpredictable state after reading" - thoughts?

[bmueller84]

Yes, exactly; just zeros at the end (of the file, in this particular case). So, yes, this would be perfect for me.