Investigate restructuring the BatchSpanProcessor queue and batching algorithm

[MrAlias]

As a result of our queueing strategy with the enque being asynchronous from the processing of the batch it is possible for spans to be dropped instead of flushed when a user calls for this flush or shutdown.

Alternatives should be discussed here.

Originally posted by @MrAlias in #1166 (comment)

[MadVikingGod]

I experimented with a batcher that uses a channel to manage state a while back here. If this looks like a promising route I can put a bit more time to double check that it works 100%, and add more comments to aid in getting this ready for a PR.

[MrAlias]

I experimented with a batcher that uses a channel to manage state a while back here. If this looks like a promising route I can put a bit more time to double check that it works 100%, and add more comments to aid in getting this ready for a PR.

Sounds good to me 👍.

[pellared]

Related issues:

• Fix panic, deadlock and race in BatchSpanProcessor #4353 (comment)
• Fix panic, deadlock and race in BatchSpanProcessor #4353 (comment)

Side note: The processing logic may be implemented similarly to PeriodicReader.