+str Add flatmapConcat with parallelism.

[He-Pin]

References #31958

@gael-ft Would you like to take a review, thanks?

[He-Pin]

There are some allocation here

[He-Pin]

before:

[info] Benchmark                                                         Mode  Cnt         Score        Error   Units
[info] FlatMapConcatBenchmark.internalSingleSource                      thrpt   10   9111519.387 锟斤拷 183996.447   ops/s
[info] FlatMapConcatBenchmark.mapBaseline                               thrpt   10  20336793.367 锟斤拷 887065.039   ops/s
[info] FlatMapConcatBenchmark.oneElementList                            thrpt   10    371981.248 锟斤拷   3765.397   ops/s
[info] FlatMapConcatBenchmark.sourceDotSingle                           thrpt   10   4578547.437 锟斤拷 479604.201   ops/s

after:

info] Benchmark                                                         Mode  Cnt          Score            Error   Units
[info] FlatMapConcatBenchmark.internalSingleSource                      thrpt   10    9027659.060 锟斤拷     136614.084   ops/s
[info] FlatMapConcatBenchmark.mapBaseline                               thrpt   10   20947850.178 锟斤拷    1039800.021   ops/s
[info] FlatMapConcatBenchmark.oneElementList                            thrpt   10    4451591.431 锟斤拷      55815.355   ops/s
[info] FlatMapConcatBenchmark.sourceDotSingle                           thrpt   10    4547444.612 锟斤拷     136653.128   ops/s

[GreyPlane]

I wonder if it is possible to write a Buffer.peek(n: Int) which peek from current to n, and by that, is it easy to impl Merge via this, just change inflightSources.peek() eq self to check through the range from 0 to breadth - 1.

[He-Pin]

@GreyPlane How about comment in line, thanks

[gael-ft]

Thank you for the implementation.
This will allow us to have more than 1 source pulled at a time keeping elements ordered from downstream perspective.

[He-Pin]

Seems should add another parameter eager too. When true will complete / fail the stream when upstream fail.

[gael-ft]

Seems should add another parameter eager too. When true will complete / fail the stream when upstream fail.

Not obvious to me.
Goal flatMapConcat is to keep elements ordered.
If order is not a concern, you could just use merge (which has the concept of eager).

So I can't see a scenario where you would like to continue after one of the upstream sources cancelled when using this operator.

That being said, if you feel that could be useful, let's go 😉

[He-Pin]

@gael-ft that's true. but I saw your PR and you can return Source.failed(...) sometime. and for the eager I mean eagerError.
If when materilization, the upstream is Source.failed would I failed the stream ASAP when the parallesm > 1?

And for most case, I think we should make the eagerError = false, even there is an inflight Source.failed(...).

I was wantted to optimize for Source.future and Source.failed, but in which way the Source can be already failed. and there are Flux.concatMapDelayError/flatMapSequentialDelayError, I think delaying error should be the default behavior as the operator only parallelism materialization but not emitting.

[gael-ft]

@He-Pin I might be a bit too much use case centric (i.e. S3 object upload where we don't want any missing part).

If when materilization, the upstream is Source.failed would I failed the stream ASAP when the parallesm > 1?

So this question can probably be a valid question in some cases.

But regarding the default value of eager I think it should be intentional to continue processing after any error occured.
As with Flux, using flatMapSequentialDelayError is intentional vs flatMapSequential.

In my understanding of flatMapConcat it guarantees that the element you process will be in order.
If for some reason you can afford skipping element N+1 and process directly N+2 that should be intentional.

Once again, just my opinion, but as long as this is parameter, users can use it as they want.

[He-Pin]

@gael-ft Yes, let's just keep it simple for now. And I saw the FailedSource which will failStage when preStartI().
And the FutureSource only trigger the failStage after be pulled. And in your code you are using the FailedSource which will cause the stream fails very soon, So I think I need adjust my code for that too.

[He-Pin]

Need take care for the upstream where fails immeditly

[johanandren]

Random is good for trying it out while developing but means that we'd have no idea what value it fails for if it fails. Parameterize over a few values or use a single random chosen and logged on test class instantiation instead, so failures can be repeated.

[He-Pin]

@johanandren How about I spliting the optimization to a dedicated PR from this one, I think this PR is too big

[johanandren]

Yes, I think that was an initial feedback from me, although I also said it might be fine.