feat:support max concurrent workers for Reconciles

[kinderyj]

By default, the concurrent workers for Reconciles is 1, user can change the concurrent works by the function WithOptions when init the builer.Builder ("sigs.k8s.io/controller-runtime"). But Flink-operator hasn't provide user the WithOptions funtion to change the concurrent workers. In this Feature, user can change the concurrent workers when deploy the operator, for example, make deploy MAX_CONCURRENT=2.

Feel free to discuss about this feature, In your prod environment, is concurrent worker=1 enough? Did anyone do some performance test about how many QPS is supported when concurrent worker=1? Thanks~.

[functicons]

Thanks for the PR! Yep, concurrent worker=1 is sufficient in my use case, and I don't see other users complain about it. Usually we just run a handful of Flink clusters, but each cluster could be very large (1k Pods). I'd like to learn what makes you want to increase the limit and how it makes a difference in your test.

[kinderyj]

Thanks for the PR! Yep, concurrent worker=1 is sufficient in my use case, and I don't see other users complain about it. Usually we just run a handful of clusters, but each cluster could be very large (1k Pods). I'd like to learn what makes you want to increase the limit and how it makes a difference in your test.

Thanks for your response.
Actually, we also haven't run into the bottleneck. Our flink cluster(job cluster) is small(20 Pods), but many users share one K8s cluster, so the concurrent depends on how many users submit flink cluster at the same time.
We will take a look about the preformace when concurrent worker=1.