gax-java: Flaky BatcherImplTest.testThrottlingBlocking - Zero interactions with this mock #1193

burkedavison · 2022-12-21T21:11:39Z

The below test failed and then passed on rerun.

Error:  Failures: 
Error:    BatcherImplTest.testThrottlingBlocking:897 
Wanted but not invoked:
apiCallContext.withOption(
    <Capturing argument>,
    <Capturing argument>
);
-> at com.google.api.gax.batching.BatcherImplTest.testThrottlingBlocking(BatcherImplTest.java:897)
Actually, there were zero interactions with this mock.

The text was updated successfully, but these errors were encountered:

meltsufin · 2022-12-22T03:27:31Z

There's been many attempts to fix it with no luck: googleapis/gax-java#1615.

blakeli0 · 2022-12-22T04:45:47Z

@burkedavison How often did you see this? I haven't seen it in a while, like Mike mentioned, there has been some attempts and it should be less frequent now. If this still happens pretty often, we should prioritize it again.

burkedavison · 2022-12-22T14:30:19Z

So far, it's happened only once.

blakeli0 · 2022-12-27T23:15:13Z

Noticed a different flaky tests in ScheduledRetryingExecutorTest.testCancelOuterFutureBeforeStart

Error:  Tests run: 30, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.38 s <<< FAILURE! - in com.google.api.gax.retrying.ScheduledRetryingExecutorTest
Error:  com.google.api.gax.retrying.ScheduledRetryingExecutorTest.testCancelOuterFutureBeforeStart[Custom retry settings: false]  Time elapsed: 0.023 s  <<< FAILURE!
org.mockito.exceptions.verification.NoInteractionsWanted: 

No interactions wanted here:
-> at com.google.api.gax.retrying.AbstractRetryingExecutorTest.testCancelOuterFutureBeforeStart(AbstractRetryingExecutorTest.java:247)
But found this interaction on mock 'tracer':
-> at com.google.api.gax.retrying.FailingCallable.call(FailingCallable.java:85)
Actually, above is the only interaction with this mock.
	at com.google.api.gax.retrying.AbstractRetryingExecutorTest.testCancelOuterFutureBeforeStart(AbstractRetryingExecutorTest.java:247)

See the exact log here

blakeli0 · 2023-01-06T19:03:11Z

Noticed another occurrence of this flaky test.

burkedavison · 2023-01-17T14:29:35Z

Another, different flaky test in HttpJsonDirectServerStreamingCallableTest.testServerStreaming

Error:  com.google.api.gax.httpjson.HttpJsonDirectServerStreamingCallableTest.testServerStreaming  Time elapsed: 10.068 s  <<< FAILURE!
value of: await(...)
expected to be true
	at com.google.api.gax.httpjson.HttpJsonDirectServerStreamingCallableTest.testServerStreaming(HttpJsonDirectServerStreamingCallableTest.java:211)

Build log

burkedavison · 2023-01-18T19:00:35Z

Noticed another occurrence of this flaky test.

burkedavison · 2023-01-19T13:49:15Z

Another flake.

burkedavison · 2023-01-24T19:17:14Z

Another, different flaky test in AbstractRetryingExecutorTest.testCancelOuterFutureBeforeStart

Error:  Failures: 
Error:    ScheduledRetryingExecutorTest>AbstractRetryingExecutorTest.testCancelOuterFutureBeforeStart:247 
No interactions wanted here:
-> at com.google.api.gax.retrying.AbstractRetryingExecutorTest.testCancelOuterFutureBeforeStart(AbstractRetryingExecutorTest.java:247)
But found this interaction on mock 'tracer':
-> at com.google.api.gax.retrying.FailingCallable.call(FailingCallable.java:85)
Actually, above is the only interaction with this mock.

Build log

burkedavison · 2023-01-24T19:54:56Z

Another flake.

burkedavison · 2023-02-02T14:49:32Z

Flaked last night on Mac

burkedavison · 2023-02-13T14:41:45Z

Flaked on nightly Ubuntu

burkedavison · 2023-03-01T19:24:44Z

Flake on Mac Java 11 and Java 17

burkedavison · 2023-03-03T18:43:56Z

Flake: #1432

blakeli0 · 2023-03-31T20:02:39Z

Two more flaky tests on nightly Mac tests: #1566 and #1579

alicejli · 2023-04-07T14:52:19Z

Java 17 on macOS-12 failed: #1614

Error:  Tests run: 25, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 39.922 s <<< FAILURE! - in com.google.api.gax.batching.BatcherImplTest
Error:  com.google.api.gax.batching.BatcherImplTest.testThrottlingBlocking  Time elapsed: 0.357 s  <<< FAILURE!
adding elements to batcher should be blocked by FlowControlled
	at com.google.api.gax.batching.BatcherImplTest.testThrottlingBlocking(BatcherImplTest.java:885)

alicejli · 2023-04-11T13:58:10Z

Nightly build for Java 11 on macOS-12 failed. #1620

Error:  Failures: 
Error:    BatcherImplTest.testThrottlingBlocking:885 adding elements to batcher should be blocked by FlowControlled
[INFO] 
Error:  Tests run: 530, Failures: 1, Errors: 0, Skipped: 0

#1620

alicejli · 2023-04-11T14:28:23Z

Nightly build for Java 17 on macOS-12 failed. #1619

Error:  Failures: 
Error:    BatcherImplTest.testThrottlingBlocking:898 
Wanted but not invoked:
apiCallContext.withOption(
    <Capturing argument>,
    <Capturing argument>
);
-> at com.google.api.gax.batching.BatcherImplTest.testThrottlingBlocking(BatcherImplTest.java:898)
Actually, there were zero interactions with this mock.

#1619

alicejli · 2023-04-14T19:17:21Z

Nightly build for Java 11 on ubuntu-22.04 failed. #1626

Error:  com.google.api.gax.batching.BatcherImplTest.testThrottlingBlocking  Time elapsed: 0.094 s  <<< FAILURE!
Wanted but not invoked:
apiCallContext.withOption(
    <Capturing argument>,
    <Capturing argument>
);
-> at com.google.api.gax.batching.BatcherImplTest.testThrottlingBlocking(BatcherImplTest.java:898)
Actually, there were zero interactions with this mock.

	at com.google.api.gax.batching.BatcherImplTest.testThrottlingBlocking(BatcherImplTest.java:898)

#1626

alicejli · 2023-07-11T15:39:44Z

Failed on Java8 for Windows:

Error:  Failures: 
Error:    BatcherImplTest.testThrottlingBlocking:899 
apiCallContext.withOption(
    <Capturing argument>,
    <Capturing argument>
);
Wanted 1 time:
-> at com.google.api.gax.batching.BatcherImplTest.testThrottlingBlocking(BatcherImplTest.java:899)
But was 0 times.

#1838

alicejli · 2023-09-06T18:50:19Z

Failed on Java8 for Windows:

Error:  Failures: 
Error:    BatcherImplTest.testThrottlingBlocking:899 
apiCallContext.withOption(
    <Capturing argument>,
    <Capturing argument>
);
Wanted 1 time:
-> at com.google.api.gax.batching.BatcherImplTest.testThrottlingBlocking(BatcherImplTest.java:899)
But was 0 times.

[INFO] 
Error:  Tests run: 552, Failures: 1, Errors: 0, Skipped: 0

alicejli · 2023-09-06T19:00:45Z

#1957
Failed on Java 11 for Windows:

Error:  Failures: 
Error:    BatcherImplTest.testThrottlingBlocking:899 
Wanted but not invoked:
apiCallContext.withOption(
    <Capturing argument>,
    <Capturing argument>
);
-> at com.google.api.gax.batching.BatcherImplTest.testThrottlingBlocking(BatcherImplTest.java:899)
Actually, there were zero interactions with this mock.

alicejli · 2023-09-06T19:04:24Z

#1955
Fialed on Java 8 for Windows:

Error:  Failures: 
Error:    BatcherImplTest.testThrottlingBlocking:899 
apiCallContext.withOption(
    <Capturing argument>,
    <Capturing argument>
);
Wanted 1 time:
-> at com.google.api.gax.batching.BatcherImplTest.testThrottlingBlocking(BatcherImplTest.java:899)
But was 0 times.

suztomo · 2023-09-13T03:13:22Z

Reproduce the issue

Install the required modules locally

~/sdk-platform-java $ mvn -am -pl gax-java/gax install

IntelliJ's repeat test N times (This didn't work)

I tried to run the test 100 times and it failed at 2nd run.

java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@497486b3[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@736caf7a[Wrapped task = com.google.api.gax.batching.BatcherImpl$PushCurrentBatchRunnable@4ce7fffa]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@63f8276e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 74]

	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:680)
	at java.base/java.util.concurrent.Executors$DelegatedScheduledExecutorService.scheduleWithFixedDelay(Executors.java:788)
	at com.google.api.gax.batching.BatcherImpl.<init>(BatcherImpl.java:206)
	at com.google.api.gax.batching.BatcherImplTest.testThrottlingBlocking(BatcherImplTest.java:855)

Maybe the test method is not meant to run twice on the same instance.

Repeat 100 times with shell script (It didn't work)

at sdk-platform-java repository clone directory:

$ repeat 100 mvn -pl gax-java/gax test -Dtest="BatcherImplTest#testThrottlingBlocking" -Dfmt.skip

It all succeeded, unfortunately. I think the sequential execution nature does not work to reproduce the issue.

Run the tests concurrently (It worked)

I created a project to run the test concurrently.

https://github.com/suztomo/gax-batcher-impl-test-reproducer/blob/main/README.md

I'm now able to reproduce the problem.

Understand the test

Next step: read the test code.

Batcher: "Represents a batching context where individual elements will be accumulated and flushed in a large batch request at some point in the future. The buffered elements can be flushed manually or when triggered by an internal threshold. This is intended to be used for high throughput scenarios at the cost of latency. Batcher instances are not thread safe. To use across different threads, create a new Batcher instance per thread."
```
new BatcherImpl<>(
          SQUARER_BATCHING_DESC_V2,
          callLabeledIntSquarer,
          labeledIntList,
          settings,
          EXECUTOR,
          flowController,
          callContext)
```

FlowController: "Provides flow control capability."

  FlowController flowController =
      new FlowController(
          FlowControlSettings.newBuilder()
              .setLimitExceededBehavior(LimitExceededBehavior.Block)
              .setMaxOutstandingElementCount(1L)
              .build());

Next step: Where should callContext.withOption(key, value) be supposed to be called?

It seems within BatcherImpl.sendOutstanding()

sdk-platform-java/gax-java/gax/src/main/java/com/google/api/gax/batching/BatcherImpl.java

Lines 288 to 292 in 4c74107

    
           ApiCallContext callContextWithOption = null; 
        
           if (callContext != null) { 
        
             callContextWithOption = 
        
                 callContext.withOption(THROTTLED_TIME_KEY, accumulatedBatch.totalThrottledTimeMs); 
        
           }

. BatcherImpl$PushCurrentBatchRunnable's run() method calls sendOutstanding(). PushCurrentBatchRunnable is submitted in BatcherImpl's constructor:

      scheduledFuture =
          executor.scheduleWithFixedDelay(runnable, delay, delay, TimeUnit.MILLISECONDS);

With debug logging, logback, and jul-to-slf4j bridge (776eb1e), I got the following logging output (a normal successful invocation):

09:51:52.154 [main] INFO  c.g.api.gax.batching.BatcherImplTest -- Starting testThrottlingBlocking
09:51:53.089 [pool-2-thread-1] DEBUG c.g.api.gax.batching.BatcherImplTest -- calling batcher.add(1)
09:51:53.113 [main] DEBUG c.g.api.gax.batching.BatcherImplTest -- Caught TimeoutException as expected
09:51:53.113 [main] DEBUG c.g.api.gax.batching.BatcherImplTest -- calling future.get(3, TimeUnit.SECONDS)
09:51:53.152 [pool-2-thread-2] DEBUG c.g.api.gax.batching.BatcherImplTest -- calling flowController.release()
09:51:53.153 [pool-2-thread-2] DEBUG c.g.api.gax.batching.BatcherImplTest -- finished flowController.release()
09:51:53.175 [pool-2-thread-1] DEBUG c.g.api.gax.batching.BatcherImplTest -- batcher.add(1) finished
09:51:53.176 [main] DEBUG c.g.api.gax.batching.BatcherImplTest -- finished future.get(3, TimeUnit.SECONDS)
09:51:53.186 [pool-1-thread-1] DEBUG c.g.api.gax.batching.BatcherImpl -- callContext.withOption called
09:51:53.193 [main] DEBUG c.g.api.gax.batching.BatcherImplTest -- Mockito.verify(callContext, ...) succeeded
09:51:53.215 [main] INFO  c.g.api.gax.batching.BatcherImplTest -- Finishing testThrottlingBlocking

There are three threads: main, pool-1-thread-1, pool-2-thread-1, and pool-2-thread-2.

BatcherImpl$PushCurrentBatchRunnable uses pool-1-thread-1. While it tries to check the need of callContext.withOption every 1 millisecond, there's no guarantee that the event on the pool-1-thread-1 thread runs before main thread's Mockito.verify(callContext, Mockito.timeout.. operation.

Simply increasing the timeout value in Mockito.verify(callContext, Mockito.timeout...) from 100 to 1000 resulted in another error:

[ERROR] Failures: 
[ERROR]   BatcherImplTest.testThrottlingBlocking:903 expected to be at least: 50
but was                : 48

What does that callContext.withOptions(total_throttled_time, 48) mean?

The Javadoc tells it's "for tracking batch total throttled time".

sdk-platform-java/gax-java/gax/src/main/java/com/google/api/gax/batching/BatcherImpl.java

Line 291 in 776eb1e

    
           callContext.withOption(THROTTLED_TIME_KEY, accumulatedBatch.totalThrottledTimeMs);

It's how much FlowController blocked.

While it got 10 milliseconds lag, there's no guarantee that batcher.add is called before Thread.sleep(throttledTime).

02:49:56.358 [gax-test06][main] INFO  c.g.api.gax.batching.BatcherImplTest -- Starting testThrottlingBlocking
02:49:59.283 [gax-test06][pool-3-thread-1] DEBUG c.g.api.gax.batching.BatcherImplTest -- calling batcher.add(1)
02:49:59.287 [gax-test06][pool-3-thread-2] DEBUG c.g.api.gax.batching.BatcherImplTest -- start sleeping 50 ms
02:49:59.293 [gax-test06][pool-3-thread-1] DEBUG c.g.api.gax.batching.BatcherImpl -- stopwatch started
02:49:59.297 [gax-test06][main] DEBUG c.g.api.gax.batching.BatcherImplTest -- Caught TimeoutException as expected
02:49:59.297 [gax-test06][main] DEBUG c.g.api.gax.batching.BatcherImplTest -- calling future.get(3, TimeUnit.SECONDS)
02:49:59.337 [gax-test06][pool-3-thread-2] DEBUG c.g.api.gax.batching.BatcherImplTest -- Sleep finished. Calling flowController.release()
02:49:59.338 [gax-test06][pool-3-thread-2] DEBUG c.g.api.gax.batching.BatcherImplTest -- finished flowController.release()
02:49:59.340 [gax-test06][pool-3-thread-1] DEBUG c.g.api.gax.batching.BatcherImpl -- stopwatch.elapsed (throttledTimeMs): 48
02:49:59.355 [gax-test06][pool-3-thread-1] DEBUG c.g.api.gax.batching.BatcherImpl -- add() called with throttledTimeMs= 48 (now totalThrottledTimeMs: 48)
02:49:59.364 [gax-test06][pool-2-thread-1] DEBUG c.g.api.gax.batching.BatcherImpl -- callContext.withOption called
02:49:59.371 [gax-test06][pool-3-thread-1] DEBUG c.g.api.gax.batching.BatcherImplTest -- batcher.add(1) finished
02:49:59.376 [gax-test06][main] DEBUG c.g.api.gax.batching.BatcherImplTest -- finished future.get(3, TimeUnit.SECONDS)
02:49:59.391 [gax-test06][main] DEBUG c.g.api.gax.batching.BatcherImplTest -- Mockito.verify(callContext, ...) succeeded
[ERROR] com.google.api.gax.batching.BatcherImplTest.testThrottlingBlocking_gax_test06 -- Time elapsed: 5.138 s <<< FAILURE!
expected to be at least: 50
but was                : 48
	at com.google.api.gax.batching.BatcherImplTest.testThrottlingBlocking_gax_test06(BatcherImplTest.java:926)

When the sleeping(50 ms) before flowController.release starts earlier than
BatcherImpl's stopwatch start time, the recorded throttledTime may
be shorter than throttledTime.

Normal case:

throttle start (stopwatch start)
sleep(50 ms) starts
sleep(50 ms) finishes and calls flowController.release
stopwatch finishes, recording more than or equal to 50 ms.

Rare case:

sleep(50 ms) starts
throttle start (stopwatch start)
sleep(50 ms) finishes and calls flowController.release
stopwatch finishes, recording less than 50 ms. assertThat(value.getValue()).isAtLeast(throttledTime) fails.

To avoid such bad operation order, we can make the "sleep(50 ms)" start only after the thread of "throttle start (stopwatch start)" becomes WAITING state:

      List<Thread> batcherAddThreadHolder = Collections.synchronizedList(new ArrayList<>());
      Future future =
          executor.submit(
              new Runnable() {
                Thread batcherAddThread;
                @Override
                public void run() {
                  batcherAddThreadHolder.add(Thread.currentThread());
                  logger.fine("calling batcher.add(1)");
                  batcher.add(1);
                  logger.fine("batcher.add(1) finished");
                }
              });
      do {
        // Wait until batcher.add blocks and starts the stopwatch.
        Thread.sleep(10);
      } while (batcherAddThreadHolder.isEmpty() || batcherAddThreadHolder.get(0).getState() != Thread.State.WAITING);

Fixes googleapis#1193

At the end of the test, it verifies that the batcher waited more than throttled time by capturing the argument of `callContext.withOption()`. There were two problems: # "Zero interactions with this mock" problem ``` Error: Failures: Error: BatcherImplTest.testThrottlingBlocking:899 apiCallContext.withOption( <Capturing argument>, <Capturing argument> ); Wanted 1 time: -> at com.google.api.gax.batching.BatcherImplTest.testThrottlingBlocking(BatcherImplTest.java:899) But was 0 times. ``` The callContext.withOption method is called by `BatcherImpl.sendOutstanding()` method via `BatcherImpl$PushCurrentBatchRunnable` in **another thread**. https://togithub.com/googleapis/sdk-platform-java/blob/4c741077d614093d08665e9ddd83fb0e332b7881/gax-java/gax/src/main/java/com/google/api/gax/batching/BatcherImpl.java#L206 Technically, there's no guarantee that the thread calls the withOption method within a certain timeframe, especially when we run the tests concurrently in a machine with many CPU cores (the nightly test setup). That's why we occasionally saw "Zero interactions with this mock" problem. In my experiment (https://togithub.com/suztomo/gax-batcher-impl-test-reproducer/blob/main/README.md), increasing 100 ms timeout to 1000 ms just worked good to prevent false positives. The assertion is not about timing, but about BatcherImpl recording the throttled time correctly. After fixing this problem, then I saw another problem below. # "expected to be at least: 50 but was: 48" problem ``` [ERROR] com.google.api.gax.batching.BatcherImplTest.testThrottlingBlocking_gax_test06 -- Time elapsed: 5.138 s <<< FAILURE! expected to be at least: 50 but was : 48 at com.google.api.gax.batching.BatcherImplTest.testThrottlingBlocking_gax_test06(BatcherImplTest.java:926) ``` At the end of the test, it verifies that the batcher waited more than throttled time by capturing the argument of `callContext.withOption()`. In ideal, usual scenarios, 1. In thread A, the batcher blocks at `batcher.add` (because of flowController) the thread and starts stopwatch to measure throttled time. 2. In thread B (created by the BatcherImplTest) sleeps 50 ms (`throttledTime`) 3. In thread B, BatcherImplTest calls `flowController.release()`, which wakes up the thread A. 4. The batcher records it was blocked more than 50 ms. However, in rare cases, these events happens in this order: 1. Thread B starts sleeping 50 ms. 2. In thread A the batcher blocks at `batcher.add` (because of flowController) and starts stopwatch to measure throttled time. 3. In thread B, after 50 ms sleep, BatcherImplTest calls `flowController.release()`, which wakes up the thread A. 5. The batcher records it was blocked **less than** 50 ms. This makes the test assertion fail with "_expected to be at least: 50 but was: 48_" message. To prevent this rare case, BatcherImplTest needs to wait before sleeping 50 ms in thread B. In this pull request, I add a do-while loop to check the thread A (the thread of `batcher.add`) is in Thread.State.WAITING state before making the sleeping of 50 ms. - [x] Make sure to open an issue as a [bug/issue](https://togithub.com/googleapis/gapic-generator-java/issues/new/choose) before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea - [x] Ensure the tests and linter pass - [x] Code coverage does not decrease (if any source code was changed) - [x] Appropriate docs were updated (if necessary) Fixes #1193 ☕️

burkedavison added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. priority: p2 Moderately-important priority. Fix may not be included in next release. labels Dec 21, 2022

alicejli self-assigned this Mar 22, 2023

emmileaf mentioned this issue Mar 27, 2023

Nightly build for Java 11 on macOS-12 failed. #1563

Closed

This was referenced Mar 31, 2023

Nightly build for Java 11 on macOS-12 failed. #1566

Closed

Nightly build for Java 11 on macOS-12 failed. #1579

Closed

Nightly build for Java 11 on macOS-12 failed. #1598

Closed

alicejli mentioned this issue Apr 7, 2023

Nightly build for Java 17 on macOS-12 failed. #1614

Closed

alicejli mentioned this issue Apr 11, 2023

Nightly build for Java 11 on macOS-12 failed. #1620

Closed

alicejli mentioned this issue Apr 11, 2023

Nightly build for Java 17 on macOS-12 failed. #1619

Closed

alicejli mentioned this issue Apr 14, 2023

Nightly build for Java 11 on ubuntu-22.04 failed. #1626

Closed

This was referenced Jul 4, 2023

Nightly build for Java 17 on windows-2022 failed. #1826

Closed

Nightly-java8 build on windows-2022 failed. #1824

Closed

Nightly-java8 build on windows-2022 failed. #1827

Closed

alicejli mentioned this issue Jul 11, 2023

Nightly-java8 build on windows-2022 failed. #1838

Closed

This was referenced Aug 31, 2023

Nightly build for Java 17 on macOS-12 failed. #1952

Closed

Nightly build for Java 17 on windows-2022 failed. #1945

Closed

Nightly-java8 build on windows-2022 failed. #1944

Closed

alicejli mentioned this issue Sep 6, 2023

Nightly-java8 build on windows-2022 failed. #1958

Closed

alicejli mentioned this issue Sep 6, 2023

Nightly build for Java 11 on windows-2022 failed. #1957

Closed

alicejli mentioned this issue Sep 6, 2023

Nightly-java8 build on windows-2022 failed. #1955

Closed

lqiu96 mentioned this issue Sep 7, 2023

gax-java(windows): BatcherImplTest.testThrottlingBlocking - Zero interactions with this mock #1967

Closed

lqiu96 changed the title ~~gax-java: Flaky BatcherImplTest.testThrottlingBlocking~~ gax-java: Flaky BatcherImplTest.testThrottlingBlocking - Zero interactions with this mock Sep 7, 2023

suztomo self-assigned this Sep 13, 2023

suztomo added a commit to suztomo/gapic-generator-java that referenced this issue Sep 18, 2023

test: fix concurrency issue in testThrottlingBlocking

339dc58

Fixes googleapis#1193

suztomo mentioned this issue Sep 18, 2023

test: fix concurrency issue in testThrottlingBlocking #2001

Merged

4 tasks

gcf-merge-on-green bot closed this as completed in #2001 Sep 18, 2023

lqiu96 mentioned this issue Sep 18, 2023

gax-java (mac): BatcherImplTest.testThrottlingBlocking- zero interactions with this mock #1813

Closed

suztomo mentioned this issue Sep 18, 2023

gax-java(mac-os): BatcherImplTest.testThrottlingBlocking- adding elements to batcher should be blocked by FlowControlled #1931

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gax-java: Flaky BatcherImplTest.testThrottlingBlocking - Zero interactions with this mock #1193

gax-java: Flaky BatcherImplTest.testThrottlingBlocking - Zero interactions with this mock #1193

burkedavison commented Dec 21, 2022

meltsufin commented Dec 22, 2022

blakeli0 commented Dec 22, 2022

burkedavison commented Dec 22, 2022

blakeli0 commented Dec 27, 2022

blakeli0 commented Jan 6, 2023

burkedavison commented Jan 17, 2023 •

edited

burkedavison commented Jan 18, 2023

burkedavison commented Jan 19, 2023

burkedavison commented Jan 24, 2023

burkedavison commented Jan 24, 2023

burkedavison commented Feb 2, 2023

burkedavison commented Feb 13, 2023

burkedavison commented Mar 1, 2023 •

edited

burkedavison commented Mar 3, 2023

blakeli0 commented Mar 31, 2023

alicejli commented Apr 7, 2023

alicejli commented Apr 11, 2023

alicejli commented Apr 11, 2023

alicejli commented Apr 14, 2023

alicejli commented Jul 11, 2023

alicejli commented Sep 6, 2023

alicejli commented Sep 6, 2023

alicejli commented Sep 6, 2023

suztomo commented Sep 13, 2023 •

edited

gax-java: Flaky BatcherImplTest.testThrottlingBlocking - Zero interactions with this mock #1193

gax-java: Flaky BatcherImplTest.testThrottlingBlocking - Zero interactions with this mock #1193

Comments

burkedavison commented Dec 21, 2022

meltsufin commented Dec 22, 2022

blakeli0 commented Dec 22, 2022

burkedavison commented Dec 22, 2022

blakeli0 commented Dec 27, 2022

blakeli0 commented Jan 6, 2023

burkedavison commented Jan 17, 2023 • edited

burkedavison commented Jan 18, 2023

burkedavison commented Jan 19, 2023

burkedavison commented Jan 24, 2023

burkedavison commented Jan 24, 2023

burkedavison commented Feb 2, 2023

burkedavison commented Feb 13, 2023

burkedavison commented Mar 1, 2023 • edited

burkedavison commented Mar 3, 2023

blakeli0 commented Mar 31, 2023

alicejli commented Apr 7, 2023

alicejli commented Apr 11, 2023

alicejli commented Apr 11, 2023

alicejli commented Apr 14, 2023

alicejli commented Jul 11, 2023

alicejli commented Sep 6, 2023

alicejli commented Sep 6, 2023

alicejli commented Sep 6, 2023

suztomo commented Sep 13, 2023 • edited

Reproduce the issue

IntelliJ's repeat test N times (This didn't work)

Repeat 100 times with shell script (It didn't work)

Run the tests concurrently (It worked)

Understand the test

burkedavison commented Jan 17, 2023 •

edited

burkedavison commented Mar 1, 2023 •

edited

suztomo commented Sep 13, 2023 •

edited