Big overhead when mocking 149 urls

[konstin]

I'm currently migrating from aiohttp/aioresponses to httpx/respx, and seeing a large regression in test times. An integration test where I mock 149 urls which took 0.5s with aioresponses now takes 2.2s with respx. build_request seems to be the major culprit. I've created a flamegraph with py-spy (full svg as gist):

Test code

@pytest.mark.asyncio
async def test_integration_json():
    with respx.mock(
        assert_all_mocked=True,
        assert_all_called=True,
    ) as respx_mock:
        for file in Path("test-data/snapshots").iterdir(): # Directory with 149 files
            respx_mock.get(
                "https://www.example.com/" + file.name.replace(".html", ""),
                content=file.read_text(),
                content_type="text/html",
            )

        # Actual test logic

[lundberg]

This is really useful findings!

You're mentioning build_request, which has been refactored/renamed to decode_request in master. There are no big changes though, we still rely on creating a httpx.Request which in the flamegraph looks to be the major thing, i.e. not respx related code.

It would be interesting to see if the flamegraph looks different with respx master and latest httpx.

[konstin]

pretty similar

[lundberg]

Yep, thats pretty much the same ;-)

It's interesting that decode_request is visible but not decode_response which is also called for each matched/mocked response. I guess httpx.Request is more expensive to instantiate than the httpx.Response.

Not sure what we can do at this moment. Caching objects for reuse in probably not an option since the request content could be a stream that we can't/shouldn't exhaust, meaning its hard to build a hash cache key. And even if possible, it's probably not being sent lots of same requests anyways.

FYI, respx works with the httpcore transports to match requests and mock responses at a lower level, for two reasons; one is to not depend on the httpx internals for easier maintainability and automatic support for future httpx client features, and secondly to allow mocking responses for other future libs using httpcore.

This is why the encode_request is used to instantiate a httpx.Request for the calls statistics. While typing this, I'm thinking we could do that lazy, meaning we actually only need to instantiate the Request on respx.calls usage in a test assertion. Would that be an option and solution to this maybe?

[lundberg]

Could you clone respx and add a return as the first line in transport.record and re-run the test to see how much of the flamegraph is affected by not instantiating the httpx.Request when recording stats.

About the matching-side, I looked at the code and we could easily move the instatiation of httpx.Request there to be done only when needed, i.e. when using callbacks.

[lundberg]

Also, please try #91 that enhances request usage in callbacks.

[konstin]

With master:

Benchmark #1: pytest tests/test_z_integration.py
  Time (mean ± σ):      2.740 s ±  0.024 s    [User: 2.806 s, System: 0.132 s]
  Range (min … max):    2.696 s …  2.776 s    10 runs

With return:

Benchmark #1: pytest tests/test_z_integration.py
  Time (mean ± σ):      2.695 s ±  0.060 s    [User: 2.775 s, System: 0.114 s]
  Range (min … max):    2.630 s …  2.831 s    10 runs

With #91:

Benchmark #1: pytest tests/test_z_integration.py
  Time (mean ± σ):     823.4 ms ±  29.0 ms    [User: 917.9 ms, System: 105.8 ms]
  Range (min … max):   795.9 ms … 884.9 ms    10 runs

That one looks much better already!

[konstin]

With #92:

Benchmark #1: pytest tests/test_z_integration.py
  Time (mean ± σ):      2.740 s ±  0.064 s    [User: 2.819 s, System: 0.118 s]
  Range (min … max):    2.624 s …  2.834 s    10 runs

[lundberg]

Super!

Will merge both PR's soon.

[lundberg]

Re-opening this issue until #92 is merged @konstin

[lundberg]

RESPX 0.14.0 now released with the fixes for this issue, among others, @konstin.

Thanks @SlavaSkvortsov for helping closing this issue.