Async and sync compatible wrapper #1036

vishes-shell · 2020-03-04T10:50:39Z

I've rebuild wrapper to support async and sync interface with support of httpx.

There is two gitlab clients: Gitlab and AsyncGitlab, initerfaces are the same, except most of the methods would return awaitable object in AsyncGitlab.

Related #1025

Make all functions async Use httpx instead of httpx Make decorators async to correctly wrap async methods

Change _http_auth on direct httpx.AsyncClient.auth property

Fix some errors that are met on the way to async

This possibly is temporary solution because httpx raises TypeError when tries to init DataField with bool value

feat: httpx based python-gitlab

on_http_error should work with sync functions that return coroutines and handle errors when coroutines will be awaited

Also provide async and sync interface to interact with GitlabList

Provide base of gitlab client along with implementations of sync and async interface

Basic principle is that we won't use constructor to create object, instead classmethod would be used that return either object or corotine that returns object

feat(async): async and sync compatible wrapper

chore: provide docstrings for gitlab client implementations

vishes-shell · 2020-03-05T16:05:52Z

Here is some sort of notable changes:

response_content no longer support chunk_size since httpx does not support this. (Allow setting chunk_size for Response.iter_bytes() etc... encode/httpx#394)
gitlab clients are moved to client.py
3 implementations of gitlab clients: gitlab.BaseGitlab abstract implementation that share common methods and store all docsrings, gitlab.Gitlab is sync implementation with httpx.Client, gitlab.AsyncGitlab is async implementation with httpx.AsyncClient.
on_http_error make nested since we can get coroutine and need to catch same error in awaiting response
RESTObject must be created (with create()), not initialized, since arguments could be coroutines, and need to be awaited, and __init__ does not support that interface
Everything should be returned, since it can be couroutin
every postprocess of response should be a function and be decorated with gitlab.utils.awaitable_postprocess, as it's done with _update_attrs and other edge cases
unit tests are run by pytest and run against async and sync client implementations
since unit tests are run with async gitlab, tests are required to be async, but fetching result is done through fixture that knows which client is used and return value if that's sync version or return await value if that's async version.
GitlabList support two versions of iteration
requests are completely removed

nejch · 2020-03-08T08:54:55Z

1. `response_content` no longer support `chunk_size` since `httpx` does not support
    this. ([encode/httpx#394](https://github.com/encode/httpx/issues/394))

I just noticed something: chunk_size is a configurable argument in several public methods (for blobs/snippet content, export download, etc) so it's possible people are using it and this would be a breaking change, right? Since you're replacing it with aiter_bytes and using chunks there, is there any way to preserve backward compatibility (I haven't checked the implementation)? Or wait for the upstream issue to resolve?

3. 3 implementations of gitlab clients: `gitlab.BaseGitlab` abstract implementation that share common methods and store all docsrings, `gitlab.Gitlab` is sync implementation with `httpx.Client`, `gitlab.AsyncGitlab` is async implementation with `httpx.AsyncClient`.
9. since unit tests are run with async gitlab, tests are required to be async, but fetching result is done through fixture that knows which client is used and `return value` if that's _sync_ version or `return await value` if that's `async` version.

That's really cool :)

max-wittig · 2020-03-08T10:42:56Z

@vishes-shell Thanks for all the work that you're doing!

People could have been using chunk_size, by providing a different requests instance, right?
(Which is also not supported anymore then?)

But we have nothing about this officially in the docs, so I would say we can solve this by mentioning it in the release notes. We anyway need to bump the version to 3.X.

nejch · 2020-03-08T11:16:43Z

@max-wittig I tried earlier and at least in some cases it can even be just:

project = gl.projects.get(1)
export = project.exports.create({})
dl = export.download(chunk_size=512)

From what I see in the current solution by @vishes-shell this would still work and wouldn't break anything, the argument would just be ignored by response_content(). But if anyone relies on chunk_size for some specific reason, it might break their behavior. I'm not sure of all the use cases but it seems useful especially for project exports: https://stackoverflow.com/questions/46205586/why-to-use-iter-content-and-chunk-size-in-python-requests/46205745

graingert · 2021-06-21T12:02:29Z

test-requirements.txt

+requests>=2.22.0
+respx>=0.12.1,<0.13
+pytest
+pytest-asyncio


can you use the anyio pytest plugin so you test on both trio and asyncio?

the latest httpx depends on anyio which bundles this plugin

darkdragon-001 · 2021-09-16T13:33:45Z

What's the current state?

darkdragon-001 · 2021-09-16T13:44:14Z

When using your library, I get the following warning:

httpx-0.18.2-py3.9.egg/httpx/_client.py:1978: UserWarning: Unclosed <httpx.AsyncClient object at 0x7ffabf61e280>. See https://www.python-httpx.org/async/#opening-and-closing-clients for details.

Apart from that, it works like a charm 😍

darkdragon-001 · 2021-09-16T15:09:21Z

Btw: chunk_size is supported by httpx now.

graingert · 2021-09-16T17:49:59Z

When using your library, I get the following warning:

httpx-0.18.2-py3.9.egg/httpx/_client.py:1978: UserWarning: Unclosed <httpx.AsyncClient object at 0x7ffabf61e280>. See https://www.python-httpx.org/async/#opening-and-closing-clients for details.

Apart from that, it works like a charm 😍

You'll need to use the client with async with AsyncGitlab() as gitlab: ...

darkdragon-001 · 2021-12-21T07:29:07Z

I am using it since a few months without any problems. chunk_size is implemented by now as well. Any other blockers? Or would it be accepted when updated to implement the latest features in the main branch?

lmilbaum · 2022-10-13T07:35:05Z

As part of onboarding myself into this project, I stumbled into this PR. Could someone provide a use case when an async wrapper is being used?

darkdragon-001 · 2022-10-13T07:59:32Z

@lmilbaum We are evaluating the jobs from the last day via script such that our QA team has a good overview about possible regressions or problems with our hardware text boxes (e.g. when the same code runs on one runner but not the other one). As network is slow, waiting for the answer of the previous request before submitting the next one takes quite a lot of time. If we can submit all requests asynchronously and just wait for all of them to finish in parallel, we have reduced the run time of our script by a multitude. Unfortunately, this MR is outdated and newer features are not available yet...

nejch · 2022-10-13T08:15:21Z

The wrapper here https://github.com/pan-net-security/aio-gitlab/ has a few points similar to what @darkdragon-001 described above. But true, this PR would need quite a bit of rework, or be used as starting point for a new one, since the library has changed a bit since then. I personally have not yet had enough need for this to work on it.

Another option would be to first switch from requests to httpx for the sync client (with a breaking change, because we're currently allowing people to bring their own requests sessions) and smooth out any kinks, then add async later, to make the transition less painful.

lmilbaum · 2022-10-13T18:41:52Z

@darkdragon-001 and @nejch thanks for your feedback. Now, I can understand better the need for an async capability.
Would it make sense to move this discussion to Discussions while closing this PR?
The first discussion point might be the migration from requests to httpx as suggested by @nejch.

nejch · 2022-10-14T09:10:29Z

@darkdragon-001 and @nejch thanks for your feedback. Now, I can understand better the need for an async capability. Would it make sense to move this discussion to Discussions while closing this PR? The first discussion point might be the migration from requests to httpx as suggested by @nejch.

@lmilbaum thanks for looking into this! I'd keep this PR open even if as draft, just so it stays on our radar as there's a lot of useful discussion and work done here. And just so we remember to credit @vishes-shell as author whether they want to still work on this or not.

I'd maybe open a new issue to plan the switch to httpx as that would be for v4.0.0. We could then get rid of types-requests and requests-toolbelt I assume, as httpx is fully typed and does multipart encoding well IIRC.

vishes-shell and others added 30 commits February 22, 2020 15:35

feat: replace requests with httpx

791de22

Make all functions async Use httpx instead of httpx Make decorators async to correctly wrap async methods

test: create TestGitlabList for async way

fb90a89

test: add pytest test dependency and remove sync TestGitlabList

3b90d09

test: refactor TestGitlabHttpMethods to async way

d9d9af8

test: fix TestGitlabAuth

c2614e2

Change _http_auth on direct httpx.AsyncClient.auth property

test: mostly transfer TestGitlab to async path

38183a6

Fix some errors that are met on the way to async

fix: remove pickability for GitlabClient

882cc2c

test: transfer all test_mixins to async

f694b20

test: rework application tests in async way

a8ef796

test: rework tests on projects snippets in async way

4352f9b

fix: response content is not async

37dbc09

fix: add missing awaits in v4 objects

1563b7d

fix: use overwrite as string in import project method

f933cd0

This possibly is temporary solution because httpx raises TypeError when tries to init DataField with bool value

test: add integration tests in async way

efb1424

test: change tox to pytest and keep integration test file name

1bf2968

Merge pull request #1 from vishes-shell/feature/httpx-based

cf8e233

feat: httpx based python-gitlab

feat(async): Make on_http_error decorator work with coroutines

0591787

on_http_error should work with sync functions that return coroutines and handle errors when coroutines will be awaited

feat(async): roll back sync cli wrapper

55f8439

feat(async): move GitlabList to types

0585d6a

Also provide async and sync interface to interact with GitlabList

feat(sync): implement sync and async gitlab clients

4528543

Provide base of gitlab client along with implementations of sync and async interface

feat(async): Provide httpx client classes to gitlab clients

c4117e1

feat(async): fullfill RESTObjectList with sync and async methods

e43a010

feat(async): clean __init__.py

46c0369

feat(async): make base object classes async compatible

933b781

Basic principle is that we won't use constructor to create object, instead classmethod would be used that return either object or corotine that returns object

feat(async): make mixins async/sync compatible

47b41d5

feat(async): fixup some errors in async implementation

a14383b

test: fix all tests that are async

34666c7

test: parametrize test_gitlab to handle both async and sync clients

ebc8327

test: test_gitlab is now sync agnostic

a2004ff

test: fix test_gitlab and move to pytest

3554a04

vishes-shell and others added 3 commits March 4, 2020 13:15

fix(async): fix typos in code

608668c

test: remove debugging from functional tests

0b4b4e4

Merge pull request #3 from vishes-shell/feature/sync-async-compatible

d8f3dec

feat(async): async and sync compatible wrapper

vishes-shell mentioned this pull request Mar 4, 2020

async support & httpx-based python-gitlab #1025

Open

vishes-shell and others added 2 commits March 5, 2020 18:19

chore: provide docstrings for gitlab client implementations

2ee7f37

Merge pull request #5 from vishes-shell/chore/client-dry-docstrings

944ee65

chore: provide docstrings for gitlab client implementations

max-wittig mentioned this pull request Mar 8, 2020

[WIP] Switch project import to requests-toolbelt #1004

Closed

florimondmanca mentioned this pull request Mar 12, 2020

Allow setting chunk_size for Response.iter_bytes() etc... encode/httpx#394

Closed

vishes-shell added 5 commits September 14, 2020 11:51

Bump httpx and respx

5eea6ae

Bump httpx version in setup.py

8ba7759

Install requires with package, not just version

8ca0521

Import unresolved exception classes in gitlab client

f94dba1

Bump httpx

fc824c2

graingert reviewed Jun 21, 2021

View reviewed changes

nejch mentioned this pull request Nov 26, 2021

Api return 404 when keyword contains point symbol #1715

Closed

lmilbaum mentioned this pull request Oct 21, 2022

feat: Initial support for httpx #2336

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Async and sync compatible wrapper #1036

Async and sync compatible wrapper #1036

vishes-shell commented Mar 4, 2020 •

edited

vishes-shell commented Mar 5, 2020 •

edited

nejch commented Mar 8, 2020

max-wittig commented Mar 8, 2020

nejch commented Mar 8, 2020

graingert Jun 21, 2021 •

edited

darkdragon-001 commented Sep 16, 2021 •

edited

darkdragon-001 commented Sep 16, 2021 •

edited

darkdragon-001 commented Sep 16, 2021

graingert commented Sep 16, 2021

darkdragon-001 commented Dec 21, 2021

lmilbaum commented Oct 13, 2022

darkdragon-001 commented Oct 13, 2022

nejch commented Oct 13, 2022

lmilbaum commented Oct 13, 2022

nejch commented Oct 14, 2022

Async and sync compatible wrapper #1036

Are you sure you want to change the base?

Async and sync compatible wrapper #1036

Conversation

vishes-shell commented Mar 4, 2020 • edited

vishes-shell commented Mar 5, 2020 • edited

nejch commented Mar 8, 2020

max-wittig commented Mar 8, 2020

nejch commented Mar 8, 2020

graingert Jun 21, 2021 • edited

Choose a reason for hiding this comment

darkdragon-001 commented Sep 16, 2021 • edited

darkdragon-001 commented Sep 16, 2021 • edited

darkdragon-001 commented Sep 16, 2021

graingert commented Sep 16, 2021

darkdragon-001 commented Dec 21, 2021

lmilbaum commented Oct 13, 2022

darkdragon-001 commented Oct 13, 2022

nejch commented Oct 13, 2022

lmilbaum commented Oct 13, 2022

nejch commented Oct 14, 2022

vishes-shell commented Mar 4, 2020 •

edited

vishes-shell commented Mar 5, 2020 •

edited

graingert Jun 21, 2021 •

edited

darkdragon-001 commented Sep 16, 2021 •

edited

darkdragon-001 commented Sep 16, 2021 •

edited