refactor: HTTPClient Re-Core #1057

alentoghostflame · 2023-05-13T03:36:26Z

Summary

Massively reworks the core of HTTP, adding support for:

Multiple authorization tokens/types with the auth kwarg, opening the way for OAuth2 extensions,
Global ratelimits per authorization, to stay within the (default) 50 requests per second per auth maximum,
Proper rate limit bucket handling, allowing multiple requests in a single bucket to run simultaneously, unlike the previous implementation,
The retry_request kwarg, allowing the user to disable auto-retries on ratelimit or Discord server errors.

Due to restructuring, there HAS been some breaking changes.

Documentation and comments will likely need to be updated across the board.
MaybeUnlock has been removed and replaced with a completely different RateLimit.
HTTPClient.request() will no longer overwrite specific provided headers, allowing you, for example, to override the auth and user-agent.
Due to HTTPClient no longer being centered around a single token, the HTTPClient.token attribute has been removed and Client._token has been added.
- This can be added back of course, but hmm.
While HTTPClient may technically not be documented, almost all of its attributes were not _'d, and thus were technically public. Since basically all of those attributes are gone/renamed, definitely a breaking change there,
HTTPClient.static_login() was modified to better fit with the rework of HTTPClient.

Notes:

Webhook and HTTPClient had their own rate limiting implementations before this PR, and with this PR still do. To avoid more delays, migrating webhooks to use HTTPClient is not in this PR.

Checklist

If code changes were made then they have been tested.
- I have updated the documentation to reflect the changes.
- I have run task pyright and fixed the relevant issues.
This PR fixes an issue.
This PR adds something new (e.g. new method or parameters).
This PR is a breaking change (e.g. methods or parameters removed/renamed)
This PR is not a code change (e.g. documentation, README, ...)

Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

… raising errors instead. Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

# Conflicts: # nextcord/http.py

MaskDuck · 2023-05-13T03:56:13Z

I demand a rename of the title

alentoghostflame · 2023-05-13T05:27:16Z

oh shoot I knew I forgot something

nextcord/http.py

…ting, adds RateLimits erroring if a bucket update unintended for them is given, now resets the bucket reset timer when pessimistic RateLimit.reset_after is updated, fixes edge cases where the RateLimit reset timer is never started, adds and changes logging messages, *_url_rate_limit() and *_url_deny() methods now take Route instead of path, improved behavior of bucket changes when using asyncio.gather(). Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

nextcord/http.py

…eLimit docstring, cleans up (Global)RateLimit checking if the global ratelimit was hit. Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

nextcord/http.py

Skelmis · 2023-05-24T09:06:54Z

Regardless I also think that kind of information is too high level / frequent for info

…ct most of token in logging when possible. Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

… of token when possible. Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

…mit time offsets, readds dispatching http_ratelimit. Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

nextcord/http.py

Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

nextcord/http.py

Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

Signed-off-by: Alex <alex@darkkingdom.lan>

… to a private function Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

nextcord/http.py

Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

EmreTech

Few more suggestions left, sorry 😭
It's all documentation stuff at this point tho

docs/api.rst

nextcord/http.py

Co-authored-by: Emre Terzioglu <50607143+EmreTech@users.noreply.github.com>

… docstring to a comment above, add real docstring Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

outdated

ooliver1

Tested by others is good 👍, merging since 3.0 is in development anyway and this is a large change.

alentoghostflame and others added 5 commits May 12, 2023 22:18

Initial Commit

a9586be

Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

style: auto fixes from pre-commit hooks

bd8f58b

Removes some commented code, attempts to fix some pyright concerns by…

179d075

… raising errors instead. Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

Merge remote-tracking branch 'origin/http_rework' into http_rework

6084c43

# Conflicts: # nextcord/http.py

style: auto fixes from pre-commit hooks

316e5ab

alentoghostflame changed the title ~~Initial Commit~~ HTTPClient Re-Core May 13, 2023

Skelmis reviewed May 14, 2023

View reviewed changes

alentoghostflame and others added 2 commits May 15, 2023 23:03

style: auto fixes from pre-commit hooks

d8d17d3

ooliver1 reviewed May 16, 2023

View reviewed changes

nextcord/http.py Outdated Show resolved Hide resolved

nextcord/http.py Outdated Show resolved Hide resolved

nextcord/http.py Outdated Show resolved Hide resolved

nextcord/http.py Outdated Show resolved Hide resolved

alentoghostflame and others added 2 commits May 17, 2023 20:30

Downgrades RateLimit reset timer increase logging to debug, moves Rat…

d373052

…eLimit docstring, cleans up (Global)RateLimit checking if the global ratelimit was hit. Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

style: auto fixes from pre-commit hooks

5a6583a

Skelmis requested changes May 24, 2023

View reviewed changes

nextcord/http.py Outdated Show resolved Hide resolved

nextcord/http.py Outdated Show resolved Hide resolved

alentoghostflame and others added 5 commits May 24, 2023 08:09

Lowers some logging messages to DEBUG at Skelmis's request, will reda…

2c0ed9c

…ct most of token in logging when possible. Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

style: auto fixes from pre-commit hooks

de13726

Changes bucket when creating GlobalRateLimit to use the redacted form…

8c6d17c

… of token when possible. Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

Changes bucket depletion logging to DEBUG, now correctly sets rate li…

c8af071

…mit time offsets, readds dispatching http_ratelimit. Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

style: auto fixes from pre-commit hooks

8eb30fa

EmreTech added s: in progress Status: the issue or PR is in development/progress p: low Priority: low - not important to be worked on t: refactor Type: refactor - this is a code change but does not fix a bug/add features labels Jun 2, 2023

alentoghostflame and others added 2 commits June 6, 2023 22:39

Adds exchange_access_code, get_current_user, auth kwarg to request.

0c425a8

Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

style: auto fixes from pre-commit hooks

c472968

github-advanced-security bot found potential problems Jun 7, 2023

View reviewed changes

nextcord/http.py Fixed Show fixed Hide fixed

EmreTech reviewed Jun 15, 2023

View reviewed changes

nextcord/http.py Show resolved Hide resolved

nextcord/http.py Outdated Show resolved Hide resolved

nextcord/http.py Outdated Show resolved Hide resolved

nextcord/http.py Outdated Show resolved Hide resolved

nextcord/http.py Outdated Show resolved Hide resolved

alentoghostflame added 2 commits June 20, 2023 21:01

Removes a pass, updates a docstring, adds the Unauthorized exception.

2d6ff6d

Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

Merge remote-tracking branch 'origin/http_rework' into http_rework

fce6d13

Addresses TAG-Epic's comments.

a23bd70

Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

teaishealthy requested changes Mar 27, 2024

View reviewed changes

nextcord/http.py Outdated Show resolved Hide resolved

nextcord/http.py Outdated Show resolved Hide resolved

nextcord/http.py Outdated Show resolved Hide resolved

nextcord/http.py Show resolved Hide resolved

EmreTech reviewed Mar 30, 2024

View reviewed changes

nextcord/http.py Outdated Show resolved Hide resolved

nextcord/http.py Outdated Show resolved Hide resolved

nextcord/http.py Outdated Show resolved Hide resolved

nextcord/http.py Outdated Show resolved Hide resolved

nextcord/http.py Outdated Show resolved Hide resolved

alentoghostflame and others added 6 commits April 1, 2024 19:38

Addresses a few comments.

f2aae71

Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

Merge branch 'refs/heads/master' into http_rework

93ee0be

remove python 3.10 unions from docs typehints

dc9ef4d

Signed-off-by: Alex <alex@darkkingdom.lan>

change dispatch docs typehint, move HTTPClient.request error handling…

e345697

… to a private function Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

remove unnecessary if from GlobalRateLimit.update

da2fbf0

Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

document the form param in HTTPClient.request

a30a03e

Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

teaishealthy self-requested a review May 11, 2024 22:12

fix _global_rate_limits typing in HTTPClient

788faa0

Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

teaishealthy previously requested changes May 11, 2024

View reviewed changes

nextcord/http.py Outdated Show resolved Hide resolved

nextcord/http.py Outdated Show resolved Hide resolved

remove unnecessary if statement in _get_logging_auth

042ff63

Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

EmreTech reviewed May 13, 2024

View reviewed changes

docs/api.rst Show resolved Hide resolved

nextcord/http.py Outdated Show resolved Hide resolved

nextcord/http.py Show resolved Hide resolved

alentoghostflame and others added 11 commits May 13, 2024 18:33

add errors.Unauthorized to docs

5cd19f5

Co-authored-by: Emre Terzioglu <50607143+EmreTech@users.noreply.github.com>

move unnecessary comment from HTTPClient._handle_http_response_errors…

82e1bc6

… docstring to a comment above, add real docstring Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

add ratelimit shedding loop to prevent memory leak

3c1ec92

Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

fix pyright unnecessary comparison issue

64b137d

Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

fix cancellation of HTTPClient._ratelimit_shed_task and await it

d65e0b6

Signed-off-by: Alex Schoenhofen <alexanderschoenhofen@gmail.com>

Merge branch 'master' into http_rework

a0a021d

Merge branch 'master' into http_rework

c5d92af

style: auto fixes from pre-commit hooks

0e08c36

Merge branch 'master' into http_rework

0d89a5e

style: auto fixes from pre-commit hooks

d651b81

sorry not too sure what happened here

e1af7f5

ooliver1 approved these changes May 17, 2024

View reviewed changes

ooliver1 merged commit cf81368 into nextcord:master May 17, 2024
10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: HTTPClient Re-Core #1057

refactor: HTTPClient Re-Core #1057

alentoghostflame commented May 13, 2023 •

edited

MaskDuck commented May 13, 2023 •

edited

alentoghostflame commented May 13, 2023

Skelmis commented May 24, 2023

EmreTech left a comment

ooliver1 left a comment

refactor: HTTPClient Re-Core #1057

refactor: HTTPClient Re-Core #1057

Conversation

alentoghostflame commented May 13, 2023 • edited

Summary

Checklist

MaskDuck commented May 13, 2023 • edited

alentoghostflame commented May 13, 2023

Skelmis commented May 24, 2023

EmreTech left a comment

Choose a reason for hiding this comment

ooliver1 left a comment

Choose a reason for hiding this comment

alentoghostflame commented May 13, 2023 •

edited

MaskDuck commented May 13, 2023 •

edited