rate-limit makes the plugin unusable

[electriquo]

Describe the bug
github rate-limit make the plugin unusable

ERROR: rpc error: code = Unknown desc = github: non-200 OK status code: 403 Forbidden body: "{\n "documentation_url": "https://docs.github.com/en/free-pro-team@latest/rest/overview/rate-limits-for-the-rest-api#about-secondary-rate-limits\",\n "message": "You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID"\n}" (SQLSTATE HV000)

and prone to getting banned by github

Continuing to make requests while you are rate limited may result in the banning of your integration.

Steampipe version (steampipe -v)
Example: v0.21.2

Plugin version (steampipe plugin list)
Example: v0.39.0

Expected behavior
github specifies exactly how to handle these status code, we should honor by implementing it.

If you exceed your primary rate limit, you will receive a 403 or 429 response, and the x-ratelimit-remaining header will be 0. You should not retry your request until after the time specified by the x-ratelimit-reset header.

If you exceed a secondary rate limit, you will receive a 403 or 429 response and an error message that indicates that you exceeded a secondary rate limit. If the retry-after response header is present, you should not retry your request until after that many seconds has elapsed. If the x-ratelimit-remaining header is 0, you should not retry your request until after the time, in UTC epoch seconds, specified by the x-ratelimit-reset header. Otherwise, wait for at least one minute before retrying. If your request continues to fail due to a secondary rate limit, wait for an exponentially increasing amount of time between retries, and throw an error after a specific number of retries.

Continuing to make requests while you are rate limited may result in the banning of your integration.

Additional context

• overcome the 5000 requests/hr limit #118
• Update plugin to handle different rate-limit error #208

[electriquo]

@cbruno10 maybe you could add more information

[ParthaI]

Hello, @electriquo. Sorry for the delayed response.

Before we proceed with the fixes, I have a couple of questions, could you please take a look?

• How many rows are you attempting to retrieve with your query?
• Could you share the specific query that's triggering the rate-limit error?

I attempted to replicate the error by querying the github_issue and github_pull_request tables, but I was unable to reproduce it. I tried the query select count(*) from github_workflow where repository_full_name in (select name_with_owner from github_my_repository) multiple times, this query returns the result successfully all the time.

+-------+
| count |
+-------+
| 893   |
+-------+

Thank you!

[electriquo]

@ParthaI Install the github modules (steampipe-mod-github-compliance, steampipe-mod-github-insights, steampipe-mod-github-sherlock) and navigate to the GitHub Default Branch Protection Report.

I have more than 2000 active (non-archived) repositories.

[ParthaI]

Thanks, @electriquo, for the detailed information, I will give it a try to reproduce the issues, and make the necessary changes.

[ParthaI]

Hello, @electriquo, following our previous discussion, I set up my local environment to try and replicate the error.

Despite my attempts, I couldn't reproduce the rate limit error. The compliance/insight mods are functioning correctly across approximately 370 repositories in our organization.

For your information, the plugin initially used the REST API to generate results, which was prone to errors, particularly those related to rate limits. As a remedy, we've implemented GraphQL query support.

Additionally, I experimented with the plugin code in my local environment, making around 2300 API calls, and it worked without issues, completing in 1,671.1 seconds.

Query Result:

> select count(*) from github_my_repository
+-------+
| count |
+-------+
| 11871 |
+-------+

Time: 1,671.1s. Rows fetched: 11,871. Hydrate calls: 0.

Thanks!

[electriquo]

@ParthaI You were experimenting, which is a reproduction best effort. Yet, it does not say that there is no issue :)
Which information/logs do you need to get to the bottom of it?

As you can see in the description, the rate limit error is clear.

[ParthaI]

Hello @electriquo,

Thank you for your patience and for highlighting the issue once more. Indeed, the rate-limit error you mentioned is a significant concern, when navigating to the GitHub Default Branch Protection Report, given the extensive number of active (non-archived) repositories you manage, exceeding 2000.

In an effort to meticulously replicate this scenario, I conducted tests involving over 2000 API calls within a single query. Regrettably, these tests did not trigger the same rate-limit error, which suggests the situation might be influenced by specific conditions or a huge number of repositories in the organization.

Understanding the importance of accurately diagnosing and resolving this issue, I'd like to delve deeper. The dashboard you referred to leverages the github_my_repository table to fetch and display data. To assist us further in isolating the cause, could you kindly confirm if executing the query select * from github_my_repository directly in your Steampipe environment results in the same rate-limit error? This information will be invaluable in our investigation.

Your cooperation and insights are greatly appreciated!

[electriquo]

@ParthaI

To assist us further in isolating the cause, could you kindly confirm if executing the query select * from github_my_repository directly in your Steampipe environment results in the same rate-limit error?

No issue here but when I open my-steampipe-host.com/github_insights.dashboard.default_branch_protection_report I get the limit-rate error.

[ParthaI]

Thank you, @electriquo, for conducting those tests and sharing your findings. Replicating the rate-limit error appears to be challenging with less set of data set in our environment.

According to the documentation, using a Personal Access Token (PAT) allows for 5000 API calls within a certain timeframe. However, running the GitHub Insight Mods seems to exceed this limit in your setup, likely due to the large number of repositories (around 2000) in your environment.

In comparison, our environment hosts approximately 330 repositories. Before executing the GraphQL query, our initial rate limit was 5000. After executing select count(*) from github_my_repository, this limit dropped to 4923, indicating that approximately 77 API calls were made (per page result limit value is set to 5) when querying the github_my_repository table.

Here's the GraphQL query to check the rate limit:

query {
  viewer {
    login
  }
  rateLimit {
    limit
    remaining
    used
    resetAt
  }
}

You can find more details on how rate limits are calculated for GraphQL queries in the GitHub documentation.

Upon reviewing the GitHub insights mod, specifically the github_insights.dashboard.default_branch_protection_report, it seems the github_my_repository table is queried multiple times, which could contribute to reaching the rate limit.

I'm further investigating the plugin and the insight mod to understand this behavior better and will update you with any progress.

Thank you once more for your assistance!

[electriquo]

using a Personal Access Token (PAT) allows for 5000 API calls within a certain timeframe.

To clear any doubts, I am using a dedicate PAT; meaning that only Steampipe is using this PAT. Thus, the rate-limit must come when using Steampipe only.

Could it be that not only the amount of repositories are the factor but the amount of metadata (such as a big Git history/commits)?

[ParthaI]

Could it be that not only the amount of repositories are the factor but the amount of metadata (such as a big Git history/commits)?

Absolutely, @electriquo, you're correct. However, it's worth noting that within the Steampipe Dashboard, API calls are not initiated until you actively click on any of the hyperlinks.

[electriquo]

My repositories are reach, they are big with many metadata.
How do you advise to address it or is this plugin is only for a small scale and thin repositories?

[ParthaI]

Hello @electriquo,

I hope this message finds you well. I am currently working on this issue aimed at reducing the number of API calls while quering the table github_my_repository I have made a few changes in the GitHub Plugin and the GitHub Insight Mod. Unfortunately, I have yet to finalize an effective solution and find myself in need of a bit more information. When you have a moment, could you kindly assist me by checking on a few details?

• Could you please share how long the query select * from github_my_repository takes to execute in your current setup? If you could follow these steps, it would be immensely helpful:

  • Execute the command steampipe query.
  • Enable timing with .timing on.
  • Run the query select * from github_my_repository.

• I have recently committed some updates to the GitHub plugin repository on the branch update-limit-for-github-my-repo-table and to the GitHub Insight mod on the branch updated-query-to-hit-cache. Could I trouble you to fetch these updates to your local setup and build the plugin from there?

Please note, these modifications are specifically for the GitHub Default Branch Protection Report dashboard. My goal is to ensure these changes perform effectively, especially with larger datasets.

If you're willing to test the scenario, here are the steps:

1. Clone the repository with git clone https://github.com/turbot/steampipe-plugin-github.git.
2. Change the directory with cd steampipe-plugin-github.
3. Switch to the relevant branch with git checkout update-limit-for-github-my-repo-table.
4. Compile the code with make.
5. Start the service with steampipe service start.
6. Return to the root directory.
7. Clone another repository with git clone https://github.com/turbot/steampipe-mod-github-insights.git.
8. Change the directory with cd steampipe-mod-github-insights.
9. Switch to the appropriate branch with git checkout updated-query-to-hit-cache.
10. Start the server with powerpipe server.
11. Finally, navigate to the GitHub Default Branch Protection Report.

Your feedback on these changes would be invaluable to me.

Thank you very much for your cooperation and assistance in this matter.

[ParthaI]

Hi @electriquo, have you had the chance to try it out yet?

[electriquo]

@ParthaI Sorry, will try it soon.

[electriquo]

Could you please share how long the query select * from github_my_repository takes to execute in your current setup?

$ steampipe query
Welcome to Steampipe v0.22.1
For more information, type .help
> .timing on
> select * from github_my_repository

Error: github: non-200 OK status code: 403 Forbidden body: "{\n  \"documentation_url\": \"https://docs.github.com/free-pro-team@latest/rest/overview/rate-limits-for-the-rest-api#about-secondary-rate-limits\",\n  \"message\": \"You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID 279A:5B4F:18264BC:2CAC1C2:65F97C91.\"\n}" (SQLSTATE HV000)

Time: 47.1s. Rows fetched: 250. Hydrate calls: 17,250.

Could I trouble you to fetch these updates to your local setup and build the plugin from there?
...
here are the steps:

Where is the install command for update-limit-for-github-my-repo-table plugin and updated-query-to-hit-cache mod?
What am I missing?

[ParthaI]

$ steampipe query
Welcome to Steampipe v0.22.1
For more information, type .help
>.timing on
>select * from github_my_repository

Error: github: non-200 OK status code: 403 Forbidden body: "{\n "documentation_url": "https://docs.github.com/free-pro-team@latest/rest/overview/rate-limits-for-the-rest-api#about-secondary-rate-limits\",\n "message": "You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID 279A:5B4F:18264BC:2CAC1C2:65F97C91."\n}" (SQLSTATE HV000)

Time: 47.1s. Rows fetched: 250. Hydrate calls: 17,250.

I believe as per our previous conversation it was working fine.

Where is the install command for update-limit-for-github-my-repo-table plugin and updated-query-to-hit-cache mod?
What am I missing?

Please follow the following steps:

Steampie Plugin changes:

1. Clone the repository with git clone https://github.com/turbot/steampipe-plugin-github.git.
2. Change the directory with cd steampipe-plugin-github.
3. Switch to the relevant branch with git checkout update-limit-for-github-my-repo-table.
4. Run the command make.
5. Start the service with steampipe service start.

Steampipe Github Insights Mod:

1. Clone the repository to a directory git clone https://github.com/turbot/steampipe-mod-github-insights.git.
2. Change the directory with cd steampipe-mod-github-insights.
3. Switch to the branch with git checkout updated-query-to-hit-cache.
4. Start the server with powerpipe server.
5. Finally, navigate to the GitHub Default Branch Protection Report.

[electriquo]

@ParthaI Haven't forgot, didn't find time this :(

[cbruno10]

Hey @electriquo , I was able to reproduce the behaviours you were seeing (albeit with a much lower repository count) with select * from github_my_repository.

I looked into the plugin code, and found our error handling code does retry secondary rate limit errors (

steampipe-plugin-github/github/errors.go

Lines 39 to 43 in ec93282

	// v4 secondary rate limit
	if strings.Contains(err.Error(), "You have exceeded a secondary rate limit.") {
	plugin.Logger(ctx).Debug("github_errors.shouldRetryError", "abuse_rate_limit_error", err)
	return true
	}

), with the plugin's retry config set to use an exponential backoff algorithm with max 10 retries (

steampipe-plugin-github/github/errors.go

Lines 54 to 62 in ec93282

	func retryConfig() *plugin.RetryConfig {
	return &plugin.RetryConfig{
	ShouldRetryErrorFunc: shouldRetryError,
	MaxAttempts: 10,
	BackoffAlgorithm: "Exponential",
	RetryInterval: 1000,
	CappedDuration: 30000,
	}
	}

); this is probably still too aggressive based on how many API calls we'd be making for that many repos.

If you're directly running a query, one suggestion is to only select the columns you want to retrieve.

If you're running a mod dashboard/benchmark/control, you could also try using rate limiters to slow down Steampipe.

For instance, you can try adding this into your github.spc:

# stay well under the 100 hydrate/list/get functions concurrently based on limits in https://docs.github.com/en/graphql/overview/rate-limits-and-node-limits-for-the-graphql-api#secondary-rate-limits
plugin "github" {
  limiter "github_global_concurrency" {
    max_concurrency = 30 
  }
}

And then see if that helps with running the dashboards or benchmarks you were trying before.

I think select * from my_repository will still be tough for the plugin to complete even with rate limiters though based on how GitHub checks for secondary rate limits, as they have some internal timeouts and we request a lot of data as part of the query when selecting all columns.

For instance, with the limiter settings above, I avoided getting the secondary abuse rate limits, but eventually got this unhelpful 502 Bad Gateway error from them with no additional information:

Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug.

I don't believe a lot of our dashboards run this type of query though.

[electriquo]

@cbruno10

found our error handling code does retry secondary rate limit errors

Does it mean that you are working on a fix and I should wait for a new release?

For instance, you can try adding this into your github.spc:

# stay well under the 100 hydrate/list/get functions concurrently based on limits in https://docs.github.com/en/graphql/overview/rate-limits-and-node-limits-for-the-graphql-api#secondary-rate-limits
plugin "github" {
  limiter "github_global_concurrency" {
    max_concurrency = 30 
  }
}

Where in the docs did you find about github_global_concurrency and max_concurrency?

[cbruno10]

@electriquo

We are looking into if we can improve the error handling for rate limit errors, but we're not actively working on an identified fix and don't have a schedule on when it will be released.

I'd suggest still trying to use a limiter and seeing if that helps your executions.

In the example I sent, the github_global_concurrency is the name I gave the limiter resource, but general documentation can be found at https://steampipe.io/docs/guides/limiter.

[electriquo]

@cbruno10

We are looking into if we can improve the error handling for rate limit errors, but we're not actively working on an identified fix and don't have a schedule on when it will be released.

Thanks

In the example I sent, the github_global_concurrency is the name I gave the limiter resource, but general documentation can be found at steampipe.io/docs/guides/limiter.

In the future, how one may know the variables that can be set in a plugin to handle concurrency and rate limiting?
If you would not have mentioned the structure and the variable, I guess that most won't know about it :|
That's why I asked

Where in the docs did you find about github_global_concurrency and max_concurrency?

Maybe it should be documented in steampipe.io/docs/guides/limiter.

---

I was able to reproduce the behaviours

you can try adding this into your github.spc:

# stay well under the 100 hydrate/list/get functions concurrently based on limits in https://docs.github.com/en/graphql/overview/rate-limits-and-node-limits-for-the-graphql-api#secondary-rate-limits
plugin "github" {
  limiter "github_global_concurrency" {
    max_concurrency = 30 
  }
}

If you apply the block in github.spc, did it mitigate the rate-limiting which you were able to reproduce?

[cbruno10]

@electriquo With that block in my github.spc, yes I did see the rate-limiting was resolved. My query execution time increased, but I was able to get all of the data for my repos (though this number is much lower than the amount of repos you have).

Were you able to give the limiter a try, either with max concurrency of 30 (or a different number)? If so, did you see any more consistency in getting results back?

Also, we should probably mention rate limiters and at least link to the doc on steampipe.io, which has some examples, so we'll add that to our backlog to see where that section belongs.

[electriquo]

Were you able to give the limiter a try, either with max concurrency of 30 (or a different number)?

It is on my list for tomorrow :)

If so, did you see any more consistency in getting results back?

Promise to keep you posted

Also, we should probably mention rate limiters and at least link to the doc on steampipe.io, which has some examples, so we'll add that to our backlog to see where that section belongs.

Awesome, thanks you :)

[electriquo]

@cbruno10

$ steampipe --version
Steampipe v0.22.2

$ steampipe plugin list --output json | jq -r '.installed[].name'
hub.steampipe.io/plugins/turbot/github@0.39.1

$ powerpipe mod list --output json | jq -r '.[].dependency_path'
github.com/turbot/steampipe-mod-github-insights@v0.4.0
github.com/turbot/steampipe-mod-github-compliance@v0.7.0
github.com/turbot/steampipe-mod-github-sherlock@v0.13.0

$ cat ~/.steampipe/config/github.spc
connection "github" {
  plugin = "hub.steampipe.io/plugins/turbot/github@v0.39.1"
}

plugin "github" {
  limiter "github_global_concurrency" {
    max_concurrency = 30
  }
}

When I navigate to the GitHub Default Branch Protection Report dashboard, the rate-limit error still pops.

But if I repeat #398 (comment) does not return a rate-limit error message

$ steampipe query
Welcome to Steampipe v0.22.2
For more information, type .help
> .timing on
> select count(*) from github_my_repository
...
Time: 7.6s. Rows fetched: 1,088. Hydrate calls: 0.

And then navigating to the GitHub Default Branch Protection Report dashboard, the rate-limit error does not pops but besides the repository list, all other columns are empty :(

I look at look at the plugin log, and I found that it seems not to use (but do honor) the max concurrency connections. Here are some log lines

2024-04-10 07:16:43.590 UTC [INFO]  waitForPluginLoad: hub.steampipe.io/plugins/turbot/github@0.39.1 is now loaded and ready (0x40007cd880)
2024-04-10 07:16:43.593 UTC [INFO]  steampipe-plugin-github.plugin: [INFO]  1712733403165: Plugin execute table name: github_my_repository quals: NONE (1712733403165)
2024-04-10 07:16:43.593 UTC [INFO]  steampipe-plugin-github.plugin: [INFO]  1712733403165: Executing for 1 connection: github
2024-04-10 07:16:43.593 UTC [INFO]  steampipe-plugin-github.plugin: [INFO]  1712733403165: Setting max concurrent connections to 25
2024-04-10 07:16:43.593 UTC [INFO]  steampipe-plugin-github.plugin: [INFO]  1712733403165: resolveRateLimiterScopeValues (github-1712733403165)
2024-04-10 07:16:43.593 UTC [INFO]  steampipe-plugin-github.plugin: [INFO]  1712733403165: resolvedRateLimiterConfig: no rate limiters (github-1712733403165)
2024-04-10 07:16:51.016 UTC [INFO]  steampipe-plugin-github.plugin: [INFO]  1712733403520: rowData chan returned nil - wait for rows to complete (github-1712733403165)
2024-04-10 07:16:51.017 UTC [INFO]  steampipe-plugin-github.plugin: [INFO]  1712733403520: streamRows - nil row, stop streaming (github-1712733403165)
2024-04-10 07:16:51.023 UTC [INFO]  steampipe-plugin-github.plugin: [INFO]  1712733403520: Plugin execute table: github_my_repository closing error chan and output chan  (1712733403165)

Also tried with max_concurrency = 10, to check whether it will still be honored and/or resolve the issue by examining the log file and the dashboard, but everything was the same :(

[electriquo]

@cbruno10 Hi, any insights?

[electriquo]

@cbruno10 Hello, do you have any new insights?

[electriquo]

@cbruno10 @ParthaI Hello, did we lost all momentum?

[misraved]

Apologies for the radio silence on this issue @electriquo !!

I think the reason why you didn't see the limiter tags getting honored could be indicative of a caching issue.

Could you please retry the queries by launching a fresh instance of Steampipe and killing any old instances?

Command that you could try before running steampipe query:

• steampipe service stop --force
• pkill -f steampipe

[electriquo]

Could you please retry the queries by launching a fresh instance of Steampipe and killing any old instances?

@misraved That's what I always did :)

I think the reason why you didn't see the limiter tags getting honored could be indicative of a caching issue.

Seems like firing in all directions rather than base things on data.
Can we get to the bottom of it or should I just drop the GitHub plugin as a whole?

[misraved]

Thanks for the clarification @electriquo !!

We are actively exploring options to enhance the rate-limiting capabilities within our plugin. Currently, there is a limited selection of solutions that offer a straightforward method for managing errors originating from the API.

[electriquo]

@misraved Could you please follow #398 (comment)?

[cbruno10]

Hi @electriquo , sorry for losing track of this thread.

• For verifying if your limiters are being used, you can run select * from steampipe_plugin_limiter (taken from Viewing and Overriding Limiters). If you run that query, are you able to see your limiter with the correct settings?
• While using a limiter, do you have issues with queries, or primarily with the dashboards?
  • If running dashboards, which ones have you tried beside GitHub Default Branch Protection Report? And have they all given you errors?
  • Dashboards in general run a lot of queries simultaneously, so it'll be more common to hit rate limiters when running dashboards

[electriquo]

@cbruno10

The rate-limit error should be handled generally regardless the component that causes it, especially when there is a clear protocol for rate-limit. Specifically, GitHub has Rate limits for the REST API.

are you able to see your limiter with the correct settings?

No.
I followed your guidance to set github_global_concurrency and executed the query. As you can see below, GitHub is not in the list.

$ steampipe query "select distinct name from steampipe_plugin_limiter"
+---------------------------------------------------+
| name                                              |
+---------------------------------------------------+
| aws_servicequotas_list_service_quotas             |
| aws_servicequotas_list_aws_default_service_quotas |
| aws_servicequotas_list_tags_for_resource          |
+---------------------------------------------------+

Why would you expect to see the limiter appear here when the plugin log clearly states so?

Dashboards in general run a lot of queries simultaneously, so it'll be more common to hit rate limiters when running dashboards

From your words, I understand that dashboards are useless :(

If running dashboards, which ones have you tried beside GitHub Default Branch Protection Report? And have they all given you errors?

Received the same error across few GitHub's dashboards.

[cbruno10]

@electriquo

• If I understand your first point above correctly, we do handle rate-limit errors today (for v3 and v4 APIs). For instance, in https://github.com/turbot/steampipe-plugin-github/blob/main/github/errors.go, we have a function to determine when to retry based on the error code or error message we get back and use a reset time (if available in the response), e.g., https://github.com/turbot/steampipe-plugin-github/blob/main/github/errors.go#L22-L37. For v4/GraphQL API calls, we handle them in

  steampipe-plugin-github/github/errors.go

  Lines 39 to 43 in 8399ae8

  	// v4 secondary rate limit
  	if strings.Contains(err.Error(), "You have exceeded a secondary rate limit.") {
  	plugin.Logger(ctx).Debug("github_errors.shouldRetryError", "abuse_rate_limit_error", err)
  	return true
  	}

  ; unfortunately, we don't get back more info from the SDK on retry times, but we do use exponential backoff when retrying requests. If I missed your original point though, please let me know.
• I was curious if the limiter would show up in that query result, as I wasn't sure in what context the plugin log you were referencing was used in. For instance, in the Steampipe plugin SDK, the log message is sent from https://github.com/turbot/steampipe-plugin-sdk/blob/5d869679770d34049cf492182213514c1f5669f2/plugin/env.go#L29-L36, but Steampipe under the hood does a lot of connection management (even before limiters existed), so it's good to check what the introspection table says as well.
• Our current dashboards may not scale well based on how many repositories you have access to. For instance, we've tested them with users who have access to 400+ repositories, but definitely not 1k+ (that's a lot)! Our goal is to make dashboards load correctly for your environment too, so we're interested in figuring out Steampipe configuration + dashboard updates to make this possible.

Looking at your limiter configurations, can you please update it to:

plugin "hub.steampipe.io/plugins/turbot/github@v0.39.1" {
  limiter "github_global_concurrency" {
    max_concurrency = 30
  }
}

Since you're using a specific plugin version, the plugin reference needs to reference that version.

Afterward, can you please restart Steampipe, steampipe service restart and then run select * from steampipe_plugin_limiter and see if the limiter then shows up?

If it does, can you please try running the single query you shared above first and see if that still executes correctly? If so, can you please then run the dashboard again? You may still get throttling errors, and if so, can you please try lowering max_concurrency to 20 (and then 10)?

Thanks!

[electriquo]

@cbruno10 Although I am cooperating, I am not the Steampipe QA team :)
I've done whatever I was asked until now, but feel that there is firing to all directions in hope to hit something, and Steampipe do not need me for it - Steampipe team can and should reproduce everything on their own to save this back and forth writing.

Since you're using a specific plugin version, the plugin reference needs to reference that version.

This doesn't sound correct for me, just another short in the dark.

After you took the toll to create and environment to:

• Reproduce the issue
• Examine the logs
• Check and test proposed solutions

I'd be happy to continue and assist.