Rate Limit Exceeded When Uploading Large Number of Files

[michael-srx]

Description

We are experiencing a rate limit issue with hs upload when used in a GitHub Action environment to handle a large number of files. The network speed in this context is significantly higher than when performed locally, leading to a Rate limit exceeded for creating files. Retry in 1 minute. error. This seems to be caused by hitting the rate limit of the underlying API endpoint.

Steps to Reproduce

1. Set up a GitHub Action using HubSpot's CLI tool to upload a large batch of files.
2. Observe that the action fails with the error: Rate limit exceeded for creating files. Retry in 1 minute.

Expected Behavior

The CLI tool should handle API rate limits with built-in rate limiting or a retry mechanism to manage bulk uploads without exceeding the API's limits.

Actual Behavior

The CLI tool fails and stops the upload process when the API's rate limit is exceeded.

Workaround

We have found a workaround by limiting the network speed using trickle:

trickle -u 2560 hs upload ...

This approach reduces the rate at which requests are made, thereby avoiding the rate limit error, but it is not an ideal solution.

Suggested Fix

It would be beneficial for the CLI tool to include an integrated rate limiter or a more sophisticated error handling mechanism that can dynamically adjust the request rate based on the API's feedback or predefined thresholds.

HubSpot CLI version: 5.2.0
Node.js version: 20