Improve the performance and latency for the cluster limiter

[absolute8511]

Issue Description

Type: feature request

Describe what feature you want

By default, the cluster limiter will have to request to token server for each request. This will increase the latency and make
too much requests to the token server. In order to solve the performance issue under cluster, the local token cache should be a proposal for this.

Describe your initial design (if present)

In order to reduce the request to server, we add a background prefetch job to period check the tokens and prefetch a batch of tokens if necessary. While the user request coming it will first check the local tokens.

Some design summary as below:

• only small request will be cached to reduce the pressure to the token server
• we allow request more than cached in some cases, and if prefetch failed for a while, most should be fall back to local limiter.

Additional context