@cduchesne commented on Sun Feb 05 2017

libStorage should have a throttling mechanism to prevent sending too many requests to storage api

@codenrhoden commented on Wed Mar 15 2017

@koensayr If you want a place to paste that throttling info, this is the place.

@codenrhoden commented on Mon Mar 20 2017

Thoughts from @cantbewong:

Consider this situation:
You need to create, mount, or unmount a volume and the operation takes a long time to complete. The reason could be that an “downstream” 3rd party API to accomplish this is:
Slow - Imposes rate limiting (rejects or fails to successfully execute requests under heavy use) Two forms of rate limiting are known to exist:
Exhibits a cap on requests per unit of time
Exhibits a cap on number of outstanding requests in progress

So, you basically have two options:

1. you will force API client to wait
2. you can immediately return some preliminary status response (202 Accepted is the convention defined by RFC 7231), and defer final status reporting to some later point

This falls into the pattern of a “long running operation” that commonly leads to:

• A desire for a cancellation API
• A mechanism to support “push” notification (websocket, message queue, etc)
• A need for state maintenance in the service

Furthermore, the downstream API, could additional exhibit non-deterministic ordering behavior. For example, when even though a caller submits an unmount request followed by a mount request, the mount request is attempted first.

While it is almost always a mistake to impose specifying an implementation demand in a functional spec, these characteristics leads to:

• A desire for state to be held in the form of a queue in order to preserve ordering of requests
• A desire for optional support for having a unified queue which combined multiple request types in a common queue to preserve ordering, e.g. unmount + mount combined into the same queue
• The queue should be implemented by libStorage itself, in order to prevent duplication of efforts in individual plugins. If libStorage API calls are submitted at a rate greater than the queue can be emptied, eventually the queue will reach capacity. There is no algorithm for implementing an unlimited size queue. When this happens, the libStorage API should return an error (429 Too Many Requests or 503 Service Unavailable would both be considered conventional)
• Because the needs of various plugins vary, these aspects of queueing and rate limiting should be configurable on a per plugin basis. A plugin will have the option to define these settings. At the plugin’s option independent or combined queues may be selected:
  ** Rate limiting (max calls to API per unit time): e.g. do not dispatch external 3rd party requests from this queue at a rate greater than 20 requests per minute
  ** Outstanding request limiting: e.g. Stop dispatching request if more than 20 active requests are pending final resolution
  ** Retry Timeout specification: e.g. if any request dispatched from this queue has been pending more than 5 minutes, attempt a retry
  ** Cancel Timeout specification: e.g. if any request dispatched from this queue has been pending more than 5 minutes, attempt a cancel
  ** Queue clear invocation for use on communication or authentication error.
  *** Certain kinds of errors such as a credential rejection, or host unreachable might be best handled by flushing all pending queued operations rather than re-encountering the error as each queue entry is processed. I mechanism will be provided to allow plugin code to flush the queue.
  ** Dispatch filter option. A plugin should have the option to look at the next impending dispatched entry from the queue and temporarily suspend dispatch.
  *** Some plugins may have a maximum of one pending operation per client (cluster node) or per volume. This feature will allow a plugin to impose appropriate limits on queue draining
  ** These defaults can be defined by a driver. Override by user config is an optional feature that could be deferred to a later release.

An Example of issue/problem (still open) in Kubernetes support for AWS:
kubernetes/kubernetes#31858

The platform API should be able to respect backend storage platforms rate limits either in a centralized throttling mechanism or something that is deferred to the driver, ex. API overloads in AWS EBS and EFS