Provides an unauthenticated plain HTTP frontend for public and private S3 buckets, and caches on the filesystem. Least Recently Used objects are evicted from the cache first.
Example:
python setup.py install
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
caching-s3-proxy &
curl localhost:8000/my_bucket/v1/my_file.txt
If you want to listen on a different port, just set the PORT variable:
PORT=9999 caching-s3-proxy
The capacity of the cache is limited to 1GB by default, and the proxy will attempt to remove cached objects to stay under this limit. If you expect to go over this limit, you can set the CAPACITY variable (in bytes):
CAPACITY=2000000000 caching-s3-proxy
Cached object files are stored by default wherever your OS leaves temporary files, but this can be modified by setting the CACHEDIR variable:
CACHEDIR=/mnt/tmp caching-s3-proxy
Alternatively, you can run under uwsgi. It's safe to use multiple worker processes (the shared file cache uses file locking to allow concurrency):
uwsgi -w proxy.wsgi --http=localhost:8000 --workers=10
If you want to put this behind Nginx or Apache, use a socket instead:
uwsgi -w proxy.wsgi -s /var/run/caching-s3-proxy.sock --workers=10
Then see http://uwsgi-docs.readthedocs.org/en/latest/Nginx.html or http://uwsgi-docs.readthedocs.org/en/latest/Apache.html