Shared storage for multi node training

[roywei]

Hi,

I love this project and great work! One question I have is what's the file storage size limit and throughput limit for a shared file system that all nodes can access (read/write)? It's not very clear from the storage doc, and what's the underlying implementation?

Thanks and can't wait to try this out!

[romilbhardwaj]

Hi @roywei, thanks for your interest!

The shared FS example in the docs uses an object store (S3/GCS/R2) as the backing persistent store. We provide access to these object stores by creating a FUSE mount using either goofys (for S3 and R2) or gcsfuse (for gcs).

Performance varies depending on your chosen VM type, the backing store and the workload. For S3, we ran some benchmarks and the detailed results can be found here. To summarize, on AWS m5.8xlarge with default EBS volume attached with 4 parallel processes reading/writing to files, we observed:

Disk Bandwidth (MB/s)

	EBS	SkyPilot Storage (S3, MOUNT mode)
Sequential Read	130	642
Sequential Write	129	1828

Disk IOPS

	EBS	SkyPilot Storage (S3, MOUNT mode)
Sequential Read	2051	8462
Sequential Write	2055	27899

Please note that performance is highly dependent on the workload. The discussion section in the results file provides cases when shared storage performance might be low (e.g., many small files).

Maximum file size, though I haven't tested it, should be similar to the limits put in by the backing object store. Here's S3's limits.

If you have a particular use case in mind, I am happy to discuss it!

[roywei]

Thanks a lot for the pointers! This is really helpful. I'm looking for how scalable is the shared storage and the perf numbers look decent. Technically it's also not hard to support lustre based high performance file system for really large scale. References in AWS, GCP, and Azure

[romilbhardwaj]

Just to add, using object stores + fuse mounting has its fair share of problems we run into:

1. Poor performance when using many small files because of increased S3 API calls
2. Lack of support for all POSIX operations (e.g., appends are not supported)

Lustre would indeed be awesome to support as a high performance FS option. Main requirements would be 1) should expose a POSIX interface (i.e., should be mountable so users don't have to change code) and 2) should be accessible across regions, in case a task gets moved. Happy to collaborate on this if it is of interest to you!

[roywei]

Sure, will create a RFC when I need this (no timeline promised :)). AWS file cache supports multi region access, but it's really expensive.

[romilbhardwaj]

Great, looking forward to it! Meanwhile if you run into any issues with SkyPilot, please let us know!