Errors when s3 caching

[btwiningfc]

I've been running a simple caching process using version 10.4 and PostgreSQL 11.X with PostGIS 2.5 for a couple of years with no issues. I used the postgis provider

I recently upgraded to Tegola 15.2, postgreSQL 14 and PostGIS 3.1.

Seed command:
/opt/tegola cache seed --bounds $BOUNDS --config $CONFIG_URL --min-zoom 14 --max-zoom 14

Config toml:

[webserver]
port = ":9090"
uri_prefix = "/tile-server"

[webserver.headers]
    Access-Control-Allow-Origin = "*"
    Cache-Control = "no-cache, no-store, must-revalidate"

[cache]                     # configure a tile cache
type = "s3"                 # will cache to the AWS S3 bucket
bucket="bucket"
aws_access_key_id="KEY_ID"
aws_secret_access_key="SECRET_KEY"


# register data providers
[[providers]]
name = "test_postgis"  
type = "postgis"       
host = "host"          
port = 5432             
database = "db"        
user = "user"             
password = "stuff"         
srid = 4326              
max_connections = 50  

  [[providers.layers]]
  #Layers and maps

During these caching runs with this updated setup, we have noticed intermittent errors when attempting to add to the s3 buckets:

Error: error seeding tile ({Z:14 X:4546 Y:6933}): RequestError: send request failed
caused by: Put "https://CACHEBUCKETNAME.s3.amazonaws.com/layers/14/4546/6933": write tcp IPADDRESS_1:50138->IPADDRESS_2:443: use of closed network connection

It would occur on different tiles, in different zoom levels. I observed it happening in approximately 10% of my processes (I parallelize the caching, running different geographical areas at the same time using AWS fargate instances).

Additionally, We noticed a similar issue on the postgis_mvt runs.

Error log:

Error: error seeding tile ({Z:16 X:14050 Y:27083}): InternalError: We encountered an internal error. Please try again.
	status code: 500, request id: , host id: 

Research indicates that the InternalError could be re-tried
https://aws.amazon.com/premiumsupport/knowledge-center/http-5xx-errors-s3/

[iwpnd]

hey @btwiningfc 👋

Given that this can be a result of too fast concurrent requests, I'd take it as a compliment for Tegola and the upgrade. 😄

Maybe it's worth a try to reduce parallelization, given the limits per prefix, and see whether you can bring that 10% down. If that is the root cause after all, there's no real change in Tegola necessary, tho not out of the equation either, but I leave that to @ARolek.
Just a general question out of interest, what do you see as a main benefit in seeding versus caching on request? Or in your case a mix of both.

Anyhows, I will try to take a look into a retry pattern, but no promises. :)