Sentry: Error uploading sourcemap file - Error: connect EADDRNOTAVAIL 35.188.42.15:443 - Local #63
Comments
|
Maybe we are being too aggressive here: |
|
Thanks for reporting @igorescobar, I'm sorry this is causing you problems. I did see that error myself but it went away on its own so I assumed it was just some sort of transient network issue. It still uploading with As a temporary workaround you could remove your My understanding is that there are around 30K ports available so I wouldn't have thought we'd be hitting that. Your setup is a bit different than mine though, I'm using serverless-webpack to bundle my files so there are only a few that get uploaded. If your setup is pulling in all of |
|
Yeah, I also noticed that after upgrading to there is no way to disable the source maps upload (besides doing what you described). the only way is rolling it back. |
|
I'm literally on a plane right now but should have a quick fix in a bit. |
|
No risk, no fun. Just published a fully untested 2.5.1 that (hopefully) fixes two problems:
🤞 |
Maybe we should do this in batches instead of calling all of them in parallel or in series.... creating batches of 50-100 requests might fix the issue. In our case it would take ages to submit all files I suppose. |
|
The best would be to upload like 3 to 5 simultaneously for best performance. 50-100 parallel connections are a lot. But I currently lack the time and ability to test this properly, so I went for the quickest fix. It's slower but should work regardless of how large the project is and how many artifacts it has to upload. |
I would also create a property to configure it if necessary. It will be hard to come up with a config that will be nice for everyone so be conservative but also allow people to be more aggressive if they have the computing power to do so 👍 |
|
I have another update 2.5.2. Can you give it a try? It will upload 5 in parallel which should not impact your performance anymore. 10, 20, 50 really shouldn't make a noticeable difference, but feel free to adjust it and try it out if you notice something. I'm open to any changes. Again, this is fully untested 🙄 |
Tests checklist v2.5.2
|
|
Already 20 minutes in and counting 😄 . No errors but as expected... it will take ages to finish 😄 |
|
😞 How long did it take before my changes to serialize the upload? |
|
without sourcemaps it took about 7 minutes. with sourcemaps we are now 32 minutes in. |
|
If you want I can try to cook a PR adding a new option where I can configure the number of files to be uploaded at the same time. I think we can easily increase it to 10 or 20. Using Keep-Alive Agent inside the super-agent library might also speed up requests considerably. |
|
Well, that sucks. I don't think we need a configuration - the plugin should use a sensible value that works for most (if not all) scenarios. I'm okay with increasing it to whatever will work with a comparable performance than before. Technically, even 50 should be a problem. Seems that many, many small files still generate too much overhead. In a perfect world we would upload all files in a single request, but I haven't reviewed the Sentry APIs in ages... So, don't know if that would even be possible. Using Keep-Alive would definitely make sense as well! If you have some availability, I'd be happy to merge in any PR that improves the situation. At least we have a working version now, even if it's unacceptable slow. |
|
Could you increase it to like 25 for now? I'm trying to avoid having to add another dependency to the project as superagent alone needs an extra library to handle keep-alive within node I suppose. |
|
moving away to |
|
(I stopped the job at 45min) |
|
Please give 2.5.3 a shot whenever you have the chance. I increased it to 50 parallel uploads which matches the default the AWS JavaScript SDK uses as well: https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/node-configuring-maxsockets.html So, this should be a sensible default. Otherwise we can still add a configuration setting or think about more complex improvements for the next update. |
|
Thanks! 🚀 |
|
|
|
With that being said... we definitely need to add some retries and grace periods into this logic. If you consider that you can have multiple builds running at the same time you don't want to fail the entire build because of this. |
|
Also... worth mentioning that even tho most requests failed because of the concurrency being too high the job finished at |
|
Something to explore: It can give you access to a list of failed items which you can re-process later at a lower concurrency. |
import { PromisePool } from '@supercharge/promise-pool'
const errors = []
const { results } = await PromisePool
.for(users)
.handleError(async (error, user) => {
// you must collect errors yourself
if (error instanceof ValidationError) {
return errors.push(error)
}
// Process error handling on specific errors
if (error instanceof ThrottleError) {
return await retryUser(user)
}
})
.process(async data => {
// the harder you work for something,
// the greater you’ll feel when you achieve it
})
await handleCollected(errors) // this may throw
return { results } |
|
Just an update on my side. We found out a few other ways @sentry recommends for source map upload and we ended up using https://github.com/getsentry/sentry-webpack-plugin which also uploads the source maps to their service and it worked nicely for us. 👍 |
Hi,
A few days from now we started to get this error on our pipelines:
This is what our dependencies look like at the moment:
We even tried to disable this functionality by adding the environment variable
SENTRY_SOURCEMAPSor adding the custom config:But it runs regardless and ignores the flag. Do you have any idea what is happening and how to work around this?
The text was updated successfully, but these errors were encountered: