Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When using timeout at high concurrency, I get a different result than I expected. #1768

Open
EmptyDreams opened this issue Aug 11, 2023 · 0 comments
Labels

Comments

@EmptyDreams
Copy link

EmptyDreams commented Aug 11, 2023

Reproduction

I'm using the following code to make a large number of GET requests to a URL, for convenience I'm repeating the request to a URL and I've found that when the number of requests exceeds a certain value it causes most of the requests to timeout.

When I set count to 100 there is no error output from the console, but when I set count to a larger value most of the requests return a timeout error, for example if I set count to 500 over 400 of them all timeout.

const fetch = require('node-fetch')

const url = 'https://cdn.jsdelivr.net/npm/twikoo@1.6.17/dist/twikoo.all.min.js'
const list = []
const count = 500
let errCount = 0

for (let i = 0; i !== count; ++i) {
    const startTime = new Date().getTime()
    const id = i
    const task = fetch(url, {
        timeout: 5000
    }).catch(err => {
        ++errCount
        const endTime = new Date().getTime()
        console.error(`fail ${id}: use time = ${endTime - startTime}, message = ${err}`)
    })
    list.push(task)
}

Promise.all(list).then(() => {
    console.log(`finish errCount = ${errCount}`)
})

Expected behavior

I suspect this is because node-fetch provides a queuing sequence to cope with high concurrency, but the timeout timer still starts when a request is queued.

If I'm right, would it be possible to provide an option to allow the timeout to start when the request actually starts? Or is it possible to provide an interface that allows the user to customise this behaviour?

Your Environment

software version
node-fetch 2.6.12
node 18.12.1
npm 9.8.1
Operating System windows 10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant