Parse raw headers

[Revadike]

• I have read the documentation and made sure this feature doesn't already exist.

So, I had this issue where a website had advanced http fingerprinting and anti-scraping techniques, including ignoring the specification where headers are treated case-insensitive, and they straight up blocked requests with lowercase headers in some cases. (I read, that some websites even check for the header order.)

I've been reading the existing issues and this has been discussed several times, but I don't think the above use case had been mentioned before. However, you said in one issue to report this to the website instead and not make an issue here. Fair enough...

So, hear me out. I believe this kinda goes against your goals of this package, but it would be really cool to have an option to just pass raw headers as 1 string to the request options, which in turn will ignore all your (nice) header processing of this package, meaning it will ignore several other options when set. (One exception maybe, is the cookies header.)

I know it's a very sensitive and error-prone option to have and perhaps best to leave out of the public docs, or at least have some warning, or both. Yet, I think it's probably a good and relatively easy-to-implement compromise. To be fair, you could suggest to me to use a different package for this, but I've just grown fond of got and its comprehensive feature-set! <3

[Revadike]

As an alternate solution, I came up with this:

        "hooks":             {
            "beforeRequest": [
                options => {
                    // Capitalize headers
                    for (let h in options.headers) {
                        let H = h.split("-").map(i => {
                            if (i === "dnt") {
                                return i.toUpperCase();
                            }
                            return i.charAt(0).toUpperCase() + i.slice(1);
                        }).join("-");
                        options.headers[H] = options.headers[h];
                        delete options.headers[h];
                    }

                    // Randomize header order (fixed = Host, Connection)
                    let hNames = new Set(Object.keys(options.headers));
                    let headers = JSON.parse(JSON.stringify(options.headers));
                    options.headers = {};

                    if (hNames.has("Host")) {
                        options.headers.Host = headers.Host;
                        hNames.delete("Host");
                    }

                    if (hNames.has("Connection")) {
                        options.headers.Connection = headers.Connection;
                        hNames.delete("Connection");
                    }

                    while (hNames.size > 0) {
                        let H = shuffle([...hNames])[0];
                        options.headers[H] = headers[H];
                        hNames.delete(H);
                    }
                }
            ]
        }

It's not ideal though. I'm using a hacky solution to capitalize headers that supposed to be all caps, like DNT. And perhaps you want a specific header to be all lowercase anyway. Also, the header order randomization only works partly. Some header order is still fixed beyond my control, like Accept-Encoding and Host.