Added man-in-the-middle attack for proper HTTPS header checking.

[synchronizing]

This is a very bootleg addition to the project to add a man-in-the-middle attack to allow ProxyBroker to check HTTPS requests and the returned headers. One of the biggest missing things on ProxyBroker was the ability to verify the return response from the servers if using HTTPS. This pull request allows that.

By creating a man-in-the-middle server ProxyBroker essentially goes from:

client <-> ProxyBroker <-> public proxy <-> server

To

client <-> ProxyBroker (server)
ProxyBroker (emulated client) <-> public proxy <-> server

By emulating the client we are able to take a peek at the responding HTTPS request coming back from the server, allowing us to ensure that the return was that actually warranted by the http_allowed_codes. This is an extremely temporary fix to the larger problem. The reality is that the entire project needs to be refactored with better usage of codes, exception throws, etc. It should also be mentioned that this PR does not include essentials like POST, SET, etc -- only GET.

If you do choose to use this implementation, you must also ensure that you either add the generated SSL certificate to your keychain or that you properly tell aiohttp or requests within your own applications to not bother checking SSL certificates. You can see a small implementation of a Python man-in-the-middle here for a better sense of the implementations here.

---

• Tests need to be expanded on for this PR.
• Add support for other HTTP methods; POST, SET, DELETE, etc.

[Alescontrela]

This PR provides a much needed upgrade

[pythoninja]

Is it good practice to add random commit as dependency? Maybe it would be better to pin latest commit in requirements.

[synchronizing]

Not good practice, no. Unfortunately http-parser on PyPi is not working, and therefore I had no choice. No excuse for PyOpenSSL.

This is a temporary fix, however, and I'm currently on the works on re-writting the entire project with the man-in-the-middle structure.

[synchronizing]

I've updated the MITM project to allow any outbound requests (GET, POST, DELETE, etc.). Planning on implementing it on a seperate project, shades, that will be used as a complete replacement to ProxyBroker due to inactivity by the main dev @constverum. Will update this Pull Request as soon as the project is partially finalized.

[bluet]

Hello @synchronizing and everyone, as @constverum is not around, I'm trying to fix bugs and add features in a separate repo https://github.com/bluet/proxybroker2

Would you be able to spend some time checking and maybe migrate the PR to the proxybroker2 repo?

And if you like, I'd be glad to add you as collaborator and we can work on improving it together :)

[synchronizing]

@bluet Hey, that sounds wonderful. I'm currently on the works to built an equivalent separate project from scratch. I'm working out a few flaws here and there, but the system would work similarly to ProxyBroker (with addons + much needed code improvements/organization).

I would not mind working together -- whether that is on a new project, or improving the current state of this project. If you have discord, my tag is sync#1871 -- if not, let me know the best way to reach you. Would love to discuss this further.

[synchronizing]

Regarding this specific PR, I would not recommend it in its current state. It's more messy than it needs to be, and it's literally a bootleg solution to the problem. It was a temporary fix to a larger problem.

[bluet]

@synchronizing Cool. Sent https://github.com/bluet/proxybroker2 collaborator invitation to you.
Actually I don't think I have enough python programming experiences to be qualified as the main maintainer, so it's also ok for me to transfer project ownership to you if you like. :-)

As I'm in need of a working solution to provide a single entrypoint and could dispatch requests to proxies in a pool, instead of starting from scratch and wait for the first working version, I'd personally prefer to fix / refactor from the current codebase step by step.
The next thing I (or, if possible, we) need to fix/implement is bluet#1 .

Sorry I don't have discord, do you use telegram? My account is BlueT_Lien