WiP: Fix looping Cloudflare challenge, Resolves #1036 #1163
Conversation
|
FWIW But after each solve there remains a chrome subtask that starts to spin up to 15% CPU and I have to manually kill them off. |
|
Another thing that I've noticed is that in the user-agent headless replacement: self.execute_cdp_cmd(
"Network.setUserAgentOverride",
{
"userAgent": self.execute_script(
"return navigator.userAgent"
).replace("Headless", "")
},
)I don't know why but If I hardcode the user-agent using the exact that my computer has like this: user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"
options.add_argument(f"--user-agent={user_agent}")it bypasses cloudflare, but if i put this to make it automatically like you have it on line 533 from So an alternative could be to setup a driver only to get the user agent: def get_user_agent(driver):
return driver.execute_script("return navigator.userAgent;").replace("Headless", "")And then pass the user-agent to the definitive driver PD: I only can tell you what I've discovered to see if we can go through the solution cuz I'm having troubles to get the project installed/set up 😅 |
|
I didnt actually use this branch, it worked fine after I switched to it. Thanks |
|
@garfield69 yea this seems to be an issue with Chrome v124. You can revert to v123 in the mean time if it's easier - #1161 Alternatively, build your own binaries, which will use Chromium v123:
|
|
@m33ts4k0z were you doing this on Windows? |
Yes on a Windows 11 VM on Unraid but it did work in the end. I updated my first post here with the cause. |
Oh cool, did not know I could build on windows. |
|
@juanfrilla sorry for the delay in replying, been busy and only got to a few quick ones on my phone. I'll have a look at the UA idea when I next get a chance, thanks. Assuming you're following the run from source instructions, what issue are you having? https://github.com/FlareSolverr/FlareSolverr#from-source-code |
|
@ilike2burnthing my main problem is that i cannot install Xvfb on MacOS |
|
Tried XQuartz? |
yessir now the project is set up, let's see what I can fix |
|
What exactly is left to do on this to get it merge? I tried to guess with the comments here and some different issues but I can't get the current status of this. It seems to be stale for quite some time, so what's needed? |
|
Well, I made my own implementation of this "new tab" idea and I was able to make it work with every website I could (ext.to, www3.yggtorrent.cool, dodi-repacks.site, hd-torrents.me/login.php, nhentai.net) on my Linux system using a VPN / socks5 proxy and also with my container image on my own remote Linux server, which was blocked by cloudflare too. Public image with my edits: 21hsmw/flaresolverr:fixlooping |
|
That's working 95% of the time on Windows for me, even with a proxy, but failing 95% of the time on Docker. Usual error: Seems it's related to |
When you say it fails on Docker, is it still on Windows or Linux? I got this error on Linux while doing my implementation, but have not been able to replicate it since. For the looping challenges, it seems to be a timing issue. Playing with the timer values can make it work in some cases, but it's not easy to know what works for everyone since it seems to take network latency into account. For example, if I use a proxy close to my location, it works 100% of the time with the sites I listed earlier, but if I use a proxy very far from me, it works 50% of the time. |
|
Linux. I'll play around with timings again (I did a bunch yesterday), see if I can get something that works both on my Docker and Windows. |
Strange then. I'm able to solve the challenges of all sites I try on my Debian and Fedora systems with different VPNs/Proxies with and without Docker involved. Here's an example with dodi-repacks.site using the docker image I shared previously: |
This comment was marked as off-topic.
This comment was marked as off-topic.
Thanks for your workaround @21hsmw Working with @aevrard the solution you provide will kill the killswitch if you're using something like gluetun... |
|
Thanks @21hsmw ! |
Worked for me on whatbox.ca services:
flaresolverr:
image: 21hsmw/flaresolverr:fixlooping
environment:
- LOG_LEVEL=${LOG_LEVEL:-info}
- LOG_HTML=${LOG_HTML:-false}
- CAPTCHA_SOLVER=${CAPTCHA_SOLVER:-none}
- TZ=UTC
- PORT=25000
- HOST=127.0.0.1
network_mode: host
pull_policy: always
restart: unless-stopped |
replacing the image of the dockerfile for this: I tested as well on a centOS server with the previous image ( |
I just tested several times with my image using the same link you provided, and I was able to pass the challenge after the tab switch is completed. Here are my observations testing with your link:
I was skeptical about the results, so I started tweaking the timers. Changed this part from 2/2 to 4/4: def switch_to_new_tab(driver: WebDriver, url: str) -> None:
logging.debug("Opening new tab...")
driver.execute_script(f"window.open('{url}', 'new tab')")
- time.sleep(2)
+ time.sleep(4)
logging.debug("Closing original tab...")
- time.sleep(2)
+ time.sleep(4)
driver.close()Then tried again:
Oddly enough, when I switch back to 2/2, the far away proxy still works, which means that if it succeeds in a challenge, it's easier to go through afterwards. Can you also try changing to 4/4 and see if that changes anything? |
yeeeeah it worked with 4/4 on the server, here is the image: |
Nice. |
|
Using your current image: With a few changes: While everything works fine on Windows, the best I could get with Docker was an equivalent to the current release, but now with various errors instead of just timing out, e.g.: As for what changes I got the best results with (there were many other failed variations): def get_correct_window(driver: WebDriver) -> WebDriver:
if len(driver.window_handles) > 1:
for window_handle in driver.window_handles:
driver.switch_to.window(window_handle)
time.sleep(1)
current_url = driver.current_url
logging.debug(f"Current URL... {current_url}")
if not current_url.startswith("devtools://devtools"):
return driver
return driver
def switch_to_new_tab(driver: WebDriver, url: str) -> None:
logging.debug("Opening new tab...")
driver.execute_script(f"window.open('{url}', 'new tab')")
time.sleep(8)
logging.debug("Closing original tab...")
driver.close()
[...]
if attempt in {4, 8, 12, 16, 20}:
switch_to_new_tab(driver, req.url)
time.sleep(1)
driver = get_correct_window(driver)
time.sleep(4) |
This comment was marked as duplicate.
This comment was marked as duplicate.
This comment was marked as duplicate.
This comment was marked as duplicate.
|
I spent all afternoon trying to find a way to bypass cloudflare on hd-torrents.me without success while I'm able to pass all others. |
|
The successor to UC is nodriver, but I'm not going to even try to change over to that, I would be immediately out of my depth. |
I've come to the same conclusions after many hours of testing, including generating a browsing history and manipulating web pages of big sites to inject an All that said, the easiest fix for this issue is to just use Firefox. I've ripped out all the uc driver related stuff, made |
|
Can you provide a PR? |
|
Unfortunately I can't because because what I run locally is a very old version that I've been patching myself to work around issues over the last year or so, and there is some custom messy stuff in there as well. And I used My from selenium.webdriver import FirefoxService
from webdriver_manager.firefox import GeckoDriverManager
from seleniumwire import webdriver
...
def get_webdriver(proxy: dict = None) -> webdriver.Firefox:
global PATCHED_DRIVER_PATH
logging.debug('Launching web browser...')
options = webdriver.FirefoxOptions()
# options.set_preference("network.proxy.type", 1) # Direct = 0, Manual = 1, PAC = 2, AUTODETECT = 4, SYSTEM = 5
# options.set_preference("network.proxy.http", "127.0.0.1")
# options.set_preference("network.proxy.http_port", 2020)
# options.set_preference("network.proxy.share_proxy_settings", True)
if get_config_headless():
if os.name == 'nt':
options.add_argument('-headless')
else:
start_xvfb_display()
driver = webdriver.Firefox(
options=options,
service=FirefoxService(
executable_path=GeckoDriverManager().install(),
),
seleniumwire_options={'disable_encoding': True},
)
driver.maximize_window()
return driver
|
I've been playing with it for an hour and it passes everything. It seems to support most of what we need for flaresolverr, so it might be an option. The only thing I don't know is if it will run correctly inside a container. If it does, I will probably start moving flaresolverr to nodriver, but that will take some time. I noticed this part in the nodriver readme (code) It could be a relatively quick way to support it without too much change until the decision is made whether or not to use it fully. |
This comment was marked as off-topic.
This comment was marked as off-topic.
|
I can access all trackers in Jackett with the latest Flaresolverr, but not with these changes. |
|
'latest' being v3.3.17? What OS? |
|
Flaresolverr v3.3.17, Linux x86_64 running on DigitalOcean. Tested on my macOS and that version also works fine with my normal IP. Any tracker I should try that you have issues with? |
|
https://github.com/search?q=repo%3AJackett%2FJackett+configuring-flaresolverr&type=code Some may not be currently using CF, there may be some missed, some only use it for login or keyword searches, but that should give you an idea. See also those sites mentioned in #1036. |
|
juanfrillaaa/flaresolverr:latest solves the challenges greatly on my ubuntu with docker! Thanks @juanfrilla! |
Update: Just tested, it works with an xvfb display inside a container. I'll see if I can find some time to implement part of nodriver for flaresolverr, but I'm not sure if I should put both undetected-chromedriver and nodriver in the same files, or if it would be better to create separate files like |
|
Separate sounds good. |
Thanks to @juanfrilla for #1036 (comment).
Unfortunately, currently this only works on Windows, and the looping challenges return if using proxies or VPNs.