ValueError: could not convert string to float: b'undefined' #241

lukastribus · 2019-05-12T23:00:30Z

Version number

$ pipenv run pip show cfscrape
Name: cfscrape
Version: 2.0.3
Summary: A simple Python module to bypass Cloudflare's anti-bot page. See https://github.com/Anorov/cloudflare-scrape for more information.
Home-page: https://github.com/Anorov/cloudflare-scrape
Author: Anorov
Author-email: anorov.vorona@gmail.com
License: UNKNOWN
Location: /home/user/.local/share/virtualenvs/project1-4cV4hs0c/lib/python3.5/site-packages
Requires: requests
Required-by:
$

Code snippet experiencing the issue

import cfscrape
cfscrape.create_scraper().get('https://pro-src.com/')

Complete exception and traceback

(If the problem doesn't involve an exception being raised, leave this blank)

>>> cfscrape.create_scraper().get('https://pro-src.com/')
Traceback (most recent call last):
  File "/home/user/.local/share/virtualenvs/project1-4cV4hs0c/lib/python3.5/site-packages/cfscrape/__init__.py", line 267, in solve_challenge
    float(result)
ValueError: could not convert string to float: b'undefined'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/.local/share/virtualenvs/project1-4cV4hs0c/lib/python3.5/site-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/home/user/.local/share/virtualenvs/project1-4cV4hs0c/lib/python3.5/site-packages/cfscrape/__init__.py", line 108, in request
    resp = self.solve_cf_challenge(resp, **kwargs)
  File "/home/user/.local/share/virtualenvs/project1-4cV4hs0c/lib/python3.5/site-packages/cfscrape/__init__.py", line 154, in solve_cf_challenge
    answer, delay = self.solve_challenge(body, domain)
  File "/home/user/.local/share/virtualenvs/project1-4cV4hs0c/lib/python3.5/site-packages/cfscrape/__init__.py", line 270, in solve_challenge
    "Cloudflare IUAM challenge returned unexpected answer. %s" % BUG_REPORT
ValueError: Cloudflare IUAM challenge returned unexpected answer. Cloudflare may have changed their technique, or there may be a bug in the script.

Please read https://github.com/Anorov/cloudflare-scrape#updates, then file a bug report at https://github.com/Anorov/cloudflare-scrape/issues."
>>>

URL of the Cloudflare-protected page

https://pro-src.com/

URL of Pastebin/Gist with HTML source of protected page

https://pastebin.com/K0LX8KDT

The text was updated successfully, but these errors were encountered:

ghost · 2019-05-13T01:59:04Z

I haven't really taken a good look at this yet but I did add the HTML source as a test fixture. It passes on python 2 and 3 so it's not immediately clear to me as to what is causing the problem.

git clone --single-branch --branch issue_241 https://github.com/pro-src/cloudflare-scrape issue_241
cd issue_241
pip install pytest responses sure
pytest tests

`

lukastribus · 2019-05-13T07:53:44Z

Well the HTML is not necessarily what cfscrape sees, since cfscrape does not output the HTML code. The pastebin merely contains the output of a parallel curl https://pro-src.com/ which may or may not be what cfscrape sees.

Are you saying you cannot reproduce this? cfscrape 2.0.3 worked fine for me until at least May 12th, 18:10 UTC and started to fail at 19:10 UTC, so I assume cloudflare changed something and cfscrape is broken for everyone?

ghost · 2019-05-13T12:21:14Z

@lukastribus I'm able to reproduce this error when using a very old version of Node.js v4. If Cloudflare updates usually they'll be a ton of issues flooding into multiple repositories. The most recent version uses more robust methods so maybe not...

(passing) Node.js v10.15.2
(passing) Node.js v6.17.1 <-- No longer supported since a couple of weeks ago
(passing) Node.js v4.9.1 <-- No longer supported since 2018-04-30
(failing) Node.js v4.0.0 <-- Seriously outdated missing crucial security updates

This works properly on any security patched version of v4 and all versions in between v4 and the most recent Node.js v12.2.0.
The reason this fails is because your version of Node.js is missing the Buffer.from related security update. ~~The error isn't being shown so I'll address that in a PR soon.~~ Logging this particular error raises a security concern.

In conclusion, you should upgrade your version of Node.js. Please read https://github.com/Anorov/cloudflare-scrape#nodejs-dependency

lukastribus · 2019-05-13T15:46:08Z

Thanks for the analysis.

This is Ubuntu 16.04 (xenial) - which is supported until 2021 and the latest nodejs package 4.2.6~dfsg-1ubuntu4.2 from the official repository. In this case, although the release is 4.2.6, Ubuntu is supposed to backport critical security fixes until the entire OS is EOS. Can you give me a pointer to the security update you are talking about, so I can file a bug against the Ubuntu package?

ghost · 2019-05-13T16:14:38Z

You're welcome. The security vulnerability was discovered by @feross and other webtorrent contributors. He created this package until the Node.js team had time to backport the proposed fix, also noted here. He raised the Node.js issue here: nodejs/node#4660

CVE-2018-7166 <- (Might just be related)
Search 'em out: https://www.cvedetails.com/vulnerability-list/vendor_id-12113/Nodejs.html

ghost · 2019-05-13T16:20:05Z

@lukastribus There is no support for v4, v6, v7 or earlier versions even if the Node.js team does receive reports about security vulernabilities, nothing will be done (No support).

Information on all security updates: https://nodejs.org/en/blog/vulnerability/

I would upgrade to bionic(18.04LTS): https://packages.ubuntu.com/search?keywords=nodejs
Or follow these instructions: https://github.com/nodesource/distributions#installation-instructions

But I don't like Ubuntu so... I use Kali.

lukastribus · 2019-05-13T19:09:13Z

Ok, that works, thanks.

Still don't really understand how this could brake cfscrape 2.0.3 from one moment to the other, only by changes on the cloudflare side.

As we suggest in README that Debian and Ubuntu users can you just apt-get install nodejs, which since yesterday is only true for Debian >= 9/Stretch and Ubuntu >= 18.04/Bionic, we should probably update README.

Can I assume that a PR updating README in that regard would be welcome?

ghost · 2019-05-13T19:14:17Z

@lukastribus Absolutely, please send one! Also, if we could pin point the exact minimum Node.js version and add that to the README with a note recommending people not to use unsupported Node.js versions for security reasons. Or simply noting v4.9.1 as being the minimum version that is known to work with cfscrape. Just suggestions. 😃

ghost · 2019-05-13T19:26:13Z

Still don't really understand how this could brake cfscrape 2.0.3 from one moment to the other, only by changes on the cloudflare side.

The IUAM JS challenge can vary from domain to domain, some domains might be locked to an older challenge. The challenges can change on a per request basis as well as the number of challenges required to solve and how often you receive those challenges with/without the cookie. Usually the changes are negligible on a per request basis but in some cases...

This library aims to maintain compatibility with all challenge versions that are currently in play.

lukastribus · 2019-05-13T19:42:44Z

This has been backported to 4.5, and first released in 4.5.0.

Confirmed by this PR here:
https://github.com/stripe/stripe-node/pull/365/files

So at least node 4.5 is required. Which explains why Ubuntu Xenial (4.2.6) is broken but Debian Stretch (4.8.2) works.

Will send a PR for this later.

As discussed in Anorov#241

Siiriion · 2019-06-07T13:10:09Z

Hi,

you can easily install Node v10.16.0 on Ubuntu 16.04.06 LTS server

using the library download here : https://nodejs.org/en/download/
and this How To here : https://github.com/nodejs/help/wiki/Installation

works for me.

vm01:/tmp$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial
vm01:/tmp$ node -v
v10.16.0

vm01:~/tmp$ pip3 show cfscrape

Metadata-Version: 2.1
Name: cfscrape
Version: 2.0.7
Summary: A simple Python module to bypass Cloudflare's anti-bot page. See https://github.com/Anorov/cloudflare-scrape for more information.
Home-page: https://github.com/Anorov/cloudflare-scrape
Author: Anorov
Author-email: anorov.vorona@gmail.com
Installer: pip
License: UNKNOWN
Location: /home//.local/lib/python3.5/site-packages
Requires: requests
Classifiers:
You are using pip version 8.1.1, however version 19.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

lukastribus added the bug label May 12, 2019

ghost closed this as completed May 13, 2019

ghost removed the bug label May 13, 2019

ghost mentioned this issue May 13, 2019

fix(cfscrape): Fix for challenge where part of formula is stored in a div SickChill/sickchill#5462

Merged

3 tasks

lukastribus added a commit to lukastribus/cloudflare-scrape that referenced this issue May 13, 2019

document nodejs 4.5 requirement

71d45e1

As discussed in Anorov#241

lukastribus mentioned this issue May 13, 2019

document nodejs 4.5 requirement #243

Closed

ghost mentioned this issue May 19, 2019

IUAM challenge returned unexpected answer #253

Closed

Anorov mentioned this issue May 19, 2019

Node.js version exception #255

Closed

ghost added the outdated deps Errors associated with using an old peer dependency label May 24, 2019

ghost mentioned this issue Jun 25, 2019

Issue with getting a page #267

Closed

This issue was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ValueError: could not convert string to float: b'undefined' #241

ValueError: could not convert string to float: b'undefined' #241

lukastribus commented May 12, 2019

ghost commented May 13, 2019

lukastribus commented May 13, 2019

ghost commented May 13, 2019 •

edited by ghost

lukastribus commented May 13, 2019 •

edited

ghost commented May 13, 2019 •

edited by ghost

ghost commented May 13, 2019 •

edited by ghost

lukastribus commented May 13, 2019

ghost commented May 13, 2019 •

edited by ghost

ghost commented May 13, 2019

lukastribus commented May 13, 2019

Siiriion commented Jun 7, 2019 •

edited

ValueError: could not convert string to float: b'undefined' #241

ValueError: could not convert string to float: b'undefined' #241

Comments

lukastribus commented May 12, 2019

Version number

Code snippet experiencing the issue

Complete exception and traceback

URL of the Cloudflare-protected page

URL of Pastebin/Gist with HTML source of protected page

ghost commented May 13, 2019

lukastribus commented May 13, 2019

ghost commented May 13, 2019 • edited by ghost

lukastribus commented May 13, 2019 • edited

ghost commented May 13, 2019 • edited by ghost

ghost commented May 13, 2019 • edited by ghost

lukastribus commented May 13, 2019

ghost commented May 13, 2019 • edited by ghost

ghost commented May 13, 2019

lukastribus commented May 13, 2019

Siiriion commented Jun 7, 2019 • edited

ghost commented May 13, 2019 •

edited by ghost

lukastribus commented May 13, 2019 •

edited

ghost commented May 13, 2019 •

edited by ghost

ghost commented May 13, 2019 •

edited by ghost

ghost commented May 13, 2019 •

edited by ghost

Siiriion commented Jun 7, 2019 •

edited