https problem on FreeBSD #32
Comments
Seems like that prerender can't connect to your headless Chrome instance. What's your Chrome version? How did you start it? |
|
wow, fast response 👍 it works fine with http based URL, for example if I do curl -v http://myserver.com:8000/http://www.somesite.com then it works fine, just doesn't like https chrome is Chromium 59.0.3071.115, it's started like this |
|
Might related to #13 |
|
Can you try prerender |
|
unfortantely this does not help, we need it to run with --headless as this is in server environment for rendering our SPA's |
|
The problem is with how we use
|
|
ok I inferred it was --headless switch from someone mentioning it on the ticket link you sent. so what you are saying its layerPainted?, but how come I can do this also other https targets have same problem, for example if I do
it seems something else is problematic with https and how prerender works with this possibly |
|
Because in prerender it tries to infer when the page has done rendering. There are several methods:
The last one can be problematic when the page constantly modify DOM like Apple homepage do. (If you SPA don't do that it should work fine) |
|
That said, I would recommend you to try to prerender your SPA instead of Apple/Google page. |
|
our SPA does not work as well, we were hoping we would be able to try it out instead of using our existing PhantomJS + Prerender.IO setup, which works fine. |
|
So you SPA has the same issue? |
|
We may have to look at puppeteer instead, https://github.com/GoogleChrome/puppeteer#readme but we will likely need to re-implement some of functionality which you already have here |
|
Yes our SPA has same issue |
|
@beyondcreed Try comment out https://github.com/bosondata/chrome-prerender/blob/master/prerender/chromerdp.py#L152 to see if it helps. Thanks! I got it work for Apple homepage. |
|
Since pip install -U https://github.com/bosondata/chrome-prerender/archive/master.zipto test it. |
|
If I comment line 152 in chromerdp.py same problem. Will try your new version now as well. |
|
If you encountered the same issue again, please post the logs here. Thanks! |
|
ok trying now Collecting https://github.com/bosondata/chrome-prerender/archive/master.zip |
|
prerender DEBUG=1 2017-08-16 19:59:22 - (sanic)[INFO]: Goin' Fast @ http://0.0.0.0:8000 2017-08-16 20:00:30,415 ERROR sanic.log:104 Traceback (most recent call last): 2017-08-16 20:00:30,419 INFO websockets.protocol.fail_connection:618 Failing the WebSocket connection: 1006 |
|
still a problem |
|
You might want to upgrade your Chromium to latest version to avoid any issue with devtools protocol the old version might have. |
|
I believe that is latest I have in my packages, let me check |
|
Chromium 59.0.3071.115 is latest I have http://www.freshports.org/www/chromium/ |
|
Thanks! I have reproduced the problem locally, I'll try to find out what is causing it. |
|
I see your website set https://github.com/bosondata/chrome-prerender/blob/master/prerender/chromerdp.py#L242 |
|
ok, btw your current version is working with https://www.apple.com |
|
I will let front-end developer know about prerenderReady. If there is any other issues they will follow up on this thread. thanks. |
|
@beyondcreed Try the current master, it should work with your website |
|
very nice, it worked. |
|
thanks for fixing that quickly |
|
For more accurate result, you should set |
|
hm, just tried it on our main site, https://www.thrinacia.com/blog/post/express-checkout-feature but it waits very long, then returns with error, debug is: 2017-08-16 20:43:29,951 INFO prerender.chromerdp.navigate:216 Page 135db0bc-8e9d-495b-b50e-10f9f4905de8 [1] navigating to https://www.thrinacia.com/blog/post/express-checkout-feature 2017-08-16 20:44:29,971 ERROR sanic.log:104 Traceback (most recent call last): 2017-08-16 20:44:29,971 WARNING prerender.app.handle_request:201 Got 504 for https://www.thrinacia.com/blog/post/express-checkout-feature in 60028ms |
|
Because counting network requests isn't very reliable when you have some external service like disqus loading lots of stuffs. Turn on |
|
yeah, I didn't write this code, talking to front-end devs to see if we can resolve, thanks for info |
|
it works ok in our existing phantomJS setup with prerender.io, but here it hangs, likely due to what you are saying. |
|
@beyondcreed I tried it locally, it works sometimes, hangs sometimes, depends on how long it takes to finish all the network requests. prerender has default timeout set to 30s, you can set envvar |
|
I think prerender.io can render it is because they disabled image loading while we didn't. #20 |
|
the problem is some sites like linkedin, facebook, etc. don't wait long, max out at 10s when crawling, so output generally needs to be pre-rendered and served relatively fast. I tried it on linkedin and it times out. going to look at #20 |
|
I looked at your website, I think it has too many javascript and css files to load, might be worth to use tools like webpack to pack vendor js/css together to reduce number of requests. |
|
we already pack it (minify and build checksums), also static resources are served off CDN. I am trying chrome with --blink-settings=imagesEnabled=false |
|
still times out |
You might be doing something wrong, this is a lot of scripts to load, ideally they should be lazy loaded on demand. |
|
On my laptop, it takes Chrome at least 10+s to load the page already. (with cache disabled)
|
|
hi, I didn't write the code, but front-end developers who did, pack it using gulp process and generate checksums so we can bust cache (you can see code is already minified if you curl the URL, e.g. spaces are removed, etc.). Front-end dev mentioned due to legacy angular framework being used code could not be easily lazily loaded, we tried it before quite a few times and there were many issues with this. |
|
phantomjs with prerender.io renders it quite fast, so I am just surprised to see this timing out, that is all |
|
also if you are loading resources from overseas (china, etc.) it may take longer to load due to cdn nodes not located there, on my chrome load is at 2.51s |
|
I understand. with |
|
22233d4#diff-b987f64b4374fdef9d36193a6f11b3ddR252 Becasue we sleep 0.2s there, manually setting |
|
ok, I can message you privately on this, may need to discuss things further. I'll email you on how we can connect |
|
ok sent you email and slack invite, thanks |
|
Hi. I upgraded to Chromium 60.0.3112.101. However same problem still persists. I run the chrome as follows: Then, I run prerender as: On console error log I am seeing 2017-08-19 15:01:53,544 INFO prerender.chromerdp.navigate:216 Page 77d8643b-7129-420f-9fd2-3000b52648f8 [1] navigating to https://www.thrinacia.com/blog/post/express-checkout-feature 2017-08-19 15:02:53,829 ERROR sanic.log:104 Traceback (most recent call last): 2017-08-19 15:02:53,829 WARNING prerender.app.handle_request:201 Got 504 for https://www.thrinacia.com/blog/post/express-checkout-feature in 60286ms |
|
I am also on slack if you need to discuss more. Thanks. |
|
@beyondcreed I left message on slack. |
I am running into issue when trying to render https based URL's. http works fine. Also chrome in headless can deal with https and spits out DOM, for example when I run:
chrome --headless --disable-gpu --dump-dom https://www.apple.comMy prerender curl request is as follows:
curl -v http://myserver.com:8000/https://www.apple.comBelow is output from prerender:
[root@node295 ~]# prerender
2017-08-16 18:37:40 - (sanic)[DEBUG]:
\u2584\u2584\u2584\u2584\u2584
\u2580\u2580\u2580\u2588\u2588\u2588\u2588\u2588\u2588\u2584\u2584\u2584 _______________
\u2584\u2584\u2584\u2584\u2584 \u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2584 /
\u2580\u2580\u2580\u2580\u2588\u2588\u2588\u2588\u2588\u258c \u2580\u2590\u2584 \u2580\u2590\u2588 | Gotta go fast! |
\u2580\u2580\u2588\u2588\u2588\u2588\u2588\u2584\u2584 \u2580\u2588\u2588\u2588\u2588\u2588\u2588\u2584\u2588\u2588 | _________________/
\u2580\u2584\u2584\u2584\u2584\u2584 \u2580\u2580\u2588\u2584\u2580\u2588\u2550\u2550\u2550\u2550\u2588\u2580 |/
\u2580\u2580\u2580\u2584 \u2580\u2580\u2588\u2588\u2588 \u2580 \u2584\u2584
\u2584\u2588\u2588\u2588\u2580\u2580\u2588\u2588\u2584\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2584 \u2584\u2580\u2580\u2580\u2580\u2580\u2580\u2588\u258c
\u2588\u2588\u2580\u2584\u2584\u2584\u2588\u2588\u2580\u2584\u2588\u2588\u2588\u2580 \u2580\u2580\u2588\u2588\u2588\u2588 \u2584\u2588\u2588
\u2584\u2580\u2580\u2580\u2584\u2588\u2588\u2584\u2580\u2580\u258c\u2588\u2588\u2588\u2588\u2592\u2592\u2592\u2592\u2592\u2592\u2588\u2588\u2588 \u258c\u2584\u2584\u2580
\u258c \u2590\u2580\u2588\u2588\u2588\u2588\u2590\u2588\u2588\u2588\u2592\u2592\u2592\u2592\u2592\u2590\u2588\u2588\u258c
\u2580\u2584\u2584\u2584\u2584\u2580 \u2580\u2580\u2588\u2588\u2588\u2588\u2592\u2592\u2592\u2592\u2584\u2588\u2588\u2580
\u2580\u2580\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2580
\u2584\u2584\u2588\u2588\u2580\u2588\u2588\u2588\u2588\u2588\u2588\u2580\u2588
\u2584\u2588\u2588\u2580 \u2580\u2580\u2580 \u2588
\u2584\u2588 \u2590\u258c
\u2584\u2584\u2584\u2584\u2588\u258c \u2580\u2588\u2584\u2584\u2584\u2584\u2580\u2580\u2584
\u258c \u2590 \u2580\u2580\u2584\u2584\u2584\u2580
\u2580\u2580\u2584\u2584\u2580
2017-08-16 18:37:40 - (sanic)[INFO]: Goin' Fast @ http://0.0.0.0:8000
2017-08-16 18:37:40,712 INFO prerender.chromerdp.new_page:47 Created new page 990b1656-05b1-4c93-9909-3299d31579b3
2017-08-16 18:37:40,723 INFO prerender.chromerdp.new_page:47 Created new page 083237b6-e7bf-4e44-97ea-a91312e636f1
2017-08-16 18:37:40,728 INFO prerender.chromerdp.new_page:47 Created new page b81c0ec4-e4a8-489c-a1be-ecd869aa861a
2017-08-16 18:37:40,735 INFO prerender.chromerdp.new_page:47 Created new page cf6d747f-d71f-4519-a74b-6a4f43e29353
2017-08-16 18:37:40,739 INFO prerender.chromerdp.new_page:47 Created new page 5272d7fd-44ed-45ff-b2c3-448fb214e29e
2017-08-16 18:37:40,747 INFO prerender.chromerdp.new_page:47 Created new page 5aac2038-452a-4b56-8e0b-a9a6fc03a8ed
2017-08-16 18:37:40,849 INFO prerender.chromerdp.new_page:47 Created new page 5d9d66ef-c68b-45b8-8e2a-711fa070a972
2017-08-16 18:37:40,863 INFO prerender.chromerdp.new_page:47 Created new page 5402ba0f-3ebd-4579-bc18-f20743deaf4e
2017-08-16 18:37:40,895 INFO prerender.chromerdp.new_page:47 Created new page ec42cd61-49ca-4319-82e4-42cffa53cd8f
2017-08-16 18:37:41,009 INFO prerender.chromerdp.new_page:47 Created new page 019aa698-ae0d-4e08-987c-63547ecd134d
2017-08-16 18:37:41,015 INFO prerender.chromerdp.new_page:47 Created new page 64d86146-6092-4b72-bbd4-c8af5e0acc66
2017-08-16 18:37:41,039 INFO prerender.chromerdp.new_page:47 Created new page fd4bb779-ee77-4ad3-8468-94348c2fda3c
2017-08-16 18:37:41,141 INFO prerender.chromerdp.new_page:47 Created new page 92221c66-8fdc-4494-a0d8-4e432fe805ee
2017-08-16 18:37:41,145 INFO prerender.chromerdp.new_page:47 Created new page 54c984c2-f04e-4b84-a267-7fb4babf9430
2017-08-16 18:37:41,198 INFO prerender.chromerdp.new_page:47 Created new page a6f6d228-4d64-4caf-906d-5ed701d019f4
2017-08-16 18:37:41,203 INFO prerender.chromerdp.new_page:47 Created new page 77111530-da40-48fc-9259-da08aa97972f
2017-08-16 18:37:41 - (sanic)[INFO]: Starting worker [95309]
2017-08-16 18:37:41,204 INFO sanic.serve:526 Starting worker [95309]
2017-08-16 18:42:07,854 INFO prerender.chromerdp.navigate:218 Page 990b1656-05b1-4c93-9909-3299d31579b3 [1] navigating to https://www.apple.com
2017-08-16 18:42:37,858 INFO websockets.protocol.fail_connection:618 Failing the WebSocket connection: 1006
2017-08-16 18:42:37,859 WARNING prerender.app._render:99 Temporary browser failure: , retry rendering https://www.apple.com in 1s
2017-08-16 18:42:37,871 ERROR asyncio.serve:527 Task exception was never retrieved
future: <Task finished coro=<Page._evaluate_prerender_ready() done, defined at /usr/local/lib/python3.6/site-packages/prerender/chromerdp.py:232> exception=AttributeError("'NoneType' object has no attribute 'send'",)>
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 234, in _evaluate_prerender_ready
res = await self.evaluate('window.prerenderReady == true')
File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 228, in evaluate
'params': {'expression': expr}
File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 178, in send
await self.websocket.send(json.dumps(payload))
AttributeError: 'NoneType' object has no attribute 'send'
2017-08-16 18:42:38,865 INFO prerender.chromerdp.navigate:218 Page 083237b6-e7bf-4e44-97ea-a91312e636f1 [1] navigating to https://www.apple.com
2017-08-16 18:43:07 - (sanic)[ERROR]: Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 143, in connection_timeout
raise RequestTimeout('Request Timeout')
sanic.exceptions.RequestTimeout: Request Timeout
2017-08-16 18:43:07,848 ERROR sanic.log:104 Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 143, in connection_timeout
raise RequestTimeout('Request Timeout')
sanic.exceptions.RequestTimeout: Request Timeout
2017-08-16 18:43:07,853 INFO websockets.protocol.fail_connection:618 Failing the WebSocket connection: 1006
2017-08-16 18:43:07,853 WARNING prerender.app.handle_request:201 Got 504 for https://www.apple.com in 60001ms
2017-08-16 18:43:07 - (sanic)[ERROR]: Connection lost before response written @ ('96.50.156.110', 18006)
2017-08-16 18:43:07,854 ERROR sanic.write_response:267 Connection lost before response written @ ('96.50.156.110', 18006)
2017-08-16 18:43:07,890 ERROR asyncio.serve:527 Task exception was never retrieved
future: <Task finished coro=<Page._evaluate_prerender_ready() done, defined at /usr/local/lib/python3.6/site-packages/prerender/chromerdp.py:232> exception=AttributeError("'NoneType' object has no attribute 'send'",)>
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 234, in _evaluate_prerender_ready
res = await self.evaluate('window.prerenderReady == true')
File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 228, in evaluate
'params': {'expression': expr}
File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 178, in send
await self.websocket.send(json.dumps(payload))
AttributeError: 'NoneType' object has no attribute 'send'
The text was updated successfully, but these errors were encountered: