Migration to Gevent #2029

Constantin1489 · 2023-12-03T13:02:09Z

If you check the https://github.com/Constantin1489/changedetection.io/actions/runs/7076892148, it shows every test failed randomly. (sometimes failed early, sometimes went further luckily. I love it.)

I turned off the monkey patch for threads. Our major threads are mostly lxml-dependent(lxml is rooted in C. https://stackoverflow.com/a/25275718/20307768). Also, as we already noticed our threads run lxml-dependent modules. But the problem is I don't know whether "I turned off the monkey patch for threads" is right. When I remove all monkeys or fully bring monkeys, the test also failed, BTW. for those cases, I assume the reason will also differ. To be honest , I don't know really this kind of the level.

I just assume and believe I know the reason. I don't know what to do currently.

Constantin1489 · 2023-12-03T13:24:31Z

Also for the longer test, there is some stackoverflow answer. But I didn't test it.

Constantin1489 · 2023-12-03T13:32:52Z

changedetectionio/__init__.py

-        eventlet.wsgi.server(eventlet.listen((host, int(port)), s_type), app)
-
+        ssl_args = {}
+    sock = WSGIServer.get_listener((host, port), family=s_type)


I tested with Netstat, ipv6 also works(?). FYI, http://[::1]:5000

Constantin1489 · 2023-12-03T13:33:08Z

changedetectionio/__init__.py

-    sys.exit()
+    cd_server.stop()
+    cd_server.close()
+    print('Shutdown Success', file=sys.stderr)


This is just for logging.

dgtlmoon · 2023-12-03T13:41:23Z

by the way, the application MUST stay as a single-threaded application

this is because the data-structure is stored as a single dict, and running separate threads in gevent/flask means each thread gets its own datastructure, which is bad.

this explains your random failures

it is more like Redis than a traditional system where it has an external database

dgtlmoon · 2023-12-03T13:44:37Z

requirements.txt

@@ -1,4 +1,4 @@
-eventlet>=0.33.3 # related to dnspython fixes
+gevent


related to dnspython fixes

also means we can maybe un-pin some package versions.. i wonder

oh the dnspython I will apply it later.

eventlet/eventlet#781 is that 'dnspython' problem i believe, but actually it may not be a 'thing' anymore

Constantin1489 · 2023-12-03T13:50:23Z

then the reason why my CD works well is I don't do massive things frequently, haha.

dgtlmoon · 2023-12-22T11:06:52Z

maybe gunicorn is an alternative too? --threads=1

Constantin1489 · 2023-12-23T02:38:45Z

@dgtlmoon gunicorn won't work with Windows. https://github.com/benoitc/gunicorn/blob/26aba9ed9d55f6465fd2271169b71bc119ea8186/pyproject.toml#L17
Also, gunicorn introduces itself as "Gunicorn 'Green Unicorn' is a Python WSGI HTTP Server for UNIX." (https://github.com/benoitc/gunicorn/blob/26aba9ed9d55f6465fd2271169b71bc119ea8186/README.rst?plain=1#L20)

Constantin1489 · 2024-01-19T05:53:58Z

I'm not sure about when I can finish this (maybe 2-3 months?).
I'm currently trying to expand my knowledge about concurrency with books and so on. (BTW I finally figured out what context the single-threaded means.)
If you have good resources, I'd be glad to look at them.

Please ignore any CI activities related to this PR until I'm prepared. Thanks.
If there is any question, please ask me to make my understanding better.

dgtlmoon · 2024-01-22T11:30:40Z

Currently I'm focusing threading and asyncio

The main problem here is that one thread is holding all the data, usually one would use something like mysql or whatever to keep the data in a resource that's accessible to any thread..

I would prefer to keep the application running without any need for mysql/mongo/etc/etc

The main solution would be that the data structure needs to be in some shared-memory resource between each thread, but this is unreliable a little bit in windows/mac.. unless theres some python library that will help

https://docs.python.org/3/library/multiprocessing.shared_memory.html for example

or move to gevent but keep exactly 1 thread only (like it is now)

i dont think there's much to win right now... better to move to multilang or some other feature perhaps (but ofcourse its up to you :) )

Constantin1489 · 2024-01-22T11:45:16Z

Yeah because Python is the single-threaded model, one process-one main thread (and C extension module..). That's something tempting. Okay, why not Babel first? I will look at it. If my intuition is right, I can lay the foundation this week?

dgtlmoon · 2024-01-22T11:46:42Z

Okay, why not Babel first? I will look at it. If my intuition is right, I can lay the foundation this week?

sure :) thanks a billion :)

changedetectionio/__init__.py

Co-authored-by: dgtlmoon <leigh@morresi.net>

dgtlmoon · 2024-01-23T15:28:29Z

https://wsgi.readthedocs.io/en/latest/definitions.html

since gevent just implements the WSGI standard, the flags are the same.

https://wsgi.readthedocs.io/en/latest/definitions.html#envvar-wsgi.multithread

https://wsgi.readthedocs.io/en/latest/definitions.html#envvar-wsgi.multiprocess

I think wsgi.multiprocess should be True because one changedetection.io process can have some child processes, like when it launches playwright

dgtlmoon · 2024-01-23T15:30:21Z

https://code.google.com/archive/p/modwsgi/wikis/ProcessesAndThreading.wiki

ahh multiprocess means something else in this context, so yeah, leave it False

dgtlmoon · 2024-01-23T15:52:57Z

Some history on the pinned dnspython version over at #1314 , it could be nice to remove the pin and let the version float - most OSs usually have this installed by default and any modern version should work

Constantin1489 · 2024-01-24T03:38:16Z

See also: https://eventlet.readthedocs.io/en/latest/migration.html#alternatives

If you really want to continue with Eventlet’s pretend-to-be-blocking approach, you can use gevent. But keep in mind that the same technical issues that make Eventlet maintenance unsustainable over the long term also apply to Gevent.

dgtlmoon · 2024-01-24T08:35:46Z

aaah... yeah thanks for the find, it is true that async code is better

Pretending to look like a blocking API while actually using an event loop underneath requires exact emulation of an ever-changing and ever-increasing API footprint, which is fundamentally unsustainable for a volunteer-driven open source project. This is why Eventlet is discouraging new users.

ok so eventlet and gevent are actually blocking-style wrappers ontop of what is really a asyncio on the "ground level" in python... so yeah thats kinda tough

dgtlmoon · 2024-01-24T08:37:41Z

I still think gevent is the way to go, it looks well maintained https://www.gevent.org/changelog.html

we can add some comment that "gevent is just a blocking wrapper ontop of what is really a asyncio interface at its heart, but due to our not-shared-across-processes datastructure (so you dont ever have to waste time setting up mysql etc) we have a single threaded model"

dgtlmoon · 2024-01-24T08:38:06Z

I'm learning more about doing fun stuff with python dicts, so i hope we can move to maybe even a sqlite backend in the future

Constantin1489 · 2024-05-16T10:26:23Z

I closed this because I saw the gevent won't work for future python version too.(there was CPython issue similar to eventlet)

dgtlmoon · 2024-05-16T10:28:10Z

I closed this because I saw the gevent won't work for future python version too.(there was CPython issue similar to eventlet)

Understood, I'm sort of thinking it may be time to start migrating to SQLite in the backend and do things more traditionally so that we can run it multithreaded

Constantin1489 added 2 commits December 3, 2023 20:02

feat: migration to gevent from eventlet

30ae7dc

fix: fix dict keys to str

e411aaf

Constantin1489 changed the title ~~[WIP] Migration to Gevent~~ [WIP] Migration to Gevent (without python3.12) Dec 3, 2023

Constantin1489 commented Dec 3, 2023

View reviewed changes

dgtlmoon reviewed Dec 3, 2023

View reviewed changes

dgtlmoon mentioned this pull request Dec 22, 2023

UI - 304 'Not Modified' cached handling causes some requests to fail, can affect reverse-proxies #2053

Open

dgtlmoon force-pushed the master branch from 723c20e to 3ac0c93 Compare January 5, 2024 17:27

Merge branch 'master' into gevent

b48f08b

Constantin1489 added 2 commits January 20, 2024 13:03

gevent: Disable gevent monkeypatch

3bcd1f0

Merge branch 'master' into gevent

b4a80b8

Constantin1489 marked this pull request as draft January 21, 2024 12:02

dgtlmoon mentioned this pull request Jan 23, 2024

mac: pip install leads to "NotImplementedError: unsupported platform" #2126

Open

dgtlmoon reviewed Jan 23, 2024

View reviewed changes

changedetectionio/__init__.py Outdated Show resolved Hide resolved

Update changedetectionio/__init__.py

9351bb9

Co-authored-by: dgtlmoon <leigh@morresi.net>

Constantin1489 added 2 commits February 16, 2024 00:55

Merge branch 'master' into gevent

eccf60b

requirements/chore: Unpin dnspython

258f227

Constantin1489 marked this pull request as ready for review February 16, 2024 10:38

Constantin1489 changed the title ~~[WIP] Migration to Gevent (without python3.12)~~ Migration to Gevent Feb 16, 2024

dgtlmoon mentioned this pull request Feb 20, 2024

requirements/chore: temporary dnspython<2.6 pin #2202

Closed

Constantin1489 closed this May 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migration to Gevent #2029

Migration to Gevent #2029

Constantin1489 commented Dec 3, 2023 •

edited

Constantin1489 commented Dec 3, 2023

Constantin1489 Dec 3, 2023 •

edited

Constantin1489 Dec 3, 2023

dgtlmoon commented Dec 3, 2023 •

edited

dgtlmoon Dec 3, 2023

Constantin1489 Dec 5, 2023

dgtlmoon Dec 22, 2023

Constantin1489 commented Dec 3, 2023 •

edited

dgtlmoon commented Dec 22, 2023

Constantin1489 commented Dec 23, 2023 •

edited

Constantin1489 commented Jan 19, 2024 •

edited

dgtlmoon commented Jan 22, 2024 •

edited

Constantin1489 commented Jan 22, 2024 •

edited

dgtlmoon commented Jan 22, 2024

dgtlmoon commented Jan 23, 2024

dgtlmoon commented Jan 23, 2024

dgtlmoon commented Jan 23, 2024

Constantin1489 commented Jan 24, 2024

dgtlmoon commented Jan 24, 2024

dgtlmoon commented Jan 24, 2024

dgtlmoon commented Jan 24, 2024

Constantin1489 commented May 16, 2024

dgtlmoon commented May 16, 2024

		@@ -1,4 +1,4 @@
		eventlet>=0.33.3 # related to dnspython fixes
		gevent

Migration to Gevent #2029

Migration to Gevent #2029

Conversation

Constantin1489 commented Dec 3, 2023 • edited

Constantin1489 commented Dec 3, 2023

Constantin1489 Dec 3, 2023 • edited

Choose a reason for hiding this comment

Constantin1489 Dec 3, 2023

Choose a reason for hiding this comment

dgtlmoon commented Dec 3, 2023 • edited

dgtlmoon Dec 3, 2023

Choose a reason for hiding this comment

Constantin1489 Dec 5, 2023

Choose a reason for hiding this comment

dgtlmoon Dec 22, 2023

Choose a reason for hiding this comment

Constantin1489 commented Dec 3, 2023 • edited

dgtlmoon commented Dec 22, 2023

Constantin1489 commented Dec 23, 2023 • edited

Constantin1489 commented Jan 19, 2024 • edited

dgtlmoon commented Jan 22, 2024 • edited

Constantin1489 commented Jan 22, 2024 • edited

dgtlmoon commented Jan 22, 2024

dgtlmoon commented Jan 23, 2024

dgtlmoon commented Jan 23, 2024

dgtlmoon commented Jan 23, 2024

Constantin1489 commented Jan 24, 2024

dgtlmoon commented Jan 24, 2024

dgtlmoon commented Jan 24, 2024

dgtlmoon commented Jan 24, 2024

Constantin1489 commented May 16, 2024

dgtlmoon commented May 16, 2024

Constantin1489 commented Dec 3, 2023 •

edited

Constantin1489 Dec 3, 2023 •

edited

dgtlmoon commented Dec 3, 2023 •

edited

Constantin1489 commented Dec 3, 2023 •

edited

Constantin1489 commented Dec 23, 2023 •

edited

Constantin1489 commented Jan 19, 2024 •

edited

dgtlmoon commented Jan 22, 2024 •

edited

Constantin1489 commented Jan 22, 2024 •

edited