Confusion about the local development flow

[svartalf]

I'm experimenting with Polylith here and it's not clear how to develop and run things locally, so I would appreciate some guidelines here.
The examples later will be based on the python-polylith-example-rye repo, commit 4353383 (latest HEAD at ab029e9 seems to be broken, see note below).

With this repository checked out locally, I'm running rye sync first:

$ rye sync
Initializing new virtualenv in /home/svartalf/projects/python-polylith-example-rye/.venv
Python version: cpython@3.12.1
Generating production lockfile: /home/svartalf/projects/python-polylith-example-rye/requirements.lock
Generating dev lockfile: /home/svartalf/projects/python-polylith-example-rye/requirements-dev.lock
Installing dependencies
… skipping a lot of information here …
Successfully built python-polylith-example-with-rye
Installing collected packages: deprecation, confluent-kafka, cloudevents, werkzeug, watchdog, uvicorn, typing-extensions, typer, tomlkit, starlette, sqlalchemy, sniffio, shellingham, ruff, rich, python-polylith-example-with-rye, pytest, pygments, pydantic-core, pydantic, polylith-cli, pluggy, packaging, mypy-extensions, mypy, mdurl, markupsafe, markdown-it-py, jinja2, itsdangerous, iniconfig, idna, h11, gunicorn, greenlet, functions-framework, flask, fastapi, click, blinker, anyio, annotated-types
Successfully installed annotated-types-0.6.0 anyio-3.7.1 blinker-1.7.0 click-8.1.7 cloudevents-1.10.1 confluent-kafka-2.3.0 deprecation-2.1.0 fastapi-0.106.0 flask-3.0.3 functions-framework-3.5.0 greenlet-3.0.3 gunicorn-21.2.0 h11-0.14.0 idna-3.6 iniconfig-2.0.0 itsdangerous-2.1.2 jinja2-3.1.3 markdown-it-py-3.0.0 markupsafe-2.1.5 mdurl-0.1.2 mypy-1.8.0 mypy-extensions-1.0.0 packaging-24.0 pluggy-1.4.0 polylith-cli-1.5.1 pydantic-2.6.4 pydantic-core-2.16.3 pygments-2.17.2 pytest-7.4.4 python-polylith-example-with-rye-0.1.0 rich-13.7.1 ruff-0.2.2 shellingham-1.5.4 sniffio-1.3.1 sqlalchemy-2.0.29 starlette-0.27.0 tomlkit-0.11.8 typer-0.12.3 typing-extensions-4.11.0 uvicorn-0.25.0 watchdog-4.0.0 werkzeug-3.0.2
Done!

Now, if I want to run a project (or base) locally to ensure that it works correctly, I'm using what this docs page states:

Example: running a FastAPI service locally

Start the service by specifying the namespace path to your entry point (i.e. the base).

uvicorn my_top_namespace.my_fastapi_service.core:app

With the example project structure, it is possible to run it as the following:

$ rye run uvicorn example.greet_api.core:app
INFO:     Started server process [111199]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)

The problem I'm having is that if I would go to the bases/example/greet_api/core.py file and add the raise ValueError() line directly at the top of it to ensure that it breaks immediately during the import:

diff --git a/bases/example/greet_api/core.py b/bases/example/greet_api/core.py
index 67ff900..7e797f5 100644
--- a/bases/example/greet_api/core.py
+++ b/bases/example/greet_api/core.py
@@ -1,3 +1,5 @@
+raise ValueError()
+
 from example import greeting
 from example.log import get_logger
 from fastapi import FastAPI

then executing the $ rye run uvicorn example.greet_api.core:app would still work correctly!

It happens because during the rye sync, the example project itself is also installed into the virtual environment, we can see it in the logs above:

Installing collected packages: … python-polylith-example-with-rye-0.1.0 …

Because of that, local development flow is not possible anymore, as any changes you make to your source code are not picked up by Python anymore, since the installed package in the virtual environment takes precedence.

Rye allows us to make the root project a virtual one:

index eeca26e..ec74e4b 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -20,6 +20,7 @@ dependencies = [
 
 [tool.rye]
 managed = true
+virtual = true
 dev-dependencies = [
   "mypy~=1.8.0",
   "polylith-cli",

That would stop us from having our own project installed into the virtual environment, but, of course, rye also can't import it anymore (virtual env was removed and re-created here):

$ rye run uvicorn example.greet_api.core:app
Traceback (most recent call last):
… redacted …
ModuleNotFoundError: No module named 'example'

I feel like I'm missing some obvious way to run things locally and at the same time being able to change the files, any help would be greatly appreciated! :)

Side note

Example repo at ab029e9 commit looks broken, with the freshly-checked-out repo, rye sync results in the following error (parts of the traceback are truncated for readability)

Show the traceback

$ rye sync
Initializing new virtualenv in /home/svartalf/projects/python-polylith-example-rye/.venv
Python version: cpython@3.12.1
Generating production lockfile: /home/svartalf/projects/python-polylith-example-rye/requirements.lock
    error: subprocess-exited-with-error
    
    × Preparing metadata (pyproject.toml) did not run successfully.
    │ exit code: 1
    ╰─> [43 lines of output]
        Traceback (most recent call last):
     … redacted …
        ValueError: Unable to determine which files to ship inside the wheel using the following heuristics: https://hatch.pypa.io/latest/plugins/builder/wheel/#default-file-selection
        
        The most likely cause of this is that there is no directory that matches the name of your project (python_polylith_example_with_rye).
        
        At least one file selection option must be defined in the `tool.hatch.build.targets.wheel` table, see: https://hatch.pypa.io/latest/config/build/
        
        As an example, if you intend to ship a directory named `foo` that resides within a `src` directory located at the root of your project, you can define the following:
        
        [tool.hatch.build.targets.wheel]
        packages = ["src/foo"]
        [end of output]
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
Traceback (most recent call last):
    … redacted …
    raise MetadataGenerationFailed(package_details=details) from error
pip._internal.exceptions.MetadataGenerationFailed: metadata generation failed
error: could not write production lockfile for project

Caused by:
    failed to generate lockfile

[DavidVujic]

Hi @svartalf and thank you for the very detailed description, this is much appreciated 🙏

Yesterday I found a bug in the Rye pyproject.toml at the root of the workspace - where I accidentally had added the hatch build-hook. After some testing and experimenting in #198 we finally found this little typo causing so much trouble. I think I had mixed things up between Rye and PDM that needs a hook at the top level too (and I am not using Rye on a regular basis).

The docs are also updated with the correct information since yesterday.

I have tried to reproduce what you describe with the raise ValueError but with the latest version I think it works as it should: after altering the base and re-running the API, it fails because of the ValueError.

You can also append the --reload to the uvicorn command to have it refresh on code changes. This should work now, but since you are having issues I am wondering if it might be a virtual environment that is not updated correctly? If so, it is probably because of the typos in the previous version of the example repo.

If you haven't already, can you delete the .venv folder (alternatively run git clean -dfx in the example repo) and run rye sync again? I think that should solve it and you should have a development environment without the source code being directed into the virtual environment (as before the latest updates in the example repo).

[svartalf]

Thank you for the prompt response, @DavidVujic!

I removed the hatch build hook and everything seems to be working correctly now :)

[svartalf]

Another update: it seems that that python-polylith-example-rye I was experienced was caused by this issue: astral-sh/rye#515

Pinning hatchling to hatchling<=1.18.0 solved the problem.

[DavidVujic]

Good find! 👏