Conflicting Second-Order Dependencies between bricks that do not interact with eachother

[bmritz]

I'd like to ask if there is an accepted solution to the following scenario:

I've got two bases that each require, through dependencies either of the base itself or from components required by the base, a different version of pydantic. Theoretically, when looking at the dependency tree from the project perspective, this would not be an issue, because the two bases do not interact with each other in the same project:

bases/
  myworkspace/
    base_1/   # requires pydantic >2, via code I wrote
      __init__.py
      core.py
    base_2/.  # requires dbt-duckdb, which requires pydantic <2
      __init__.py
      core.py
projects/
  project_1/  # requires base_1, but not base_2
  project_2/  # requires base_2, but not base 1

My problem is--how do I develop on the repo in this situation? What should I put into the top level pyproject.toml? I originally had pydantic > 2 in there, but when I go to add in dbt-core, I get a dependency conflict. From what I can tell here, this situation stems from having the mono-repo, and trying to have a single dev environment for multiple projects.

I was wondering if maybe I'm missing something that would help in this situation. There might be some kind of solution having to do with multiple venvs and different versions of pydantic in different dependency groups, or maybe vendoring in the conflicting repos, but I don't have the details worked out in my mind yet.

[DavidVujic]

I don't know any other way than excluding the third-party dependencies that aren't compatible with the other dependencies. Poetry will always resolve dependencies, even if they are in a optional dependency group and in this case that would mean that the Pydantic versions will be in conflict.

Ideally, all code and dependencies should be in the top-level pyproject.toml for a nice developer experience, but it is not a requirement. If you want to run the code that need the specific dependency, I would probably comment/uncomment the brick(s) and dependency when needed. This is not perfect, but we are limited in the way dependencies are resolved in a virtual environment.

Another option, maybe better, is to navigate to the actual project and create the virtual environment there. In the happy path case, the projects should be an implementation detail and not really used during development, but it could solve this particular issue. It would be just like how we would run code in different repos.

Another option is - if the dbt package will stay like that for some time and the - is to align with the limitations of it across all projects, having all projects use Pydantic 1.*.

[bmritz]

Thanks for the thoughtful reply David! I didn't think about the potential for developing in the project-centric venv. That would help this particular case, although it's a little ad hoc to have to go into the project to develop on that specific feature. I wanted to make sure I wasn't missing something obvious on how to use multiple versions. I was exploring around a little more on being able to install a third party package under a different top-level namespace, but that doesn't look possible with python packages from my googling. Thanks for the response!