Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store compiled dependencies In _build by version #12520

Open
jwdotjs opened this issue Apr 11, 2023 · 7 comments
Open

Store compiled dependencies In _build by version #12520

jwdotjs opened this issue Apr 11, 2023 · 7 comments

Comments

@jwdotjs
Copy link

jwdotjs commented Apr 11, 2023

Elixir and Erlang/OTP versions

erlang 25.2
elixir 1.14.3-otp-25

Operating system

MacOS - Ventura 13.1 - Apple M1 Max

Current behavior

As an engineer, when I switch between branches to provide peer review to other engineers, if dependencies changed across branches, I need to re-fetch and re-compile those dependencies. On larger projects this process can take 5-20 minutes, which can lead to slowdowns in pair programming and getting new pull requests tested, especially if I keep switching back to a local branch that I'm working on where dependencies have changed.

Reproduction steps:

  1. Start on a project that is branched off main
  2. On that branch change dependencies in mix.exs (either add dependencies, remove dependencies, or upgrade dependencies)
  3. Run mix deps.get
  4. Run iex -S mix phx.server to compile dependencies and the project
  5. Run git checkout main on that project
  6. Run iex -S mix phx.server and observe an error saying the lockfile doesn't match the compiled dependencies and to run mix deps.get
  7. Run mix deps.get
  8. Run iex -S mix phx.server and observe dependencies are re-compiled

Expected behavior

Here is a proposal for deps and _build that would be a breaking change to how the project is compiled and starts up, but would speed up development by caching the compiled versions of each dependency at the cost of more disk space (which is likely an acceptable tradeoff for most).

Sample filesystem structure:

project
  deps
     name_of_dependency
         1.13.0
         1.29.3
  _build
     lib
        name_of_dependency
           1.13.0
           1.29.3

The expected outcome of this change would be when I switch between a branch that has dependency changes and a branch where I already compiled dependencies with different versions, there is no time-cost associated with compiling dependencies when I switch between those two branches after they've been compiled once each

@jwdotjs jwdotjs changed the title Explore: Store Compiled Dependencies In _build Explore: Store Compiled Dependencies In _build By Version Apr 11, 2023
@ericmj
Copy link
Member

ericmj commented Apr 11, 2023

For deps/ we can do this. But for _build/ individual dependency compilation artifacts cannot be stored by version since the version of dependency A can affect dependency B, if B calls A at compile time. Instead we would have to group all dependencies together like this:

  _build
     lib
        SOME_VERSION_1
          dep_a
          dep_b
        SOME_VERSION_2
          dep_a
          dep_b

I am not sure what version they should be grouped by though. Maybe the hash of the lock file?

@josevalim
Copy link
Member

Good call @ericmj. Perhaps the solution is to have branched builds. Something like this:

_build/
  dev/
  dev-branch/

Every time a new branch is started, it copies from the canonical "_build/dev". However, compile-time config will be an issue. When you change a compile-time config, the dep is not automatically recompiled, so if we copy from _build/dev and there is a new configuration, compilation will fail telling you that you need to explicitly recompile that dependency. Maybe we could detect those but there is additional work.

Note we also already have MIX_TARGET which could be used for this purpose but I think introducing a custom option would be fine. So, roughly, I think we need this:

  1. Add version to dependencies in deps/
  2. Allow build_per_branch: "branch_name"
  3. build_per_branch will copy from the non-branched name but it must consider compile-time configuration

@josevalim
Copy link
Member

josevalim commented Apr 19, 2023

However, compile-time config will be an issue. When you change a compile-time config, the dep is not automatically recompiled, so if we copy from _build/dev and there is a new configuration

I am working on this part of the issue on a branch. EDIT: This has been merged.

@josevalim josevalim changed the title Explore: Store Compiled Dependencies In _build By Version Store compiled dependencies In _build by version May 4, 2023
@Nezteb
Copy link
Contributor

Nezteb commented Dec 22, 2023

I am working on this part of the issue on a branch.

@josevalim Is that branch public by chance? Or possibly already merged? I couldn't find any obvious branches related to this issue. 😄

@josevalim
Copy link
Member

It is local but not really in a state to be shared yet, sorry. :(

@teamon
Copy link

teamon commented May 26, 2024

I just wanted to highlight that adding a version to deps would make NODE_PATH a lot harder when importing js files from dependencies (e.g. in the context of phoenix)

@josevalim
Copy link
Member

@teamon that's an excellent point. We could have symlinks, so deps/foo/current always point to the current version, but it is clear the behaviour would have to be opt-in.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

5 participants