Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Pixi as an option for installation #3130

Merged
merged 10 commits into from May 19, 2024

Conversation

pablovela5620
Copy link
Contributor

This PR adds the option to include Pixi as an installation option. This still keeps the normal recommended installation option working with no modifications.

Why Pixi

I think its best explained to show what it takes to normally install with the recommended method on linux.

Conda

  1. Make sure that conda is installed (4 commands based on this)
mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh
  1. Create a conda and activate a conda environment (3 commands)
conda create --name nerfstudio -y python=3.8
conda activate nerfstudio
python -m pip install --upgrade pip
  1. Uninstall and reinstall pytorch (2 commands)
pip uninstall torch torchvision functorch tinycudann
pip install torch==2.1.2+cu118 torchvision==0.16.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118
  1. add cuda toolkit (1 command)
conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit
  1. Install tcnn (1 command)
pip install ninja git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch

Optional (if using custom data)
7. Install colmap (if from source maybe another 7 commands?)
8. Install hloc (another 3 commands)

in total (if we include optional things that folks have a lot of issues with) this comes up to around 21 separate commands that a user has to run, and make sure they do correctly

Pixi

  1. make sure pixi is installed (2 commands)
curl -fsSL https://pixi.sh/install.sh | bash
source ~/.bashrc
  1. run examples (2 commands)
pixi run post-install
pixi run train-example-splat

this version does everything the above does but in just 4 commands, all while making sure that the user has the right version of pytorch cuda, cudatoolkit, tinycudann, colmap hloc etc....

pixi uses the conda and pip ecosystem under the hood, but includes a bunch of extra really nice features like reproducibility via lock files, a SUPER powerful task system (this is what made installing things like tinycudann easy)

I think this would make things for user much MUCH easier to get started.

if one wants just the same experience as creating a conda and activating a conda environment, and just running the typical cli commands, all that's required is

pixi run post-install
pixi shell

and this will activate the underlying conda environment like usual but with everything setup

Let me know if ya'll have any questions! I already have a working implementation of this for the in DN-Splatter repo that's based on nerfstudio if you'd like to see another example

pablovela5620 and others added 4 commits May 7, 2024 18:14
The commit updates the download and training commands for the dozer data in the `pixi.toml` file. It renames the `download-sample-data` command to `download-dozer-data` and modifies the `train-example-splat` and `train-example-nerf` commands to include the dozer data as a dependency.

This makes it easy to run an example training for both nerfacto and splatfacto
pixi.toml Outdated Show resolved Hide resolved
@pablovela5620
Copy link
Contributor Author

I also added install + minimal usage instructions to the PR. I wanted to add a few caveats

  1. Currently this is only for linux (I don't have a windows machine to test on)
  2. Colmap should technically install the GPU version if the underlying system has cuda 11.8 installed. I don't (I have 12.1) and for some reason the cuda version installed directly doesn't seem to want to accept the GPU version of colmap. So in my case for colmap to work I have to use the --no-gpu flag. My understanding is that this is only for SIFT, so I also included hloc for more modern matches and detectors. I had to do this manually v.s. doing it in the conda ecosystem as hloc requires third party libraries that are not included missing thirdparty, cannot use matchers like lightglue/superglue conda-forge/hloc-feedstock#2
  3. This is a minimal implementation, there's probably a better way of doing it, but I thought just getting something up would be helpful to folks. I can look into seeing if this works for windows later if someone has a windows PC they're willing to help me debug with

Copy link
Collaborator

@maturk maturk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@pablovela5620, don't worry about the colmap thing, this seems to be outside the scope of a Pixi install. One thing to note is that this also does not work with macos, due to the platforms = ["linux-64"] flag and cuda dependency.

For the future, it would be interesting the support both Windows and macos with a Pixi install.

@maturk maturk merged commit 8d5502d into nerfstudio-project:main May 19, 2024
2 checks passed
@brentyi
Copy link
Collaborator

brentyi commented May 20, 2024

I just learned about pixi through this, cool work @pablovela5620 @maturk!

One question: the version string in the pixi.toml now looks out-of-date relative to the pyproject.toml one. Should the match? Is it necessary to maintain both? Perhaps we add comment to the .toml files saying the versions should be manually synchronized?

@ruben-arts
Copy link

@pablovela5620 Thanks for sharing the work and using pixi for what it is built for!

@brentyi Pixi dev here, currently the version in the pixi.toml is not used, so removing it would make sense in this case.

That said it is also possible to add the pixi information to the pyproject.toml by prefixing all TOML tables with [tool.pixi.xxx] Pixi will understand this and will also automatically make use of the (optional-)dependencies. But this can always be done later when you get used to using pixi.

@pablovela5620 pablovela5620 deleted the pixi branch May 20, 2024 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants