Welcome to dvc Discussions!

[efiop]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[AdamFermier]

Hi Folks - We're starting to explore the use of DVC in our data science work and enjoying what we're finding thus far. My one question is we've always been taught to 'move the analytics to the data and not the data to the analytics' as it is cheaper to move the modeling code. However - as I'm understanding it- DVC pulls the dependent data sources down to the local machines when running and can cause a bit of trouble if your pulling in 200GB or something of the sort. Can you help clarify - it could be a basic user training issue :) Adam

[efiop]

Hi @AdamFermier !

dvc can pull part of your repo. For example, if you have 200GB dataset data and have a small model model, you could only dvc pull model and not have to pull the rest of the artifacts.

[shcheklein]

@efiop @AdamFermier should we lock this conversation? and move this to a separate topic?

@AdamFermier moving 200GB might make or not make sense. DVC is very flexible:

1. You can setup a machine with DVC cache where multiple users do their training/processing w/o affecting each other, w/o duplicating data, and still having versioning https://dvc.org/doc/use-cases/shared-development-server
2. Or that shared cache can be on NFS that is being attached to everyones machine - this way it looks like you have data in your workspace, but those are actually symlinks to a NAS storage - quite efficient and convenient since you still deal with files within your project.
3. In some cases we don't advice versioning raw data at all. DVC is not a versioning tool for the "big data" world. I would not recommend using it to version a multi-PT Hadoop cluster. In this case data science team versions artifacts - slices of data, end results (models), etc.

I guess bottom line here is - could you clarify how does your workflow look like? :) what kind of data, how do you process it, etc.