I would like to configure Polyaxon in a way to avoid asking data-scientists to configure pre-emptible node-pools or request TPUs on their own

[polyaxon-team]

From slack

I am currently defining some machines configuration using machine-env1.yaml, machine-env2.yaml which basically contains node selectors and CPU, GPU, and TPU requests configuration, and then running:

polyaxon run -f polyaxonfile.yaml -f machine-env1.yaml

I have two problems with this approach:

1. I need to copy the env files to all our git repos, which means if I make a change I need to perform several pull requests
2. I need to tell the data-scientits to pull the last commit, sometimes that's not possible because they can not merge/rebase the changes.

Based on those two issues, in the end we tell data-scientists to just use:

environment:
  nodeSelector:
    nodes: large-pool
...
run:
  ...
  container:
      resources:
        limits:
          cpu: 3000m
          memory: 6000Mi
        requests:
          cpu: 2000m
          memory: 4000Mi

Which is error prone and confusing for them, and make the files bigger and difficult to change.

Any elegant way to abstract this type of configuration from the data-scientists?

[polyaxon-team]

We have already shared a resource on how to configure the environments in this guide

if you are using multiple git repos and you do not want to replicate the yaml files in all repos you can register those files as presets:

• Users will be able to use --presets machine1 or --presets=env1

Note that in the example in that link, it shows that it defines a queue but you do not have to define a queue, a preset is just any YAML file that can be used with the override operator -f main.yaml -f override1.yaml -f override2.yaml in this case override1.yaml and override2.yaml it can be saved as organization presets using the UI.

More info from the intro section about presets and the UI section

Also, when you define presets you can use them directly on the operation or component

presets: [preset1, preset2]

This is similar to the CLI command

polyaxon run -f polyaxon.yaml --presets preset1,preset2