Are Tensorboad services preventing CPU nodes to scale down?

[polyaxon-team]

From slack

When I schedule a tensorboard from the UI does it always schedule on the polyaxon "core nodes", or does it schedule on any cpu machine that is available?

We have an issue sometimes with cpu nodes not scaling down, and I am wondering if it is because tensorboards are running on them?

[polyaxon-team]

By default, Tensorboard service does not use any node scheduling, so it probably uses the default nodes.
If you need to use a different node selectors or attach presets. When you click start tensorboard or run downstreamOp, the modal shows an editor which is basically prefilled a polyaxonfile:

you can just assign a preset if you have one:

presets: [tensorboard-cpu-preset]

Or patch the run with a node selector:

runPatch:
  environment:
    nodeSelector:
      polyaxon_pool_type: polyaxon-tensorboard

Note that for Tensorboad, by default it loads the component from here: Tensorboard Component
You can create your own Tensorboad component by following this tutorial
In that case you will use:

hubRef: ORG/tensorboard:VERSION

Finally you can add this preset if you think that you might forget about a services running forever: https://polyaxon.com/docs/core/scheduling-presets/services-timeout/

Or adding it before clicking start:

termination:
  timeout: 86400 # 24 hours