Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug] pipeline local Docker Executor sending extra args #10788

Open
chinwobble opened this issue May 6, 2024 · 0 comments
Open

[bug] pipeline local Docker Executor sending extra args #10788

chinwobble opened this issue May 6, 2024 · 0 comments

Comments

@chinwobble
Copy link

chinwobble commented May 6, 2024

Environment

  • How do you deploy Kubeflow Pipelines (KFP)?
  • KFP version: N/A
  • KFP SDK version: 2.7.0
  • python docker package: 6.0.1

Steps to reproduce

  1. Use the local runner to run with docker
  2. Run the pipeline below with the following config (below)
  3. Get a docker error because one of the python packages is injecting an extra parameter.
    See error below
    launch_experiment.py: error: unrecognized arguments: python katib_experiment_launcher.yaml
  1. Looking at the docker container that was executed it has the following args.
"Path": "python",
	"Args": [
		"src/launch_experiment.py",
		"python",
		"src/launch_experiment.py",
		"--experiment-name",
		"experiment-fashion",
		"--experiment-namespace",
		"kubeflow-zeroone-gg",
		"--experiment-spec",
                "..."
	]

The ENTRYPOINT in the dockerfile is being prepended to the docker command. The implementation.container.command should override this.
Instead the container is running

python src/launch_experiment.py python src/launch_experiment.py ....
import kfp.dsl as dsl
import kfp
import kfp.local as local
import kfp.components as components
import yaml

local.init(runner=local.DockerRunner())
NAMESPACE = 'kubeflow-zeroone-gg'
katib_experiment_launcher_op = components.load_component_from_file('try_kubeflow/components/katib_experiment_launcher.yaml')

def get_experiment():
    with open('try_kubeflow/experiments/experiment.yaml') as f:
        return yaml.load(f, Loader=yaml.CLoader)

@dsl.pipeline()
def my_pipeline():
    op = katib_experiment_launcher_op(
        experiment_name='experiment-fashion',
        experiment_namespace=NAMESPACE,
        experiment_spec=get_experiment()
    ).set_display_name('train')

if __name__ == '__main__':
    my_pipeline()

Copied from an example component in this repo.

name: Katib - Launch Experiment
description: Katib Experiment launcher
inputs:
- {name: Experiment Name,            type: String,       default: '',        description: 'Experiment name'}
- {name: Experiment Namespace,       type: String,       default: anonymous, description: 'Experiment namespace'}
- {name: Experiment Spec,            type: JsonObject,   default: '{}',      description: 'Experiment specification in dict format'}
- {name: Experiment Timeout Minutes, type: Integer,      default: 1440,      description: 'Time in minutes to wait for the Experiment to complete'}
- {name: Delete Finished Experiment, type: Bool,         default: 'True',    description: 'Whether to delete the Experiment after it is finished'}
outputs:
- {name: Best Parameter Set,         type: JsonObject,                       description: 'The hyperparameter set of the best Experiment Trial'}
implementation:
  container:
    image: docker.io/kubeflowkatib/kubeflow-pipelines-launcher
    command: [python, src/launch_experiment.py]
    args: [
      --experiment-name,            {inputValue: Experiment Name},
      --experiment-namespace,       {inputValue: Experiment Namespace},
      --experiment-spec,            {inputValue: Experiment Spec},
      --experiment-timeout-minutes, {inputValue: Experiment Timeout Minutes},
      --delete-after-done,          {inputValue: Delete Finished Experiment},
      --output-file,                {outputPath: Best Parameter Set},
    ]

Expected result

Running locally should run in the same way as running the pipeline on the api-server.
The component yaml file has specified a implementation.container.command. This field should override whatever set as the entrypoint in the Dockerfile.

Materials and reference

Labels

/area sdk


Impacted by this bug? Give it a 👍.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant