Dataflow changes

[jiya-zhang]

I changed a few things in this PR:

1. Add Dockerfile entrypoint for Dataflow (this is needed for Dataflow worker to start up successfully)
2. Mount gcloud config folder to docker container (so that container will have the credentials to access GCS, etc.)
3. Format command to avoid long commands and strip trailing newlines

[markblee]

Thanks! Some initial comments/questions.

[markblee]

Thanks! A few additional suggestions while testing this PR:

1. On L175, let's change to cfg.command = f"{cfg.command.strip()} {dataflow_flags}" so that trailing newlines in the command do not cause dataflow_flags to be interpreted as a separate command.
2. L249, similar change to cmd = f"{cfg.setup_command.strip()} && {cmd}".
3. You may want to rebase onto latest main (which fixes the bundler specs) for local testing.

[markblee]

Thanks, also update the PR description?

[jiya-zhang]

@markblee Do you mind trigger the CI tests again? The error should disappear now that dependency issue was fixed and merged. (I don't have permission to rerun CI tests)

[markblee]

@markblee Do you mind trigger the CI tests again? The error should disappear now that dependency issue was fixed and merged. (I don't have permission to rerun CI tests)

It failed again with the same issue. Did you rebase?