Thanks for raising this @rodrigocaus . I was fighting this exact same issue earlier and managed to fix it by adding the scopes attribute to default() 🙌.
I was actually initially trying using federated workload identity, which isn't working either (failing with invalid argument error). My next attend was via SA key which got me investigating the issue you are fixing here.

Awesome timing and I keep digging to eventually get federated workload identity working too (feels great not having to share SA keys around)

Hi @rodrigocaus, @tuliodesouza !

Thank you for reporting this. If you have a proposal, please submit a PR and request @connor-mccarthy and @chensun to review it.

For anyone here with the exact same issue and use case. You can work around this by getting the credentials directly and feeding them into the client as an auth argument (though if you do this it wont work locally unless you set the env var)

from google.oauth2 import service_account
from kfp.registry import RegistryClient 
import os

credentials = service_account.Credentials.from_service_account_file(
    os.environ['GOOGLE_APPLICATION_CREDENTIALS'],
    scopes=['https://www.googleapis.com/auth/cloud-platform']
)
client = RegistryClient(host=KFP_REGISTRY_URL, auth=credentials)
template_name, version_name = client.upload_pipeline(
  file_name="pipeline.yaml",
  tags=["latest"],
)

I'm not sure if the scopes is needed though

We faced same issue when using the stack Github Actions + Vertex AI + KFP v2. We were able to solve it by using Oauth2 token for authentication.

In github workflow we get access token:

      - id: 'auth'
        name: 'Authenticate to Google Cloud'
        uses: 'google-github-actions/auth@v1'
        with:
          token_format: access_token
          workload_identity_provider: ....
          service_account: ....

We pass it to the step running the python code using by referring to ${{ steps.auth.outputs.access_token }}.

The actual python code then is:

from kfp.registry import RegistryClient, ApiAuth

client = RegistryClient(host=KFP_REGISTRY_URL, auth=ApiAuth(access_token))
template_name, version_name = client.upload_pipeline(
  file_name="pipeline.yaml",
  tags=["latest"],
)

In case of using WIF and service accounts to provide Github actions access to GCP, it seems RegistryClient cannot use default Google credentials from current environment (unlike Vertex AI SDK). OAuth2 token needs to be provided explicitly. However, when one is authenticated using personal GCP-account and runs the Python script from localhost it seems it works without OAuth2 token.

We faced same issue when using the stack Github Actions + Vertex AI + KFP v2. We were able to solve it by using Oauth2 token for authentication.

In github workflow we get access token:

      - id: 'auth'
        name: 'Authenticate to Google Cloud'
        uses: 'google-github-actions/auth@v1'
        with:
          token_format: access_token
          workload_identity_provider: ....
          service_account: ....

We pass it to the step running the python code using by referring to ${{ steps.auth.outputs.access_token }}.

The actual python code then is:

from kfp.registry import RegistryClient, ApiAuth

client = RegistryClient(host=KFP_REGISTRY_URL, auth=ApiAuth(access_token))
template_name, version_name = client.upload_pipeline(
  file_name="pipeline.yaml",
  tags=["latest"],
)

In case of using WIF and service accounts to provide Github actions access to GCP, it seems RegistryClient cannot use default Google credentials from current environment (unlike Vertex AI SDK). OAuth2 token needs to be provided explicitly. However, when one is authenticated using personal GCP-account and runs the Python script from localhost it seems it works without OAuth2 token.

Thank you so much for this. We are facing same and your experience has helped us

I'm facing the same problem.
Wednesday was running without any problem.
Since yesterday I'm facing it, as decribed below.

I'm receiving the errors

• HTTPError: 400 Client Error: Bad Request for url : when I try running without auth.
• HTTPError: 401 Client Error: Unauthorized for url : when I run with auth, using personal GCP account and with SA.

I tried changing kfp version to the latest, but the problem continued.

What should I do?

I would also like to bump this issue.
Ideally, to run KFP in enterprise setting, it is important to support authentication with artifact registry for respective cloud provider.

In the case of google cloud, if KFP could authenticate with Google Artifact registry using KFP @dsl . It will be cleanest solution because during auth, it can check for the service account and it's permissions.

Otherwise, the only other option left is to build your own custom docker container and use it as the base image for KFP

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Closing this issue, there is a workaround documented and it's platform specific.

/close

@rimolive: Closing this issue.