Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Temporal fails to connect to Google Cloud SQL Postgres when password contains certain characters #5729

Open
NAjustin opened this issue Apr 15, 2024 · 0 comments

Comments

@NAjustin
Copy link

Expected Behavior

Temporal should connect correctly regardless of what characters are contained in the password.

Actual Behavior

Temporal actually has mixed results when certain characters are present:

  1. The temporal-sql-tool appears to run correctly. For example, it creates the temporal and temporal_visibility databases, as well as creates/updates the schemas. This also proves that the password is, in fact, valid.
  2. But that's where success ends. Once the creation/update is handled, connections begin to error: sql schema version compatibility check failed: pq: password authentication failed for user "example-user"

Here's the full section of logs containing errors:

[Fx] ERROR  Failed to initialize custom logger: could not build arguments for function "go.uber.org/fx".(*App).constructCustomLogger.func2
/go/pkg/mod/go.uber.org/fx@v1.18.2/app.go:414:
failed to build fxevent.Logger:
could not build arguments for function "go.temporal.io/server/temporal".glob..func8
/home/builder/temporal/temporal/fx.go:1025:
failed to build log.Logger:
received non-nil error from function "go.temporal.io/server/temporal".ServerOptionsProvider
/home/builder/temporal/temporal/fx.go:159:
sql schema version compatibility check failed: pq: password authentication failed for user "example-user"
Unable to create server. Error: could not build arguments for function "go.uber.org/fx".(*App).constructCustomLogger.func2 (/go/pkg/mod/go.uber.org/fx@v1.18.2/app.go:414): failed to build fxevent.Logger: could not build arguments for function "go.temporal.io/server/temporal".glob..func8 (/home/builder/temporal/temporal/fx.go:1025): failed to build log.Logger: received non-nil error from function "go.temporal.io/server/temporal".ServerOptionsProvider (/home/builder/temporal/temporal/fx.go:159): sql schema version compatibility check failed: pq: password authentication failed for user "example-user".

Steps to Reproduce the Problem

  1. Use special characters in your password. Verified (failed) example: y{0zzu2p3\t*az<g
  2. Deploy Temporal (I used Helm)
  3. Enjoy the errors

Changing the password to not include the above characters makes the identical setup work correctly (and also fixes the surrounding errors). I didn't have the time to hunt down the offending character sequence, but my money is on the \t getting turned into a tab somewhere in the execution (e.g. before or after the password gets turned into a DSN)—but because the temporal-sql-tool scripts run in the same deployment, I think it's safe to rule out environmental factors. It may just be that net/url.QueryEscape isn't the right tool for the job to always generate a valid DSN, or it may be somewhere else in that code flow.

(It doesn't appear that this code flow has changed in more recent versions than what's being supplied in my upstream.)

Specifications

  • Version: 1.20.1 (tested via temporalio/auto-setup:1.20.1)
  • Platform: GKE Autopilot (password passed via env variable from Kubernetes Secret) using the Postgres plugin/DB provider to connect to Google Cloud SQL (Postgres 13).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant