[Bug] temporal_visibility schema mismatch. Looking for v1.16 but only 1.9 is available

[dannyoka]

What are you really trying to do?

Trying to run temporal in a K8s cluster

Describe the bug

Several temporal services that check the DB for schema versions. I'm using postgres as my DB and running the setup tool from the temporal repo. It's updating the temporal schema to v 1.16, and temporal_visibility to 1.9, but the code is complaining that it's expecting temporal_visibility to be at 1.16 as well.

sql schema version compatibility check failed: version mismatch for keyspace/database: "temporal_visibility". Expected version: 1.16 cannot be greater than Actual version: 1.9
Unable to create server. Error: could not build arguments for function "go.uber.org/fx".(*module).constructCustomLogger.func2 (/home/runner/go/pkg/mod/go.uber.org/[email protected]/module.go:294): failed to build fxevent.Logger: could not build arguments for function "go.temporal.io/server/temporal".init.func7 (/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:1008): failed to build log.Logger: received non-nil error from function "go.temporal.io/server/temporal".ServerOptionsProvider (/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:182): sql schema version compatibility check failed: version mismatch for keyspace/database: "temporal_visibility". Expected version: 1.16 cannot be greater than Actual version: 1.9.

Minimal Reproduction

1. Use the temporal-sql-tool with a postgres target.
2. Spin up temporal helm chart pointing to that postgres instance
3. View logs of temporaltest-frontend and temporaltest-history, temporaltest-matching and temporaltest-worker

Environment/Versions

Using in kubernetes

Additional context