We couldn’t reproduce Keycloak’s most famous upgrade failure. Then we found the two things that actually cause it.
Here is the story every Keycloak operator has heard: an upgrade dies partway
through the schema migration, Liquibase's DATABASECHANGELOGLOCK row is never
released, and every subsequent start hangs waiting for a lock nobody holds —
including the start you attempt in order to roll back. It is the canonical
upgrade horror story, and it is repeated as fact.
We set out to reproduce the stuck lock. By the two textbook methods — killing
the process, and aborting its SQL — we could not. What
we found instead is that the half-migrated database is real, and it is produced
not by a crash or a stuck lock but by two ordinary configurations: a
statement_timeout shorter than deadlock_timeout, and a connection pooler in
transaction mode.
We tried to break it two ways and failed, then found it twice by accident.
The environment
Keycloak 26.0.0 → 26.7.1, official image, start against an external
PostgreSQL 16.15, single node, stock tuning. Realm sizes from 1,002 to
2,000,003 users. Hetzner CCX33 (8 vCPU, 30 GB RAM, local NVMe), JVM -Xms1g -Xmx4g. This is a lab, not a customer; the numbers are ours and reproducible.
First we killed it. Twice.
We SIGKILLed Keycloak mid-migration. The first time we killed it the instant "Updating database" appeared, before anything was applied. The second time we waited until the migration was genuinely half-applied — 165 of 211 changesets committed — and killed it there.
Both times, the databasechangeloglock table showed zero held locks afterwards.
Both times, the next start completed the migration by itself in 13 seconds,
ending at the correct schema version with all 100,002 users intact.
A half-migrated schema completed itself on the next start. No manual lock clearing, no intervention, no data loss — Liquibase simply resumed from the last committed changeset, which is exactly what per-changeset commits are for.
We also started two instances against an un-migrated schema at the same moment. Both came up; the schema ended correct.
One honest caveat, and it matters. We never caught the lock in a held state — our polling sampled every 200–400 ms against a migration that finishes in about 3.5 seconds. We are not claiming Keycloak does not lock. We are claiming we never observed a held lock, and, more importantly, the schema recovered by itself every time.
Why the crash window is so narrow
The reason these migrations are only ~3.5 seconds is itself a finding. Keycloak
skips index creation on tables above 300,000 rows. The one piece of DDL that
would take real time on a large table — a new index on EVENT_ENTITY, say — is
deliberately declined above that threshold, with only a WARN in the startup log
and a DATABASECHANGELOG row still marked EXECUTED.
We ran that upgrade eleven times against the same 100,000-user realm, varying
only the row count in EVENT_ENTITY. At 300,000 rows the index is created. At
301,000 it is not. Time-to-ready stayed flat from 0 to 5,000,000 rows (a 2.1 GB
database at the top), precisely because the expensive work is declined.
That is load-bearing for the crash test: the window in which a kill can hurt you is narrow because Keycloak works to keep it narrow. A deployment with a data-rewriting changeset on a large table has a wider window, and this result may not carry. (More on that in "what we did not test".)
Then we aborted its SQL — and got it wrong before we got it right
The second textbook route to a half-migrated schema is statement_timeout: set a
tight timeout on the role, the migration's SQL gets cancelled partway, and the
schema is left half-applied.
Our first measurement said that was impossible. We instrumented Postgres with
log_min_duration_statement=0 and ranked every statement the migration
issued at 2,000,003 users. The longest single statement took 2.142 ms. At a
2 ms timeout Keycloak died in changelog parse having applied nothing; at 3 ms it
completed. The band that would leave a schema half-applied looked empty.
We published that. It was wrong.
Re-running it, the same version pair at the same scale, a statement_timeout of
500 ms, 200 ms and 50 ms all half-migrated the database — DATABASECHANGELOG
at 157 of 211, migration_model still 26.0.0, the server refusing to
start. Seven times in ten identical runs.
The statement that dies is not the index build we were watching for. It is this:
ALTER TABLE public.CREDENTIAL ADD VERSION INTEGER DEFAULT 0ADD COLUMN … DEFAULT 0 is metadata-only on Postgres 11+ — no table rewrite,
however many rows. It executes in 0.177 ms. What it needs is an
AccessExclusiveLock on a 2,000,002-row table, and an autovacuum worker is
holding it. Postgres will evict a blocking autovacuum — but only after
deadlock_timeout, which defaults to 1 second. The wait is inside the
statement, and statement_timeout is charged for the whole thing.
The statement is not slow. The queue is. And its length is set by a Postgres parameter almost nobody connects to upgrades.
We confirmed the mechanism three independent ways: turn autovacuum off and the
same 500 ms timeout is survivable; drop deadlock_timeout to 100 ms and the
statement's duration drops to 100.485 ms, exactly deadlock_timeout; catch the
migration at a moment when no worker is on CREDENTIAL and the statement is back
to 0.177 ms. Ten repetitions at 500 ms split 7 failures / 3 successes — and the
presence of the autovacuum worker predicted the outcome in all ten.
Why we got it wrong is itself worth writing down: we measured one uncontended
run, ranked its statements, and reasoned from the top of the list. The run
happened to have no autovacuum worker on CREDENTIAL when the migration arrived.
Generalising from one sample of a race is what produced the wrong conclusion.
Anything whose outcome depends on a background process needs repetition before a
negative result is recorded.
The practical band is (500 ms, 1 s]: a statement_timeout above
deadlock_timeout survives the eviction; one below it does not. And the failure
is likeliest immediately after a restore or bulk load — which is the state of
every staging rehearsal, and of a production database restored for a DR test.
The pooler is the other route, and it is deterministic
The route that reproduces the half-migrated state every time is not a crash and
not a timeout. It is a PgBouncer in transaction pooling mode — the default
mode nearly everywhere, because it is the mode that actually saves connections.
Upgrading 26.0.0 → 26.7.1 on a 1,002-user realm through it, three times out of three, Keycloak died 16–19 seconds in:
ERROR: Cannot invoke "CustomLockService.waitForLock(DBLockProvider$Namespace)"
because "this.lockService" is nullKeycloak's DBLockProvider needs a session-scoped lock. Transaction pooling
hands out a different backend for every transaction, so the lock service is never
established and the server dereferences a null. The same pooler in session mode
upgraded cleanly in 18 seconds — the pooler is not the problem; the pooling mode
is.
The state it leaves behind is the dangerous part:
DATABASECHANGELOG | 211 rows — the full migration, committed |
migration_model | 26.0.0 — unchanged |
The schema moved. The version stamp did not.
The payoff: every verification built on the changelog is wrong
This is the through-line of both findings, and the thing worth remembering.
The timeout route leaves DATABASECHANGELOG at 157 of 211 — visibly incomplete,
but only if you check the count, and a schema-version query against the
changelog table's metadata will happily report the old version. The pooler route
is worse: the changelog is completely correct at 211 rows, the schema diff
matches the new version, and the upgrade has plainly failed. Only one column
knows the truth:
select version, update_time from migration_model order by update_time desc limit 1;DATABASECHANGELOG records what Liquibase did. migration_model records what
Keycloak thinks it is. They can disagree, and when they do, the changelog is
the one that lies. (The index-skip finding is the same lesson one layer down: a
skipped index is recorded EXECUTED, not skipped.)
And because it recovers on the next start — the pooler run came up healthy in 14 seconds, still through transaction-mode PgBouncer — under Kubernetes this presents as one CrashLoopBackOff followed by a healthy pod, which nobody investigates. The failed attempt's log is the only place the truth ever appears.
What to check before your window
pool_mode. Ask your pooler.transactionmeans the upgrade will fail on the first start. PointKC_DB_URLstraight at Postgres for the migration, put the pooler back afterwards.statement_timeoutagainstdeadlock_timeout. For the role that runs the migration,statement_timeoutmust be abovedeadlock_timeout— orVACUUMthe large tables beforehand (or disable autovacuum for the window) so there is nothing to contend with.- Verify with
migration_model, never the changelog. The query above.
What we did not test
- Postgres only. Every locking, ownership and pooling result above is Postgres-specific; MySQL, MariaDB and Oracle have different semantics.
- Postgres HA — only the crash case. A later run failed a streaming-replica primary over mid-migration (2026-08-26-ha-failover-mid-migration): the replica held a committed-prefix snapshot, the changelog lock did not survive the crash, and Keycloak resumed on the promoted replica — consistent with the crash-recovery result above. Synchronous replication and a clean (maintenance) failover remain untested.
- One version pair. 26.0.0 → 26.7.1. A pair with a genuinely data-rewriting changeset on a large table would widen the crash window beyond what we saw.
- Serving (not upgrading) through a transaction-mode pooler was not tested — this covers startup and migration only.
- Other poolers (Odyssey, RDS Proxy, Cloud SQL connectors) were not tested.
We would rather publish a negative result — "we could not reproduce the stuck lock" — than repeat folklore. If you have a reproducible stuck lock, we want the configuration; we will run it and publish what happens.
Keycloak Advisory Watch — one email per advisory batch, within 72 hours of publication, listing the patched versions per maintained minor line. Double opt-in, no tracking pixels, no click tracking, public archive. Subscribe.
Self-hosted Keycloak is one of the things we keep patched, upgraded and owned for teams that run it but have nobody to run it. Talk to us about your project.
Sources: 2026-08-25-s1-s2-lock-failure-modes, 2026-08-26-s3-revisited-autovacuum-lock-wait (supersedes the conclusion of 2026-08-25-s3-statement-timeout), 2026-08-25-s8-pgbouncer-transaction-mode, 2026-08-25-index-skip-threshold.
