Three of our fifteen Keycloak tests refuted the thing they were written to prove.

Keycloak

13 Oct

We ran fifteen adverse-condition scenarios against a Keycloak upgrade, plus a handful of follow-up runs. The most useful records in the set are the negative ones — the tests that failed to demonstrate what they were written to demonstrate. Three of the fifteen did. Here they are.

The stuck lock we could not reproduce

The canonical Keycloak upgrade horror story is a stuck DATABASECHANGELOGLOCK: an upgrade dies partway, the lock row is never released, and every subsequent start hangs waiting for a lock nobody holds, including the rollback. We tried to produce it by SIGKILLing Keycloak mid-migration, the second time with 165 of 211 changesets already committed.

Both times the lock table showed zero held locks afterwards, and both times the next start completed the migration by itself in 13 seconds with all 100,002 users intact. We could not reproduce the story. Our polling never caught the lock in a held state, so this bounds where the folklore applies rather than disproving it. (We did, eventually, produce a half-migrated schema, and a crash was not what did it. It took a transaction-mode connection pooler, or a statement_timeout racing an autovacuum worker.) A third run used a Postgres streaming-replica pair. We killed the primary mid-migration and promoted the replica. It recovered the same way: a committed-prefix snapshot, no stuck lock, clean resume (2026-08-26-ha-failover-mid-migration).

The per-million heap floor that does not exist

We set out to find the heap a Keycloak upgrade needs per million users. There is no such thing. A 1,002-user realm and a 2,000,003-user realm have the same floor. 160 MB works, 128 MB dies, because the thing that runs out of memory is the Quarkus build step, which runs before the database is touched. The user count never entered the picture.

The attribute loss the upgrade did not cause

We expected crossing 23 → 24, where the declarative user profile became default, to make existing custom attributes invisible. It does not. An in-place upgrade preserves them, because the migration sets the realm's unmanagedAttributePolicy to ENABLED. What loses attributes is creating a realm on a new version — which we nearly published as an upgrade regression before checking where the realm came from.

And one we refuted twice — against ourselves

The statement_timeout scenario deserves its own sentence. We first measured it and concluded no timeout could half-migrate the schema, because every statement executes in a couple of milliseconds. That conclusion was wrong, and the revisit overturned it: a timeout below deadlock_timeout half-migrates the database, because the statement waits a second behind an autovacuum worker's lock and the timeout is charged for the wait. Seven failures in ten runs, at a 500 ms timeout.

We published the first conclusion and corrected it in the next record.

Two fixtures we almost published as Keycloak findings

The failures were not only in the predictions. Two of our own test fixtures were one step from being published as Keycloak behaviour:

The LDAP fixture that pointed at nothing. Our LDAP federation scenario reported as coverage while its component pointed at a host defined in no compose file. The "survives intact" result was about a fixture that never existed.
The attribute invisibility that was our own doing. Our large test realms were built on 26.0.0, so their unmanagedAttributePolicy was never set, and the attributes: null symptom we observed came from the version our realms were born on. The upgrade had nothing to do with it.

Both were found only by building the thing the fixture claimed to test.

Keycloak Advisory Watch — one email per advisory batch, within 72 hours of publication, listing the patched versions per maintained minor line. Double opt-in, no tracking pixels, no click tracking, public archive. Subscribe.

Self-hosted Keycloak is one of the things we keep patched, upgraded and owned for teams that run it but have nobody to run it. Talk to us about your project.

Sources, one per refutation above.