One Keycloak to Rule Them All - Our Products Started Talking to Each Other. Security Was Not Amused.
in Programming on Programming, Keycloack, Auth
Client Credentials, Token Exchange, and why forwarding JWTs everywhere is probably not the architecture we want. Products work individually -> products need each other’s APIs -> user tokens aren’t always appropriate -> introduce service identities and delegation. 😵
In the previous post, we finally got our products to use Keycloak for authentication.
Users could log in once and access multiple products. We had shared identities, SSO sessions, and a reasonably clean separation between authentication and business logic.
Things were looking good.
Then the Support team had an idea.
“What if we let users access their processed documents directly from Support?”
Sounds useful.
The documents already exist in the Documents service. We just need Support to call its API.
Alice → Support API → Documents API
An HTTP request. Some JSON. Maybe a couple of headers.
What could possibly go wrong? 🙃
Can’t we just forward the token?
Alice is already logged into Support.
Her browser sends an access token to the Support API.
So the obvious approach is to forward that same token to Documents.
const response = await fetch(
"https://documents.example.com/api/documents",
{
headers: {
Authorization: `Bearer ${userAccessToken}`,
},
},
);
Done.
Well, assuming Documents accepts the token.
And that’s where things get interesting.
An access token isn’t a universal VIP pass to every service in the ecosystem.
It’s issued with a particular set of claims, including the intended audience.
Consider this simplified token:
{
"iss": "https://auth.example.com/realms/my-platform",
"sub": "alice-123",
"aud": ["support-api"],
"azp": "support-web",
"exp": 1790000000
}
The token is intended for support-api.
Now Documents receives it.
It checks the signature. Valid.
Expiration? Fine.
Issuer? Correct.
Audience?
support-api.
Hmm. 🤨
Documents isn’t the intended audience.
If Documents blindly accepts every correctly signed token from our realm, we’ve effectively turned those tokens into ecosystem-wide access passes.
Not ideal.
Just because Alice can authenticate to Support doesn’t mean every other API should accept that exact credential.
Wait, who’s actually making the request?
Before fixing the problem, we need to answer a more important question.
Is Support calling Documents as itself, or on behalf of Alice?
Those are two different scenarios.
And OAuth gives us different tools for each.
Scenario A: Support acts as itself (machine-2-machine)
Suppose Support runs a background task that retrieves document-processing status every few minutes.
No user is actively making the request.
We don’t need Alice’s identity.
We need the identity of the Support service.
This is where Client Credentials comes in.
Scenario B: Support acts on behalf of Alice (user-2-machine)
Suppose Alice clicks “View my documents” inside Support.
Now her identity matters.
Documents needs to know which user requested the documents so it can enforce the appropriate access rules.
For this scenario, we can use Token Exchange.
Same two services.
Very different security requirements.
And mixing them up is a great way to create an authorization bug that nobody notices until the security review. 💀
When machines talk to machines
Let’s start with the easier case.
We register a confidential Keycloak client for the Support backend.
Client ID: support-service
Client authentication: On
Service accounts roles: Enabled
Then we enable the necessary service-account permissions and configure the intended API audience.
The Support backend can request an access token:
POST https://auth.example.com/realms/my-platform/protocol/openid-connect/token
Content-Type: application/x-www-form-urlencoded
grant_type=client_credentials # No user involved. Just two services having a private conversation.
&client_id=support-service
&client_secret=<CLIENT_SECRET> # The secret handshake. Please don't commit this to Git. 🙃
Keycloak authenticates the client and returns a token.
A simplified payload might look like this:
{
"iss": "https://auth.example.com/realms/my-platform",
"sub": "service-account-user-id",
"aud": ["documents-api"],
"azp": "support-service",
"scope": "documents:status"
}
The audience and scope shown here require the appropriate Keycloak client-scope, mapper, and permission configuration. They don’t appear just because we gave our client a convincing name.
Now Support can call Documents:
GET /api/processing-status
Authorization: Bearer <service-access-token>
Documents validates the token and checks whether the service identity has permission to access the endpoint.
No user session required.
No human password involved.
And no reason to put the client secret in the browser, unless we’re actively trying to make the security team cry.
Keep it on the backend.
But Alice wants her documents
Now let’s return to the original requirement.
Alice clicks a button in Support:
“Show my processed documents.”
Support calls Documents, but Documents needs to authorize the request using Alice’s identity.
We could use a machine token and include Alice’s user ID in a header:
GET /api/documents
Authorization: Bearer <service-token>
X-User-ID: alice-123
Technically, we can build a trusted internal delegation system around this.
But the receiving service would need a reliable way to determine whether Support is allowed to make that claim.
Otherwise, one service could simply say:
“Hello, I am Alice.”
Or Bob.
Or the CEO.
Very convenient. Very concerning. 😬
Instead, we can use Keycloak’s standard Token Exchange support.
Token Exchange: Can I have a token for that API?
The basic idea is straightforward.
Support receives Alice’s access token and exchanges it for another token intended for Documents.
Alice
|
| Access Token
v
Support API
|
| Token Exchange Request
v
Keycloak
|
| Token for Documents API
v
Support API
|
| Exchanged Token
v
Documents API
First, we configure support-service as a confidential client authorized to perform standard token exchange.
We also configure the required audiences and client scopes.
For Keycloak’s standard token exchange, the original subject token needs to contain support-service as an audience, unless it is a token the requesting client originally obtained for itself.
That detail is easy to overlook.
Ask me how many 403 responses an incorrectly configured audience can produce. 😩
The exchange request looks like this:
POST https://auth.example.com/realms/my-platform/protocol/openid-connect/token
Content-Type: application/x-www-form-urlencoded
client_id=support-service # This time I'm acting on someone's behalf.
&client_secret=<CLIENT_SECRET> # Secret handshake. Still not something to put on GitHub.
&grant_type=urn:ietf:params:oauth:grant-type:token-exchange # One token for another.
&subject_token=<USER_ACCESS_TOKEN> # Here's the user's token.
&subject_token_type=urn:ietf:params:oauth:token-type:access_token # Just confirming it's an access token, not a random string I found.
&audience=documents-api # Make this token valid for Documents API.
&scope=documents:read # Read-only access. Look, but don't touch.
This assumes documents:read has been configured as an available optional client scope, together with the corresponding audience and permission mappings.
If the configuration allows the exchange, Keycloak issues a new access token suitable for the target API.
Its claims might look approximately like this:
{
"sub": "alice-123",
"aud": ["documents-api"],
"azp": "support-service",
"scope": "documents:read"
}
Alice remains the subject, while the token is now intended for Documents.
Importantly, the resulting claims depend on the Keycloak configuration. Standard token exchange doesn’t automatically provide a complete delegation history or guarantee arbitrary permission downscoping just because we requested it.
And asking for documents:read doesn’t magically give Alice that business permission.
Documents still needs to enforce its own authorization rules.
So what does Documents validate?
At minimum, Documents should validate the token’s signature, issuer, expiration, and intended audience.
It also needs to enforce the appropriate authorization policy.
A simplified TypeScript example using jose:
import { createRemoteJWKSet, jwtVerify } from "jose";
const issuer =
"https://auth.example.com/realms/my-platform";
const jwks = createRemoteJWKSet(
new URL(`${issuer}/protocol/openid-connect/certs`),
);
async function verifyAccessToken(token: string) {
const { payload } = await jwtVerify(token, jwks, {
issuer,
audience: "documents-api",
});
const scopes = String(payload.scope ?? "").split(" ");
if (!scopes.includes("documents:read")) {
throw new Error("Insufficient scope");
}
return payload;
}
This verifies a basic token boundary.
Production APIs also need to validate the caller context, tenant membership, resource ownership, and any relevant business permissions.
For example, Alice may have permission to read documents in Acme but not Globex.
A valid token doesn’t make that distinction disappear.
Also, jose verifies cryptographic validity and standard claims. It won’t read your mind and implement the organization’s data-access policy.
Unreasonable, I know. 🙃
Why not give every product one giant audience?
Here’s another tempting shortcut.
What if we configure every token to include every API?
{
"aud": [
"support-api",
"documents-api",
"workflows-api"
]
}
Then we could forward tokens anywhere.
Problem solved!
Well, technically we’ve just weakened audience isolation across our product ecosystem.
A token intended for one application is now potentially acceptable to several services.
That increases the impact if a token is leaked or accidentally forwarded.
For a small tightly coupled system, broad audiences might be a deliberate trade-off.
But for independently managed products, we prefer narrower tokens.
The rule we try to follow is simple:
Give each service the credentials and permissions it actually needs, not everything we might need someday.
A few things we learned the hard way
First, Client Credentials and Token Exchange solve different problems.
If there’s no user context, use a service identity.
If the operation genuinely depends on the user’s identity, use an appropriate delegated access mechanism.
Second, token audiences matter.
Signature verification alone is not enough.
Third, don’t treat scopes as a complete business authorization system.
A scope can permit a category of operation, while the application still determines whether the caller may access a particular resource.
Fourth, token exchange isn’t completely automatic.
The requesting client, audience mappings, available scopes, and original token claims must all be configured correctly.
And finally, avoid exchanging tokens on every API call when a safely cached, still-valid token is suitable. Token caching, expiration handling, and concurrency deserve some engineering attention.
Because replacing an authentication problem with a Keycloak performance problem would be a particularly creative achievement .
Everything is finally connected
After some experiments, several questionable configuration attempts, and considerably more time examining JWT payloads than we’d like to admit, our internal products can finally communicate securely.
Background jobs use service identities.
User-driven cross-product requests use appropriately scoped tokens.
Each API validates its own audience and permissions.
Nice 😘.
We have an authentication system.
We have internal service authorization.
We even understand why the aud claim exists.
Then someone asks:
“What if our customers want to connect their own applications? You know, like Google integrations?”
Ah, yes.
Because apparently the outside world would like to join the party. 🙃
Next: Part 3 — Everybody Wants Access. Nobody Wants to Ask Permission