Skip to main content

ML Agent Authentication: ML-Agent to MLflow

This page covers how ML-Agent (agent_based_ml_platform) authenticates to MLflow when tracking experiments, runs, and models. Unlike the BFF-to-ML-Agent hop (see ML Agent Auth — Trino), this leg does not involve the end user's identity at all — ML-Agent talks to MLflow as itself, using its own Keycloak service account.

End-to-End Flow​

Step 1: The User's JWT Never Reaches MLflow​

mlAgentRoute.js / mlAgentService.js forward the human user's Authorization header to ML-Agent's own HTTP API (/upload, /run-workflow, /run-workflow-trino, /generate-synthetic-data, etc.) as-is — the same raw-forwarding pattern documented in ML Agent Auth — Trino. The user's email is attached as form data purely for ML-Agent's own run bookkeeping (e.g. naming an experiment automl_<user>_<workspace>) — it is never used for authentication.

Everything MLflow ever sees comes from ML-Agent's own credentials, decoupled from which user triggered the run. MLflow's access logs always show the same identity (service-account-<workspace>-agent-based-ml-platform) regardless of who submitted the job.

Step 2: ML-Agent's Own Service Account​

ML-Agent authenticates to MLflow using a client_credentials grant against its own Keycloak OAuth client (aml-oauth-client in KCL), using KEYCLOAK_CLIENT_ID / KEYCLOAK_CLIENT_SECRET / KEYCLOAK_REALM / KEYCLOAK_URL from its own secret.

Two details matter here that are easy to get wrong:

  • aud is realm-wide, not per-call. The resulting token's aud claim lists every app client in the realm (a shared, realm-level audience mapper) — including MLflow's own client ID. This is what lets a single service-account token from ML-Agent be presented to multiple downstream services without a separate token-exchange step per target.
  • Realm roles vs. client roles. The service account needs a role that satisfies MLflow's group-authorization gate (see Step 3). That role must be a genuine Keycloak realm role (e.g. ml_engineer) — never a client role of the same name. Client roles (like MLflow's own ml_platform_user, defined on MLflow's OAuth client) are namespaced per-client and only ever appear under the token's resource_access.<client>.roles, never in the flat roles claim. A same-named role on a different client is a different role entirely.

Step 3: MLflow's OIDC Plugin (mlflow-oidc-auth)​

MLflow validates incoming bearer tokens via the mlflow-oidc-auth plugin, configured entirely through env vars on the MLflow Helm release (see Configuration).

  1. Token validation. OIDC_AUDIENCE and OIDC_ISSUER, when set, are checked as essential JWT claims — the token is rejected outright if either doesn't match.
  2. Group-authorization gate. The plugin reads OIDC_GROUPS_ATTRIBUTE (set to roles — the flat, realm-role-only claim) and requires at least one value to appear in OIDC_GROUP_NAME.
  3. Permission-record provisioning. MLflow's own permission model normally provisions a user record only on interactive browser OIDC login. A bearer-token identity that never logs in via browser — like ML-Agent's service account — has no such record by default, and any resource it creates would be denied ownership (no permission record yet, cannot own the created resource) even though the resource itself gets committed. OIDC_PROVISION_ON_BEARER_AUTH opts into auto-provisioning a record on first bearer authentication, gated behind OIDC_AUDIENCE + OIDC_ISSUER both being set (so only scoped tokens can provision).
  4. Admin is never inferred from a token unless OIDC_TRUST_BEARER_GROUP_CLAIMS is also set, and only then if the role matches OIDC_ADMIN_GROUP_NAME.

Once provisioned as an ordinary (non-admin) user, MLflow's own creation hook grants that identity MANAGE ownership of whatever resource it just created — this is standard MLflow behavior, not specific to service accounts.

Configuration​

Config keyPurpose
OIDC_DISCOVERY_URLMLflow's own OIDC metadata endpoint (browser login flow)
OIDC_CLIENT_ID / OIDC_CLIENT_SECRETMLflow's own OAuth client credentials
OIDC_AUDIENCEExpected aud claim — MLflow's own client ID (present in every token in the realm)
OIDC_ISSUERExpected iss claim — the realm's issuer URL
OIDC_GROUPS_ATTRIBUTEroles — reads the flat, realm-role-only claim
OIDC_GROUP_NAMERealm roles allowed to authenticate (ml_platform_user,platform_admin,ml_engineer)
OIDC_ADMIN_GROUP_NAMERealm role treated as MLflow admin (platform_admin)
OIDC_PROVISION_ON_BEARER_AUTHAuto-provision a permission record for bearer/service-account identities
OIDC_TRUST_BEARER_GROUP_CLAIMSWhether a bearer token's role claim can confer admin
KEYCLOAK_CLIENT_ID / KEYCLOAK_CLIENT_SECRET / KEYCLOAK_REALM / KEYCLOAK_URLML-Agent's own client credentials, used for the client_credentials grant against Keycloak

Go Deeper​