Spark Team
The Spark Team capability provisions an isolated Kubernetes namespace for a group of Spark users. One instance is deployed per team — multiple instances can coexist in the same cluster.
What It Provisions
| Resource | Description |
|---|---|
| Namespace | A dedicated Kubernetes namespace named after the team (e.g. spark-team-a) |
| IRSA role | An IAM role bound to the team's service account — grants full S3 read/write access across all workspace buckets |
| RBAC | Role bindings for Spark cluster operations within the team namespace |
| OAuth secret | Keycloak OAuth credentials scoped to the namespace for authenticated Spark job submissions |
| Observability config | Prometheus scrape targets wired up for Spark jobs running in the namespace |
Isolation Model
Each team gets its own namespace, service account, and IRSA role. Spark jobs submitted by team members run as pods in that namespace with that service account's credentials — S3 access is scoped to those credentials.
This means two teams' Spark jobs cannot access each other's credentials, even though they may share the same underlying Kubernetes cluster and Spark operator.
Go Deeper
- Spark Operator — the operator that manages SparkApplication resources across all team namespaces
- JupyterHub — notebook sessions where team members submit Spark jobs
- Celeborn — the shared remote shuffle service used by Spark jobs across teams