In a recent Kubernetes version update, they announced Fine-Grained Kubelet API Authorization is available to use, and it can prevent an issue that comes with nodes/proxy permission. That permission makes remote code execution (RCE) possible even with read only access. The detailed explanation can be found here.
How kubelet runs
Before we try the feature, it’s a good idea to understand how kubelet works. As we know, kubelet is the main component that acts as the node agent on each worker node. It handles the lifecycle of containers within the cluster. A common setup to bootstrap a cluster is kubeadm.
Once we have a cluster ready, we can connect to the worker node via SSH and verify that kubelet runs as a systemd service and that it always listens on port 10250.
systemctl status kubeletkubelet runs as a single binary on the node, and it ships with some files at /etc/kubernetes.
- kubelet.conf
- manifests
- patches
- pki
kubelet.conf works like the typical kubeconfigs you already know. It pins the kubelet client certificate under /var/lib/kubelet/. The next block prints that certificate with openssl.
openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -text -nooutCertificate:
Data:
Version: 3 (0x2)
Serial Number:
dd:86:4a:60:8d:11:4c:9c:1b:83:39:23:ee:d2:2a:c3
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = kubernetes
Validity
Not Before: May 7 09:09:12 2026 GMT
Not After : May 7 09:09:12 2027 GMT
Subject: O = system:nodes, CN = system:node:timbernetes-md-0-jplhj-rznt7-jgkc7
...system:nodes is a built-in Kubernetes group that all kubelets are automatically bound to. It defines the exact permissions kubelet needs to function on its assigned node. Based on the kubelet kubeconfig we have, we know that another data directory used by kubelet is /var/lib/kubelet/, and config.yaml is the main configuration. Since v1.10, anonymous authentication is set to false, and if you’re wondering, the PR can be found here.
Kubernetes v1.10 reached End-Of-Life (EOL) on 13 February 2019. Since then, a related attack on kubelet no longer works in the wild, because it depends on misconfigured kubelet anonymous authentication (true). Those kubelet settings belong to the same authorization tightening that entrenched Webhook authorization on kubelet. With Webhook, kubelet evaluates unauthenticated requests against policy and denies them when they would not be permitted.
For Kubelet API endpoints, see this Stack Overflow answer.
Making the kubelet insecure
To make anonymous authentication work, mirror the next configuration block.
authentication:
anonymous:
enabled: true
...
authorization:
mode: AlwaysAllowSave the configuration and restart the kubelet service so the change takes effect.
AlwaysAllow can allow all requests in kubelet.
When authorization is left at its defaults, calling the Kubelet API directly instead of routing through kubectl can yield Forbidden. The next block shows one representative line.
Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource(s)=[pods proxy])The Forbidden line names the resource tied to the denial. If you run kubectl get po through kube-apiserver without chaining through kubelet or node proxies you should still see JSON like the next sample. That body should reference pods, not nodes.
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group \"\" at the cluster scope",
"reason": "Forbidden",
"details": {
"kind": "pods"
},
"code": 403
}At least pods should appear in that message, not nodes, right? I briefly imagined the API server was handing the call straight into kubelet API.
func NewStorage(optsGetter generic.RESTOptionsGetter, k client.ConnectionInfoGetter, proxyTransport http.RoundTripper, podDisruptionBudgetClient policyclient.PodDisruptionBudgetsGetter, authorizer authorizer.UnconditionalAuthorizer) (PodStorage, error) {
store := &genericregistry.Store{
NewFunc: func() runtime.Object { return &api.Pod{} },
NewListFunc: func() runtime.Object { return &api.PodList{} },
...The function shows the API server reading a pod list from etcd on this path.
Source code: kubernetes/pkg/registry/core/pod/storage/storage.go near line 76
func encodePods(pods []*v1.Pod) (data []byte, err error) {
podList := new(v1.PodList)
for _, pod := range pods {
podList.Items = append(podList.Items, *pod)
}
// TODO: this needs to be parameterized to the kubelet, not hardcoded. Depends on Kubelet
// as API server refactor.
// TODO: Locked to v1, needs to be made generic
codec := legacyscheme.Codecs.LegacyCodec(schema.GroupVersion{Group: v1.GroupName, Version: "v1"})
return runtime.Encode(codec, podList)
}Source code: kubernetes/pkg/kubelet/server/server.go near line 894
We now have a clearer picture of what happens when a call reaches the kubelet API directly: kubelet does not embed its own authorization engine. It delegates those checks to the API server, as implemented in pkg/kubelet/server/auth.go. That layer inspects the incoming path and maps it to a Kubernetes resource and subresource pair for the access review flow.
Paths like /stats/, /metrics/, and /logs/ get their own dedicated subresources. But anything that doesn’t match a recognized pattern, including /pods, falls into the default case and gets assigned nodes/proxy as the subresource. nodes/proxy is commonly understood as the permission for reaching a node through the API server’s own proxy endpoint but because of this default fallback, it also covers every unrecognized path on the direct kubelet API. That includes pod listing, exec sessions, port-forwards, and anything else the kubelet serves without an explicit subresource mapping all reachable on port 10250.
Based on the previous configuration, direct API access to the kubelet will now give us a response in JSON format, since no authorization is enabled.
curl -k https://localhost:10250/podsWhy KubeletFineGrainedAuthz exists?
Now you can see why nodes/proxy is far from a read-only permission. Under the principle of least privilege, it’s not a permission you want to hand out broadly especially to open source tools like monitoring agents that only need access to the /metrics endpoint on each node.
If your nodes do not already have this feature turned on (for example on an older Kubernetes version or an older node image), open config.yaml file on each node and add a featureGates block like the one below to the existing file. Restart kubelet on every node so the change takes effect, because the kubelet config file is local to each machine.
featureGates:
KubeletFineGrainedAuthz: trueWith KubeletFineGrainedAuthz enabled by default, there is no need to pass extra flags or bake the feature gate into a custom node image. Previously, enabling it on an autoscaling cluster meant every new node had to be provisioned with the flag explicitly set either through a custom image. Going forward, well-maintained Helm charts should start dropping nodes/proxy from their RBAC manifests and replacing it with the narrower subresource permissions.
The kube-prometheus-stack Helm chart is a good example. In this commit, the maintainers explicitly dropped nodes/proxy from the chart’s ClusterRole, narrowing the monitoring stack’s permissions to only what it actually needs.
To verify what permissions a service account actually holds by prometheus chart, we can use below command.
kubectl auth can-i --list --as=system:serviceaccount:monitoring:prometheus-serverOn version 28.7.0, nodes/proxy is no longer present in the output.
selfsubjectreviews.authentication.k8s.io [] [] [create]
selfsubjectaccessreviews.authorization.k8s.io [] [] [create]
selfsubjectrulesreviews.authorization.k8s.io [] [] [create]
configmaps [] [] [get list watch]
endpoints [] [] [get list watch]
ingresses [] [] [get list watch]
nodes/metrics [] [] [get list watch]
nodes [] [] [get list watch]
pods [] [] [get list watch]
services [] [] [get list watch]
endpointslices.discovery.k8s.io [] [] [get list watch]
ingresses.networking.k8s.io/status [] [] [get list watch]
ingresses.networking.k8s.io [] [] [get list watch]
[/.well-known/openid-configuration/] [] [get]
[/.well-known/openid-configuration] [] [get]
[/api/*] [] [get]
[/api] [] [get]
[/apis/*] [] [get]
[/apis] [] [get]
[/healthz] [] [get]
[/healthz] [] [get]
[/livez] [] [get]
[/livez] [] [get]
[/metrics] [] [get]
[/openapi/*] [] [get]
[/openapi] [] [get]
[/openid/v1/jwks/] [] [get]
[/openid/v1/jwks] [] [get]
[/readyz] [] [get]
[/readyz] [] [get]
[/version/] [] [get]
[/version/] [] [get]
[/version] [] [get]
[/version] [] [get]On version 28.6.1 and below, nodes/proxy appears explicitly:
Resources Non-Resource URLs Resource Names Verbs
selfsubjectreviews.authentication.k8s.io [] [] [create]
selfsubjectaccessreviews.authorization.k8s.io [] [] [create]
selfsubjectrulesreviews.authorization.k8s.io [] [] [create]
configmaps [] [] [get list watch]
endpoints [] [] [get list watch]
ingresses [] [] [get list watch]
nodes/metrics [] [] [get list watch]
nodes/proxy [] [] [get list watch]
nodes [] [] [get list watch]
pods [] [] [get list watch]
services [] [] [get list watch]
endpointslices.discovery.k8s.io [] [] [get list watch]
ingresses.networking.k8s.io/status [] [] [get list watch]
ingresses.networking.k8s.io [] [] [get list watch]
[/.well-known/openid-configuration/] [] [get]
[/.well-known/openid-configuration] [] [get]
[/api/*] [] [get]
[/api] [] [get]
[/apis/*] [] [get]
[/apis] [] [get]
[/healthz] [] [get]
[/healthz] [] [get]
[/livez] [] [get]
[/livez] [] [get]
[/metrics] [] [get]
[/openapi/*] [] [get]
[/openapi] [] [get]
[/openid/v1/jwks/] [] [get]
[/openid/v1/jwks] [] [get]
[/readyz] [] [get]
[/readyz] [] [get]
[/version/] [] [get]
[/version/] [] [get]
[/version] [] [get]
[/version] [] [get]Confirming that 28.7.0 is the version where the Prometheus chart dropped nodes/proxy from its service account permissions.
Thoughts
Starting from Kubernetes v1.36.0, nodes/proxy in audit logs becomes a much stronger signal. With legitimate tooling dropping it, any workload still holding that permission is worth questioning whether it was granted intentionally or just never cleaned up. For defenders, a permission that once hid behind legitimate monitoring tools is now easier to flag as suspicious.