Kubernetes role-based authorisation for controller applications

There are many scenarios where an application running inside a Kubernetes environment may need to interact with its API.

For example, an application running inside a Pod may need to retrieve real time information about the availability of other applications' endpoints.

This may be a form of service discovery that integrates or extend the native Kubernetes internal service discovery. In most cases, DNS records are associated to a Service and provide the list of active Endpoints for that Service, with a proper TTL. There are situations though where those DNS records are not available, an application is not able to use them directly, or what's needed is more than the private IP addresses associated with the Endpoints.

If interacting with the Kubernetes API from inside an application is needed, then there are two main areas to consider: Authentication and Authorisation.

Every Pod has a Service Account associated to it, and applications running inside that Pod can use that Service Account. Without a specific configuration the Service Account will default to a generic namespace and generic authorisation.

It's possible instead to define a more specific Service Account, with fine grained permissions to access the API. This Service Account can then be linked to a Pod with a Role-based approach.

A reference: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/

You can get a list of available Service Accounts with an intuitive:

# kubectl get serviceaccounts

which is likely to show you a single 'default' service.

Service Accounts may have a namespace scope. You can check what Service Accounts are associated to a pod with a command like:

# kubectl -n NAMESPACE get pods/PODNAME -o yaml | grep serviceAccountName

Service Account authentication can use the token reachable from inside a Pod, in the /var/run/secrets/kubernetes.io/serviceaccount directory, under the namespace-specific directory, e.g. /var/run/secrets/kubernetes.io/serviceaccount/NAMESPACE/token.

Those tokens are also visible as Mounts in the related containers.

For internal requests, Kubernetes provides a local default HTTPS endpoint at https://kubernetes.default.svc - so a way to discover the details of a Service Account for a given namespace could be:

#!/bin/bash

# Point to the internal API server hostname

APISERVER=https://kubernetes.default.svc

# Path to ServiceAccount token

SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount

# Read this Pod's namespace

NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)

# Read the ServiceAccount bearer token

TOKEN=$(cat ${SERVICEACCOUNT}/token)

# Reference the internal certificate authority (CA)

CACERT=${SERVICEACCOUNT}/ca.crt

# Explore the API with TOKEN

curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api/v1/namespaces/${NAMESPACE}/pods

Accessing the internal API programmatically

With common python libraries as https://github.com/kubernetes-client/python (available on debian with the 'python3-kubernetes' package) it's extremely easy to automate the invocation of the internal APIs.

Most of the examples assume you're running your program as a user, and refer to the local kube config file, but when running inside a container it's possible to inherit the Service Account token associated with the hosting Pod.

For this, instead of using

config.load_kube_config()

use

config.load_incluster_config()

Then you can instantiate your API client object:

v1 = client.CoreV1Api()

and either do a single request, like getting a list of all pods inside any namespace:

ret = v1.list_pod_for_all_namespaces(watch=False)

or a list of pods belonging to a namespace and matching a specific application label, like:

ret = v1.list_namespaced_pod(namespace, label_selector=app_name, watch=False)

or you can "watch" some resources, which basically means subscribing to such resource updates and getting a notification at each change:

w = watch.Watch()

for event in w.stream(v1.list_namespace, _request_timeout=60):

    ...

Each event can be ADDED, DELETED and MODIFIED, and carries a rich set of information associated to the current status of the resource.

Defining your specific ServiceAccount

Before getting to that point, though, you need to define your non-default Service Account and assign specific permissions to it. To achieve this, the role-based approach can be used.

First of all define a ServiceAccount resource:

apiVersion: v1

kind: ServiceAccount

metadata:

  labels:

    app.kubernetes.io/component: mycomponent

    name: mycomponent-serviceaccount

Then define a role associated to this resource:

apiVersion: rbac.authorization.k8s.io/v1

kind: Role

metadata:

  name: myrole

  namespace: mynamespace

labels:

[...]

annotations:

[...]

rules:

[...]

  - apiGroups:

  - ""

  resources:

    - pods

    - endpoints

    verbs: ["get", "list", "watch"]

This example adds the permission to get, list, or watch the list of pods and endpoints in the given namespace.

Create a role binding:

apiVersion: rbac.authorization.k8s.io/v1

kind: RoleBinding

metadata:

name: myrolebinding

namespace: mynamespace

labels:

[...]

annotations:

[...]

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: Role

  name: myrole

subjects:

  [...]

  - kind: ServiceAccount

    name: mycomponent-serviceaccount

    namespace: mynamespace

Whenever a Role cannot be restricted to a namespace, for example if it needs to access cluster-wide resources like Nodes, then the ClusterRole resource is available.

References and other sources

"Access clusters using the Kubernetes API", https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-api/

An interesting post about asynchronous watches with Python: https://medium.com/@sebgoa/kubernets-async-watches-b8fa8a7ebfd4

"Kubernetes Patterns", an ebook by Redhat: https://www.redhat.com/cms/managed-files/cm-oreilly-kubernetes-patterns-ebook-f19824-201910-en.pdf