Getting Started with Tetragon
Quickly get started and learn how to install, deploy and configure it
Getting started
This is the multi-page printable view of this section. Click here to print.
Cilium Tetragon component enables powerful realtime, eBPF-based Security Observability and Runtime Enforcement.
Tetragon detects and is able to react to security-significant events, such as
When used in a Kubernetes environment, Tetragon is Kubernetes-aware - that is, it understands Kubernetes identities such as namespaces, pods and so-on - so that security event detection can be configured in relation to individual workloads.
Tetragon Overview Diagram
Tetragon is a runtime security enforcement and observability tool. What this means is Tetragon applies policy and filtering directly in eBPF in the kernel. It performs the filtering, blocking, and reacting to events directly in the kernel instead of sending events to a user space agent.
For an observability use case, applying filters directly in the kernel drastically reduces observation overhead. By avoiding expensive context switching and wake-ups, especially for high frequency events, such as send, read, or write operations, eBPF reduces required resources. Instead, Tetragon provides rich filters (file, socket, binary names, namespace/capabilities, etc.) in eBPF, which allows users to specify the important and relevant events in their specific context, and pass only those to the user-space agent.
Tetragon can hook into any function in the Linux kernel and filter on its arguments, return value, associated metadata that Tetragon collects about processes (e.g., executable names), files, and other properties. By writing tracing policies users can solve various security and observability use cases. We provide a number of examples for these in the repository and highlight some below in the ‘Getting Started Guide’, but users are encouraged to create new policies that match their use cases. The examples are just that, jumping off points that users can then use to create new and specific policy deployments even potentially tracing kernel functions we did not consider. None of the specifics about which functions are traced and what filters are applied are hard-coded in the engine itself.
Critically, Tetragon allows hooking deep in the kernel where data structures can not be manipulated by user space applications avoiding common issues with syscall tracing where data is incorrectly read, maliciously altered by attackers, or missing due to page faults and other user/kernel boundary errors.
Many of the Tetragon developers are also kernel developers. By leveraging this knowledge base Tetragon has created a set of tracing policies that can solve many common observability and security use cases.
Tetragon, through eBPF, has access to the Linux kernel state. Tetragon can then join this kernel state with Kubernetes awareness or user policy to create rules enforced by the kernel in real time. This allows annotating and enforcing process namespace and capabilities, sockets to processes, process file descriptor to filenames and so on. For example, when an application changes its privileges we can create a policy to trigger an alert or even kill the process before it has a chance to complete the syscall and potentially run additional syscalls.
If you don’t have a Kubernetes Cluster yet, you can use the instructions below to create a Kubernetes cluster locally or using a managed Kubernetes service:
The following commands create a single node Kubernetes cluster using Google
Kubernetes Engine. See
Installing Google Cloud SDK for
instructions on how to install gcloud
and prepare your account.
export NAME="$(whoami)-$RANDOM"
export ZONE="us-west2-a"
gcloud container clusters create "${NAME}" --zone ${ZONE} --num-nodes=1
gcloud container clusters get-credentials "${NAME}" --zone ${ZONE}
The following commands create a single node Kubernetes cluster using Azure
Kubernetes Service. See
Azure Cloud CLI
for instructions on how to install az
and prepare your account.
export NAME="$(whoami)-$RANDOM"
export AZURE_RESOURCE_GROUP="${NAME}-group"
az group create --name "${AZURE_RESOURCE_GROUP}" -l westus2
az aks create --resource-group "${AZURE_RESOURCE_GROUP}" --name "${NAME}"
az aks get-credentials --resource-group "${AZURE_RESOURCE_GROUP}" --name "${NAME}"
The following commands create a Kubernetes cluster with eksctl
using Amazon Elastic
Kubernetes Service. See eksctl installation
for instructions on how to install eksctl
and prepare your account.
export NAME="$(whoami)-$RANDOM"
eksctl create cluster --name "${NAME}"
kind create cluster
Tetragon’s correct operation depends on access to the host /proc filesystem. The following steps configure kind and Tetragon accordingly when using a Linux system.
cat <<EOF > kind-config.yaml
apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
nodes:
- role: control-plane
extraMounts:
- hostPath: /proc
containerPath: /procHost
EOF
kind create cluster --config kind-config.yaml
EXTRA_HELM_FLAGS="--set tetragon.hostProcPath=/procHost" # flags for helm install
To install and deploy Tetragon, run the following commands:
helm repo add cilium https://helm.cilium.io
helm repo update
helm install tetragon ${EXTRA_HELM_FLAGS} cilium/tetragon -n kube-system
kubectl rollout status -n kube-system ds/tetragon -w
By default, Tetragon will filter kube-system events to reduce noise in the event logs. See concepts and advanced configuration to configure these parameters.
To explore Tetragon its helpful to have a sample workload. Here we use the Cilium HTTP application, but any workload would work equally well.
To use our demo application
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.15.0-pre.1/examples/minikube/http-sw-app.yaml
Before going forward, verify that all pods are up and running - it might take several seconds for some pods until they satisfy all the dependencies:
kubectl get pods
The output should be similar to this:
NAME READY STATUS RESTARTS AGE
deathstar-6c94dcc57b-7pr8c 1/1 Running 0 10s
deathstar-6c94dcc57b-px2vw 1/1 Running 0 10s
tiefighter 1/1 Running 0 10s
xwing 1/1 Running 0 10s
Check for execution events.
This guide has been tested on Ubuntu 22.04 and 22.10 with respectively kernel
5.15.0
and 5.19.0
on amd64 and arm64 but
any recent distribution
shipping with a relatively recent kernel should work. See the FAQ for further details on
the recommended kernel versions.
Note that you cannot run Tetragon using Docker Desktop on macOS because of a limitation of the Docker Desktop Linux virtual machine. Learn more about this issue and how to run Tetragon on a Mac computer in this section of the FAQ page.
The easiest way to start experimenting with Tetragon is to run it via Docker using the released container images.
docker run --name tetragon-container --rm --pull always \
--pid=host --cgroupns=host --privileged \
-v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf \
quay.io/cilium/tetragon-ci:latest
This will start Tetragon in a privileged container. Priviliges are required to load and attach BPF programs. See Installation section for more details.
Check for execution events.
At the core of Tetragon is the tracking of all executions in a kubernetes cluster, virtual machines, and baremetal systems. This creates the foundation that allows Tetragon to attribute all system behavior back to a specific binary and its associated metadata (container, Pod, Node, and cluster).
Tetragon exposes the execution trace over JSON logs and GRPC stream. The user can then observe all executions in the system.
The following command can be used to observe exec events.
kubectl exec -ti -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact --pods xwing
docker exec tetragon-container tetra getevents -o compact
This will print a compact form of the exec logs. For an example we do the following with the demo application.
kubectl exec -ti xwing -- bash -c 'curl https://ebpf.io/applications/#tetragon'
curl https://ebpf.io/applications/#tetragon
The CLI will print a compact form of the event to the terminal similar to the following output.
🚀 process default/xwing /bin/bash -c "curl https://ebpf.io/applications/#tetragon"
🚀 process default/xwing /usr/bin/curl https://ebpf.io/applications/#tetragon
💥 exit default/xwing /usr/bin/curl https://ebpf.io/applications/#tetragon 60
The compact exec event contains the event type, the pod name, the binary and the args. The exit event will include the return code, in the case of curl 60
above.
For the complete exec event in JSON format remove the compact option.
kubectl exec -ti -n kube-system ds/tetragon -c tetragon -- tetra getevents --pods xwing
docker exec tetragon-container tetra getevents
This will include a lot more details related the binary and event. A full example of the above curl is hown here, In a Kubernetes environment this will include the Kubernetes metadata include the Pod, Container, Namespaces, and Labels among other useful metadata.
{
"process_exec": {
"process": {
"exec_id": "Z2tlLWpvaG4tNjMyLWRlZmF1bHQtcG9vbC03MDQxY2FjMC05czk1OjEzNTQ4Njc0MzIxMzczOjUyNjk5",
"pid": 52699,
"uid": 0,
"cwd": "/",
"binary": "/usr/bin/curl",
"arguments": "https://ebpf.io/applications/#tetragon",
"flags": "execve rootcwd",
"start_time": "2023-10-06T22:03:57.700327580Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://551e161c47d8ff0eb665438a7bcd5b4e3ef5a297282b40a92b7c77d6bd168eb3",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-10-06T21:52:41Z",
"pid": 49
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
},
"workload": "xwing"
},
"docker": "551e161c47d8ff0eb665438a7bcd5b4",
"parent_exec_id": "Z2tlLWpvaG4tNjMyLWRlZmF1bHQtcG9vbC03MDQxY2FjMC05czk1OjEzNTQ4NjcwODgzMjk5OjUyNjk5",
"tid": 52699
},
"parent": {
"exec_id": "Z2tlLWpvaG4tNjMyLWRlZmF1bHQtcG9vbC03MDQxY2FjMC05czk1OjEzNTQ4NjcwODgzMjk5OjUyNjk5",
"pid": 52699,
"uid": 0,
"cwd": "/",
"binary": "/bin/bash",
"arguments": "-c \"curl https://ebpf.io/applications/#tetragon\"",
"flags": "execve rootcwd clone",
"start_time": "2023-10-06T22:03:57.696889812Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://551e161c47d8ff0eb665438a7bcd5b4e3ef5a297282b40a92b7c77d6bd168eb3",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-10-06T21:52:41Z",
"pid": 49
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
},
"workload": "xwing"
},
"docker": "551e161c47d8ff0eb665438a7bcd5b4",
"parent_exec_id": "Z2tlLWpvaG4tNjMyLWRlZmF1bHQtcG9vbC03MDQxY2FjMC05czk1OjEzNTQ4NjQ1MjQ1ODM5OjUyNjg5",
"tid": 52699
}
},
"node_name": "gke-john-632-default-pool-7041cac0-9s95",
"time": "2023-10-06T22:03:57.700326678Z"
}
Execution events are the most basic event Tetragon can produce. To see how to use tracing policies to enable file monitoring see the File Access Monitoring quickstart. To see a network policy check the Networking Monitoring quickstart.
Tracing Policies can be added to Tetragon through YAML configuration files that extend Tetragon’s base execution tracing capabilities. These policies do filtering in kernel to ensure only interesting events are published to userspace from the BPF programs running in kernel. This ensures overhead remains low even on busy systems.
The following extends the example from Execution Tracing with a policy to
monitor sensitive files in Linux. The policy used is the
file_monitoring.yaml
it can be reviewed and extended as needed. Files monitored here serve as a good
base set of files.
To apply the policy Kubernetes uses a CRD that can be applied with kubectl. Uses the same YAML configuration as Kuberenetes, but loaded through a file on disk.
kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/quickstart/file_monitoring.yaml
wget https://raw.githubusercontent.com/cilium/tetragon/main/examples/quickstart/file_monitoring.yaml
docker stop tetragon-container
docker run --name tetragon-container --rm --pull always \
--pid=host --cgroupns=host --privileged \
-v ${PWD}/file_monitoring.yaml:/etc/tetragon/tetragon.tp.d/file_monitoring.yaml \
-v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf \
quay.io/cilium/tetragon-ci:latest
With the file applied we can attach tetra to observe events again:
kubectl exec -ti -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact --pods xwing
docker exec tetragon-container tetra getevents -o compact
Then reading a sensitive file:
kubectl exec -ti xwing -- bash -c 'cat /etc/shadow'
cat /etc/shadow
This will generate a read event (Docker events will omit Kubernetes metadata),
🚀 process default/xwing /bin/bash -c "cat /etc/shadow"
🚀 process default/xwing /bin/cat /etc/shadow
📚 read default/xwing /bin/cat /etc/shadow
💥 exit default/xwing /bin/cat /etc/shadow 0
Attempts to write in sensitive directories will similar create an event. For
example attempting to write in /etc
.
kubectl exec -ti xwing -- bash -c 'echo foo >> /etc/bar'
cat /etc/shadow
Will result in the following output in the tetra CLI.
🚀 process default/xwing /bin/bash -c "echo foo >> /etc/bar"
📝 write default/xwing /bin/bash /etc/bar
📝 write default/xwing /bin/bash /etc/bar
💥 exit default/xwing /bin/bash -c "echo foo >> /etc/bar
To explore tracing policies for networking try the Networking Monitoring quickstart. To dive into the details of policies and events please see Concepts section.
This adds a network policy on top of execution and file tracing already deployed in the quick start. In this case we monitor all network traffic outside the Kubernetes pod CIDR and service CIDR.
First we must find the pod CIDR and service CIDR in use. The pod IP CIDR can be found relatively easily in many cases.
export PODCIDR=`kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'`
The services CIDR can then be fetched depending on environment. We require environment variables ZONE and NAME from install steps.
export SERVICECIDR=$(gcloud container clusters describe ${NAME} --zone ${ZONE} | awk '/servicesIpv4CidrBlock/ { print $2; }')
export SERVICECIDR=$(kubectl describe pod -n kube-system kube-apiserver-kind-control-plane | awk -F= '/--service-cluster-ip-range/ {print $2; }')
First we apply a policy that includes the podCIDR
and serviceIP
list as
filters to avoid filter out cluster local traffic. To apply the policy:
wget https://raw.githubusercontent.com/cilium/tetragon/main/examples/quickstart/network_egress_cluster.yaml
envsubst < network_egress_cluster.yaml | kubectl apply -f -
With the file applied we can attach tetra to observe events again:
kubectl exec -ti -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact --pods xwing --processes curl
Then execute a curl command in the xwing pod to curl one of our favorite sites.
kubectl exec -ti xwing -- bash -c 'curl https://ebpf.io/applications/#tetragon'
A connect will be observed in the tetra shell:
🚀 process default/xwing /usr/bin/curl https://ebpf.io/applications/#tetragon
🔌 connect default/xwing /usr/bin/curl tcp 10.32.0.19:33978 -> 104.198.14.52:443
💥 exit default/xwing /usr/bin/curl https://ebpf.io/applications/#tetragon 60
We can confirm in-kernel BPF filters are not producing events for in cluster traffic by issuing a curl to one of our services and noting there is no connect event.
kubectl exec -ti xwing -- bash -c 'curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing'
The output should be similar to:
Ship landed
And as expected no new events:
🚀 process default/xwing /usr/bin/curl https://ebpf.io/applications/#tetragon
🔌 connect default/xwing /usr/bin/curl tcp 10.32.0.19:33978 -> 104.198.14.52:443
💥 exit default/xwing /usr/bin/curl https://ebpf.io/applications/#tetragon 60
This example also works easily for local docker users as well except it is not as
obvious to the quickstart authors what IP address CIDR will be useful. The policy
by default will filter all local IPs 127.0.0.1
from the event log. So we can
demo that here.
Set env variables to local loopback IP.
export PODCIDR="127.0.0.1/32"
export SERVICECIDR="127.0.0.1/32"
To create the policy,
wget https://raw.githubusercontent.com/cilium/tetragon/main/examples/quickstart/network_egress_cluster.yaml
envsubst < network_egress_cluster.yaml > network_egress_cluster_subst.yaml
Start Tetragon with the new policy:
docker stop tetragon-container
docker run --name tetragon-container --rm --pull always \
--pid=host --cgroupns=host --privileged \
-v ${PWD}/network_egress_cluster_subst.yaml:/etc/tetragon/tetragon.tp.d/network_egress_cluster_subst.yaml \
-v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf \
quay.io/cilium/tetragon-ci:latest
Attach to the tetragon output
docker exec tetragon-container tetra getevents -o compact
Now remote TCP connections will be logged and local IPs filters. Executing a curl to generate a remote TCP connect.
curl https://ebpf.io/applications/#tetragon
Produces the following output:
🚀 process /usr/bin/curl https://ebpf.io/applications/#tetragon
🔌 connect /usr/bin/curl tcp 192.168.1.190:36124 -> 104.198.14.52:443
💥 exit /usr/bin/curl https://ebpf.io/applications/#tetragon 0
So far we have installed Tetragon and shown a couple policies to monitor sensitive files and provide network auditing for connections outside our own cluster and node. Both these cases highlight the value of in kernel filtering. Another benefit of in-kernel filtering is we can add enforcement to the policies to not only alert, but block the operation in kernel and/or kill the application attempting the operation.
To learn more about policies and events Tetragon can implement review the Concepts section.
This adds a network and file policy enforcement on top of execution, file tracing
and networking policy already deployed in the quick start. In this use case we use
a namespace filter to limit the scope of the enforcement policy to just the default
namespace we installed the demo application in from the
Quick Kubernetes Install.
This highlights two important concepts of Tetragon. First in kernel filtering
provides a key performance improvement by limiting events from kernel to user
space. But, also allows for enforcing policies in the kernel. By issueing a
SIGKILL
to the process at this point the application will be stopped from
continuing to run. If the operation is triggered through a syscall this means
the application will not return from the syscall and will be terminated.
Second, by including kubernetes filters, such as namespace and labels we can segment a policy to apply to targeted namespaces and pods. This is critical for effective policy segmentation.
For implementation details see the Enforcement concept section.
The following section is layed out with the following:
First we will deploy the Network Monitoring
policy with enforcement on. For this case the policy is written to only apply
against the empire
namespace. This limits the scope of the policy for the
getting started guide.
Ensure we have the proper Pod CIDRs
export PODCIDR=`kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'`
and Service CIDRs configured.
export SERVICECIDR=$(gcloud container clusters describe ${NAME} --zone ${ZONE} | awk '/servicesIpv4CidrBlock/ { print $2; }')
export SERVICECIDR=$(kubectl describe pod -n kube-system kube-apiserver-kind-control-plane | awk -F= '/--service-cluster-ip-range/ {print $2; }')
Then we can apply the egress cluster enforcement policy
wget https://raw.githubusercontent.com/cilium/tetragon/main/examples/quickstart/network_egress_cluster_enforce.yaml
envsubst < network_egress_cluster_enforce.yaml | kubectl apply -n default -f -
With the enforcement policy applied we can attach tetra to observe events again:
kubectl exec -ti -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact --pods xwing
And once again execute a curl command in the xwing:
kubectl exec -ti xwing -- bash -c 'curl https://ebpf.io/applications/#tetragon'
The command returns an error code because the egress TCP connects are blocked shown here.
command terminated with exit code 137
Connect inside the cluster will work as expected,
kubectl exec -ti xwing -- bash -c 'curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing'
The Tetra CLI will print the curl and annotate that the process that was issued a Sigkill. The successful internal connect is filtered and will not be shown.
🚀 process default/xwing /bin/bash -c "curl https://ebpf.io/applications/#tetragon"
🚀 process default/xwing /usr/bin/curl https://ebpf.io/applications/#tetragon
🔌 connect default/xwing /usr/bin/curl tcp 10.32.0.28:45200 -> 104.198.14.52:443
💥 exit default/xwing /usr/bin/curl https://ebpf.io/applications/#tetragon SIGKILL
🚀 process default/xwing /bin/bash -c "curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing"
🚀 process default/xwing /usr/bin/curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
The following extends the example from File Access Monitoring
with enforcement to ensure sensitive files are not read. The policy used is the
file_monitoring_enforce.yaml
it can be reviewed and extended as needed. The only difference between the
observation policy and the enforce policy is the addition of an action block
to sigkill the application and return an error on the op.
To apply the policy:
kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/quickstart/file_monitoring.yaml
kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/quickstart/file_monitoring_enforce.yaml
wget https://raw.githubusercontent.com/cilium/tetragon/main/examples/quickstart/file_monitoring.yaml
docker stop tetragon-container
docker run --name tetragon-container --rm --pull always \
--pid=host --cgroupns=host --privileged \
-v ${PWD}/file_monitoring.yaml:/etc/tetragon/tetragon.tp.d/file_monitoring_enforce.yaml \
-v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf \
quay.io/cilium/tetragon-ci:latest
With the file applied we can attach tetra to observe events again,
kubectl exec -ti -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact --pods xwing
docker exec tetragon-container tetra getevents -o compact
Then reading a sensitive file,
kubectl exec -ti xwing -- bash -c 'cat /etc/shadow'
cat /etc/shadow
The command will fail with an error code because this is one of our sensitive files,
kubectl exec -ti xwing -- bash -c 'cat /etc/shadow'
The output should be similar to:
command terminated with exit code 137
This will generate a read event (Docker events will omit Kubernetes metadata),
🚀 process default/xwing /bin/bash -c "cat /etc/shadow"
🚀 process default/xwing /bin/cat /etc/shadow
📚 read default/xwing /bin/cat /etc/shadow
📚 read default/xwing /bin/cat /etc/shadow
📚 read default/xwing /bin/cat /etc/shadow
💥 exit default/xwing /bin/cat /etc/shadow SIGKILL
Writes and reads to files not part of the enforced file policy will not be impacted.
🚀 process default/xwing /bin/bash -c "echo foo >> bar; cat bar"
🚀 process default/xwing /bin/cat bar
💥 exit default/xwing /bin/cat bar 0
💥 exit default/xwing /bin/bash -c "echo foo >> bar; cat bar" 0
The completes the quick start guides. At this point we should be able to observe execution traces in a Kubernetes cluster and extend the base deployment of Tetragon with policies to observe and enforce different aspects of a Kubernetes system.
The rest of the docs provide further documentation about installation and using policies. Some useful links:
To explore details of writing and implementing policies the Concepts is a good jumping off point. For installation into production environments we recommend reviewing Advanced Installations. Finally Use Cases and Tutorials cover different uses and deployment concerns related to Tetragon.
The recommended way to deploy Tetragon on a Kubernetes cluster is to use the Helm chart with Helm 3. Tetragon uses the helm.cilium.io repository to release the helm chart.
To install the latest release of the Tetragon helm chart, use the following command.
--set KEY1=VALUE1,KEY2=VALUE2
.helm repo add cilium https://helm.cilium.io
helm repo update
helm install tetragon cilium/tetragon -n kube-system
To wait until Tetragon deployment is ready, use the following kubectl
command:
kubectl rollout status -n kube-system ds/tetragon -w
You can then make modifications to the Tetragon configuration using helm upgrade
, see the following example.
helm upgrade tetragon cilium/tetragon -n kube-system --set tetragon.grpc.address=localhost:1337
You can also edit the tetragon-config
ConfigMap directly and restart the
Tetragon daemonset with:
kubectl edit cm tetragon-config -n kube-system
kubectl rollout restart ds/tetragon -n kube-system
Upgrade Tetragon using a new specific version of the helm chart.
helm upgrade tetragon cilium/tetragon -n kube-system --version 0.9.0
Uninstall Tetragon using the following command.
helm uninstall tetragon -n kube-system
To run a stable version, please check Tetragon quay repository
and select which version you want. For example if you want to run the latest
version which is v1.0.0
currently.
docker run --name tetragon --rm -d \
--pid=host --cgroupns=host --privileged \
-v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf \
quay.io/cilium/tetragon:v1.0.0
To run unstable development versions of Tetragon, use the
latest
tag from Tetragon-CI quay repository.
This will run the image that was built from the latest commit available on the
Tetragon main branch.
docker run --name tetragon --rm -d \
--pid=host --cgroupns=host --privileged \
-v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf \
quay.io/cilium/tetragon-ci:latest
There are multiple ways to set configuration options:
Append Tetragon controlling settings at the end of the command
As an example set the file where to export JSON events with --export-filename
argument:
docker run --name tetragon --rm -d \
--pid=host --cgroupns=host --privileged \
-v /sys/kernel:/sys/kernel \
quay.io/cilium/tetragon:v1.0.0 \
/usr/bin/tetragon --export-filename /var/log/tetragon/tetragon.log
For a complete list of CLI arguments, please check Tetragon daemon configuration.
Environment variables
docker run --name tetragon --rm -d \
--pid=host --cgroupns=host --privileged \
--env "TETRAGON_EXPORT_FILENAME=/var/log/tetragon/tetragon.log" \
-v /sys/kernel:/sys/kernel \
quay.io/cilium/tetragon:v1.0.0
Every controlling setting can be set using environment variables. Prefix it with the key word TETRAGON_
then upper case the controlling setting. As an example to set where to export JSON events: --export-filename
will be TETRAGON_EXPORT_FILENAME
.
For a complete list of all controlling settings, please check tetragon daemon configuration.
Configuration files mounted as volumes
On the host machine set the configuration drop-ins inside /etc/tetragon/tetragon.conf.d/
directory according to the configuration examples, then mount it as volume:
docker run --name tetragon --rm -d \
--pid=host --cgroupns=host --privileged \
-v /sys/kernel:/sys/kernel \
-v /etc/tetragon/tetragon.conf.d/:/etc/tetragon/tetragon.conf.d/ \
quay.io/cilium/tetragon:v1.0.0
This will map the /etc/tetragon/tetragon.conf.d/
drop-in directory from the host into the container.
See Tetragon daemon configuration reference for further details.
Tetragon will be managed as a systemd service. Tarballs are built and distributed along the assets in the releases.
First download the latest binary tarball, using curl
for example to download the amd64
release:
curl -LO https://github.com/cilium/tetragon/releases/download/v1.0.0/tetragon-v1.0.0-amd64.tar.gz
Extract the downloaded archive, and start the install script to install Tetragon. Feel free to inspect the script before starting it.
tar -xvf tetragon-v1.0.0-amd64.tar.gz
cd tetragon-v1.0.0-amd64/
sudo ./install.sh
If Tetragon was successfully installed, the final output should be similar to:
Tetragon installed successfully!
Finally, you can check the Tetragon systemd service.
sudo systemctl status tetragon
The output should be similar to:
● tetragon.service - Tetragon eBPF-based Security Observability and Runtime Enforcement
Loaded: loaded (/lib/systemd/system/tetragon.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2023-01-23 20:08:16 CET; 5s ago
Docs: https://github.com/cilium/tetragon/
Main PID: 138819 (tetragon)
Tasks: 17 (limit: 18985)
Memory: 151.7M
CPU: 913ms
CGroup: /system.slice/tetragon.service
└─138819 /usr/local/bin/tetragon
The default Tetragon configuration shipped with the Tetragon package will be
installed in /usr/local/lib/tetragon/tetragon.conf.d/
. Local administrators
can change the configuration by adding drop-ins inside
/etc/tetragon/tetragon.conf.d/
to override the default settings or use the
command line flags. To restore default settings, remove any added configuration
inside /etc/tetragon/tetragon.conf.d/
.
See Tetragon daemon configuration for further details.
To upgrade Tetragon:
Download the new tarball.
curl -LO https://github.com/cilium/tetragon/releases/download/v1.0.0/tetragon-v1.0.0-amd64.tar.gz
Stop the Tetragon service.
sudo systemctl stop tetragon
Remove the old Tetragon version.
sudo rm -fr /usr/lib/systemd/system/tetragon.service
sudo rm -fr /usr/local/bin/tetragon
sudo rm -fr /usr/local/lib/tetragon/
Install the upgraded Tetragon version.
tar -xvf tetragon-v1.0.0-amd64.tar.gz
cd tetragon-v1.0.0-amd64/
sudo ./install.sh
To completely remove Tetragon run the uninstall.sh
script that is provided
inside the tarball.
sudo ./uninstall.sh
Or remove it manually.
sudo systemctl stop tetragon
sudo systemctl disable tetragon
sudo rm -fr /usr/lib/systemd/system/tetragon.service
sudo systemctl daemon-reload
sudo rm -fr /usr/local/bin/tetragon
sudo rm -fr /usr/local/bin/tetra
sudo rm -fr /usr/local/lib/tetragon/
To just purge custom settings:
sudo rm -fr /etc/tetragon/
unix:///var/run/tetragon/tetragon.sock
To access the gRPC API with tetra
client, set --server-address
to point to the corresponding address:
sudo tetra --server-address unix:///var/run/tetragon/tetragon.sock getevents
See restrict gRPC API access for further details.
By default JSON events are logged to /var/log/tetragon/tetragon.log
unless this location is changed.
Logs are always rotated into the same directory.
To read real-time JSON events, tailing the logs file is enough.
sudo tail -f /var/log/tetragon/tetragon.log
Tetragon also ships a gRPC client that can be used to receive events.
To print events in json
format using tetra
gRPC client:
sudo tetra --server-address "unix:///var/run/tetragon/tetragon.sock" getevents
To print events in human compact format:
sudo tetra --server-address "unix:///var/run/tetragon/tetragon.sock" getevents -o compact
See Explore security observability events to learn more about how to see the Tetragon events.
This guide presents various methods to install tetra
in your environment.
This shell script autodetects the OS and the architecture, downloads the archive of the binary and its SHA 256 digest, compares that the actual digest with the supposed one, installs the binary, and removes the download artifacts.
curl(1)
, and the
sha256sum(1)
utilities. For Go, see how to install the latest Go release
and for the curl and checksum utility, it is usually distributed in common
Linux distribution but you can usually find them respectively under the package
curl
and coreutils
.GOOS=$(go env GOOS)
GOARCH=$(go env GOARCH)
curl -L --remote-name-all https://github.com/cilium/tetragon/releases/latest/download/tetra-${GOOS}-${GOARCH}.tar.gz{,.sha256sum}
sha256sum --check tetra-${GOOS}-${GOARCH}.tar.gz.sha256sum
sudo tar -C /usr/local/bin -xzvf tetra-${GOOS}-${GOARCH}.tar.gz
rm tetra-${GOOS}-${GOARCH}.tar.gz{,.sha256sum}
This installation method retrieves the adapted archived for your environment,
extract it and install it in the /usr/local/bin
directory.
curl(1)
that should be already
installed in your environment, otherwise you can usually find it under the
curl
package.curl -L https://github.com/cilium/tetragon/releases/latest/download/tetra-linux-amd64.tar.gz | tar -xz
sudo mv tetra /usr/local/bin
curl -L https://github.com/cilium/tetragon/releases/latest/download/tetra-linux-arm64.tar.gz | tar -xz
sudo mv tetra /usr/local/bin
curl -L https://github.com/cilium/tetragon/releases/latest/download/tetra-darwin-amd64.tar.gz | tar -xz
sudo mv tetra /usr/local/bin
curl -L https://github.com/cilium/tetragon/releases/latest/download/tetra-darwin-arm64.tar.gz | tar -xz
sudo mv tetra /usr/local/bin
curl -L https://github.com/cilium/tetragon/releases/latest/download/tetra-windows-amd64.tar.gz
tar -xz tetra-windows-amd64.tar.gz
# move the binary in a directory in your PATH
curl -L https://github.com/cilium/tetragon/releases/latest/download/tetra-windows-arm64.tar.gz
tar -xz tetra-windows-arm64.tar.gz
# move the binary in a directory in your PATH
Homebrew is a package manager for macOS and Linux.
A formulae is available to
fetch precompiled binaries. You can also use it to build from sources (using the
--build-from-source
flag) with a Go dependency.
brew install tetra
You can retrieve the release of tetra
along the release of Tetragon on GitHub
at the following URL: https://github.com/cilium/tetragon/releases.
To download a specific release you can use the following script, replacing the
OS
, ARCH
and TAG
values with your desired options.
OS=linux
ARCH=amd64
TAG=v0.9.0
curl -LO https://github.com/cilium/tetragon/releases/download/${TAG}/tetra-${OS}-${ARCH}.tar.gz | tar -xz
Tetragon’s events are exposed to the system through either the gRPC endpoint or JSON logs. Commands in this section assume the Getting Started guide was used, but are general other than the namespaces chosen and should work in most environments.
The first way is to observe the raw json output from the stdout container log:
kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f
The raw JSON events provide Kubernetes API, identity metadata, and OS
level process visibility about the executed binary, its parent and the execution
time. A base Tetragon installation will produce process_exec
and process_exit
events encoded in JSON as shown here,
{
"process_exec": {
"process": {
"exec_id": "Z2tlLWpvaG4tNjMyLWRlZmF1bHQtcG9vbC03MDQxY2FjMC05czk1OjEzNTQ4Njc0MzIxMzczOjUyNjk5",
"pid": 52699,
"uid": 0,
"cwd": "/",
"binary": "/usr/bin/curl",
"arguments": "https://ebpf.io/applications/#tetragon",
"flags": "execve rootcwd",
"start_time": "2023-10-06T22:03:57.700327580Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://551e161c47d8ff0eb665438a7bcd5b4e3ef5a297282b40a92b7c77d6bd168eb3",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-10-06T21:52:41Z",
"pid": 49
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
},
"workload": "xwing"
},
"docker": "551e161c47d8ff0eb665438a7bcd5b4",
"parent_exec_id": "Z2tlLWpvaG4tNjMyLWRlZmF1bHQtcG9vbC03MDQxY2FjMC05czk1OjEzNTQ4NjcwODgzMjk5OjUyNjk5",
"tid": 52699
},
"parent": {
"exec_id": "Z2tlLWpvaG4tNjMyLWRlZmF1bHQtcG9vbC03MDQxY2FjMC05czk1OjEzNTQ4NjcwODgzMjk5OjUyNjk5",
"pid": 52699,
"uid": 0,
"cwd": "/",
"binary": "/bin/bash",
"arguments": "-c \"curl https://ebpf.io/applications/#tetragon\"",
"flags": "execve rootcwd clone",
"start_time": "2023-10-06T22:03:57.696889812Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://551e161c47d8ff0eb665438a7bcd5b4e3ef5a297282b40a92b7c77d6bd168eb3",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-10-06T21:52:41Z",
"pid": 49
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
},
"workload": "xwing"
},
"docker": "551e161c47d8ff0eb665438a7bcd5b4",
"parent_exec_id": "Z2tlLWpvaG4tNjMyLWRlZmF1bHQtcG9vbC03MDQxY2FjMC05czk1OjEzNTQ4NjQ1MjQ1ODM5OjUyNjg5",
"tid": 52699
}
},
"node_name": "gke-john-632-default-pool-7041cac0-9s95",
"time": "2023-10-06T22:03:57.700326678Z"
}
Will only highlight a few important fields here. For a full specification of events see the Reference
section. All events in Tetragon contain a process_exec
block to identify the process generating the
event. For execution events this is the primary block. For Tracing Policy events the
hook that generated the event will attach further data to this. The process_exec
event provides
a cluster wide unique id the process_exec.exec_id
for this process along with the metadata expected
in a Kubernetes cluster process_exec.process.pod
. The binary and args being executed are part of
the event here process_exec.process.binary
and process_exec.process.args
. Finarlly a node_name
and time
provide the location and time for the event and will be present in all event types.
A default deployment writes the JSON log to /var/run/cilium/tetragon/tetragon.log
where it can
be exported through normal log collection tooling, e.g. ‘fluentd’, logstash, etc.. The file will
be rotated and compressed by default. See [Helm Options] for details on how to customize this location.
tetra
CLIA second way is to use the tetra
CLI. This
has the advantage that it can also be used to filter and pretty print the output. The tool
allows filtering by process, pod, and other fields. To install tetra see the
Tetra Installation Guide
To start printing events run:
kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | tetra getevents -o compact
The tetra
CLI is also available inside tetragon
container.
kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact
This was used in the quick start and generates a pretty printing of the events, To further filter by a specific binary and/or pod do the following,
kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | tetra getevents -o compact --processes curl --pod xwing
Will filter and report just the relevant events.
🚀 process default/xwing /usr/bin/curl https://ebpf.io/applications/#tetragon
💥 exit default/xwing /usr/bin/curl https://ebpf.io/applications/#tetragon 60
In addition Tetragon can expose a gRPC endpoint listeners may attach to. The
gRPC is exposed by default helm install on localhost:54321
, but the address
can be configured with the --server-address
option. This can be
set from helm with the tetragon.grpc.address
flag or disabled completely if
needed with tetragon.grpc.enabled
.
helm install tetragon cilium/tetragon -n kube-system --set tetragon.grpc.enabled=true --set tetragon.grpc.address=localhost:54321
An example gRPC endpoint is the Tetra CLI when its not piped JSON output directly,
kubectl exec -ti -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact
Tetragon’s metrics are exposed to the system through an HTTP endpoint. These are used to expose event summaries and information about the state of the Tetragon agent.
Tetragon pods exposes a metrics endpoint by default. The chart also creates a
service named tetragon
that exposes metrics on the specified port.
Check if the tetragon
service exists:
kubectl get services tetragon -n kube-system
The output should be similar to:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
tetragon ClusterIP 10.96.54.218 <none> 2112/TCP 3m
To forward the metrics port locally, use kubectl port forward
:
kubectl -n kube-system port-forward service/tetragon 2112:2112
By default, metrics are disabled when using release packages to install locally. The
metrics can be enabled using --metrics-server
flag to specify the address.
Alternatively, the examples/configuration/tetragon.yaml
file contains example entries showing the defaults for the address of
metrics-server. Local overrides can be created by editing and copying this file
into /etc/tetragon/tetragon.yaml
, or by editing and copying “drop-ins” from
the examples/configuration/tetragon.conf.d
directory into the /etc/tetragon/tetragon.conf.d/
subdirectory. The latter is
generally recommended.
Run sudo tetragon --metrics-server localhost:2112
to set metrics address to localhost:2112
and export metrics.
sudo tetragon --metrics-server localhost:2112
The output should be similar to this:
time="2023-09-21T13:17:08+05:30" level=info msg="Starting tetragon"
version=v0.11.0
time="2023-09-21T13:17:08+05:30" level=info msg="config settings"
config="mapeased
time="2023-09-22T23:16:24+05:30" level=info msg="Starting metrics server"
addr="localhost:2112"
[...]
time="2023-09-21T13:17:08+05:30" level=info msg="Listening for events..."
Alternatively, a file named server-address
can be created in etc/tetragon/tetragon.conf.d/metrics-server
with content specifying
a port like this localhost:2112
, or any port of your choice as mentioned
above.
After the metrics are exposed, either by port forwarding in case of
Kubernetes installation or by setting metrics address in case of Package
installation, the metrics can be fetched using
curl
on localhost:2112/metrics
:
curl localhost:2112/metrics
The output should be similar to this:
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
# HELP tetragon_errors_total The total number of Tetragon errors. For internal use only.
# TYPE tetragon_errors_total counter
[...]
Tetragon’s TracingPolicy
is a user-configurable Kubernetes custom resource (CR) that
allows users to trace arbitrary events in the kernel and optionally define
actions to take on a match. Policies consist of a hook point (kprobes,
tracepoints, and uprobes are supported), and selectors for in-kernel filtering
and specifying actions. For more details, see
hook points page and the
selectors page.
TracingPolicy
allows for powerful, yet low-level configuration and, as such,
requires knowledge about the Linux kernel and containers to avoid unexpected
issues such as TOCTOU bugs.For the complete custom resource definition (CRD) refer to the YAML file
cilium.io_tracingpolicies.yaml
.
One practical way to explore the CRD is to use kubectl explain
against a
Kubernetes API server on which it is installed, for example kubectl explain tracingpolicy.spec.kprobes
provides field-specific documentation and details
on kprobe spec.
Tracing Policies can be loaded and unloaded at runtime in Tetragon, or on startup using flags.
kubectl
to add and remove a TracingPolicy
.tetra
gRPC CLI to add and remove a TracingPolicy
.--tracing-policy
and --tracing-policy-dir
flags to statically add policies at
startup time, see more in the daemon configuration page.Hence, even though Tracing Policies are structured as a Kubernetes CR, they can also be used in non-Kubernetes environments using the last two loading methods.
To discover TracingPolicy
, let’s understand via an example that will be
explained, part by part, in this document:
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
name: "fd-install"
spec:
kprobes:
- call: "fd_install"
syscall: false
args:
- index: 0
type: "int"
- index: 1
type: "file"
selectors:
- matchArgs:
- index: 1
operator: "Equal"
values:
- "/tmp/tetragon"
matchActions:
- action: Sigkill
The policy checks for file descriptors being created, and sends a SIGKILL
signal to any process that
creates a file descriptor to a file named /tmp/tetragon
. We discuss the policy in more detail
next.
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
name: "fd-install"
The first part follows a common pattern among all Cilium Policies or more widely Kubernetes object. It first declares the Kubernetes API used, then the kind of Kubernetes object it is in this API and an arbitrary name for the object that has to comply with Kubernetes naming convention.
spec:
kprobes:
- call: "fd_install"
syscall: false
args:
- index: 0
type: "int"
- index: 1
type: "file"
The beginning of the specification describes the hook point to use. Here we are
using a kprobe, hooking on the kernel function fd_install
. That’s the kernel
function that gets called when a new file descriptor is created. We
indicate that it’s not a syscall, but a regular kernel function. We then
specify the function arguments, so that Tetragon’s BPF code will extract
and optionally perform filtering on them.
See the hook points page for further information on the various hook points available and arguments.
selectors:
- matchArgs:
- index: 1
operator: "Equal"
values:
- "/tmp/tetragon"
matchActions:
- action: Sigkill
Selectors allow you to filter on the events to extract only a subset of the events based on different properties and optionally take an action.
In the example, we filter on the argument at index 1, passing a file
struct
to the function. Tetragon has the knowledge on how to apply the Equal
operator over a Linux kernel file
struct and match on the
path of the file.
Then we add the Sigkill
action, meaning, that any match of the selector
should send a SIGKILL signal to the process that initiated the event.
Learn more about the various selectors in the dedicated selectors page.
First, let’s create the /tmp/tetragon
file with some content:
echo eBPF! > /tmp/tetragon
You can save the policy in an example.yaml
file, compile Tetragon locally, and start Tetragon:
sudo ./tetragon --bpf-lib bpf/objs --tracing-policy example.yaml
(See Quick Kubernetes Install and Quick Local Docker Install for other ways to start Tetragon.)
Once the Tetragon starts, you can monitor events using tetra
, the tetragon CLI:
./tetra tetra getevents -o compact
Reading the /tmp/tetragon
file with cat
:
cat /tmp/tetragon
Results in the following events:
🚀 process /usr/bin/cat /tmp/tetragon
📬 open /usr/bin/cat /tmp/tetragon
💥 exit /usr/bin/cat /tmp/tetragon SIGKILL
And the shell where the cat
command was performed will return:
Killed
For more examples of tracing policies, take a look at the examples/tracingpolicy folder in the Tetragon repository. Also read the following sections on hook points and selectors.
Tetragon can hook into the kernel using kprobes
and tracepoints
, as well as in user-space
programs using uprobes
. Users can configure these hook points using the correspodning sections of
the TracingPolicy
specification (.spec
). These hook points include arguments and return values
that can be specified using the args
and returnArg
fields as detailed in the following sections.
Kprobes enables you to dynamically hook into any kernel function and execute BPF code. Because kernel functions might change across versions, kprobes are highly tied to your kernel version and, thus, might not be portable across different kernels.
Conveniently, you can list all kernel symbols reading the /proc/kallsyms
file. For example to search for the write
syscall kernel function, you can
execute sudo grep sys_write /proc/kallsyms
, the output should be similar to
this, minus the architecture specific prefixes.
ffffdeb14ea712e0 T __arm64_sys_writev
ffffdeb14ea73010 T ksys_write
ffffdeb14ea73140 T __arm64_sys_write
ffffdeb14eb5a460 t proc_sys_write
ffffdeb15092a700 d _eil_addr___arm64_sys_writev
ffffdeb15092a740 d _eil_addr___arm64_sys_write
You can see that the exact name of the symbol for the write syscall on our
kernel version is __arm64_sys_write
. Note that on x86_64
, the prefix would
be __x64_
instead of __arm64_
.
Kernel symbols contain an architecture specific prefix when they refer to syscall symbols. To write portable tracing policies, i.e. policies that can run on multiple architectures, just use the symbol name without the prefix.
For example, instead of writing call: "__arm64_sys_write"
or call: "__x64_sys_write"
, just write call: "sys_write"
, Tetragon will adapt and add
the correct prefix based on the architecture of the underlying machine. Note
that the event generated as output currently includes the prefix.
In our example, we will explore a kprobe
hooking into the
fd_install
kernel function. The fd_install
kernel function is called each time a file
descriptor is installed into the file descriptor table of a process, typically
referenced within system calls like open
or openat
. Hooking fd_install
has its benefits and limitations, which are out of the scope of this guide.
spec:
kprobes:
- call: "fd_install"
syscall: false
syscall
field, specific to a kprobe
spec, with default value
false
, that indicates whether Tetragon will hook a syscall or just a regular
kernel function. Tetragon needs this information because syscall and kernel
function use a different ABI.Kprobes calls can be defined independently in different policies, or together in the same Policy. For example, we can define trace multiple kprobes under the same tracing policy:
spec:
kprobes:
- call: "sys_read"
syscall: true
# [...]
- call: "sys_write"
syscall: true
# [...]
Tracepoints are statically defined in the kernel and have the advantage of being stable across kernel versions and thus more portable than kprobes.
To see the list of tracepoints available on your kernel, you can list them
using sudo ls /sys/kernel/debug/tracing/events
, the output should be similar
to this.
alarmtimer ext4 iommu page_pool sock
avc fib ipi pagemap spi
block fib6 irq percpu swiotlb
bpf_test_run filelock jbd2 power sync_trace
bpf_trace filemap kmem printk syscalls
bridge fs_dax kvm pwm task
btrfs ftrace libata qdisc tcp
cfg80211 gpio lock ras tegra_apb_dma
cgroup hda mctp raw_syscalls thermal
clk hda_controller mdio rcu thermal_power_allocator
cma hda_intel migrate regmap thermal_pressure
compaction header_event mmap regulator thp
cpuhp header_page mmap_lock rpm timer
cros_ec huge_memory mmc rpmh tlb
dev hwmon module rseq tls
devfreq i2c mptcp rtc udp
devlink i2c_slave napi sched vmscan
dma_fence initcall neigh scmi wbt
drm interconnect net scsi workqueue
emulation io_uring netlink signal writeback
enable iocost oom skb xdp
error_report iomap page_isolation smbus xhci-hcd
You can then choose the subsystem that you want to trace, and look the
tracepoint you want to use and its format. For example, if we choose the
netif_receive_skb
tracepoints from the net
subsystem, we can read its
format with sudo cat /sys/kernel/debug/tracing/events/net/netif_receive_skb/format
,
the output should be similar to the following.
name: netif_receive_skb
ID: 1398
format:
field:unsigned short common_type; offset:0; size:2; signed:0;
field:unsigned char common_flags; offset:2; size:1; signed:0;
field:unsigned char common_preempt_count; offset:3; size:1; signed:0;
field:int common_pid; offset:4; size:4; signed:1;
field:void * skbaddr; offset:8; size:8; signed:0;
field:unsigned int len; offset:16; size:4; signed:0;
field:__data_loc char[] name; offset:20; size:4; signed:0;
print fmt: "dev=%s skbaddr=%px len=%u", __get_str(name), REC->skbaddr, REC->len
Similarly to kprobes, tracepoints can also hook into system calls. For more
details, see the raw_syscalls
and syscalls
subysystems.
An example of tracepoints TracingPolicy
could be the following, observing all
syscalls and getting the syscall ID from the argument at index 4:
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
name: "raw-syscalls"
spec:
tracepoints:
- subsystem: "raw_syscalls"
event: "sys_enter"
args:
- index: 4
type: "int64"
This hook point method lacks documentation, see issue #878.
Kprobes, uprobes and tracepoints all share a needed arguments fields called args
. It is a list of
arguments to include in the trace output. Tetragon’s BPF code requires
information about the types of arguments to properly read, print and
filter on its arguments. This information needs to be provided by the user under the
args
section. For the available
types,
check the TracingPolicy
CRD.
Following our example, here is the part that defines the arguments:
args:
- index: 0
type: "int"
- index: 1
type: "file"
Each argument can optionally include a ’label’ parameter, which will be included in the output. This can be used to annotate the arguments to help with understanding and processing the output. As an example, here is the same definition, with an appropriate label on the int argument:
args:
- index: 0
type: "int"
label: "FD"
- index: 1
type: "file"
To properly read and hook onto the fd_install(unsigned int fd, struct file *file)
function, the YAML snippet above tells the BPF code that the first
argument is an int
and the second argument is a file
, which is the
struct file
of the kernel. In this way, the BPF code and its printer can properly collect
and print the arguments.
These types are sorted by the index
field, where you can specify the order.
The indexing starts with 0. So, index: 0
means, this is going to be the first
argument of the function, index: 1
means this is going to be the second
argument of the function, etc.
Note that for some args types, char_buf
and char_iovec
, there are
additional fields named returnCopy
and sizeArgIndex
available:
returnCopy
indicates that the corresponding argument should be read later (when
the kretprobe for the symbol is triggered) because it might not be populated
when the kprobe is triggered at the entrance of the function. For example, a
buffer supplied to read(2)
won’t have content until kretprobe is triggered.sizeArgIndex
indicates the (1-based, see warning below) index of the arguments
that represents the size of the char_buf
or iovec
. For example, for
write(2)
, the third argument, size_t count
is the number of char
element that we can read from the const void *buf
pointer from the second
argument. Similarly, if we would like to capture the __x64_sys_writev(long, iovec *, vlen)
syscall, then iovec
has a size of vlen
, which is going to
be the third argument.sizeArgIndex
is inconsistent at the moment and does not take the index, but
the number of the index (or index + 1). So if the size is the third argument,
index 2, the value should be 3.These flags can be combined, see the example below.
- call: "sys_write"
syscall: true
args:
- index: 0
type: "int"
- index: 1
type: "char_buf"
returnCopy: true
sizeArgIndex: 3
- index: 2
type: "size_t"
Note that you can specify which arguments you would like to print from a
specific syscall. For example if you don’t care about the file descriptor,
which is the first int
argument with index: 0
and just want the char_buf
,
what is written, then you can leave this section out and just define:
args:
- index: 1
type: "char_buf"
returnCopy: true
sizeArgIndex: 3
- index: 2
type: "size_t"
This tells the printer to skip printing the int
arg because it’s not useful.
For char_buf
type up to the 4096 bytes are stored. Data with bigger size are
cut and returned as truncated bytes.
You can specify maxData
flag for char_buf
type to read maximum possible data
(currently 327360 bytes), like:
args:
- index: 1
type: "char_buf"
maxData: true
sizeArgIndex: 3
- index: 2
type: "size_t"
This field is only used for char_buff
data. When this value is false (default),
the bpf program will fetch at most 4096 bytes. In later kernels (>=5.4) tetragon
supports fetching up to 327360 bytes if this flag is turned on.
The maxData
flag does not work with returnCopy
flag at the moment, so it’s
usable only for syscalls/functions that do not require return probe to read the
data.
A TracingPolicy
spec can specify that the return value should be reported in
the tracing output. To do this, the return
parameter of the call needs to be
set to true
, and the returnArg
parameter needs to be set to specify the
type
of the return argument. For example:
- call: "sk_alloc"
syscall: false
return: true
args:
- index: 1
type: int
label: "family"
returnArg:
type: sock
In this case, the sk_alloc
hook is specified to return a value of type sock
(a pointer to a struct sock
). Whenever the sk_alloc
hook is hit, not only
will it report the family
parameter in index 1, it will also report the socket
that was created.
A unique feature of a sock
being returned from a hook such as sk_alloc
is that
the socket it refers to can be tracked. Most networking hooks in the network stack
are run in a context that is not that of the process that owns the socket for which
the actions relate; this is because networking happens asynchronously and not
entirely in-line with the process. The sk_alloc
hook does, however, occur in the
context of the process, such that the task, the PID, and the TGID are of the process
that requested that the socket was created.
Specifying socket tracking tells Tetragon to store a mapping between the socket
and the process’ PID and TGID; and to use that mapping when it sees the socket in a
sock
argument in another hook to replace the PID and TGID of the context with the
process that actually owns the socket. This can be done by adding a returnArgAction
to the call. Available actions are TrackSock
and UntrackSock
.
See TrackSock
and UntrackSock
.
- call: "sk_alloc"
syscall: false
return: true
args:
- index: 1
type: int
label: "family"
returnArg:
type: sock
returnArgAction: TrackSock
Socket tracking is only available on kernels >=5.3.
It’s possible to define list of functions and use it in the kprobe’s call
field.
Following example traces all sys_dup[23]
syscalls.
spec:
lists:
- name: "dups"
type: "syscalls"
values:
- "sys_dup"
- "sys_dup2"
- "sys_dup3"
kprobes:
- call: "list:dups"
It is basically a shortcut for following policy:
spec:
kprobes:
- call: "sys_dup"
syscall: true
- call: "sys_dup2"
syscall: true
- call: "sys_dup3"
syscall: true
As shown in subsequent examples, its main benefit is allowing a single definition for calls that have the same filters.
The list is defined under lists
field with arbitrary values for name
and values
fields.
spec:
lists:
- name: "dups"
type: "syscalls"
values:
- "sys_dup"
- "sys_dup2"
- "sys_dup3"
...
It’s possible to define multiple lists.
spec:
lists:
- name: "dups"
type: "syscalls"
values:
- "sys_dup"
- "sys_dup2"
- "sys_dup3"
name: "another"
- "sys_open"
- "sys_close"
Specific list can be referenced in kprobe’s call
field with "list:NAME"
value.
spec:
lists:
- name: "dups"
...
kprobes:
- call: "list:dups"
The kprobe definition creates a kprobe for each item in the list and shares the rest of the config specified for kprobe.
List can also specify type
field that implies extra checks on the values (like for syscall
type)
or denote that the list is generated automatically (see below).
User must specify syscall
type for list with syscall functions. Also syscall
functions
can’t be mixed with regular functions in the list.
The additional selector configuration is shared with all functions in the list. In following example we create 3 kprobes that share the same pid filter.
spec:
lists:
- name: "dups"
type: "syscalls"
values:
- "sys_dup"
- "sys_dup2"
- "sys_dup3"
kprobes:
- call: "list:dups"
selectors:
- matchPIDs:
- operator: In
followForks: true
isNamespacePID: false
values:
- 12345
It’s possible to use argument filter together with the list
.
It’s important to understand that the argument will be retrieved by using the specified argument type for all the functions in the list.
Following example adds argument filter for first argument on all functions in dups list to match value 9999.
spec:
lists:
- name: "dups"
type: "syscalls"
values:
- "sys_dup"
- "sys_dup2"
- "sys_dup3"
kprobes:
- call: "list:dups"
args:
- index: 0
type: int
selectors:
- matchArgs:
- index: 0
operator: "Equal"
values:
- 9999
There are two additional special types of generated lists.
The generated_syscalls
type of list that generates list with all possible
syscalls on the system.
Following example traces all syscalls for /usr/bin/kill
binary.
spec:
lists:
- name: "all-syscalls"
type: "generated_syscalls"
kprobes:
- call: "list:all-syscalls"
selectors:
- matchBinaries:
- operator: "In"
values:
- "/usr/bin/kill"
The generated_ftrace
type of list that generates functions from ftrace available_filter_functions
file with specified filter. The filter is specified with pattern
field and expects regular expression.
Following example traces all kernel ksys_*
functions for /usr/bin/kill
binary.
spec:
lists:
- name: "ksys"
type: "generated_ftrace"
pattern: "^ksys_*"
kprobes:
- call: "list:ksys"
selectors:
- matchBinaries:
- operator: "In"
values:
- "/usr/bin/kill"
It’s possible to pass options through spec file as an array of name and value pairs:
spec:
options:
- name: "option-1"
value: "True"
- name: "option-2"
value: "10"
Options array is passed and processed by each hook used in the spec file that supports options. At the moment it’s availabe for kprobe hooks.
Kprobe Options
: options for kprobe hooks.disable-kprobe-multi
: disable kprobe multi linkThis option disables kprobe multi link interface for all the kprobes defined in the spec file. If enabled, all the defined kprobes will be atached through standard kprobe interface. It stays enabled for another spec file without this option.
It takes boolean as value, by default it’s false.
Example:
options:
- name: "disable-kprobe-multi"
value: "1"
Selectors are a way to perform in-kernel BPF filtering on the events to export, or on the events on which to apply an action.
A TracingPolicy
can contain from 0 to 5 selectors. A selector is composed of
1 or more filters. The available filters are the following:
matchArgs
: filter on the value of arguments.matchReturnArgs
: filter on the return value.matchPIDs
: filter on PID.matchBinaries
: filter on binary path.matchNamespaces
: filter on Linux namespaces.matchCapabilities
: filter on Linux capabilities.matchNamespaceChanges
: filter on Linux namespaces changes.matchCapabilityChanges
: filter on Linux capabilities changes.matchActions
: apply an action on selector matching.Arguments filters can be specified under the matchArgs
field and provide
filtering based on the value of the function’s argument.
In the next example, a selector is defined with a matchArgs
filter that tells
the BPF code to process only the function call for which the second argument,
index equal to 1, concerns the file under the path /etc/passwd
or
/etc/shadow
. It’s using the operator Equal
to match against the value of
the argument.
Note that conveniently, we can match against a path directly when the argument
is of type file
.
selectors:
- matchArgs:
- index: 1
operator: "Equal"
values:
- "/etc/passwd"
- "/etc/shadow"
The available operators for matchArgs
are:
Equal
NotEqual
Prefix
Postfix
Mask
Further examples
In the previous example, we used the operator Equal
, but we can also use the
Prefix
operator and match against all files under /etc
with:
selectors:
- matchArgs:
- index: 1
operator: "Prefix"
values:
- "/etc"
In this situation, an event will be created every time a process tries to
access a file under /etc
.
Although it makes less sense, you can also match over the first argument, to only detect events that will use the file descriptor 4, which is usually the first that come afters stdin, stdout and stderr in process. And combine that with the previous example.
- matchArgs:
- index: 0
operator: "Equal"
values:
- "3"
- index: 1
operator: "Prefix"
values:
- "/etc"
Arguments filters can be specified under the returnMatchArgs
field and
provide filtering based on the value of the function return value. It allows
you to filter on the return value, thus success, error or value returned by a
kernel call.
matchReturnArgs:
- operator: "NotEqual"
values:
- 0
The available operators for matchReturnArgs
are:
Equal
NotEqual
Prefix
Postfix
A use case for this would be to detect the failed access to certain files, like
/etc/shadow
. Doing cat /etc/shadow
will use a openat
syscall that will
returns -1
for a failed attempt with an unprivileged user.
PIDs filters can be specified under the matchPIDs
field and provide filtering
based on the value of host pid of the process. For example, the following
matchPIDs
filter tells the BPF code that observe only hooks for which the
host PID is equal to either pid1
or pid2
or pid3
:
- matchPIDs:
- operator: "In"
followforks: true
values:
- "pid1"
- "pid2"
- "pid3"
The available operators for matchPIDs
are:
In
NotIn
Further examples
Another example can be to collect all processes not associated with a
container’s init PID, which is equal to 1. In this way, we are able to detect
if there was a kubectl exec
performed inside a container because processes
created by kubectl exec
are not children of PID 1.
- matchPIDs:
- operator: NotIn
followforks: false
isNamespacePID: true
values:
- 1
Binary filters can be specified under the matchBinaries
field and provide
filtering based on the value of a certain binary name. For example, the
following matchBinaries
selector tells the BPF code to process only system
calls and kernel functions that are coming from cat
or tail
.
- matchBinaries:
- operator: "In"
values:
- "/usr/bin/cat"
- "/usr/bin/tail"
Currently, only the In
operator type is supported and the values
field has
to be a map of strings
. The default behaviour is followForks: true
, so all
the child processes are followed. The current limitation is 4 values.
Further examples
One example can be to monitor all the sys_write
system calls which are
coming from the /usr/sbin/sshd
binary and its child processes and writing to
stdin/stdout/stderr
.
This is how we can monitor what was written to the console by different users
during different ssh sessions. The matchBinaries
selector in this case is the
following:
- matchBinaries:
- operator: "In"
values:
- "/usr/sbin/sshd"
while the whole kprobe
call is the following:
- call: "sys_write"
syscall: true
args:
- index: 0
type: "int"
- index: 1
type: "char_buf"
sizeArgIndex: 3
- index: 2
type: "size_t"
selectors:
# match to /sbin/sshd
- matchBinaries:
- operator: "In"
values:
- "/usr/sbin/sshd"
# match to stdin/stdout/stderr
matchArgs:
- index: 0
operator: "Equal"
values:
- "1"
- "2"
- "3"
Namespaces filters can be specified under the matchNamespaces
field and
provide filtering of calls based on Linux namespace. You can specify the
namespace inode or use the special host_ns
keyword, see the example and
description for more information.
An example syntax is:
- matchNamespaces:
- namespace: Pid
operator: In
values:
- "4026531836"
- "4026531835"
This will match if: [Pid
namespace is 4026531836
] OR
[Pid
namespace is
4026531835
]
namespace
can be: Uts
, Ipc
, Mnt
, Pid
, PidForChildren
, Net
,
Cgroup
, or User
. Time
and TimeForChildren
are also available in Linux
>= 5.6.operator
can be In
or NotIn
values
can be raw numeric values (i.e. obtained from lsns
) or "host_ns"
which will automatically be translated to the appropriate value.Limitations
values
. These can be both numeric and host_ns
inside
a single namespace
.namespace
values under matchNamespaces
in Linux
kernel < 5.3. In Linux >= 5.3 we can have up to 10 values (i.e. the maximum
number of namespaces that modern kernels provide).Further examples
We can have multiple namespace filters:
selectors:
- matchNamespaces:
- namespace: Pid
operator: In
values:
- "4026531836"
- "4026531835"
- namespace: Mnt
operator: In
values:
- "4026531833"
- "4026531834"
This will match if: ([Pid
namespace is 4026531836
] OR
[Pid
namespace is
4026531835
]) AND
([Mnt
namespace is 4026531833
] OR
[Mnt
namespace
is 4026531834
])
Use cases examples
Generate a kprobe event if
/etc/shadow
was opened by/bin/cat
which either had hostNet
orMnt
namespace access
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
name: "example_ns_1"
spec:
kprobes:
- call: "fd_install"
syscall: false
args:
- index: 0
type: int
- index: 1
type: "file"
selectors:
- matchBinaries:
- operator: "In"
values:
- "/bin/cat"
matchArgs:
- index: 1
operator: "Equal"
values:
- "/etc/shadow"
matchNamespaces:
- namespace: Mnt
operator: In
values:
- "host_ns"
- matchBinaries:
- operator: "In"
values:
- "/bin/cat"
matchArgs:
- index: 1
operator: "Equal"
values:
- "/etc/shadow"
matchNamespaces:
- namespace: Net
operator: In
values:
- "host_ns"
This example has 2 selectors
. Note that each selector starts with -
.
Selector 1:
- matchBinaries:
- operator: "In"
values:
- "/bin/cat"
matchArgs:
- index: 1
operator: "Equal"
values:
- "/etc/shadow"
matchNamespaces:
- namespace: Mnt
operator: In
values:
- "host_ns"
Selector 2:
- matchBinaries:
- operator: "In"
values:
- "/bin/cat"
matchArgs:
- index: 1
operator: "Equal"
values:
- "/etc/shadow"
matchNamespaces:
- namespace: Net
operator: In
values:
- "host_ns"
We have [
Selector1 OR
Selector2]
. Inside each selector we have filters
.
Both selectors have 3 filters (i.e. matchBinaries
, matchArgs
, and
matchNamespaces
) with different arguments. Adding a -
in the beginning of a
filter will result in a new selector.
So the previous CRD will match if:
[
binary == /bin/cat AND
arg1 == /etc/shadow AND
MntNs == host]
OR
[
binary == /bin/cat AND
arg1 == /etc/shadow AND
NetNs is host]
We can modify the previous example as follows:
Generate a kprobe event if
/etc/shadow
was opened by/bin/cat
which has hostNet
andMnt
namespace access
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
name: "example_ns_2"
spec:
kprobes:
- call: "fd_install"
syscall: false
args:
- index: 0
type: int
- index: 1
type: "file"
selectors:
- matchBinaries:
- operator: "In"
values:
- "/bin/cat"
matchArgs:
- index: 1
operator: "Equal"
values:
- "/etc/shadow"
matchNamespaces:
- namespace: Mnt
operator: In
values:
- "host_ns"
- namespace: Net
operator: In
values:
- "host_ns"
Here we have a single selector. This CRD will match if:
[
binary == /bin/cat AND
arg1 == /etc/shadow AND
(
MntNs == host AND
NetNs == host)
]
Capabilities filters can be specified under the matchCapabilities
field and
provide filtering of calls based on Linux capabilities in the specific sets.
An example syntax is:
- matchCapabilities:
- type: Effective
operator: In
values:
- "CAP_CHOWN"
- "CAP_NET_RAW"
This will match if: [Effective
capabilities contain CAP_CHOWN
] OR
[Effective
capabilities contain CAP_NET_RAW
]
type
can be: Effective
, Inheritable
, or Permitted
.operator
can be In
or NotIn
values
can be any supported capability. A list of all supported
capabilities can be found in /usr/include/linux/capability.h
.Limitations
values
.type
field can be specified under matchCapabilities
.Namespace changes filter can be specified under the matchNamespaceChanges
field and provide filtering based on calls that are changing Linux namespaces.
This filter can be useful to track execution of code in a new namespace or even
container escapes that change their namespaces.
For instance, if an unprivileged process creates a new user namespace, it gains full privileges within that namespace. This grants the process the ability to perform some privileged operations within the context of this new namespace that would otherwise only be available to privileged root user. As a result, such filter is useful to track namespace creation, which can be abused by untrusted processes.
To keep track of the changes, when a process_exec
happens, the namespaces of
the process are recorded and these are compared with the current namespaces on
the event with a matchNamespaceChanges
filter.
matchNamespaceChanges:
- operator: In
values:
- "Mnt"
The unshare
command, or executing in the host namespace using nsenter
can
be used to test this feature. See a
demonstration example
of this feature.
Capability changes filter can be specified under the matchCapabilityChanges
field and provide filtering based on calls that are changing Linux capabilities.
To keep track of the changes, when a process_exec
happens, the capabilities
of the process are recorded and these are compared with the current
capabilities on the event with a matchCapabilityChanges
filter.
matchCapabilityChanges:
- type: Effective
operator: In
isNamespaceCapability: false
values:
- "CAP_SETUID"
See a demonstration example of this feature.
Actions filters are a list of actions that execute when an appropriate selector
matches. They are defined under matchActions
and currently, the following
action
types are supported:
Sigkill
, Override
, FollowFD
, UnfollowFD
, CopyFD
, Post
,
TrackSock
and UntrackSock
are
executed directly in the kernel BPF code while GetUrl
and DnsLookup
are
happening in userspace after the reception of events.Sigkill
action terminates synchronously the process that made the call that
matches the appropriate selectors from the kernel. In the example below, every
sys_write
system call with a PID not equal to 1 or 0 attempting to write to
/etc/passwd
will be terminated. Indeed when using kubectl exec
, a new
process is spawned in the container PID namespace and is not a child of PID 1.
- call: "sys_write"
syscall: true
args:
- index: 0
type: "fd"
- index: 1
type: "char_buf"
sizeArgIndex: 3
- index: 2
type: "size_t"
selectors:
- matchPIDs:
- operator: NotIn
followForks: true
isNamespacePID: true
values:
- 0
- 1
matchArgs:
- index: 0
operator: "Prefix"
values:
- "/etc/passwd"
matchActions:
- action: Sigkill
Signal
action sends specified signal to current process. The signal number
is specified with argSig
value.
Following example is equivalent to the Sigkill action example above.
The difference is to use the signal action with SIGKILL(9)
signal.
- call: "sys_write"
syscall: true
args:
- index: 0
type: "fd"
- index: 1
type: "char_buf"
sizeArgIndex: 3
- index: 2
type: "size_t"
selectors:
- matchPIDs:
- operator: NotIn
followForks: true
isNamespacePID: true
values:
- 0
- 1
matchArgs:
- index: 0
operator: "Prefix"
values:
- "/etc/passwd"
matchActions:
- action: Signal
argSig: 9
Override
action allows to modify the return value of call. While Sigkill
will terminate the entire process responsible for making the call, Override
will run in place of the original kprobed function and return the value
specified in the argError
field. It’s then up to the code path or the user
space process handling the returned value to whether stop or proceed with the
execution.
For example, you can create a TracingPolicy
that intercepts sys_symlinkat
and will make it return -1
every time the first argument is equal to the
string /etc/passwd
:
kprobes:
- call: "sys_symlinkat"
syscall: true
args:
- index: 0
type: "string"
- index: 1
type: "int"
- index: 2
type: "string"
selectors:
- matchArgs:
- index: 0
operator: "Equal"
values:
- "/etc/passwd\0"
matchActions:
- action: Override
argError: -1
Override
uses the kernel error injection framework and is only available
on kernels compiled with CONFIG_BPF_KPROBE_OVERRIDE
configuration option.
Overriding system calls is the primary use case, but there are other kernel
functions that support error injections too. These functions are annotated
with ALLOW_ERROR_INJECTION()
in the kernel source, and can be identified by
reading the file /sys/kernel/debug/error_injection/list
.
Starting from kernel version 5.7
overriding security_
hooks is also possible.
The FollowFD
action allows to create a mapping using a BPF map between file
descriptors numbers and filenames. It however needs to maintain a state
correctly, see UnfollowFD
and
CopyFD
related actions.
The fd_install
kernel function is called each time a file descriptor must be
installed into the file descriptor table of a process, typically referenced
within system calls like open
or openat
. It is a good place for tracking
file descriptor and filename matching.
Let’s take a look at the following example:
- call: "fd_install"
syscall: false
args:
- index: 0
type: int
- index: 1
type: "file"
selectors:
- matchPIDs:
# [...]
matchArgs:
# [...]
matchActions:
- action: FollowFD
argFd: 0
argName: 1
This action uses the dedicated argFd
and argName
fields to get respectively
the index of the file descriptor argument and the index of the name argument in
the call.
The UnfollowFD
action takes a file descriptor from a system call and deletes
the corresponding entry from the BPF map, where it was put under the FollowFD
action.
Let’s take a look at the following example:
- call: "sys_close"
syscall: true
args:
- index: 0
type: "int"
selectors:
- matchPIDs:
- operator: NotIn
followForks: true
isNamespacePID: true
values:
- 0
- 1
matchActions:
- action: UnfollowFD
argFd: 0
Similar to the FollowFD
action, the index of the file descriptor is described
under argFd
:
matchActions:
- action: UnfollowFD
argFd: 0
In this example, argFD
is 0. So, the argument from the sys_close
system
call at index: 0
will be deleted from the BPF map whenever a sys_close
is
executed.
- index: 0
type: "int"
FollowFD
block,
there should be a matching UnfollowFD
block, otherwise the BPF map will be
broken.The CopyFD
action is specific to duplication of file descriptor use cases.
Similary to FollowFD
, it takes an argFd
and argName
arguments. It can
typically be used tracking the dup
, dup2
or dup3
syscalls.
See the following example for illustration:
- call: "sys_dup2"
syscall: true
args:
- index: 0
type: "fd"
- index: 1
type: "int"
selectors:
- matchPIDs:
# [...]
matchActions:
- action: CopyFD
argFd: 0
argName: 1
- call: "sys_dup3"
syscall: true
args:
- index: 0
type: "fd"
- index: 1
type: "int"
- index: 2
type: "int"
selectors:
- matchPIDs:
# [...]
matchActions:
- action: CopyFD
argFd: 0
argName: 1
The GetUrl
action can be used to perform a remote interaction such as
triggering Thinkst canaries or any system that can be triggered via an URL
request. It uses the argUrl
field to specify the URL to request using GET
method.
matchActions:
- action: GetUrl
argUrl: http://ebpf.io
The DnsLookup
action can be used to perform a remote interaction such as
triggering Thinkst canaries or any system that can be triggered via an DNS
entry request. It uses the argFqdn
field to specify the domain to lookup.
matchActions:
- action: DnsLookup
argFqdn: ebpf.io
The Post
action allows an event to be transmitted to the agent, from
kernelspace to userspace. By default, all TracingPolicy
hook will create an
event with the Post
action except in those situations:
NoPost
action was specified in a matchActions
;This action allows you to specify parameters for the Post
action.
Post
takes the rateLimit
parameter with a time value. This value defaults
to seconds, but post-fixing ’m’ or ‘h’ will cause the value to be interpreted
in minutes or hours. When this parameter is specified for an action, that
action will check if the same action has fired within the time window, with the
same inspected arguments. (Only the first 16 bytes of each inspected argument
is used in the matching. Only supported on kernels v5.3 onwards.)
For example, you can specify a selector to only generate an event every 5 minutes with adding the following action and its paramater:
matchActions:
- action: Post
rateLimit: 5m
Post
takes the stackTrace
parameter, when turned to true
(by default to
false
) it enables dump of the kernel stack trace to the hook point in kprobes
events. For example, the following kprobe hook can be used to retrieve the
kernel stack to kfree_skb_reason
, the function called in the kernel to drop
kernel socket buffers.
kprobes:
- call: kfree_skb_reason
selectors:
- matchActions:
- action: post
stackTrace: true
By default Tetragon does not expose the kernel addresses, you need to enable
the flag --expose-kernel-addresses
to get the addresses along the rest.
Note that the Tetragon agent is using its privilege to read the kernel symbols and their address. Being able to retrieve kernel symbols address can be used to break kernel address space layout randomization (KASLR) so only privileged users should be able to enable this feature and read events containing stack traces.
Once loaded, events created from this policy will contain a new stack_trace
field on the process_kprobe
event with an output similar to:
{
"address": "18446670264256120896",
"offset": "272",
"symbol": "skb_release_head_state"
},
{
"address": "18446670264257081512",
"offset": "88",
"symbol": "ip_protocol_deliver_rcu"
},
{
"address": "18446670264257082152",
"offset": "88",
"symbol": "ip_local_deliver_finish"
},
[...]
The “address” is the kernel function address, “offset” is the offset into the native instruction for the function and “symbol” is the function symbol name.
This output can be enhanced in a more human friendly using the tetra getevents -o compact
command. Indeed, by default, it will print the stack trace along
the compact output of the event similarly to this:
❓ syscall /usr/bin/curl kfree_skb_reason
0xffffbcdee5bf4840: skb_release_head_state+0x110
0xffffbcdee5be3678: __sys_connect_file+0x88
0xffffbcdee5be376c: __sys_connect+0xbc
0xffffbcdee5be37bc: __arm64_sys_connect+0x28
0xffffbcdee4dfcd68: invoke_syscall+0x78
0xffffbcdee4dfcf70: el0_svc_common.constprop.0+0x180
0xffffbcdee4dfcfa4: do_el0_svc+0x30
0xffffbcdee5e861e8: el0_svc+0x48
0xffffbcdee5e881b4: el0t_64_sync_handler+0xa4
0xffffbcdee4de1e38: el0t_64_sync+0x1a4
The printing format is "0x%x: %s+0x%x", address, symbol, offset
.
0x0
, see the above note on
--expose-kernel-addresses
for more info.The NoPost
action can be used to suppress the event to be generated, but at
the same time all its defined actions are performed.
It’s useful when you are not interested in the event itself, just in the action being performed.
Following example override openat syscall for “/etc/passwd” file but does not generate any event about that.
- call: "sys_openat"
return: true
syscall: true
args:
- index: 0
type: int
- index: 1
type: "string"
- index: 2
type: "int"
returnArg:
type: "int"
selectors:
- matchPIDs:
matchArgs:
- index: 1
operator: "Equal"
values:
- "/etc/passwd"
matchActions:
- action: Override
argError: -2
- action: NoPost
The TrackSock
action allows to create a mapping using a BPF map between sockets
and processes. It however needs to maintain a state
correctly, see UntrackSock
related action. TrackSock
works similarly to FollowFD
, specifying the argument with the sock
type using
argSock
instead of specifying the FD argument with argFd
.
It is however more likely that socket tracking will be performed on the return
value of sk_alloc
as described above.
Socket tracking is only available on kernel >=5.3.
The UntrackSock
action takes a struct sock pointer from a function call and deletes
the corresponding entry from the BPF map, where it was put under the TrackSock
action.
Let’s take a look at the following example:
- call: "__sk_free"
syscall: false
args:
- index: 0
type: sock
selectors:
- matchActions:
- action: UntrackSock
argSock: 0
Similar to the TrackSock
action, the index of the sock is described under argSock
:
- matchActions:
- action: UntrackSock
argSock: 0
In this example, argSock
is 0. So, the argument from the __sk_free
function
call at index: 0
will be deleted from the BPF map whenever a __sk_free
is
executed.
- index: 0
type: "sock"
TrackSock
block,
there should be a matching UntrackSock
block, otherwise the BPF map will be
broken.Socket tracking is only available on kernel >=5.3.
The NotifyKiller
action notifies the killer program to kill or override a syscall.
It’s meant to be used on systems with kernel that lacks multi kprobe feature, that allows to attach many kprobes quickly). To workaround that the killer sensor uses the raw syscall tracepoint and attaches simple program to syscalls that we need to kill or override.
The specs needs to have killer
program definition, that instructs tetragon to load
the killer
program and attach it to specified syscalls.
spec:
killers:
- syscalls:
- "list:dups"
The syscalls expects list of syscalls or list:XXX
pointer to list.
Note that currently only single killer definition is allowed.
The NotifyKiller
action takes 2 arguments.
matchActions:
- action: "NotifyKiller"
argError: -1
argSig: 9
If specified the argError will be passed to bpf_override_return
helper to override the syscall return value.
If specified the argSig will be passed to bpf_send_signal
helper to override the syscall return value.
The following is spec for killing /usr/bin/bash
program whenever it calls sys_dup
or sys_dup2
syscalls.
spec:
lists:
- name: "dups"
type: "syscalls"
values:
- "sys_dup"
- "sys_dup2"
killers:
- syscalls:
- "list:dups"
tracepoints:
- subsystem: "raw_syscalls"
event: "sys_enter"
args:
- index: 4
type: "uint64"
selectors:
- matchArgs:
- index: 0
operator: "InMap"
values:
- "list:dups"
matchBinaries:
- operator: "In"
values:
- "/usr/bin/bash"
matchActions:
- action: "NotifyKiller"
argSig: 9
Note as mentioned above the NotifyKiller
with killer program is meant to be used only on kernel versions
with no support for fast attach of multiple kprobes (kprobe_multi
link).
With kprobe_multi
link support the above example can be easily replaced with:
spec:
lists:
- name: "syscalls"
type: "syscalls"
values:
- "sys_dup"
- "sys_dup2"
kprobes:
- call: "list:syscalls"
selectors:
- matchBinaries:
- operator: "In"
values:
- "/usr/bin/bash"
matchActions:
- action: "Sigkill"
The selector
semantics of the CiliumTracingPolicy
follows the standard
Kubernetes semantics and the principles that are used by Cilium to create a
unified policy definition.
To explain deeper the structure and the logic behind it, let’s consider first the following example:
selectors:
- matchPIDs:
- operator: In
followForks: true
values:
- pid1
- pid2
- pid3
matchArgs:
- index: 0
operator: "Equal"
values:
- fdString1
In the YAML above matchPIDs
and matchArgs
are logically AND
together
giving the expression:
(pid in {pid1, pid2, pid3} AND arg0=fdstring1)
When multiple values are given, we apply the OR
operation between them. In
case of having multiple values under the matchPIDs
selector, if any value
matches with the given pid from pid1
, pid2
or pid3
then we accept the
event:
pid==pid1 OR pid==pid2 OR pid==pid3
As an example, we can filter for sys_read()
syscalls that were not part of
the container initialization and the main pod process and tried to read from
the /etc/passwd
file by using:
selectors:
- matchPIDs:
- operator: NotIn
followForks: true
values:
- 0
- 1
matchArgs:
- index: 0
operator: "Equal"
values:
- "/etc/passwd"
Similarly, we can use multiple values under the matchArgs
selector:
(pid in {pid1, pid2, pid3} AND arg0={fdstring1, fdstring2})
If any value matches with fdstring1
or fdstring2
, specifically
(string==fdstring1 OR string==fdstring2)
then we accept the event.
For example, we can monitor sys_read()
syscalls accessing both the
/etc/passwd
or the /etc/shadow
files:
selectors:
- matchPIDs:
- operator: NotIn
followForks: true
values:
- 0
- 1
matchArgs:
- index: 0
operator: "Equal"
values:
- "/etc/passwd"
- "/etc/shadow"
When multiple operators are supported under matchPIDs
or matchArgs
, they
are logically AND
together. In case if we have multiple operators under
matchPIDs
:
selectors:
- matchPIDs:
- operator: In
followForks: true
values:
- pid1
- operator: NotIn
followForks: true
values:
- pid2
then we would build the following expression on the BPF side:
(pid == 0[following forks]) && (pid != 1[following forks])
In case of having multiple matchArgs
:
selectors:
- matchPIDs:
- operator: In
followForks: true
values:
- pid1
- pid2
- pid3
matchArgs:
- index: 0
operator: "Equal"
values:
- 1
- index: 2
operator: "lt"
values:
- 500
Then we would build the following expression on the BPF side
(pid in {pid1, pid2, pid3} AND arg0=1 AND arg2 < 500)
There are different types supported for each operator. In case of matchArgs
:
The operator types Equal
and NotEqual
are used to test whether the certain
argument of a system call is equal to the defined value in the CR.
For example, the following YAML snippet matches if the argument at index 0 is
equal to /etc/passwd
:
matchArgs:
- index: 0
operator: "Equal"
values:
- "/etc/passwd"
Both Equal
and NotEqual
are set operations. This means if multiple values
are specified, they are OR
d together in case of Equal
, and AND
d together
in case of NotEqual
.
For example, in case of Equal
the following YAML snippet matches if the
argument at index 0 is in the set of {arg0, arg1, arg2}
.
matchArgs:
- index: 0
operator: "Equal"
values:
- "arg0"
- "arg1"
- "arg2"
The above would be executed in the kernel as
arg == arg0 OR arg == arg1 OR arg == arg2
In case of NotEqual
the following YAML snippet matches if the argument at
index 0 is not in the set of {arg0, arg1}
.
matchArgs:
- index: 0
operator: "NotEqual"
values:
- "arg0"
- "arg1"
The above would be executed in the kernel as
arg != arg0 AND arg != arg1
The operator type Mask
performs and bitwise operation on the argument value
and defined values. The argument type needs to be one of the value types.
For example in following YAML snippet we match second argument for bits 1 and 9 (0x200 value). We could use single value 0x201 as well.
matchArgs:
- index: 2
operator: "Mask"
values:
- 1
- 0x200
The above would be executed in the kernel as
arg & 1 OR arg & 0x200
The value can be specified as hexadecimal (with 0x prefix) octal (with 0 prefix) or decimal value (no prefix).
The operator Prefix
checks if the certain argument starts with the defined value,
while the operator Postfix
compares if the argument matches to the defined value
as trailing.
The operators relating to ports, addresses and protocol are used with sock or skb
types. Port operators can accept a range of ports specified as min:max
as well
as lists of individual ports. Address operators can accept IPv4/6 CIDR ranges as well
as lists of individual addresses.
The Protocol
operator can accept integer values to match against, or the equivalent
IPPROTO_ enumeration. For example, UDP can be specified as either IPPROTO_UDP
or 17;
TCP can be specified as either IPPROTO_TCP
or 6.
The Family
operator can accept integer values to match against or the equivalent
AF_ enumeration. For example, IPv4 can be specified as either AF_INET
or 2; IPv6
can be specified as either AF_INET6
or 10.
The State
operator can accept integer values to match against or the equivalent
TCP_ enumeration. For example, an established socket can be matched with
TCP_ESTABLISHED
or 1; a closed socket with TCP_CLOSE
or 7.
In case of matchPIDs
:
The operator types In
and NotIn
are used to test whether the pid of a
system call is found in the provided values
list in the CR. Both In
and
NotIn
are set operations, which means if multiple values are specified they
are OR
d together in case of In
and AND
d together in case of NotIn
.
For example, in case of In
the following YAML snippet matches if the pid of a
certain system call is being part of the list of {0, 1}
:
- matchPIDs:
- operator: In
followForks: true
isNamespacePID: true
values:
- 0
- 1
The above would be executed in the kernel as
pid == 0 OR pid == 1
In case of NotIn
the following YAML snippet matches if the pid of a certain
system call is not being part of the list of {0, 1}
:
- matchPIDs:
- operator: NotIn
followForks: true
isNamespacePID: true
values:
- 0
- 1
The above would be executed in the kernel as
pid != 0 AND pid != 1
In case of matchBinaries
:
The In
operator type is used to test whether a binary name of a system call
is found in the provided values
list. For example, the following YAML snippet
matches if the binary name of a certain system call is being part of the list
of {binary0, binary1, binary2}
:
- matchBinaries:
- operator: "In"
values:
- "binary0"
- "binary1"
- "binary2"
When multiple selectors are configured they are logically OR
d together.
selectors:
- matchPIDs:
- operator: In
followForks: true
values:
- pid1
- pid2
- pid3
matchArgs:
- index: 0
operator: "Equal"
values:
- 1
- index: 2
operator: "lt"
values:
- 500
- matchPIDs:
- operator: In
followForks: true
values:
- pid1
- pid2
- pid3
matchArgs:
- index: 0
operator: "Equal"
values:
- 2
The above would be executed in kernel as:
(pid in {pid1, pid2, pid3} AND arg0=1 AND arg2 < 500) OR
(pid in {pid1, pid2, pid3} AND arg0=2)
Those limitations might be outdated, see issue #709.
Because BPF must be bounded we have to place limits on how many selectors can exist.
Tetragon is configured via TracingPolicies. Broadly speaking, TracingPolicies define what situations Tetragon should react to and how. The what can be, for example, specific system calls with specific argument values. The how defines what action the Tetragon agent should perform when the specified situation occurs. The most common action is generating an event, but there are others (e.g., returning an error without executing a function, or killing the corresponding process).
Here we discuss how to apply tracing policies only on a subset of pods running on the system via two mechanisms: namespaced policies, and pod-label filters. Tetragon implements both mechanisms in-kernel via eBPF. This is important for both observability and enforcement use-cases. For observability, copying only the relevant events from kernel- to user-space reduces overhead. For enforcement, performing the enforcement action in the kernel avoids the race-condition of doing it in user-space. For example, let us consider the case where we want to block an application from performing a system call. Performing the filtering in-kernel means that the application will never finish executing the system call, which is not possible if enforcement happens in user-space (after the fact).
To ensure that namespaced tracing policies are always correctly applied, Tetragon needs to perform actions before containers start executing. Tetragon supports this via OCI runtime hooks. If such hooks are not added, Tetragon will apply policies in a best-effort manner using information from the k8s API server.
For namespace filtering we use TracingPolicyNamespaced
which has the same contents as a
TracingPolicy
, but it is defined in a specific namespace and it is only applied to pods of that
namespace.
For pod label filters, we use the PodSelector
field of tracing policies to select the pods that
the policy is applied to.
For this demo, we use containerd and configure appropriate run-time hooks using minikube.
First, let us start minikube, build and load images, and install Tetragon and OCI hooks:
minikube start --container-runtime=containerd
./contrib/rthooks/minikube-containerd-install-hook.sh
make image image-operator
minikube image load --daemon=true cilium/tetragon:latest cilium/tetragon-operator:latest
minikube ssh -- sudo mount bpffs -t bpf /sys/fs/bpf
helm install --namespace kube-system \
--set tetragonOperator.image.override=cilium/tetragon-operator:latest \
--set tetragon.image.override=cilium/tetragon:latest \
--set tetragon.enablePolicyFilter="true" \
--set tetragon.grpc.address="unix:///var/run/cilium/tetragon/tetragon.sock" \
tetragon ./install/kubernetes
Once the tetragon pod is up and running, we can get its name.
tetragon_pod=$(kubectl -n kube-system get pods -l app.kubernetes.io/name=tetragon -o custom-columns=NAME:.metadata.name --no-headers)
Next, we check the tetragon-operator logs and tetragon agent logs to ensure that everything is in order.
First, we check if the operator installed the TracingPolicyNamespaced CRD.
kubectl -n kube-system logs -c tetragon-operator $tetragon_pod
The expected output is:
level=info msg="Tetragon Operator: " subsys=tetragon-operator
level=info msg="CRD (CustomResourceDefinition) is installed and up-to-date" name=TracingPolicy/v1alpha1 subsys=k8s
level=info msg="Creating CRD (CustomResourceDefinition)..." name=TracingPolicyNamespaced/v1alpha1 subsys=k8s
level=info msg="CRD (CustomResourceDefinition) is installed and up-to-date" name=TracingPolicyNamespaced/v1alpha1 subsys=k8s
level=info msg="Initialization complete" subsys=tetragon-operator
Next, we check that policyfilter (the low level mechanism that implements the desired functionality) is indeed enabled.
kubectl -n kube-system logs -c tetragon $tetragon_pod
The output should include:
level=info msg="Enabling policy filtering"
For illustration purposes, we will use the lseek system call with an invalid argument. Specifically a file descriptor (the first argument) of -1. Normally, this operation would return a “Bad file descriptor error”.
Let us start a pod in the default namespace:
kubectl -n default run test --image=python -it --rm --restart=Never -- python
Above command will result in the following python shell:
If you don't see a command prompt, try pressing enter.
>>>
There is no policy installed so attempting to do the lseek operation will just return an error. So using the python shell we can execute an lseek and see the returned error.
>>> import os
>>> os.lseek(-1,0,0)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OSError: [Errno 9] Bad file descriptor
>>>
In another terminal, we install a policy in the default namespace:
cat << EOF | kubectl apply -n default -f -
apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
name: "lseek-namespaced"
spec:
kprobes:
- call: "sys_lseek"
syscall: true
args:
- index: 0
type: "int"
selectors:
- matchArgs:
- index: 0
operator: "Equal"
values:
- "-1"
matchActions:
- action: Sigkill
EOF
The above tracing policy will kill the process that performs an lseek system call with a file
descriptor of -1
. Note that we use a SigKill
action only for illustration purposes because it’s
easier to observe its effects.
Then, attempting the lseek operation on the previous terminal, will result in the process getting killed:
>>> os.lseek(-1, 0, 0)
pod "test" deleted
pod default/test terminated (Error)
The same is true for a newly started container:
kubectl -n default run test --image=python -it --rm --restart=Never -- python
If you don't see a command prompt, try pressing enter.
>>> import os
>>> os.lseek(-1, 0, 0)
pod "test" deleted
pod default/test terminated (Error)
Doing the same on another namespace:
kubectl create namespace test
kubectl -n test run test --image=python -it --rm --restart=Never -- python
Will not kill the process and result in an error:
If you don't see a command prompt, try pressing enter.
>>> import os
>>> os.lseek(-1, 0, 0)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OSError: [Errno 9] Bad file descriptor
Let’s install a tracing policy with a pod label filter.
cat << EOF | kubectl apply -f -
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
name: "lseek-podfilter"
spec:
podSelector:
matchLabels:
app: "lseek-test"
kprobes:
- call: "sys_lseek"
syscall: true
args:
- index: 0
type: "int"
selectors:
- matchArgs:
- index: 0
operator: "Equal"
values:
- "-1"
matchActions:
- action: Sigkill
EOF
Pods without the label will not be affected:
kubectl run test --image=python -it --rm --restart=Never -- python
If you don't see a command prompt, try pressing enter.
>>> import os
>>> os.lseek(-1, 0, 0)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OSError: [Errno 9] Bad file descriptor
>>>
But pods with the label will:
kubectl run test --labels "app=lseek-test" --image=python -it --rm --restart=Never -- python
If you don't see a command prompt, try pressing enter.
>>> import os
>>> os.lseek(-1, 0, 0)
pod "test" deleted
pod default/test terminated (Error)
Tetragon allows enforcing events in the kernel inline with the operation itself. This document describes the types of enforcement provided by Tetragon and concerns policy implementors must be aware of.
There are two ways that Tetragon performs enforcement: overriding the return value of a function and
sending a signal (e.g., SIGKILL
) to the process.
Override the return value of a call means that the function will never be executed and, instead, a value (typically an error) will be returned to the caller. Generally speaking, only system calls and security check functions allow to change their return value in this manner. Details about how users can configure tracing policies to override the return value can be found in the Override action documentation.
Another type of enforcement is signals. For example, users can write a TracingPolicy (details can be
found in the Signal action
documentation) that sends a SIGKILL
to a process matching certain criteria and thus terminate it.
In contrast with overriding the return value, sending a SIGKILL
signal does not always stop the
operation being performed by the process that triggered the operation. For example, a SIGKILL
sent
in a write()
system call does not guarantee that the data will not be written to the file.
However, it does ensure that the process is terminated synchronously (and any threads will be
stopped). In some cases it may be sufficient to ensure the process is stopped and the process does
not handle the return of the call. To ensure the operation is not completed, though, the Signal
action should be combined with the Override
action.
Audit BPF program loads and BPFFS interactions
Understanding BPF programs loaded in a cluster and interactions between applications and programs can identify bugs and malicious or unexpected BPF activity.
jq 'select(.process_kprobe != null) | select(.process_kprobe.function_name | test("bpf_check")) | "\(.time) \(.process_kprobe.process.binary) \(.process_kprobe.process.arguments) programType:\(.process_kprobe.args[0].bpf_attr_arg.ProgType) programInsn:\(.process_kprobe.args[0].bpf_attr_arg.InsnCnt)"
"2023-11-01T02:56:54.926403604Z /usr/bin/bpftool prog list programType:BPF_PROG_TYPE_SOCKET_FILTER programInsn:2"
Audit loading of kernel modules
Understanding exactly what kernel modules are running in the cluster is crucial to understand attack surface and any malicious actors loading unexpected modules.
jq 'select(.process_kprobe != null) | select(.process_kprobe.function_name | test("security_kernel_module_request")) | "\(.time) \(.process_kprobe.process.binary) \(.process_kprobe.process.arguments) module:\(.process_kprobe.args[0].string_arg)"'
"2023-11-01T04:11:38.390880528Z /sbin/iptables -A OUTPUT -m cgroup --cgroup 1 -j LOG module:ipt_LOG"
Monitor loading of libraries
Understanding the exact versions of shared libraries that binaries load and use is crucial to understand use of vulnerable or deprecated library versions or attacks such as shared library hijacking.
jq 'select(.process_loader != null) | "\(.time) \(.process_loader.process.pod.namespace) \(.process_loader.process.binary) \(.process_loader.process.arguments) \(.process_loader.path)"
"2023-10-31T19:42:33.065233159Z default/xwing /usr/bin/curl https://ebpf.io /usr/lib/x86_64-linux-gnu/libssl.so.3"
Monitor execution of a binary in the /tmp directory.
Preventing execution of executables in /tmp
is a common best-practice as several canned exploits rely on writing and then executing malicious binaries in the /tmp
directory. A common best-practice to enforce this is to mount the /tmp
filesystem with the noexec
flag. This observability policy is used to monitor for violations of this best practice.
No policy needs to be loaded, standard process execution observability is sufficient.
jq 'select(.process_exec != null) | select(.process_exec.process.binary | contains("/tmp/")) | "\(.time) \(.process_exec.process.pod.namespace) \(.process_exec.process.pod.name) \(.process_exec.process.binary) \(.process_exec.process.arguments)"'
"2023-10-31T18:44:22.777962637Z default/xwing /tmp/nc ebpf.io 1234"
Monitor sudo invocations
sudo is used to run executables with particular privileges. Creating a audit log of sudo invocations is a common best-practice.
No policy needs to be loaded, standard process execution observability is sufficient.
jq 'select(.process_exec != null) | select(.process_exec.process.binary | contains("sudo")) | "\(.time) \(.process_exec.process.pod.namespace) \(.process_exec.process.binary) \(.process_exec.process.arguments)"'
"2023-10-31T19:03:35.273111185Z null /usr/bin/sudo -i"
Monitor execution of SUID “Set User ID” binaries.
The “Set User Identity” and “Set Group Identity” are permission flags. When set on a binary file, the binary will execute with the permissions of the owner or group associated with the executable file, rather than the user executing it. Usually it is used to run programs with elevated privileges to perform specific tasks.
Detecting the execution of setuid
and setgid
binaries is a common
best-practice as attackers may abuse such binaries, or even create them
during an exploit for subsequent execution.
Run Tetragon with enable-process-creds
setting set to enable visibility
into process_credentials and binary_properties.
kubectl edit cm -n kube-system tetragon-config
# Change "enable-process-cred" from "false" to "true", then save and exit
# Restart Tetragon daemonset
kubectl rollout restart -n kube-system ds/tetragon
docker run --name tetragon --rm -d \
--pid=host --cgroupns=host --privileged \
-v /sys/kernel:/sys/kernel \
-v /var/log/tetragon:/var/log/tetragon \
quay.io/cilium/tetragon:v1.0.0 \
/usr/bin/tetragon --enable-process-cred \
--export-filename /var/log/tetragon/tetragon.log
# Write to the drop-in file /etc/tetragon/tetragon.conf.d/enable-process-cred true
# Run the following as a privileged user then restart tetragon service
echo "true" > /etc/tetragon/tetragon.conf.d/enable-process-cred
systemctl restart tetragon
No policy needs to be loaded, standard process execution observability is sufficient.
jq 'select(.process_exec != null) | select(.process_exec.process.binary_properties != null) | select(.process_exec.process.binary_properties.setuid != null or .process_exec.process.binary_properties.setgid != null) | "\(.time) \(.process_exec.process.pod.namespace) \(.process_exec.process.pod.name) \(.process_exec.process.binary) \(.process_exec.process.arguments) uid=\(.process_exec.process.process_credentials.uid) euid=\(.process_exec.process.process_credentials.euid) gid=\(.process_exec.process.process_credentials.gid) egid=\(.process_exec.process.process_credentials.egid)"'
"2023-11-13T08:20:43.615672640Z null null /usr/bin/sudo id uid=1000 euid=0 gid=1000 egid=1000"
"2023-11-13T08:20:45.591560940Z null null /usr/bin/wall hello uid=1000 euid=1000 gid=1000 egid=5"
"2023-11-13T08:20:47.036760043Z null null /usr/bin/su - uid=1000 euid=0 gid=1000 egid=1000"
Monitor execution of the setuid() system calls family.
The setuid() and setgid() system calls family allow to change the effective user ID and group ID of the calling process.
Detecting setuid() and setgid() calls that set the user ID or group ID to root is a common best-practice to identify when privileges are raised.
The privileges-setuid-root.yaml is a catch all to the various interfaces of setuid()
and setgid()
to root.
jq 'select(.process_kprobe != null) | select(.process_kprobe.policy_name | test("privileges-setuid-root")) | "\(.time) \(.process_kprobe.process.pod.namespace) \(.process_kprobe.process.pod.name) \(.process_kprobe.process.binary) \(.process_kprobe.process.arguments) \(.process_kprobe.function_name) \(.process_kprobe.args)"'
"2023-11-12T22:17:40.754680857Z null null /usr/bin/sudo id __sys_setresgid [{\"int_arg\":-1},{\"int_arg\":0},{\"int_arg\":-1}]"
"2023-11-12T22:17:40.754730285Z null null /usr/bin/sudo id __sys_setresuid [{\"int_arg\":-1},{\"int_arg\":0},{\"int_arg\":-1}]"
"2023-11-12T22:17:40.758125709Z null null /usr/bin/sudo id __sys_setgid [{\"int_arg\":0}]"
"2023-11-12T22:17:40.758747395Z null null /usr/bin/sudo id __sys_setresuid [{\"int_arg\":0},{\"int_arg\":0},{\"int_arg\":0}]"
Monitor sessions to SSHd
It is best practice to audit remote connections into a shell server.
jq 'select(.process_kprobe != null) | select(.process_kprobe.function_name | test("tcp_close")) | "\(.time) \(.process_kprobe.process.binary) \(.process_kprobe.process.arguments) \(.process_kprobe.args[0].sock_arg.family) \(.process_kprobe.args[0].sock_arg.type) \(.process_kprobe.args[0].sock_arg.protocol) \(.process_kprobe.args[0].sock_arg.saddr):\(.process_kprobe.args[0].sock_arg.sport)"'
"2023-11-01T04:51:20.109146920Z /usr/sbin/sshd default/xwing AF_INET SOCK_STREAM IPPROTO_TCP 127.0.0.1:22"
Monitor all cluster egress connections
Connections made outside a Kubernetes cluster can be audited to provide insights into any unexpected or malicious reverse shells.
PODCIDR=`kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'`
SERVICECIDR=$(gcloud container clusters describe ${NAME} --zone ${ZONE} | awk '/servicesIpv4CidrBlock/ { print $2; }')
SERVICECIDR=$(kubectl describe pod -n kube-system kube-apiserver-kind-control-plane | awk -F= '/--service-cluster-ip-range/ {print $2; }')
jq 'select(.process_kprobe != null) | select(.process_kprobe.function_name | test("tcp_connect")) | "\(.time) \(.process_kprobe.process.binary) \(.process_kprobe.process.arguments) \(.process_kprobe.args[0].sock_arg.saddr):\(.process_kprobe.args[0].sock_arg.sport) -> \(.process_kprobe.args[0].sock_arg.daddr):\(.process_kprobe.args[0].sock_arg.dport)"'
"2023-11-01T05:25:14.837745007Z /usr/bin/curl http://ebpf.io 10.168.0.45:48272 -> 104.198.14.52:80"
By default, Tetragon monitors process lifecycle, learn more about that in the dedicated use cases.
For more advanced use cases, Tetragon can observe tracepoints and arbitrary
kernel calls via kprobes. For that, Tetragon must be extended and configured
with custom resources objects named TracingPolicy.
It can then generates process_tracepoint
and process_kprobes
events.
Tetragon observes process creation and termination with default configuration
and generates process_exec
and process_exit
events:
process_exec
events include useful information about the execution of
binaries and related process information. This includes the binary image that
was executed, command-line arguments, the UID context the process was
executed with, the process parent information, the capabilities that a
process had while executed, the process start time, the Kubernetes Pod,
labels and more.process_exit
events, as the process_exec
event shows how and when a
process started, indicate how and when a process is removed. The information
in the event includes the binary image that was executed, command-line
arguments, the UID context the process was executed with, process parent
information, process start time, the status codes and signals on process
exit. Understanding why a process exited and with what status code helps
understand the specifics of that exit.Both these events include Linux-level metadata (UID, parents, capabilities, start time, etc.) but also Kubernetes-level metadata (Kubernetes namespace, labels, name, etc.). This data make the connection between node-level concepts, the processes, and Kubernetes or container environments.
These events enable a full lifecycle view into a process that can aid an incident investigation, for example, we can determine if a suspicious process is still running in a particular environment. For concrete examples of such events, see the next use case on process execution.
process_exec
and process_exit
This first use case is monitoring process execution, which can be observed with
the Tetragon process_exec
and process_exit
JSON events.
These events contain the full lifecycle of processes, from fork/exec to
exit, including metadata such as:
As a first step, let’s start monitoring the events from the xwing
pod:
kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | tetra getevents -o compact --namespace default --pod xwing
Then in another terminal, let’s kubectl exec
into the xwing
pod and execute
some example commands:
kubectl exec -it xwing -- /bin/bash
whoami
If you observe, the output in the first terminal should be:
🚀 process default/xwing /bin/bash
🚀 process default/xwing /usr/bin/whoami
💥 exit default/xwing /usr/bin/whoami 0
Here you can see the binary names along with its arguments, the pod info, and return codes in a compact one-line view of the events.
For more details use the raw JSON events to get detailed information, you can stop
the Tetragon CLI by Crl-C
and parse the tetragon.log
file by executing:
kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | jq 'select(.process_exec.process.pod.name=="xwing" or .process_exit.process.pod.name=="xwing")'
Example process_exec
and process_exit
events can be:
{
"process_exec": {
"process": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjExNDI4NjE1NjM2OTAxOjUxNTgz",
"pid": 51583,
"uid": 0,
"cwd": "/",
"binary": "/usr/bin/whoami",
"arguments": "--version",
"flags": "execve rootcwd clone",
"start_time": "2022-05-11T12:54:45.615Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://1fb931d2f6e5e4cfdbaf30fdb8e2fdd81320bdb3047ded50120a4f82838209ce",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2022-05-11T10:07:33Z",
"pid": 50
}
},
"docker": "1fb931d2f6e5e4cfdbaf30fdb8e2fdd",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjkwNzkyMjU2MjMyNjk6NDM4NzI=",
"refcnt": 1
},
"parent": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjkwNzkyMjU2MjMyNjk6NDM4NzI=",
"pid": 43872,
"uid": 0,
"cwd": "/",
"binary": "/bin/bash",
"flags": "execve rootcwd clone",
"start_time": "2022-05-11T12:15:36.225Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://1fb931d2f6e5e4cfdbaf30fdb8e2fdd81320bdb3047ded50120a4f82838209ce",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2022-05-11T10:07:33Z",
"pid": 43
}
},
"docker": "1fb931d2f6e5e4cfdbaf30fdb8e2fdd",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjkwNzkxODU5NTMzOTk6NDM4NjE=",
"refcnt": 1
}
},
"node_name": "kind-control-plane",
"time": "2022-05-11T12:54:45.615Z"
}
{
"process_exit": {
"process": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjExNDI4NjE1NjM2OTAxOjUxNTgz",
"pid": 51583,
"uid": 0,
"cwd": "/",
"binary": "/usr/bin/whoami",
"arguments": "--version",
"flags": "execve rootcwd clone",
"start_time": "2022-05-11T12:54:45.615Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://1fb931d2f6e5e4cfdbaf30fdb8e2fdd81320bdb3047ded50120a4f82838209ce",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2022-05-11T10:07:33Z",
"pid": 50
}
},
"docker": "1fb931d2f6e5e4cfdbaf30fdb8e2fdd",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjkwNzkyMjU2MjMyNjk6NDM4NzI="
},
"parent": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjkwNzkyMjU2MjMyNjk6NDM4NzI=",
"pid": 43872,
"uid": 0,
"cwd": "/",
"binary": "/bin/bash",
"flags": "execve rootcwd clone",
"start_time": "2022-05-11T12:15:36.225Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://1fb931d2f6e5e4cfdbaf30fdb8e2fdd81320bdb3047ded50120a4f82838209ce",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2022-05-11T10:07:33Z",
"pid": 43
}
},
"docker": "1fb931d2f6e5e4cfdbaf30fdb8e2fdd",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjkwNzkxODU5NTMzOTk6NDM4NjE="
}
},
"node_name": "kind-control-plane",
"time": "2022-05-11T12:54:45.616Z"
}
Advanced process execution can be performed by using Tracing Policies to monitor the execve system call path.
If we want to monitor execution of Executable and Linkable Format (ELF) or flat binaries before they are actually executed. Then the process-exec-elf-begin tracing policy is a good first choice.
The process-exec-elf-begin tracing policy, will not report the different binary format handlers or scripts being executed, but will report the final ELF or flat binary, like the shebang handler.
To report those another tracing policy can be used.
Before going forward, verify that all pods are up and running, ensure you deploy our Demo Application to explore the Security Observability Events:
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.15.0-pre.1/examples/minikube/http-sw-app.yaml
It might take several seconds for some pods until they satisfy all the dependencies:
kubectl get pods -A
The output should be similar to:
NAMESPACE NAME READY STATUS RESTARTS AGE
default deathstar-54bb8475cc-6c6lc 1/1 Running 0 2m54s
default deathstar-54bb8475cc-zmfkr 1/1 Running 0 2m54s
default tiefighter 1/1 Running 0 2m54s
default xwing 1/1 Running 0 2m54s
kube-system tetragon-sdwv6 2/2 Running 0 27m
Let’s apply the process-exec-elf-begin Tracing Policy.
kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-exec/process-exec-elf-begin.yaml
Then start monitoring events with the tetra
CLI:
kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents
In another terminal, kubectl exec
into the xwing Pod:
kubectl exec -it xwing -- /bin/bash
And execute some commands:
id
The tetra
CLI will generate the following ProcessKprobe events:
{
"process_kprobe": {
"process": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE2NjY0MDI4MTA4MzcxOjM2NDk5",
"pid": 36499,
"uid": 0,
"cwd": "/",
"binary": "/bin/bash",
"flags": "execve",
"start_time": "2023-08-02T11:58:53.618461573Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://775beeb1a25a95e10dc149d6eb166bf45dd5e6039e8af3b64e8fb4d29669f349",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-08-02T07:24:54Z",
"pid": 13
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
}
},
"docker": "775beeb1a25a95e10dc149d6eb166bf",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE2NjYyNzg3ODI1MTQ4OjM2NDkz",
"refcnt": 1,
"tid": 36499
},
"parent": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE2NjYyNzg3ODI1MTQ4OjM2NDkz",
"pid": 36493,
"uid": 0,
"cwd": "/",
"binary": "/bin/bash",
"flags": "execve rootcwd clone",
"start_time": "2023-08-02T11:58:52.378178852Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://775beeb1a25a95e10dc149d6eb166bf45dd5e6039e8af3b64e8fb4d29669f349",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-08-02T07:24:54Z",
"pid": 13
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
}
},
"docker": "775beeb1a25a95e10dc149d6eb166bf",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE2NjYyNzE2OTU0MjgzOjM2NDg0",
"tid": 36493
},
"function_name": "security_bprm_creds_from_file",
"args": [
{
"file_arg": {
"path": "/bin/busybox"
}
}
],
"action": "KPROBE_ACTION_POST"
},
"node_name": "kind-control-plane",
"time": "2023-08-02T11:58:53.624096751Z"
}
In addition to the Kubernetes Identity and process metadata, ProcessKprobe events contain the binary being executed. In the above case they are:
function_name
: where we are hooking into the kernel to read the binary that is being executed.file_arg
: that includes the path
being executed, and here it is /bin/busybox
that is the real
binary being executed, since on the xwing
pod the container is running busybox.
The binary /usr/bin/id -> /bin/busybox
points to busybox.To disable the process-exec-elf-being Tracing Policy run:
kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-exec/process-exec-elf-begin.yaml
Tetragon also provides the ability to check process capabilities and kernel namespaces access.
This information would help us determine which process or Kubernetes pod has started or gained access to privileges or host namespaces that it should not have. This would help us answer questions like:
Which Kubernetes pods are running with
CAP_SYS_ADMIN
in my cluster?
Which Kubernetes pods have host network or pid namespace access in my cluster?
As a first step let’s enable visibility to capability and namespace changes via
the configmap by setting enable-process-cred
and enable-process-ns
from
false
to true
:
kubectl edit cm -n kube-system tetragon-config
# change "enable-process-cred" from "false" to "true"
# change "enable-process-ns" from "false" to "true"
# then save and exit
Restart the Tetragon daemonset:
kubectl rollout restart -n kube-system ds/tetragon
As a second step, let’s start monitoring the Security Observability events from the privileged test-pod
workload:
kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | tetra getevents --namespace default --pod test-pod
In another terminal let’s apply the privileged PodSpec:
kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/testdata/specs/testpod.yaml
If you observe the output in the first terminal, you can see the container start with CAP_SYS_ADMIN
:
🚀 process default/test-pod /bin/sleep 365d 🛑 CAP_SYS_ADMIN
🚀 process default/test-pod /usr/bin/jq -r .bundle 🛑 CAP_SYS_ADMIN
🚀 process default/test-pod /usr/bin/cp /kind/product_name /kind/product_uuid /run/containerd/io.containerd.runtime.v2.task/k8s.io/7c7e513cd4d506417bc9d97dd9af670d94d9e84161c8c8 fdc9fa3a678289a59/rootfs/ 🛑 CAP_SYS_ADMIN
This page shows how you can create a tracing policy to monitor filename access. For general information about tracing policies, see the tracing policy page.
There are two aspects of the tracing policy: (i) what hooks you can use to monitor specific types of access, and (ii) how you can filter at the kernel level for only specific events.
There are different ways applications can access and modify files, and for this tracing policy we focus in three different types.
The first is read and write accesses, which the most common way that files are accessed by
applications. Applications can perform this type of accesses with a variety of different system
calls: read
and write
, optimized system calls such as copy_file_range
and sendfile
, as well
as asynchronous I/O system call families such as the ones provided by aio
and io_uring
. Instead
of monitoring every system call, we opt to hook into the security_file_permission
hook, which is a
common execution point for all the above system calls.
Applications can also access files by mapping them directly into their virtual address space. Since
it is difficult to caught the accesses themselves in this case, our policy will instead monitor the
point when the files are mapped into the application’s virtual memory. To do so, we use the
security_mmap_file
hook.
Lastly, there is a family of system calls (e.g,. truncate
) that allow to indirectly modify the
contents of the file by changing its size. To catch these types of accesses we will hook into
security_path_truncate
.
Using the hooks above, you can monitor all accesses in the system. This will create a large number of events, however, and it is frequently the case that you are only interested in a specific subset those events. It is possible to filter the events after their generation, but this induces unnecessary overhead. Tetragon, using BPF, allows filtering these events directly in the kernel.
For example, the following snippet shows how you can limit the events from the
security_file_permission
hook only for the /etc/passwd
file. For this, you need to specify the
arguments of the function that you hooking into, as well as their type.
- call: "security_file_permission"
syscall: false
args:
- index: 0
type: "file" # (struct file *) used for getting the path
- index: 1
type: "int" # 0x04 is MAY_READ, 0x02 is MAY_WRITE
selectors:
- matchArgs:
- index: 0
operator: "Equal"
values:
- "/etc/passwd" # filter by filename (/etc/passwd)
- index: 1
operator: "Equal"
values:
- "2" # filter by type of access (MAY_WRITE)
The previous example uses the Equal
operator. Similarly, you can use the Prefix
operator to
filter events based on the prefix of a filename.
In this example, we monitor if a process inside a Kubernetes workload performs an read or write in
the /etc/
directory. The policy may be extended with additional directories or specific files if
needed.
As a first step, we apply the following policy that uses the three hooks mentioned previously as well as appropriate filtering:
kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/filename_monitoring.yaml
Next, you can start monitoring the events from the xwing
pod:
kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | tetra getevents -o compact --namespace default --pod xwing
In another terminal, kubectl exec
into the xwing
pod:
kubectl exec -it xwing -- /bin/bash
and edit the /etc/passwd
file:
vi /etc/passwd
If you observe, the output in the first terminal should be:
🚀 process default/xwing /bin/bash
📚 read default/xwing /bin/bash /etc/passwd
📚 read default/xwing /bin/bash /etc/passwd
💥 exit default/xwing /bin/bash 127
🚀 process default/xwing /usr/bin/vi /etc/passwd
📚 read default/xwing /usr/bin/vi /etc/passwd
📚 read default/xwing /usr/bin/vi /etc/passwd
📝 write default/xwing /usr/bin/vi /etc/passwd
📝 write default/xwing /usr/bin/vi /etc/passwd
📝 trunc default/xwing /usr/bin/vi /etc/passwd
💥 exit default/xwing /usr/bin/vi /etc/passwd 0
Note, that read and writes are only generated for /etc/
files based on BPF in-kernel filtering
specified in the policy. The default CRD additionally filters events associated with the pod init
process to filter init noise from pod start.
Similarly to the previous example, reviewing the JSON events provides additional data. An example
process_kprobe
event observing a write can be:
{
"process_kprobe": {
"process": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjMyMDY2OTEyNDkyMTA4MToyMDk3ODM=",
"pid": 209783,
"uid": 0,
"cwd": "/",
"binary": "/usr/bin/vi",
"arguments": "/etc/passwd",
"flags": "execve rootcwd clone",
"start_time": "2023-06-26T11:23:43.774969054Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://1f1d1ed09d56f04c17857cc5154f11d22d738b04b28941accd08569183caa4b5",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-06-26T11:11:01Z",
"pid": 37
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
}
},
"docker": "1f1d1ed09d56f04c17857cc5154f11d",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjMyMDY2MDM3OTEzODY1MzoyMDk3NTE=",
"refcnt": 1,
"tid": 209783
},
"parent": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjMyMDY2MDM3OTEzODY1MzoyMDk3NTE=",
"pid": 209751,
"uid": 0,
"cwd": "/",
"binary": "/bin/bash",
"flags": "execve rootcwd clone",
"start_time": "2023-06-26T11:23:35.029187062Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://1f1d1ed09d56f04c17857cc5154f11d22d738b04b28941accd08569183caa4b5",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-06-26T11:11:01Z",
"pid": 28
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
}
},
"docker": "1f1d1ed09d56f04c17857cc5154f11d",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjMyMDY2MDM0NTk5MzM2NToyMDk3NDE=",
"refcnt": 2,
"tid": 209751
},
"function_name": "security_file_permission",
"args": [
{
"file_arg": {
"path": "/etc/passwd"
}
},
{
"int_arg": 2
}
],
"return": {
"int_arg": 0
},
"action": "KPROBE_ACTION_POST"
},
"node_name": "kind-control-plane",
"time": "2023-06-26T11:23:49.019160599Z"
}
In addition to the Kubernetes Identity
and process metadata from exec events, process_kprobe
events contain
the arguments of the observed system call. In the above case they are
file_arg.path
: the observed file pathint_arg
: is the type of the operation (2 for a write and 4 for a read)return.int_arg
: is 0 if the operation is allowedTo disable the TracingPolicy
run:
kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/filename_monitoring.yaml
Another example of a similar policy can be found in our examples folder.
Note that this policy has certain limitations because it matches on the filename that the application uses to accesses. If an application accesses the same file via a hard link or a different bind mount, no event will be generated.
To view TCP connect events, apply the example TCP connect TracingPolicy
:
kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/tcp-connect.yaml
To start monitoring events in the xwing
pod run the Tetragon CLI:
kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | tetra getevents -o compact --namespace default --pod xwing
In another terminal, start generate a TCP connection. Here we use curl.
kubectl exec -it xwing -- curl http://cilium.io
The output in the first terminal will capture the new connect and write,
🚀 process default/xwing /usr/bin/curl http://cilium.io
🔌 connect default/xwing /usr/bin/curl tcp 10.244.0.6:34965 -> 104.198.14.52:80
📤 sendmsg default/xwing /usr/bin/curl tcp 10.244.0.6:34965 -> 104.198.14.52:80 bytes 73
🧹 close default/xwing /usr/bin/curl tcp 10.244.0.6:34965 -> 104.198.14.52:80
💥 exit default/xwing /usr/bin/curl http://cilium.io 0
To disable the TracingPolicy run:
kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/tcp-connect.yaml
On Linux each process has various associated user, group IDs, capabilities, secure management flags, keyring, LSM security that are used part of the security checks upon acting on other objects. These are called the task privileges or process credentials.
Changing the process credentials is a standard operation to perform privileged actions or to execute commands as another user. The obvious example is sudo that allows to gain high privileges and run commands as root or another user. An other example is services or containers that can gain high privileges during execution to perform restricted operations.
execve(2)
.execve(2)
, especially when a binary is executed that will execute as
UID 0.These govern the way the UIDs/GIDs and capabilities are manipulated and
inherited over certain operations such as execve(2)
.
The LSM framework provides a mechanism for various security checks to be hooked by new kernel extensions. Tasks can have extra controls part of LSM on what operations they are allowed to perform.
Monitoring Linux process credentials is a good practice to idenfity programs
running with high privileges. Tetragon allows retrieving Linux process credentials
as a process_credentials
object.
Changes to credentials can be monitored either in system calls or in internal kernel functions.
Generally it is better to monitor in internal kernel functions. For further details please read Advantages and disadvantages of kernel layer monitoring compared to the system call layer section.
Tetragon can hook at the system calls that directly manipulate the credentials. This allows us to determine which process is trying to change its credentials and the new credentials that could be applied by the kernel.
This answers the questions:
Which process or container is trying to change its UIDs/GIDs in my cluster?
Which process or container is trying to change its capabilities in my cluster?
Before going forward, verify that all pods are up and running, ensure you deploy our Demo Application to explore the Security Observability Events:
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.15.0-pre.1/examples/minikube/http-sw-app.yaml
It might take several seconds for some pods until they satisfy all the dependencies:
kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default deathstar-54bb8475cc-6c6lc 1/1 Running 0 2m54s
default deathstar-54bb8475cc-zmfkr 1/1 Running 0 2m54s
default tiefighter 1/1 Running 0 2m54s
default xwing 1/1 Running 0 2m54s
kube-system tetragon-sdwv6 2/2 Running 0 27m
We use the
process.credentials.changes.at.syscalls
Tracing Policy that hooks the
setuid
system calls
family:
setuid
setgid
setfsuid
setfsgid
setreuid
setregid
setresuid
setresgid
Let’s apply the process.credentials.changes.at.syscalls Tracing Policy.
kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-credentials/process.credentials.changes.at.syscalls.yaml
Then start monitoring events with the tetra
CLI:
kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents
In another terminal, kubectl exec
into the xwing Pod:
kubectl exec -it xwing -- /bin/bash
And execute su as this will call the
related setuid
system calls:
su root
The tetra
CLI will generate the following ProcessKprobe events:
{
"process_kprobe": {
"process": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQwNzk4ODc2MDI2NTk4OjEyNTc5OA==",
"pid": 125798,
"uid": 0,
"cwd": "/",
"binary": "/bin/su",
"arguments": "root",
"flags": "execve rootcwd clone",
"start_time": "2023-07-05T19:14:30.918693157Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://55936e548de63f77ceb595d64966dd8e267b391ff0ef63b26c17eb8c2f6510be",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-07-05T18:45:16Z",
"pid": 19
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
}
},
"docker": "55936e548de63f77ceb595d64966dd8",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQwNzk1NjYyMDM3MzMyOjEyNTc5Mg==",
"refcnt": 1,
"tid": 125798
},
"parent": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQwNzk1NjYyMDM3MzMyOjEyNTc5Mg==",
"pid": 125792,
"uid": 0,
"cwd": "/",
"binary": "/bin/bash",
"flags": "execve rootcwd clone",
"start_time": "2023-07-05T19:14:27.704703805Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://55936e548de63f77ceb595d64966dd8e267b391ff0ef63b26c17eb8c2f6510be",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-07-05T18:45:16Z",
"pid": 13
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
}
},
"docker": "55936e548de63f77ceb595d64966dd8",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQwNzk1NjE2MTU0NzA2OjEyNTc4Mw==",
"refcnt": 2,
"tid": 125792
},
"function_name": "__x64_sys_setgid",
"args": [
{
"int_arg": 0
}
],
"action": "KPROBE_ACTION_POST"
},
"node_name": "kind-control-plane",
"time": "2023-07-05T19:14:30.918977160Z"
}
{
"process_kprobe": {
"process": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQwNzk4ODc2MDI2NTk4OjEyNTc5OA==",
"pid": 125798,
"uid": 0,
"cwd": "/",
"binary": "/bin/su",
"arguments": "root",
"flags": "execve rootcwd clone",
"start_time": "2023-07-05T19:14:30.918693157Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://55936e548de63f77ceb595d64966dd8e267b391ff0ef63b26c17eb8c2f6510be",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-07-05T18:45:16Z",
"pid": 19
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
}
},
"docker": "55936e548de63f77ceb595d64966dd8",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQwNzk1NjYyMDM3MzMyOjEyNTc5Mg==",
"refcnt": 1,
"tid": 125798
},
"parent": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQwNzk1NjYyMDM3MzMyOjEyNTc5Mg==",
"pid": 125792,
"uid": 0,
"cwd": "/",
"binary": "/bin/bash",
"flags": "execve rootcwd clone",
"start_time": "2023-07-05T19:14:27.704703805Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://55936e548de63f77ceb595d64966dd8e267b391ff0ef63b26c17eb8c2f6510be",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-07-05T18:45:16Z",
"pid": 13
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
}
},
"docker": "55936e548de63f77ceb595d64966dd8",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQwNzk1NjE2MTU0NzA2OjEyNTc4Mw==",
"refcnt": 2,
"tid": 125792
},
"function_name": "__x64_sys_setuid",
"args": [
{
"int_arg": 0
}
],
"action": "KPROBE_ACTION_POST"
},
"node_name": "kind-control-plane",
"time": "2023-07-05T19:14:30.918990583Z"
}
In addition to the Kubernetes Identity and process metadata from exec events, ProcessKprobe events contain the arguments of the observed system call. In the above case they are:
function_name
: the system call, __x64_sys_setuid
or
__x64_sys_setgid
int_arg
: the uid
or gid
to use, in our case it’s 0 which corresponds to
the root user.To disable the process.credentials.changes.at.syscalls Tracing Policy run:
kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-credentials/process.credentials.changes.at.syscalls.yaml
Monitoring Process Credentials changes at the kernel layer is also possible.
This allows to capture the new process_credentials
that should be applied.
This process-creds-installed tracing policy can be used to answer the following questions:
Which process or container is trying to change its own UIDs/GIDs in the cluster?
Which process or container is trying to change its own capabilities in the cluster?
In which user namespace the credentials are being changed?
How to monitor
process_credentials
changes?
The main advantages of monitoring at the kernel layer compared to the system call layer:
Not vulnerable to user space arguments tampering.
Ability to display the full new credentials to be applied.
It is more reliable since it has full context on where and how the new credentials should be applied including the user namespace.
A catch all layer for all system calls, and every normal kernel path that manipulate credentials.
One potential disadvantage is that this approach may generate a lot of events, so appropriate filtering must be applied to reduce the noise.
First, verify that your k8s environment is all setup and that all pods are up and running, and deploy the Demo Application:
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.15.0-pre.1/examples/minikube/http-sw-app.yaml
It might take several seconds until all pods are Running:
kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default deathstar-54bb8475cc-6c6lc 1/1 Running 0 2m54s
default deathstar-54bb8475cc-zmfkr 1/1 Running 0 2m54s
default tiefighter 1/1 Running 0 2m54s
default xwing 1/1 Running 0 2m54s
kube-system tetragon-sdwv6 2/2 Running 0 27m
We use the process-creds-installed Tracing Policy that hooks the kernel layer when credentials are being installed.
So let’s apply the process-creds-installed Tracing Policy.
kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-credentials/process-creds-installed.yaml
Then we start monitoring for events with tetra
cli:
kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents
In another terminal, inside a pod and as a non root user we will execute a setuid binary (suid):
/tmp/su -
The tetra
cli will generate the following ProcessKprobe events:
{
"process_kprobe": {
"process": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE0MDczMDQyODk3MTc6MjIzODY=",
"pid": 22386,
"uid": 11,
"cwd": "/",
"binary": "/tmp/su",
"arguments": "-",
"flags": "execve rootcwd clone",
"start_time": "2023-07-25T12:04:59.359333454Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://2e58c8357465961fd96f758e87d0269dfb5f97c536847485de9d7ec62be34a64",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-07-25T11:44:48Z",
"pid": 43
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
}
},
"docker": "2e58c8357465961fd96f758e87d0269",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjEzOTU0MDY5OTA1ODc6MjIzNTI=",
"refcnt": 1,
"tid": 22386
},
"parent": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjEzOTU0MDY5OTA1ODc6MjIzNTI=",
"pid": 22352,
"uid": 11,
"cwd": "/",
"binary": "/bin/sh",
"flags": "execve rootcwd",
"start_time": "2023-07-25T12:04:47.462035587Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://2e58c8357465961fd96f758e87d0269dfb5f97c536847485de9d7ec62be34a64",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-07-25T11:44:48Z",
"pid": 41
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
}
},
"docker": "2e58c8357465961fd96f758e87d0269",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjEzOTU0MDQ3NzY5NzI6MjIzNTI=",
"refcnt": 2,
"tid": 22352
},
"function_name": "commit_creds",
"args": [
{
"process_credentials_arg": {
"uid": 0,
"gid": 0,
"euid": 0,
"egid": 0,
"suid": 0,
"sgid": 0,
"fsuid": 0,
"fsgid": 0,
"caps": {
"permitted": [
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_NET_BIND_SERVICE",
"CAP_NET_RAW",
"CAP_SYS_CHROOT",
"CAP_MKNOD",
"CAP_AUDIT_WRITE",
"CAP_SETFCAP"
],
"effective": [
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_NET_BIND_SERVICE",
"CAP_NET_RAW",
"CAP_SYS_CHROOT",
"CAP_MKNOD",
"CAP_AUDIT_WRITE",
"CAP_SETFCAP"
]
},
"user_ns": {
"level": 0,
"uid": 0,
"gid": 0,
"ns": {
"inum": 4026531837,
"is_host": true
}
}
}
}
],
"action": "KPROBE_ACTION_POST"
},
"node_name": "kind-control-plane",
"time": "2023-07-25T12:05:01.410834172Z"
}
In addition to the Kubernetes Identity and process metadata from exec events, ProcessKprobe events contain the arguments of the observed system call. In the above case they are:
function_name
: the kernel commit_creds()
function to install new credentials.process_credentials_arg
: the new process_credentials
to be installed
on the current process. It includes the UIDs/GIDs, the capabilities and the target user namespace.Here we can clearly see that the suid binary is being executed by a user ID 11
in order to elevate its privileges to user ID 0
including capabilities.
To disable the process-creds-installed Tracing Policy run:
kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-credentials/process-creds-installed.yaml
Some pods need to change the host system or kernel parameters in order to perform administrative tasks, obvious examples are pods loading a kernel module to extend the operating system functionality, or pods managing the network.
However, there are also other cases where a compromised container may want to load a kernel module to hide its behaviour.
In this aspect, monitoring such host system changes helps to identify pods and containers that affect the host system.
A kernel module is a code that can be loaded into the kernel image at runtime, without rebooting. These modules, which can be loaded by pods and containers, can modify the host system. The Monitor Linux kernel modules guide will assist you in observing such events.
Monitoring kernel modules helps to identify processes that load kernel modules to add features, to the operating system, to alter host system functionality or even hide their behaviour. This can be used to answer the following questions:
Which process or container is changing the kernel?
Which process or container is loading or unloading kernel modules in the cuslter?
Which process or container requested a feature that triggered the kernel to automatically load a module?
Are the loaded kernel modules signed?
After deploying Tetragon, use the monitor-kernel-modules tracing policy which generates ProcessKprobe events to trace kernel module operations.
Apply the monitor-kernel-modules tracing policy:
kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/host-changes/monitor-kernel-modules.yaml
Then start monitoring for events with tetra
CLI:
kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents
When loading an out of tree module named kernel_module_hello.ko
with the command insmod
,
tetra
CLI will generate the following ProcessKprobe events:
{
"process_kprobe": {
"process": {
"exec_id": "OjEzMTg4MTQwNDUwODkwOjgyMDIz",
"pid": 82023,
"uid": 0,
"cwd": "/home/tixxdz/tetragon",
"binary": "/usr/sbin/insmod",
"arguments": "contrib/tester-progs/kernel_module_hello.ko",
"flags": "execve clone",
"start_time": "2023-08-30T11:01:22.846516679Z",
"auid": 1000,
"parent_exec_id": "OjEzMTg4MTM4MjY2ODQyOjgyMDIy",
"refcnt": 1,
"tid": 82023
},
"parent": {
"exec_id": "OjEzMTg4MTM4MjY2ODQyOjgyMDIy",
"pid": 82022,
"uid": 1000,
"cwd": "/home/tixxdz/tetragon",
"binary": "/usr/bin/sudo",
"arguments": "insmod contrib/tester-progs/kernel_module_hello.ko",
"flags": "execve",
"start_time": "2023-08-30T11:01:22.844332959Z",
"auid": 1000,
"parent_exec_id": "OjEzMTg1NTE3MTgzNDM0OjgyMDIx",
"refcnt": 1,
"tid": 0
},
"function_name": "security_kernel_read_file",
"args": [
{
"file_arg": {
"path": "/home/tixxdz/tetragon/contrib/tester-progs/kernel_module_hello.ko"
}
},
{
"int_arg": 2
}
],
"return": {
"int_arg": 0
},
"action": "KPROBE_ACTION_POST"
},
"time": "2023-08-30T11:01:22.847554295Z"
}
In addition to the process metadata from exec events, ProcessKprobe events contain the arguments of the observed call. In the above case they are:
security_kernel_read_file
: the kernel security hook when the kernel loads file specified by user space.file_arg
: the full path of the kernel module on the file system.{
"process_kprobe": {
"process": {
"exec_id": "OjEzMTg4MTQwNDUwODkwOjgyMDIz",
"pid": 82023,
"uid": 0,
"cwd": "/home/tixxdz/tetragon",
"binary": "/usr/sbin/insmod",
"arguments": "contrib/tester-progs/kernel_module_hello.ko",
"flags": "execve clone",
"start_time": "2023-08-30T11:01:22.846516679Z",
"auid": 1000,
"parent_exec_id": "OjEzMTg4MTM4MjY2ODQyOjgyMDIy",
"refcnt": 1,
"tid": 82023
},
"parent": {
"exec_id": "OjEzMTg4MTM4MjY2ODQyOjgyMDIy",
"pid": 82022,
"uid": 1000,
"cwd": "/home/tixxdz/tetragon",
"binary": "/usr/bin/sudo",
"arguments": "insmod contrib/tester-progs/kernel_module_hello.ko",
"flags": "execve",
"start_time": "2023-08-30T11:01:22.844332959Z",
"auid": 1000,
"parent_exec_id": "OjEzMTg1NTE3MTgzNDM0OjgyMDIx",
"refcnt": 1,
"tid": 0
},
"function_name": "do_init_module",
"args": [
{
"module_arg": {
"name": "kernel_module_hello",
"tainted": [
"TAINT_OUT_OF_TREE_MODULE",
"TAINT_UNSIGNED_MODULE"
]
}
}
],
"action": "KPROBE_ACTION_POST"
},
"time": "2023-08-30T11:01:22.847638990Z"
}
This ProcessKprobe event contains:
do_init_module
: the function call where the module is finaly loaded.module_arg
: the kernel module information, it contains:name
: the name of the kernel module as a string.tainted
: the module tainted flags that will be applied on the kernel. In the example above, it indicates we are loading an out-of-tree module, that is unsigned module which may compromise the integrity of our system.Kernels compiled with CONFIG_MODULE_SIG
option will check if the modules being loaded were cryptographically signed.
This allows to assert that:
If the module being loaded is signed, the kernel has its key and the signature verification succeeded.
The integrity of the system or the kernel was not compromised.
After deploying Tetragon, use the monitor-signed-kernel-modules tracing policy which generates ProcessKprobe events to identify if kernel modules are signed or not.
Apply the monitor-signed-kernel-modules tracing policy:
kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/host-changes/monitor-signed-kernel-modules.yaml
Before going forward, deploy the test-pod
into the demo-app namespace, which has its security context set to privileged.
This allows to run the demo by mountig an xfs
file system inside the test-pod
which requires privileges,
but will also trigger an automatic xfs
module loading operation.
kubectl create namespace demo-app
kubectl apply -n demo-app -f https://raw.githubusercontent.com/cilium/tetragon/main/testdata/specs/testpod.yaml
Start monitoring for events with tetra
CLI:
kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents
In another terminal, kubectl exec into the test-pod
and run the following commands to create an xfs
filesystem:
kubectl exec -it -n demo-app test-pod -- /bin/sh
apk update
dd if=/dev/zero of=loop.xfs bs=1 count=0 seek=32M
ls -lha loop.xfs
apk add xfsprogs
mkfs.xfs -q loop.xfs
mkdir /mnt/xfs.volume
mount -o loop -t xfs loop.xfs /mnt/xfs.volume/
losetup -a | grep xfs
Now the xfs filesystem should be mounted at /mnt/xfs.volume
. To unmount it and release the loop device run:
umount /mnt/xfs.volume/
tetra
CLI will generate the following events:
First the mount
command will trigger an automatic operation to load the xfs
kernel module.
{
"process_kprobe": {
"process": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQxMjc1NTA0OTk5NTcyOjEzMDg3Ng==",
"pid": 130876,
"uid": 0,
"cwd": "/",
"binary": "/bin/mount",
"arguments": "-o loop -t xfs loop.xfs /mnt/xfs.volume/",
"flags": "execve rootcwd clone",
"start_time": "2023-09-09T23:27:42.732039059Z",
"auid": 4294967295,
"pod": {
"namespace": "demo-app",
"name": "test-pod",
"container": {
"id": "containerd://1e910d5cc8d8d68c894934170b162ef93aea5652867ed6bd7c620c7e3f9a10f1",
"name": "test-pod",
"image": {
"id": "docker.io/cilium/starwars@sha256:f92c8cd25372bac56f55111469fe9862bf682385a4227645f5af155eee7f58d9",
"name": "docker.io/cilium/starwars:latest"
},
"start_time": "2023-09-09T22:46:09Z",
"pid": 45672
},
"workload": "test-pod"
},
"docker": "1e910d5cc8d8d68c894934170b162ef",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQxMjYyOTc1MjI1MDkzOjEzMDgwOQ==",
"refcnt": 1,
"tid": 130876
},
"parent": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQxMjYyOTc1MjI1MDkzOjEzMDgwOQ==",
"pid": 130809,
"uid": 0,
"cwd": "/",
"binary": "/bin/sh",
"flags": "execve rootcwd clone",
"start_time": "2023-09-09T23:27:30.202263472Z",
"auid": 4294967295,
"pod": {
"namespace": "demo-app",
"name": "test-pod",
"container": {
"id": "containerd://1e910d5cc8d8d68c894934170b162ef93aea5652867ed6bd7c620c7e3f9a10f1",
"name": "test-pod",
"image": {
"id": "docker.io/cilium/starwars@sha256:f92c8cd25372bac56f55111469fe9862bf682385a4227645f5af155eee7f58d9",
"name": "docker.io/cilium/starwars:latest"
},
"start_time": "2023-09-09T22:46:09Z",
"pid": 45612
},
"workload": "test-pod"
},
"docker": "1e910d5cc8d8d68c894934170b162ef",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQxMjYyOTEwMjM3OTQ2OjEzMDgwMA==",
"tid": 130809
},
"function_name": "security_kernel_module_request",
"args": [
{
"string_arg": "fs-xfs"
}
],
"return": {
"int_arg": 0
},
"action": "KPROBE_ACTION_POST"
},
"node_name": "kind-control-plane",
"time": "2023-09-09T23:27:42.751151233Z"
}
In addition to the process metadata from exec events, ProcessKprobe event contains the arguments of the observed call. In the above case they are:
security_kernel_module_request
: the kernel security hook where modules are loaded on-demand.string_arg
: the name of the kernel module. When modules are automatically loaded, for security reasons,
the kernel prefixes the module with the name of the subsystem that requested it. In our case, it’s requested
by the file system subsystem, hence the name is fs-xfs
.The kernel will then call user space modprobe
to load the kernel module.
{
"process_exec": {
"process": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQxMjc1NTI0MjYzMjIxOjEzMDg3Nw==",
"pid": 130877,
"uid": 0,
"cwd": "/",
"binary": "/sbin/modprobe",
"arguments": "-q -- fs-xfs",
"flags": "execve rootcwd clone",
"start_time": "2023-09-09T23:27:42.751301124Z",
"auid": 4294967295,
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
"tid": 130877
},
"parent": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
"pid": 0,
"uid": 0,
"binary": "<kernel>",
"flags": "procFS",
"start_time": "2023-09-09T11:59:47.227037763Z",
"auid": 0,
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
"tid": 0
}
},
"node_name": "kind-control-plane",
"time": "2023-09-09T23:27:42.751300984Z"
}
The ProcessExec event where modprobe
tries to load the xfs
module.
modprobe
is started in the initial Linux host namespaces, outside of the container namespaces. When kernel
modules are loaded on-demand, the kernel will spawn a user space process modprobe
that finds and load the appropriate
module from the host file system. This is done on behalf of the container and since its originate from the kernel then
the inherited Linux namespaces including the file system are eventually from the host.modprobe
will read the passed xfs
kernel module from the host file system.
{
"process_kprobe": {
"process": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQxMjc1NTI0MjYzMjIxOjEzMDg3Nw==",
"pid": 130877,
"uid": 0,
"cwd": "/",
"binary": "/sbin/modprobe",
"arguments": "-q -- fs-xfs",
"flags": "execve rootcwd clone",
"start_time": "2023-09-09T23:27:42.751301124Z",
"auid": 4294967295,
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
"refcnt": 1,
"tid": 130877
},
"parent": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
"pid": 0,
"uid": 0,
"binary": "<kernel>",
"flags": "procFS",
"start_time": "2023-09-09T11:59:47.227037763Z",
"auid": 0,
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
"tid": 0
},
"function_name": "security_kernel_read_file",
"args": [
{
"file_arg": {
"path": "/usr/lib/modules/6.2.0-32-generic/kernel/fs/xfs/xfs.ko"
}
},
{
"int_arg": 2
}
],
"return": {
"int_arg": 0
},
"action": "KPROBE_ACTION_POST"
},
"node_name": "kind-control-plane",
"time": "2023-09-09T23:27:42.752425825Z"
}
This ProcessKprobe event contains:
security_kernel_read_file
: the kernel security hook when the kernel loads file specified by user space.file_arg
: the full path of the kernel module on the host file system.The final event is when the kernel is parsing the module sections. If all succeed the module will be loaded.
{
"process_kprobe": {
"process": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQxMjc1NTI0MjYzMjIxOjEzMDg3Nw==",
"pid": 130877,
"uid": 0,
"cwd": "/",
"binary": "/sbin/modprobe",
"arguments": "-q -- fs-xfs",
"flags": "execve rootcwd clone",
"start_time": "2023-09-09T23:27:42.751301124Z",
"auid": 4294967295,
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
"refcnt": 1,
"tid": 130877
},
"parent": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
"pid": 0,
"uid": 0,
"binary": "<kernel>",
"flags": "procFS",
"start_time": "2023-09-09T11:59:47.227037763Z",
"auid": 0,
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
"tid": 0
},
"function_name": "find_module_sections",
"args": [
{
"module_arg": {
"name": "xfs",
"signature_ok": true
}
}
],
"action": "KPROBE_ACTION_POST"
},
"node_name": "kind-control-plane",
"time": "2023-09-09T23:27:42.760880332Z"
}
This ProcessKprobe event contains the module argument.
find_module_sections
: the function call where the kernel parses the module sections.module_arg
: the kernel module information, it contains:name
: the name of the kernel module as a string.signature_ok
: a boolean value, if set to true
then module signature was successfully verified by the kernel. If it is false
or missing then the signature verification was not performed or probably failed. In all cases this means the integrity of the system has been compromised. Depends on kernels compiled with CONFIG_MODULE_SIG
option.Using the same monitor-kernel-modules tracing policy allows to monitor unloading of kernel modules.
The following ProcessKprobe event will be generated:
{
"process_kprobe": {
"process": {
"exec_id": "OjMzNzQ4NzY1MDAyNDk5OjI0OTE3NQ==",
"pid": 249175,
"uid": 0,
"cwd": "/home/tixxdz/tetragon",
"binary": "/usr/sbin/rmmod",
"arguments": "kernel_module_hello",
"flags": "execve clone",
"start_time": "2023-08-30T16:44:03.471068355Z",
"auid": 1000,
"parent_exec_id": "OjMzNzQ4NzY0MjQ4MTY5OjI0OTE3NA==",
"refcnt": 1,
"tid": 249175
},
"parent": {
"exec_id": "OjMzNzQ4NzY0MjQ4MTY5OjI0OTE3NA==",
"pid": 249174,
"uid": 1000,
"cwd": "/home/tixxdz/tetragon",
"binary": "/usr/bin/sudo",
"arguments": "rmmod kernel_module_hello",
"flags": "execve",
"start_time": "2023-08-30T16:44:03.470314558Z",
"auid": 1000,
"parent_exec_id": "OjMzNzQ2MjA5OTUxODI4OjI0OTE3Mw==",
"refcnt": 1,
"tid": 0
},
"function_name": "free_module",
"args": [
{
"module_arg": {
"name": "kernel_module_hello",
"tainted": [
"TAINT_OUT_OF_TREE_MODULE",
"TAINT_UNSIGNED_MODULE"
]
}
}
],
"action": "KPROBE_ACTION_POST"
},
"time": "2023-08-30T16:44:03.471984676Z"
}
To disable the monitor-kernel-modules run:
kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/host-changes/monitor-kernel-modules.yaml
Tetragon is able to observe various security events and even enforce security policies.
The Record Linux Capabilities Usage guide shows how to monitor and record Capabilities checks conducted by the kernel on behalf of applications during privileged operations. This can be used to inspect and produce security profiles for pods and containers.
When the kernel needs to perform a privileged operation on behalf of a process, it checks the Capabilities of the process and issues a verdict to allow or deny the operation.
Tetragon is able to record these checks performed by the kernel. This can be used to answer the following questions:
What is the capabilities profile of pods or containters running in the cluster?
What capabilities to add or remove when configuring a security context for a pod or container?
First, verify that your k8s environment is set up and that all pods are up and running, and deploy the demo application:
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.15.0-pre.1/examples/minikube/http-sw-app.yaml
It might take several seconds until all pods are Running:
kubectl get pods -A
The output should be similar to:
NAMESPACE NAME READY STATUS RESTARTS AGE
default deathstar-54bb8475cc-6c6lc 1/1 Running 0 2m54s
default deathstar-54bb8475cc-zmfkr 1/1 Running 0 2m54s
default tiefighter 1/1 Running 0 2m54s
default xwing 1/1 Running 0 2m54s
kube-system tetragon-sdwv6 2/2 Running 0 27m
We use the creds-capability-usage tracing policy which generates ProcessKprobe events.
Tracing policies as the one used here, may emit a high number of events.
To reduce events, the creds-capability-usage rate limits events to 1 minute. More details about rate-limiting can be found in the tracing policy documentation.
Apply the creds-capability-usage policy:
kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-credentials/creds-capability-usage.yaml
Start monitoring for events with tetra
cli, but match only events of xwing
pod:
kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents --namespaces default --pods xwing
In another terminal, kubectl exec into the xwing pod:
kubectl exec -it xwing -- /bin/bash
As an example execute dmesg to print the kernel ring buffer. This requires the special capability CAP_SYSLOG
:
dmesg
The output should be similar to:
dmesg: klogctl: Operation not permitted
The tetra
cli will generate the following ProcessKprobe events:
{
"process_kprobe": {
"process": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjEyODQyNzgzMzUwNjg0OjczODYw",
"pid": 73860,
"uid": 0,
"cwd": "/",
"binary": "/bin/dmesg",
"flags": "execve rootcwd clone",
"start_time": "2023-07-06T10:13:33.834390020Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://cfb961400ff25811d22d139a10f6a62efef53c2ecc11af47bc911a7f9a2ac1f7",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-07-06T08:07:30Z",
"pid": 171
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
}
},
"docker": "cfb961400ff25811d22d139a10f6a62",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjEyODQyMTI3MTIwOTcyOjczODUw",
"refcnt": 1,
"ns": {
"uts": {
"inum": 4026534655
},
"ipc": {
"inum": 4026534656
},
"mnt": {
"inum": 4026534731
},
"pid": {
"inum": 4026534732
},
"pid_for_children": {
"inum": 4026534732
},
"net": {
"inum": 4026534512
},
"time": {
"inum": 4026531834,
"is_host": true
},
"time_for_children": {
"inum": 4026531834,
"is_host": true
},
"cgroup": {
"inum": 4026534733
},
"user": {
"inum": 4026531837,
"is_host": true
}
},
"tid": 73860
},
"parent": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjEyODQyMTI3MTIwOTcyOjczODUw",
"pid": 73850,
"uid": 0,
"cwd": "/",
"binary": "/bin/bash",
"flags": "execve rootcwd clone",
"start_time": "2023-07-06T10:13:33.178160018Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://cfb961400ff25811d22d139a10f6a62efef53c2ecc11af47bc911a7f9a2ac1f7",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-07-06T08:07:30Z",
"pid": 165
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
}
},
"docker": "cfb961400ff25811d22d139a10f6a62",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjEyODQyMDgxNTA3MzUzOjczODQx",
"refcnt": 2,
"tid": 73850
},
"function_name": "cap_capable",
"args": [
{
"user_ns_arg": {
"level": 0,
"uid": 0,
"gid": 0,
"ns": {
"inum": 4026531837,
"is_host": true
}
}
},
{
"capability_arg": {
"value": 34,
"name": "CAP_SYSLOG"
}
}
],
"return": {
"int_arg": -1
},
"action": "KPROBE_ACTION_POST"
},
"node_name": "kind-control-plane",
"time": "2023-07-06T10:13:33.834882128Z"
}
In addition to the Kubernetes Identity and process metadata from exec events, ProcessKprobe events contain the arguments of the observed system call. In the above case they are:
function_name
: that is the cap_capable
kernel function.
user_ns_arg
: is the user namespace where the capability is required.
level
: is the nested level of the user namespace. Here it is zero which indicates the initial user namespace.uid
: is the user ID of the owner of the user namespace.gid
: is the group ID of the owner of the user namespace.ns
: details the information about the namespace. is_host
indicates that the target user namespace where the capability is required is the host namespace.capability_arg
: is the capability required to perform the operation. In this example reading the kernel ring buffer.
value
: is the integer number of the required capability.name
: is the name of the required capability. Here it is the CAP_SYSLOG
.return
: indicates via the int_arg
if the capability check succeeded or failed. 0
means it succeeded and the access was granted while -1
means it failed and the operation was denied.
To disable the creds-capability-usage run:
kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-credentials/creds-capability-usage.yaml
When debugging, it might be useful to change the log level. The default log level is controlled by the log-level option:
Enable debug level with --log-level=debug
Enable trace level with --log-level=trace
It is possible to change the log level dynamically by sending the corresponding signal to tetragon process.
Change log level to debug level by sending the SIGRTMIN+20
signal to tetragon pid:
sudo kill -s SIGRTMIN+20 $(pidof tetragon)
Change log level to trace level by sending the SIGRTMIN+21
signal to tetragon pid:
sudo kill -s SIGRTMIN+21 $(pidof tetragon)
To Restore the original log level send the SIGRTMIN+22
signal to tetragon pid:
sudo kill -s SIGRTMIN+22 $(pidof tetragon)
This guide has been tested on Ubuntu 22.04 and 22.10 with respectively kernel
5.15.0
and 5.19.0
on amd64 and arm64 but
any recent distribution
shipping with a relatively recent kernel should work. See the FAQ for further details on
the recommended kernel versions.
Note that you cannot run Tetragon using Docker Desktop on macOS because of a limitation of the Docker Desktop Linux virtual machine. Learn more about this issue and how to run Tetragon on a Mac computer in this section of the FAQ page.
The easiest way to start experimenting with Tetragon is to run it via Docker using the released container images. If you prefer running directly the binaries without Docker, please refer to the development setup to see how to build the binaries.
docker run --name tetragon-container --rm --pull always \
--pid=host --cgroupns=host --privileged \
-v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf \
quay.io/cilium/tetragon-ci:latest
Let’s break down the previous command:
--name tetragon-container
names our container so that we can refer to it
later via a simple name.--rm
will make Docker delete the container once it exists.--pull always
will force Docker to download the latest existing image
corresponding to the latest
mutable tag.--pid=host
is needed to read the procfs for gathering context on processes
started before Tetragon.--cgroupns=host
is needed to detect container feature.--privileged
is needed for Tetragon to load its BPF programs.-v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf
mounts the BTF file inside
the Tetragon container filesystem. Tetragon needs a BTF file corresponding to
your kernel version to load its BPF programs.quay.io/cilium/tetragon-ci:latest
is the latest version of Tetragon built
from the main branch.With this default configuration, Tetragon already loaded its base sensors to perform process lifecycle observability.
To quickly see the events, you can use the tetra
CLI already shipped in the
Tetragon container that was just started, it will connect to the Tetragon gRPC
server listening on localhost:54321
. In another terminal, type the following
command:
docker exec tetragon-container tetra getevents -o compact
In a different terminal, you can start a shell like sh
and start executing
programs. Here, the commands executed in the shell where ls
, uname -a
,
whoami
, and exit
, the output should be similar to this:
🚀 process /usr/bin/sh
🚀 process /usr/bin/ls
💥 exit /usr/bin/ls 0
🚀 process /usr/bin/uname -a
💥 exit /usr/bin/uname -a 0
🚀 process /usr/bin/whoami
💥 exit /usr/bin/whoami 0
💥 exit /usr/bin/sh 0
You can see the start of sh
first, then the execution of the multiple
programs, ls
, uname
and whoami
, and finally the exit of sh
.
Tetragon by default observes only process lifecycles. More advanced use-cases, require configuring Tetragon via TracingPolicies. A TracingPolicy is a user-configurable Kubernetes custom resource (CR) that allows users to access additional functionality, such as tracing arbitrary events in the kernel and, optionally, define actions to take on a match. These actions can be, for example, used to implement enforcement.
Let’s start with a simple policy:
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
name: "cat-open"
spec:
kprobes:
- call: "sys_openat"
syscall: true
args:
- index: 0
type: "int"
- index: 1
type: "string"
- index: 2
type: "int"
selectors:
- matchBinaries:
- operator: In
values:
- "/usr/bin/cat"
Copy this policy in a file named tracing_policy.yaml
in your current working
directory and restart Tetragon with the following command:
docker run --name tetragon-container --rm --pull always \
--pid=host --cgroupns=host --privileged \
-v $PWD/tracing_policy.yaml:/tracing_policy.yaml \
-v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf \
quay.io/cilium/tetragon-ci:latest \
--tracing-policy /tracing_policy.yaml
We added two flags to our commands:
-v $(pwd)/tracing_policy.yaml:/tracing_policy.yaml
mounts the created local
file into the container filesystem.--tracing-policy /tracing_policy.yaml
indicates Tetragon to load the policy in
the provided file.Now again, in another terminal, open the tetra
CLI to see the new events
generated by Tetragon, monitoring the openat
system call.
docker exec tetragon-container tetra getevents -o compact
And in the third terminal, just read the tracing_policy.yaml
file we created
using cat with:
cat tracing_policy.yaml
The output from the tetra
CLI should be similar to:
🚀 process /usr/bin/cat tracingpolicy.yaml
📬️ openat /usr/bin/cat /etc/ld.so.cache
📬️ openat /usr/bin/cat /lib/aarch64-linux-gnu/libc.so.6
📬️ openat /usr/bin/cat tracingpolicy.yaml
💥 exit /usr/bin/cat tracingpolicy.yaml 0
openat
and not open
because the glibc and most of
the implementations of the standard C library use the openat
syscall for
their open
wrapper. So cat
on Linux is using openat
.We can see that during the execution of cat
, because it is dynamically
linked, it used the openat
syscall to open:
ld.so.cache
;libc.so.6
library;tracing_policy.yaml
.This is the kind of information we could obtain by executing the program with
strace
for example, but here Tetragon obtains this information directly from
the kernel, transparently for the executed program.
Please note that tetra
CLI, with the -o compact
output format,
automatically tried to decode the second argument as a string and printed it
because it has the hardcoded knowledge that the openat
syscall has an
interesting string as second argument.
You can use tetra getevents
to retrieve the whole JSON events and see the
complete fields and values retrieved from the event. In our situation the
output should be similar to the following.
{
"process_kprobe": {
"process": {
"exec_id": "OjEwMTgzOTUyNjg1NjcwNDoyNjIzNw==",
"pid": 26237,
"uid": 1000,
"cwd": "/home/ubuntu",
"binary": "/usr/bin/cat",
"arguments": "tracing_policy.yaml",
"flags": "execve clone",
"start_time": "2023-04-06T18:30:03.405730761Z",
"auid": 1000,
"parent_exec_id": "OjEwMTczMDEyNTQ3MjUxMDoyNjIwOA==",
"refcnt": 1
},
"parent": {
"exec_id": "OjEwMTczMDEyNTQ3MjUxMDoyNjIwOA==",
"pid": 26208,
"uid": 1000,
"cwd": "/home/ubuntu",
"binary": "/bin/bash",
"flags": "execve clone",
"start_time": "2023-04-06T18:28:14.004347210Z",
"auid": 1000,
"parent_exec_id": "OjEwMTczMDEwMjc2NTg0NjoyNjIwNw==",
"refcnt": 2
},
"function_name": "__x64_sys_openat",
"args": [
{
"int_arg": -100
},
{
"string_arg": "tracing_policy.yaml"
},
{
"int_arg": 0
}
],
"action": "KPROBE_ACTION_POST"
},
"time": "2023-04-06T18:30:03.406205829Z"
}
You can see that Tetragon collected information on the process itself, with for
example, the uid, the binary name, the start_time and much more. It also
indicates process parent, /bin/bash
in this case. This is the common base of
information we also get from the process lifecycle sensors. Finally, we can see
that Tetragon hooked the function __x64_sys_openat
(for an arm64 host it
would be __arm64_sys_openat
) and that it also collected the arguments’ value.
Let’s see how we can modify the previous TracingPolicy to do enforcement with Tetragon.
openat
syscall can
take relative paths as arguments and even absolute path filtering can often be
bypassed by pseudo filesystems like procfs, for example using a prefix like
/proc/self/root/
to access /
.First, let’s say we don’t want to stop any program that would try to open the
tracing_policy.yaml
but just stop them from actually using the syscall to
open the file, as if they were not unauthorized to access the file. We could
write the following TracingPolicy.
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
name: "file-access"
spec:
kprobes:
- call: "sys_openat"
syscall: true
args:
- index: 0
type: "int"
- index: 1
type: "string"
selectors:
- matchBinaries:
- operator: In
values:
- "/usr/bin/cat"
matchArgs:
- index: 1
operator: "Equal"
values:
- "tracing_policy.yaml\0"
matchActions:
- action: Override
argError: -1
In the above TracingPolicy, we removed the third argument because we don’t
need to extract it specifically, and we added two filters in our existing
selector. One matchArgs
to filter on the value of the index 1 argument, which
contains the pathname to access, and one matchActions
to perform on action on
a such match.
Replace the content of the policy in the tracing_policy.yaml
file with the
above and restart Tetragon with the same command as before. If we open tetra
with docker exec tetragon-container tetra getevents -o compact
and perform a
cat tracing_policy.yaml
on the side, we should now observe an output similar
to:
🚀 process /usr/bin/cat tracing_policy.yaml
📬️ openat /usr/bin/cat tracing_policy.yaml
💥 exit /usr/bin/cat tracing_policy.yaml 1
And the cat tracing_policy.yaml
should return the following error:
cat: tracing_policy.yaml: Operation not permitted
Now, let’s say that accessing this file in this manner is critical and we want
to stop any process that tries to perform such action immediately. We just need
to modify the action
in the matchActions
filter from Override
to
Sigkill
.
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
name: "file-access"
spec:
kprobes:
- call: "sys_openat"
syscall: true
args:
- index: 0
type: "int"
- index: 1
type: "string"
selectors:
- matchBinaries:
- operator: In
values:
- "/usr/bin/cat"
matchArgs:
- index: 1
operator: "Equal"
values:
- "tracing_policy.yaml\0"
matchActions:
- action: Sigkill
Again, replace the content of the policy in the tracing_policy.yaml
file with
the above and restart Tetragon with the same command as before. If we open
tetra with docker exec tetragon-container tetra getevents -o compact
and
perform a cat tracing_policy.yaml
on the side, we should now observe an
output similar to:
🚀 process /usr/bin/cat tracing_policy.yaml
📬️ openat /usr/bin/cat tracing_policy.yaml
💥 exit /usr/bin/cat tracing_policy.yaml SIGKILL
And the cat tracing_policy.yaml
should return an error (that is actually
returned by the shell that invoked the program):
Killed
Now the process will not have the time to catch the error and will be synchronously terminated as soon as he triggers the kernel function we hooked with our specified argument values.
cat ./tracing_policy.yaml
will bypass the policies presented here.You will need to install cosign.
Since version 0.8.4, all Tetragon container images are signed using cosign.
Let’s verify a Tetragon image’s signature using the cosign verify
command:
COSIGN_EXPERIMENTAL=1 cosign verify --certificate-github-workflow-repository cilium/tetragon --certificate-oidc-issuer https://token.actions.githubusercontent.com <Image URL> | jq
Note
COSIGN_EXPERIMENTAL=1
is used to allow verification of images signed in
KEYLESS mode. To learn more about keyless signing, please refer to Keyless
Signatures.
A Software Bill of Materials (SBOM) is a complete, formally structured list of components that are required to build a given piece of software. SBOM provides insight into the software supply chain and any potential concerns related to license compliance and security that might exist.
Starting with version 0.8.4, all Tetragon images include an SBOM. The SBOM is generated in SPDX format using the bom tool. If you are new to the concept of SBOM, see what an SBOM can do for you.
The SBOM can be downloaded from the supplied Tetragon image using the cosign download sbom
command.
cosign download sbom --output-file sbom.spdx <Image URL>
To ensure the SBOM is tamper-proof, its signature can be verified using the
cosign verify
command.
COSIGN_EXPERIMENTAL=1 cosign verify --certificate-github-workflow-repository cilium/tetragon --certificate-oidc-issuer https://token.actions.githubusercontent.com --attachment sbom <Image URL> | jq
It can be validated that the SBOM image was signed using Github Actions in the
Cilium repository from the Issuer
and Subject
fields of the output.
We’re happy you’re interested in contributing to the Tetragon project.
This section of the Tetragon documentation will help you make sure you have an environment capable of testing changes to the Tetragon source code, and that you understand the workflow of getting these changes reviewed and merged upstream.
Make sure you have a GitHub account.
Fork the Tetragon repository to your GitHub user or organization.
Turn off GitHub actions for your fork as described in the GitHub Docs.
This is recommended to avoid unnecessary CI notification failures on the fork.
Clone your ${YOUR_GITHUB_USERNAME_OR_ORG}/tetragon
fork into your
GOPATH
, and set up the base repository as upstream
remote:
mkdir -p "${GOPATH}/src/github.com/cilium"
cd "${GOPATH}/src/github.com/cilium"
git clone https://github.com/${YOUR_GITHUB_USERNAME_OR_ORG}/tetragon.git
cd tetragon
git remote add upstream https://github.com/cilium/tetragon.git
Prepare your Development setup, see section below.
Check the GitHub issues for good tasks to get started.
Follow the steps in Making changes to start contributing. Welcome :)!
For local development, you will likely want to build and run bare-metal Tetragon.
Requirements:
libcap
and libelf
(in Debian systems, e.g., install libelf-dev
and
libcap-dev
)You can build Tetragon as follows:
make
If you want to use podman
instead of docker
, you can do the following (assuming you
need to use sudo
with podman
):
CONTAINER_ENGINE='sudo podman' make
You can ignore /bin/sh: docker: command not found
in the output.
To build using the local clang, you can use:
CONTAINER_ENGINE='sudo podman' LOCAL_CLANG=1 LOCAL_CLANG_FORMAT=1 make
See
Dockerfile.clang
for the minimal required version of clang
.
You should now have a ./tetragon
binary, which can be run as follows:
sudo ./tetragon --bpf-lib bpf/objs
Notes:
The --bpf-lib
flag tells Tetragon where to look for its compiled BPF
programs (which were built in the make
step above).
If Tetragon fails with an error "BTF discovery: candidate btf file does not exist"
, then make sure that your kernel support BTF,
otherwise place a BTF file where Tetragon can read it and specify its path
with the --btf
flag.
Tetragon uses code generation based on protoc to generate large amounts of boilerplate code based on our protobuf API. We similarly use automatic generation to maintain our k8s CRDs. Whenever you make changes to these files, you will be required to re-run code generation before your PR can be accepted.
To run codegen from protoc, run the following command from the root of the repository:
make codegen
And to run k8s CRD generation, run the following command from the root of the repository:
make generate
Finally, should you wish to modify any of the resulting codegen files (ending
in .pb.go
), do not modify them directly. Instead, you can edit the files in
cmd/protoc-gen-go-tetragon
and then re-run make codegen
.
The base kernel should support BTF
or a BTF file should be bind mounted on top of /var/lib/tetragon/btf
inside
container.
To build Tetragon image:
make image
To run the image:
docker run --name tetragon \
--rm -it -d --pid=host \
--cgroupns=host --privileged \
-v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf \
cilium/tetragon:latest \
bash -c "/usr/bin/tetragon"
Run the tetra
binary to get Tetragon events:
docker exec -it tetragon \
bash -c "/usr/bin/tetra getevents -o compact"
To build Tetragon tarball:
make tarball
The scripts in contrib/localdev will help you run Tetragon locally in a kind cluster. First, ensure that docker, kind, kubectl, and helm are installed on your system. Then, run the following commands:
# Build Tetragon agent and operator images
make LOCAL_CLANG=0 image image-operator
# Bootstrap the cluster
contrib/localdev/bootstrap-kind-cluster.sh
# Install Tetragon
contrib/localdev/install-tetragon.sh --image cilium/tetragon:latest --operator cilium/tetragon-operator:latest
Verify that Tetragon is installed by running:
kubectl get pods -n kube-system
If you are on a Mac, use Vagrant to create a dev VM:
vagrant up
vagrant ssh
make
If you are getting an error, you can try to run sudo launchctl load /Library/LaunchDaemons/org.virtualbox.startup.plist
(from a Stackoverflow
answer).
You can also run the tetragon agent directly (instead of in a pod). Here we describe how this can be done in minikube:
minikube start --driver=kvm2
minikube mount $HOME:$HOME # so that we can use .kube/config
./tetragon-operator --kube-config ~/.kube/config
make STATIC=1 tetragon
minikube ssh -- 'sudo mkdir -p /var/run/cilium/tetragon'
minikube ssh sudo "sh -c 'NODE_NAME=minikube /home/kkourt/src/tetragon/tetragon --bpf-lib /home/kkourt/src/tetragon/bpf/objs --server-address unix:///var/run/cilium/tetragon/tetragon.sock --enable-k8s-api --k8s-kubeconfig-path /home/kkourt/.kube/config'"
Make sure the main branch of your fork is up-to-date:
git fetch upstream
git checkout main
git merge upstream/main
For further reference read github syncing a fork documentation.
Create a PR branch with a descriptive name, branching from main:
git switch -c pr/${GITHUB_USERNAME_OR_ORG}/changes-to-something main
Make the changes you want.
Separate the changes into logical commits.
(git commit -s)
.
See the section Developer’s Certificate of OriginMake sure your changes meet the following criteria:
Run git diff --check
to catch obvious white space violations
Build Tetragon with your changes included.
See https://github.com/cilium/tetragon/tree/main/tests/vmtests.
Contributions must be submitted in the form of pull requests against the upstream GitHub repository at https://github.com/cilium/tetragon.
Fork the Tetragon repository.
Push your changes to the topic branch in your fork of the repository.
Submit a pull request on https://github.com/cilium/tetragon.
Before hitting the submit button, please make sure that the following requirements have been met:
Each commit compiles and is functional on its own to allow for bisecting of commits.
All code is covered by unit and/or runtime tests where feasible.
All changes have been tested and checked for regressions by running the existing testsuite against your changes.
All commits contain a well written commit description including a title,
description and a Fixes: #XXX
line if the commit addresses a particular
GitHub issue identified by its number. Note that the GitHub issue will be
automatically closed when the commit is merged.
doc: add contribution guideline and how to submit pull requests
Tetragon Open Source project was just released and it does not include
default contributing guidelines.
This patch fixes this by adding:
1. CONTRIBUTING.md file in the root directory as suggested by github documentation: https://docs.github.com/en/communities/setting-up-your-project-for-healthy-contributions/setting-guidelines-for-repository-contributors
2. Development guide under docs directory with a section on how to submit pull requests.
3. Moves the DEVELOP.md file from root directory to the `docs/contributing/development/` one.
Fixes: #33
Signed-off-by: Djalal Harouni <djalal@cilium.io>
Note: Make sure to include a blank line in between commit title and commit description.
All commits are signed off. See the section Developer’s Certificate of Origin.
All important steps in Making changes have been followed.
To improve tracking of who did what, we’ve introduced a “sign-off” procedure, make sure to read and apply the Developer’s Certificate of Origin.
Tetragon - eBPF-based Security Observability and Runtime Enforcement
Usage:
tetragon [flags]
Flags:
--bpf-lib string Location of Tetragon libs (btf and bpf files) (default "/var/lib/tetragon/")
--btf string Location of btf
--config-dir string Configuration directory that contains a file for each option
--data-cache-size int Size of the data events cache (default 1024)
-d, --debug Enable debug messages. Equivalent to '--log-level=debug'
--disable-kprobe-multi Allow to disable kprobe multi interface
--enable-export-aggregation Enable JSON export aggregation
--enable-k8s-api Access Kubernetes API to associate Tetragon events with Kubernetes pods
--enable-msg-handling-latency Enable metrics for message handling latency
--enable-pid-set-filter Enable pidSet export filters. Not recommended for production use
--enable-pod-info Enable PodInfo custom resource
--enable-policy-filter Enable policy filter code (beta)
--enable-policy-filter-debug Enable policy filter debug messages
--enable-process-ancestors Include ancestors in process exec events (default true)
--enable-process-cred Enable process_cred events
--enable-process-ns Enable namespace information in process_exec and process_kprobe events
--event-queue-size uint Set the size of the internal event queue. (default 10000)
--export-aggregation-buffer-size uint Aggregator channel buffer size (default 10000)
--export-aggregation-window-size duration JSON export aggregation time window (default 15s)
--export-allowlist string JSON export allowlist
--export-denylist string JSON export denylist
--export-file-compress Compress rotated JSON export files
--export-file-max-backups int Number of rotated JSON export files to retain (default 5)
--export-file-max-size-mb int Size in MB for rotating JSON export files (default 10)
--export-file-perm string Access permissions on JSON export files (default "600")
--export-file-rotation-interval duration Interval at which to rotate JSON export files in addition to rotating them by size
--export-filename string Filename for JSON export. Disabled by default
--export-rate-limit int Rate limit (per minute) for event export. Set to -1 to disable (default -1)
--expose-kernel-addresses Expose real kernel addresses in events stack traces
--field-filters string Field filters for event exports
--force-large-progs Force loading large programs, even in kernels with < 5.3 versions
--force-small-progs Force loading small programs, even in kernels with >= 5.3 versions
--gops-address string gops server address (e.g. 'localhost:8118'). Disabled by default
-h, --help help for tetragon
--k8s-kubeconfig-path string Absolute path of the kubernetes kubeconfig file
--kernel string Kernel version
--kmods strings List of kernel modules to load symbols from
--log-format string Set log format (default "text")
--log-level string Set log level (default "info")
--metrics-label-filter string Comma-separated list of enabled metric labels. (e.g. "namespace,workload,pod,binary") By default all labels are enabled.
--metrics-server string Metrics server address (e.g. ':2112'). Disabled by default
--netns-dir string Network namespace dir (default "/var/run/docker/netns/")
--process-cache-size int Size of the process cache (default 65536)
--procfs string Location of procfs to consume existing PIDs (default "/proc/")
--rb-queue-size string Set size of channel between ring buffer and sensor go routines (default 65k, allows K/M/G suffix) (default "65535")
--rb-size string Set perf ring buffer size for single cpu (default 65k, allows K/M/G suffix) (default "0")
--rb-size-total string Set perf ring buffer size in total for all cpus (default 65k per cpu, allows K/M/G suffix) (default "0")
--release-pinned-bpf Release all pinned BPF programs and maps in Tetragon BPF directory. Enabled by default. Set to false to disable (default true)
--server-address string gRPC server address (e.g. 'localhost:54321' or 'unix:///var/run/tetragon/tetragon.sock' (default "localhost:54321")
--tracing-policy string Tracing policy file to load at startup
--tracing-policy-dir string Directory from where to load Tracing Policies (default "/etc/tetragon/tetragon.tp.d")
--verbose int set verbosity level for eBPF verifier dumps. Pass 0 for silent, 1 for truncated logs, 2 for a full dump
The Tetragon API is an independant Go module that can be found in the Tetragon repository under api. The version 1 of this API is defined in github.com/cilium/tetragon/api/v1/tetragon.
Name | Number | Description |
---|---|---|
CAP_CHOWN | 0 | In a system with the [_POSIX_CHOWN_RESTRICTED] option defined, this overrides the restriction of changing file ownership and group ownership. |
DAC_OVERRIDE | 1 | Override all DAC access, including ACL execute access if [_POSIX_ACL] is defined. Excluding DAC access covered by CAP_LINUX_IMMUTABLE. |
CAP_DAC_READ_SEARCH | 2 | Overrides all DAC restrictions regarding read and search on files and directories, including ACL restrictions if [_POSIX_ACL] is defined. Excluding DAC access covered by "$1"_LINUX_IMMUTABLE. |
CAP_FOWNER | 3 | Overrides all restrictions about allowed operations on files, where file owner ID must be equal to the user ID, except where CAP_FSETID is applicable. It doesn't override MAC and DAC restrictions. |
CAP_FSETID | 4 | Overrides the following restrictions that the effective user ID shall match the file owner ID when setting the S_ISUID and S_ISGID bits on that file; that the effective group ID (or one of the supplementary group IDs) shall match the file owner ID when setting the S_ISGID bit on that file; that the S_ISUID and S_ISGID bits are cleared on successful return from chown(2) (not implemented). |
CAP_KILL | 5 | Overrides the restriction that the real or effective user ID of a process sending a signal must match the real or effective user ID of the process receiving the signal. |
CAP_SETGID | 6 | Allows forged gids on socket credentials passing. |
CAP_SETUID | 7 | Allows forged pids on socket credentials passing. |
CAP_SETPCAP | 8 | Without VFS support for capabilities: Transfer any capability in your permitted set to any pid, remove any capability in your permitted set from any pid With VFS support for capabilities (neither of above, but) Add any capability from current's capability bounding set to the current process' inheritable set Allow taking bits out of capability bounding set Allow modification of the securebits for a process |
CAP_LINUX_IMMUTABLE | 9 | Allow modification of S_IMMUTABLE and S_APPEND file attributes |
CAP_NET_BIND_SERVICE | 10 | Allows binding to ATM VCIs below 32 |
CAP_NET_BROADCAST | 11 | Allow broadcasting, listen to multicast |
CAP_NET_ADMIN | 12 | Allow activation of ATM control sockets |
CAP_NET_RAW | 13 | Allow binding to any address for transparent proxying (also via NET_ADMIN) |
CAP_IPC_LOCK | 14 | Allow mlock and mlockall (which doesn't really have anything to do with IPC) |
CAP_IPC_OWNER | 15 | Override IPC ownership checks |
CAP_SYS_MODULE | 16 | Insert and remove kernel modules - modify kernel without limit |
CAP_SYS_RAWIO | 17 | Allow sending USB messages to any device via /dev/bus/usb |
CAP_SYS_CHROOT | 18 | Allow use of chroot() |
CAP_SYS_PTRACE | 19 | Allow ptrace() of any process |
CAP_SYS_PACCT | 20 | Allow configuration of process accounting |
CAP_SYS_ADMIN | 21 | Allow everything under CAP_BPF and CAP_PERFMON for backward compatibility |
CAP_SYS_BOOT | 22 | Allow use of reboot() |
CAP_SYS_NICE | 23 | Allow setting cpu affinity on other processes |
CAP_SYS_RESOURCE | 24 | Control memory reclaim behavior |
CAP_SYS_TIME | 25 | Allow setting the real-time clock |
CAP_SYS_TTY_CONFIG | 26 | Allow vhangup() of tty |
CAP_MKNOD | 27 | Allow the privileged aspects of mknod() |
CAP_LEASE | 28 | Allow taking of leases on files |
CAP_AUDIT_WRITE | 29 | Allow writing the audit log via unicast netlink socket |
CAP_AUDIT_CONTROL | 30 | Allow configuration of audit via unicast netlink socket |
CAP_SETFCAP | 31 | Set or remove capabilities on files |
CAP_MAC_OVERRIDE | 32 | Override MAC access. The base kernel enforces no MAC policy. An LSM may enforce a MAC policy, and if it does and it chooses to implement capability based overrides of that policy, this is the capability it should use to do so. |
CAP_MAC_ADMIN | 33 | Allow MAC configuration or state changes. The base kernel requires no MAC configuration. An LSM may enforce a MAC policy, and if it does and it chooses to implement capability based checks on modifications to that policy or the data required to maintain it, this is the capability it should use to do so. |
CAP_SYSLOG | 34 | Allow configuring the kernel's syslog (printk behaviour) |
CAP_WAKE_ALARM | 35 | Allow triggering something that will wake the system |
CAP_BLOCK_SUSPEND | 36 | Allow preventing system suspends |
CAP_AUDIT_READ | 37 | Allow reading the audit log via multicast netlink socket |
CAP_PERFMON | 38 | Allow system performance and observability privileged operations using perf_events, i915_perf and other kernel subsystems |
CAP_BPF | 39 | CAP_BPF allows the following BPF operations: - Creating all types of BPF maps - Advanced verifier features - Indirect variable access - Bounded loops - BPF to BPF function calls - Scalar precision tracking - Larger complexity limits - Dead code elimination - And potentially other features - Loading BPF Type Format (BTF) data - Retrieve xlated and JITed code of BPF programs - Use bpf_spin_lock() helper CAP_PERFMON relaxes the verifier checks further: - BPF progs can use of pointer-to-integer conversions - speculation attack hardening measures are bypassed - bpf_probe_read to read arbitrary kernel memory is allowed - bpf_trace_printk to print kernel memory is allowed CAP_SYS_ADMIN is required to use bpf_probe_write_user. CAP_SYS_ADMIN is required to iterate system wide loaded programs, maps, links, BTFs and convert their IDs to file descriptors. CAP_PERFMON and CAP_BPF are required to load tracing programs. CAP_NET_ADMIN and CAP_BPF are required to load networking programs. |
CAP_CHECKPOINT_RESTORE | 40 | Allow writing to ns_last_pid |
Name | Number | Description |
---|---|---|
SecBitNotSet | 0 | |
SecBitNoRoot | 1 | When set UID 0 has no special privileges. When unset, inheritance of root-permissions and suid-root executable under compatibility mode is supported. If the effective uid of the new process is 0 then the effective and inheritable bitmasks of the executable file is raised. If the real uid is 0, the effective (legacy) bit of the executable file is raised. |
SecBitNoRootLocked | 2 | Make bit-0 SecBitNoRoot immutable |
SecBitNoSetUidFixup | 4 | When set, setuid to/from uid 0 does not trigger capability-"fixup". When unset, to provide compatiblility with old programs relying on set*uid to gain/lose privilege, transitions to/from uid 0 cause capabilities to be gained/lost. |
SecBitNoSetUidFixupLocked | 8 | Make bit-2 SecBitNoSetUidFixup immutable |
SecBitKeepCaps | 16 | When set, a process can retain its capabilities even after transitioning to a non-root user (the set-uid fixup suppressed by bit 2). Bit-4 is cleared when a process calls exec(); setting both bit 4 and 5 will create a barrier through exec that no exec()'d child can use this feature again. |
SecBitKeepCapsLocked | 32 | Make bit-4 SecBitKeepCaps immutable |
SecBitNoCapAmbientRaise | 64 | When set, a process cannot add new capabilities to its ambient set. |
SecBitNoCapAmbientRaiseLocked | 128 | Make bit-6 SecBitNoCapAmbientRaise immutable |
Field | Type | Label | Description |
---|---|---|---|
setuid | google.protobuf.UInt32Value | If set then this is the set user ID used for execution | |
setgid | google.protobuf.UInt32Value | If set then this is the set group ID used for execution |
Field | Type | Label | Description |
---|---|---|---|
permitted | CapabilitiesType | repeated | Permitted set indicates what capabilities the process can use. This is a limiting superset for the effective capabilities that the thread may assume. It is also a limiting superset for the capabilities that may be added to the inheritable set by a thread without the CAP_SETPCAP in its effective set. |
effective | CapabilitiesType | repeated | Effective set indicates what capabilities are active in a process. This is the set used by the kernel to perform permission checks for the thread. |
inheritable | CapabilitiesType | repeated | Inheritable set indicates which capabilities will be inherited by the current process when running as a root user. |
Field | Type | Label | Description |
---|---|---|---|
id | string | Identifier of the container. | |
name | string | Name of the container. | |
image | Image | Image of the container. | |
start_time | google.protobuf.Timestamp | Start time of the container. | |
pid | google.protobuf.UInt32Value | Process identifier in the container namespace. | |
maybe_exec_probe | bool | If this is set true, it means that the process might have been originated from a Kubernetes exec probe. For this field to be true, the following must be true: 1. The binary field matches the first element of the exec command list for either liveness or readiness probe excluding the basename. For example, "/bin/ls" and "ls" are considered a match. 2. The arguments field exactly matches the rest of the exec command list. |
CreateContainer informs the agent that a container was created This is intented to be used by OCI hooks (but not limited to them) and corresponds to the CreateContainer hook: https://github.com/opencontainers/runtime-spec/blob/main/config.md#createcontainer-hooks.
Field | Type | Label | Description |
---|---|---|---|
cgroupsPath | string | cgroupsPath is the cgroups path for the container. The path is expected to be relative to the cgroups mountpoint. See: https://github.com/opencontainers/runtime-spec/blob/58ec43f9fc39e0db229b653ae98295bfde74aeab/specs-go/config.go#L174 | |
rootDir | string | rootDir is the absolute path of the root directory of the container. See: https://github.com/opencontainers/runtime-spec/blob/main/specs-go/config.go#L174 | |
annotations | CreateContainer.AnnotationsEntry | repeated | annotations are the run-time annotations for the container see https://github.com/opencontainers/runtime-spec/blob/main/config.md#annotations |
Field | Type | Label | Description |
---|---|---|---|
key | string | ||
value | string |
Field | Type | Label | Description |
---|---|---|---|
event_set | HealthStatusType | repeated |
Field | Type | Label | Description |
---|---|---|---|
health_status | HealthStatus | repeated |
Field | Type | Label | Description |
---|---|---|---|
event | HealthStatusType | ||
status | HealthStatusResult | ||
details | string |
Field | Type | Label | Description |
---|---|---|---|
id | string | Identifier of the container image composed of the registry path and the sha256. | |
name | string | Name of the container image composed of the registry path and the tag. |
Field | Type | Label | Description |
---|---|---|---|
name | string | Kernel module name | |
signature_ok | google.protobuf.BoolValue | If true the module signature was verified successfully. Depends on kernels compiled with CONFIG_MODULE_SIG option, for details please read: https://www.kernel.org/doc/Documentation/admin-guide/module-signing.rst | |
tainted | TaintedBitsType | repeated | The module tainted flags that will be applied on the kernel. For further details please read: https://docs.kernel.org/admin-guide/tainted-kernels.html |
Field | Type | Label | Description |
---|---|---|---|
string_arg | string | ||
int_arg | int32 | ||
skb_arg | KprobeSkb | ||
size_arg | uint64 | ||
bytes_arg | bytes | ||
path_arg | KprobePath | ||
file_arg | KprobeFile | ||
truncated_bytes_arg | KprobeTruncatedBytes | ||
sock_arg | KprobeSock | ||
cred_arg | KprobeCred | ||
long_arg | int64 | ||
bpf_attr_arg | KprobeBpfAttr | ||
perf_event_arg | KprobePerfEvent | ||
bpf_map_arg | KprobeBpfMap | ||
uint_arg | uint32 | ||
user_namespace_arg | KprobeUserNamespace | Deprecated. | |
capability_arg | KprobeCapability | ||
process_credentials_arg | ProcessCredentials | ||
user_ns_arg | UserNamespace | ||
module_arg | KernelModule | ||
label | string |
Field | Type | Label | Description |
---|---|---|---|
ProgType | string | ||
InsnCnt | uint32 | ||
ProgName | string |
Field | Type | Label | Description |
---|---|---|---|
MapType | string | ||
KeySize | uint32 | ||
ValueSize | uint32 | ||
MaxEntries | uint32 | ||
MapName | string |
Field | Type | Label | Description |
---|---|---|---|
value | google.protobuf.Int32Value | ||
name | string |
Field | Type | Label | Description |
---|---|---|---|
permitted | CapabilitiesType | repeated | |
effective | CapabilitiesType | repeated | |
inheritable | CapabilitiesType | repeated |
Field | Type | Label | Description |
---|---|---|---|
mount | string | ||
path | string | ||
flags | string |
Field | Type | Label | Description |
---|---|---|---|
mount | string | ||
path | string | ||
flags | string |
Field | Type | Label | Description |
---|---|---|---|
KprobeFunc | string | ||
Type | string | ||
Config | uint64 | ||
ProbeOffset | uint64 |
Field | Type | Label | Description |
---|---|---|---|
hash | uint32 | ||
len | uint32 | ||
priority | uint32 | ||
mark | uint32 | ||
saddr | string | ||
daddr | string | ||
sport | uint32 | ||
dport | uint32 | ||
proto | uint32 | ||
sec_path_len | uint32 | ||
sec_path_olen | uint32 | ||
protocol | string | ||
family | string |
Field | Type | Label | Description |
---|---|---|---|
family | string | ||
type | string | ||
protocol | string | ||
mark | uint32 | ||
priority | uint32 | ||
saddr | string | ||
daddr | string | ||
sport | uint32 | ||
dport | uint32 | ||
cookie | uint64 | ||
state | string |
Field | Type | Label | Description |
---|---|---|---|
bytes_arg | bytes | ||
orig_size | uint64 |
Field | Type | Label | Description |
---|---|---|---|
level | google.protobuf.Int32Value | ||
owner | google.protobuf.UInt32Value | ||
group | google.protobuf.UInt32Value | ||
ns | Namespace |
Field | Type | Label | Description |
---|---|---|---|
inum | uint32 | Inode number of the namespace. | |
is_host | bool | Indicates if namespace belongs to host. |
Field | Type | Label | Description |
---|---|---|---|
uts | Namespace | Hostname and NIS domain name. | |
ipc | Namespace | System V IPC, POSIX message queues. | |
mnt | Namespace | Mount points. | |
pid | Namespace | Process IDs. | |
pid_for_children | Namespace | Process IDs for children processes. | |
net | Namespace | Network devices, stacks, ports, etc. | |
time | Namespace | Boot and monotonic clocks. | |
time_for_children | Namespace | Boot and monotonic clocks for children processes. | |
cgroup | Namespace | Cgroup root directory. | |
user | Namespace | User and group IDs. |
Field | Type | Label | Description |
---|---|---|---|
namespace | string | Kubernetes namespace of the Pod. | |
name | string | Name of the Pod. | |
labels | string | repeated | Deprecated. Cilium identity labels of the Pod. This field is deprecated. Use pod_labels instead. |
container | Container | Container of the Pod from which the process that triggered the event originates. | |
pod_labels | Pod.PodLabelsEntry | repeated | Contains all the labels of the pod. Note that the labels field contains Cilium identity labels, which is a subset of pod labels. |
workload | string | Kubernetes workload of the Pod. | |
workload_kind | string | Kubernetes workload kind (e.g. "Deployment", "DaemonSet") of the Pod. |
Field | Type | Label | Description |
---|---|---|---|
key | string | ||
value | string |
Field | Type | Label | Description |
---|---|---|---|
exec_id | string | Exec ID uniquely identifies the process over time across all the nodes in the cluster. | |
pid | google.protobuf.UInt32Value | Process identifier from host PID namespace. | |
uid | google.protobuf.UInt32Value | User identifier associated with the process. | |
cwd | string | Current working directory of the process. | |
binary | string | Absolute path of the executed binary. | |
arguments | string | Arguments passed to the binary at execution. | |
flags | string | Flags are for debugging purposes only and should not be considered a reliable source of information. They hold various information about which syscalls generated events, use of internal Tetragon buffers, errors and more. - execve This event is generated by an execve syscall for a new process. See procFs for the other option. A correctly formatted event should either set execve or procFS (described next). - procFS This event is generated from a proc interface. This happens at Tetragon init when existing processes are being loaded into Tetragon event buffer. All events should have either execve or procFS set. - truncFilename Indicates a truncated processes filename because the buffer size is too small to contain the process filename. Consider increasing buffer size to avoid this. - truncArgs Indicates truncated the processes arguments because the buffer size was too small to contain all exec args. Consider increasing buffer size to avoid this. - taskWalk Primarily useful for debugging. Indicates a walked process hierarchy to find a parent process in the Tetragon buffer. This may happen when we did not receive an exec event for the immediate parent of a process. Typically means we are looking at a fork that in turn did another fork we don't currently track fork events exactly and instead push an event with the original parent exec data. This flag can provide this insight into the event if needed. - miss An error flag indicating we could not find parent info in the Tetragon event buffer. If this is set it should be reported to Tetragon developers for debugging. Tetragon will do its best to recover information about the process from available kernel data structures instead of using cached info in this case. However, args will not be available. - needsAUID An internal flag for Tetragon to indicate the audit has not yet been resolved. The BPF hooks look at this flag to determine if probing the audit system is necessary. - errorFilename An error flag indicating an error happened while reading the filename. If this is set it should be reported to Tetragon developers for debugging. - errorArgs An error flag indicating an error happened while reading the process args. If this is set it should be reported to Tetragon developers for debugging - needsCWD An internal flag for Tetragon to indicate the current working directory has not yet been resolved. The Tetragon hooks look at this flag to determine if probing the CWD is necessary. - noCWDSupport Indicates that CWD is removed from the event because the buffer size is too small. Consider increasing buffer size to avoid this. - rootCWD Indicates that CWD is the root directory. This is necessary to inform readers the CWD is not in the event buffer and is '/' instead. - errorCWD An error flag indicating an error occurred while reading the CWD of a process. If this is set it should be reported to Tetragon developers for debugging. - clone Indicates the process issued a clone before exec*. This is the general flow to exec* a new process, however its possible to replace the current process with a new process by doing an exec* without a clone. In this case the flag will be omitted and the same PID will be used by the kernel for both the old process and the newly exec'd process. | |
start_time | google.protobuf.Timestamp | Start time of the execution. | |
auid | google.protobuf.UInt32Value | Audit user ID, this ID is assigned to a user upon login and is inherited by every process even when the user's identity changes. For example, by switching user accounts with su - john. | |
pod | Pod | Information about the the Kubernetes Pod where the event originated. | |
docker | string | The 15 first digits of the container ID. | |
parent_exec_id | string | Exec ID of the parent process. | |
refcnt | uint32 | Reference counter from the Tetragon process cache. | |
cap | Capabilities | Set of capabilities that define the permissions the process can execute with. | |
ns | Namespaces | Linux namespaces of the process, disabled by default, can be enabled by the --enable-process-ns flag. | |
tid | google.protobuf.UInt32Value | Thread ID, note that for the thread group leader, tid is equal to pid. | |
process_credentials | ProcessCredentials | Process credentials | |
binary_properties | BinaryProperties | Executed binary properties. This field is only available on ProcessExec events. |
Field | Type | Label | Description |
---|---|---|---|
uid | google.protobuf.UInt32Value | The real user ID | |
gid | google.protobuf.UInt32Value | The real group ID | |
euid | google.protobuf.UInt32Value | The effective user ID | |
egid | google.protobuf.UInt32Value | The effective group ID | |
suid | google.protobuf.UInt32Value | The saved user ID | |
sgid | google.protobuf.UInt32Value | The saved group ID | |
fsuid | google.protobuf.UInt32Value | the filesystem user ID | |
fsgid | google.protobuf.UInt32Value | The filesystem group ID | |
securebits | SecureBitsType | repeated | Secure management flags |
caps | Capabilities | Set of capabilities that define the permissions the process can execute with. | |
user_ns | UserNamespace | User namespace where the UIDs, GIDs and capabilities are relative to. |
Field | Type | Label | Description |
---|---|---|---|
process | Process | Process that triggered the exec. | |
parent | Process | Immediate parent of the process. | |
ancestors | Process | repeated | Ancestors of the process beyond the immediate parent. |
Field | Type | Label | Description |
---|---|---|---|
process | Process | Process that triggered the exit. | |
parent | Process | Immediate parent of the process. | |
signal | string | Signal that the process received when it exited, for example SIGKILL or SIGTERM (list all signal names with kill -l ). If there is no signal handler implemented for a specific process, we report the exit status code that can be found in the status field. | |
status | uint32 | Status code on process exit. For example, the status code can indicate if an error was encountered or the program exited successfully. | |
time | google.protobuf.Timestamp | Date and time of the event. |
Field | Type | Label | Description |
---|---|---|---|
process | Process | Process that triggered the kprobe. | |
parent | Process | Immediate parent of the process. | |
function_name | string | Symbol on which the kprobe was attached. | |
args | KprobeArgument | repeated | Arguments definition of the observed kprobe. |
return | KprobeArgument | Return value definition of the observed kprobe. | |
action | KprobeAction | Action performed when the kprobe matched. | |
stack_trace | StackTraceEntry | repeated | Kernel stack trace to the call. |
policy_name | string | Name of the Tracing Policy that created that kprobe. |
loader sensor event triggered for loaded binary/library
Field | Type | Label | Description |
---|---|---|---|
process | Process | ||
path | string | ||
buildid | bytes |
Field | Type | Label | Description |
---|---|---|---|
process | Process | Process that triggered the tracepoint. | |
parent | Process | Immediate parent of the process. | |
subsys | string | Subsystem of the tracepoint. | |
event | string | Event of the subsystem. | |
args | KprobeArgument | repeated | Arguments definition of the observed tracepoint. TODO: once we implement all we want, rename KprobeArgument to GenericArgument |
policy_name | string | Name of the policy that created that tracepoint. | |
action | KprobeAction | Action performed when the tracepoint matched. |
Field | Type | Label | Description |
---|---|---|---|
process | Process | ||
parent | Process | ||
path | string | ||
symbol | string | ||
policy_name | string | Name of the policy that created that uprobe. |
RuntimeHookRequest synchronously propagates information to the agent about run-time state.
Field | Type | Label | Description |
---|---|---|---|
createContainer | CreateContainer |
Field | Type | Label | Description |
---|---|---|---|
address | uint64 | address is the kernel function address. | |
offset | uint64 | offset is the offset into the native instructions for the function. | |
symbol | string | symbol is the symbol name of the function. |
Field | Type | Label | Description |
---|---|---|---|
arg0 | uint64 | ||
arg1 | uint64 | ||
arg2 | uint64 | ||
arg3 | uint64 |
Field | Type | Label | Description |
---|---|---|---|
level | google.protobuf.Int32Value | Nested level of the user namespace. Init or host user namespace is at level 0. | |
uid | google.protobuf.UInt32Value | The owner user ID of the namespace | |
gid | google.protobuf.UInt32Value | The owner group ID of the namepace. | |
ns | Namespace | The user namespace details that include the inode number of the namespace. |
Name | Number | Description |
---|---|---|
HEALTH_STATUS_UNDEF | 0 | |
HEALTH_STATUS_RUNNING | 1 | |
HEALTH_STATUS_STOPPED | 2 | |
HEALTH_STATUS_ERROR | 3 |
Name | Number | Description |
---|---|---|
HEALTH_STATUS_TYPE_UNDEF | 0 | |
HEALTH_STATUS_TYPE_STATUS | 1 |
Name | Number | Description |
---|---|---|
KPROBE_ACTION_UNKNOWN | 0 | Unknown action |
KPROBE_ACTION_POST | 1 | Post action creates an event (default action). |
KPROBE_ACTION_FOLLOWFD | 2 | Post action creates a mapping between file descriptors and file names. |
KPROBE_ACTION_SIGKILL | 3 | Sigkill action synchronously terminates the process. |
KPROBE_ACTION_UNFOLLOWFD | 4 | Post action removes a mapping between file descriptors and file names. |
KPROBE_ACTION_OVERRIDE | 5 | Override action modifies the return value of the call. |
KPROBE_ACTION_COPYFD | 6 | Post action dupplicates a mapping between file descriptors and file names. |
KPROBE_ACTION_GETURL | 7 | GetURL action issue an HTTP Get request against an URL from userspace. |
KPROBE_ACTION_DNSLOOKUP | 8 | GetURL action issue a DNS lookup against an URL from userspace. |
KPROBE_ACTION_NOPOST | 9 | NoPost action suppresses the transmission of the event to userspace. |
KPROBE_ACTION_SIGNAL | 10 | Signal action sends specified signal to the process. |
KPROBE_ACTION_TRACKSOCK | 11 | TrackSock action tracks socket. |
KPROBE_ACTION_UNTRACKSOCK | 12 | UntrackSock action un-tracks socket. |
KPROBE_ACTION_NOTIFYKILLER | 13 | NotifyKiller action notifies killer sensor. |
Tainted bits to indicate if the kernel was tainted. For further details: https://docs.kernel.org/admin-guide/tainted-kernels.html
Name | Number | Description |
---|---|---|
TAINT_UNSET | 0 | |
TAINT_PROPRIETARY_MODULE | 1 | A proprietary module was loaded. |
TAINT_FORCED_MODULE | 2 | A module was force loaded. |
TAINT_FORCED_UNLOAD_MODULE | 4 | A module was force unloaded. |
TAINT_STAGED_MODULE | 1024 | A staging driver was loaded. |
TAINT_OUT_OF_TREE_MODULE | 4096 | An out of tree module was loaded. |
TAINT_UNSIGNED_MODULE | 8192 | An unsigned module was loaded. Supported only on kernels built with CONFIG_MODULE_SIG option. |
TAINT_KERNEL_LIVE_PATCH_MODULE | 32768 | The kernel has been live patched. |
TAINT_TEST_MODULE | 262144 | Loading a test module. |
AggregationInfo contains information about aggregation results.
Field | Type | Label | Description |
---|---|---|---|
count | uint64 | Total count of events in this aggregation time window. |
AggregationOptions defines configuration options for aggregating events.
Field | Type | Label | Description |
---|---|---|---|
window_size | google.protobuf.Duration | Aggregation window size. Defaults to 15 seconds if this field is not set. | |
channel_buffer_size | uint64 | Size of the buffer for the aggregator to receive incoming events. If the buffer becomes full, the aggregator will log a warning and start dropping incoming events. |
Field | Type | Label | Description |
---|---|---|---|
event_set | EventType | repeated | Event types to filter or undefined to filter over all event types. |
fields | google.protobuf.FieldMask | Fields to include or exclude. | |
action | FieldFilterAction | Whether to include or exclude fields. | |
invert_event_set | google.protobuf.BoolValue | Whether or not the event set filter should be inverted. |
Field | Type | Label | Description |
---|---|---|---|
binary_regex | string | repeated | |
namespace | string | repeated | |
health_check | google.protobuf.BoolValue | ||
pid | uint32 | repeated | |
pid_set | uint32 | repeated | Filter by the PID of a process and any of its descendants. Note that this filter is intended for testing and development purposes only and should not be used in production. In particular, PID cycling in the OS over longer periods of time may cause unexpected events to pass this filter. |
event_set | EventType | repeated | |
pod_regex | string | repeated | Filter by process.pod.name field using RE2 regular expression syntax: https://github.com/google/re2/wiki/Syntax |
arguments_regex | string | repeated | Filter by process.arguments field using RE2 regular expression syntax: https://github.com/google/re2/wiki/Syntax |
labels | string | repeated | Filter events by pod labels using Kubernetes label selector syntax: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors Note that this filter never matches events without the pod field (i.e. host process events). |
Field | Type | Label | Description |
---|---|---|---|
allow_list | Filter | repeated | allow_list specifies a list of filters to apply to only return certain events. If multiple filters are specified, at least one of them has to match for an event to be included in the results. |
deny_list | Filter | repeated | deny_list specifies a list of filters to apply to exclude certain events from the results. If multiple filters are specified, at least one of them has to match for an event to be excluded. If both allow_list and deny_list are specified, the results contain the set difference allow_list - deny_list. |
aggregation_options | AggregationOptions | aggregation_options configures aggregation options for this request. If this field is not set, responses will not be aggregated. Note that currently only process_accept and process_connect events are aggregated. Other events remain unaggregated. | |
field_filters | FieldFilter | repeated | Fields to include or exclude for events in the GetEventsResponse. Omitting this field implies that all fields will be included. Exclusion always takes precedence over inclusion in the case of conflicts. |
Field | Type | Label | Description |
---|---|---|---|
process_exec | ProcessExec | ProcessExec event includes information about the execution of binaries and other related process metadata. | |
process_exit | ProcessExit | ProcessExit event indicates how and when a process terminates. | |
process_kprobe | ProcessKprobe | ProcessKprobe event contains information about the pre-defined functions and the process that invoked them. | |
process_tracepoint | ProcessTracepoint | ProcessTracepoint contains information about the pre-defined tracepoint and the process that invoked them. | |
process_loader | ProcessLoader | ||
process_uprobe | ProcessUprobe | ||
test | Test | ||
rate_limit_info | RateLimitInfo | ||
node_name | string | Name of the node where this event was observed. | |
time | google.protobuf.Timestamp | Timestamp at which this event was observed. For an aggregated response, this field to set to the timestamp at which the event was observed for the first time in a given aggregation time window. | |
aggregation_info | AggregationInfo | aggregation_info contains information about aggregation results. This field is set only for aggregated responses. |
Field | Type | Label | Description |
---|---|---|---|
number_of_dropped_process_events | uint64 |
Represents the type of a Tetragon event.
NOTE: EventType constants must be in sync with the numbers used in the GetEventsResponse event oneof.
Name | Number | Description |
---|---|---|
UNDEF | 0 | |
PROCESS_EXEC | 1 | |
PROCESS_EXIT | 5 | |
PROCESS_KPROBE | 9 | |
PROCESS_TRACEPOINT | 10 | |
PROCESS_LOADER | 11 | |
PROCESS_UPROBE | 12 | |
TEST | 40000 | |
RATE_LIMIT_INFO | 40001 |
Determins the behaviour of a field filter
Name | Number | Description |
---|---|---|
INCLUDE | 0 | |
EXCLUDE | 1 |
Field | Type | Label | Description |
---|---|---|---|
address | uint64 | ||
symbol | string |
Field | Type | Label | Description |
---|---|---|---|
addresses | StackAddress | repeated |
Field | Type | Label | Description |
---|---|---|---|
key | string | ||
count | uint64 |
Field | Type | Label | Description |
---|---|---|---|
address | StackAddress | ||
count | uint64 | ||
labels | StackTraceLabel | repeated | |
children | StackTraceNode | repeated |
Field | Type | Label | Description |
---|---|---|---|
yaml | string |
Field | Type | Label | Description |
---|---|---|---|
name | string |
Field | Type | Label | Description |
---|---|---|---|
name | string |
Field | Type | Label | Description |
---|---|---|---|
name | string |
Field | Type | Label | Description |
---|---|---|---|
name | string |
Field | Type | Label | Description |
---|---|---|---|
name | string |
Field | Type | Label | Description |
---|---|---|---|
name | string |
Field | Type | Label | Description |
---|---|---|---|
root | StackTraceNode |
Field | Type | Label | Description |
---|---|---|---|
version | string |
Field | Type | Label | Description |
---|---|---|---|
sensors | SensorStatus | repeated |
Field | Type | Label | Description |
---|---|---|---|
policies | TracingPolicyStatus | repeated |
Field | Type | Label | Description |
---|---|---|---|
name | string |
Field | Type | Label | Description |
---|---|---|---|
name | string | name is the name of the sensor | |
enabled | bool | enabled marks whether the sensor is enabled | |
collection | string | collection is the collection the sensor belongs to (typically a tracing policy) |
Field | Type | Label | Description |
---|---|---|---|
id | uint64 | id is the id of the policy | |
name | string | name is the name of the policy | |
namespace | string | namespace is the namespace of the policy (or empty of the policy is global) | |
info | string | info is additional information about the policy | |
sensors | string | repeated | sensors loaded in the scope of this policy |
enabled | bool | indicating if the policy is enabled | |
filter_id | uint64 | filter ID of the policy used for k8s filtering | |
error | string | potential error of the policy |
Method Name | Request Type | Response Type | Description |
---|---|---|---|
GetEvents | GetEventsRequest | GetEventsResponse stream | |
GetHealth | GetHealthStatusRequest | GetHealthStatusResponse | |
AddTracingPolicy | AddTracingPolicyRequest | AddTracingPolicyResponse | |
DeleteTracingPolicy | DeleteTracingPolicyRequest | DeleteTracingPolicyResponse | |
RemoveSensor | RemoveSensorRequest | RemoveSensorResponse | |
ListTracingPolicies | ListTracingPoliciesRequest | ListTracingPoliciesResponse | |
EnableTracingPolicy | EnableTracingPolicyRequest | EnableTracingPolicyResponse | |
DisableTracingPolicy | DisableTracingPolicyRequest | DisableTracingPolicyResponse | |
ListSensors | ListSensorsRequest | ListSensorsResponse | |
EnableSensor | EnableSensorRequest | EnableSensorResponse | |
DisableSensor | DisableSensorRequest | DisableSensorResponse | |
GetStackTraceTree | GetStackTraceTreeRequest | GetStackTraceTreeResponse | |
GetVersion | GetVersionRequest | GetVersionResponse | |
RuntimeHook | RuntimeHookRequest | RuntimeHookResponse |
.proto Type | Notes | C++ | Java | Python | Go | C# | PHP | Ruby |
---|---|---|---|---|---|---|---|---|
double | double | double | float | float64 | double | float | Float | |
float | float | float | float | float32 | float | float | Float | |
int32 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) |
int64 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. | int64 | long | int/long | int64 | long | integer/string | Bignum |
uint32 | Uses variable-length encoding. | uint32 | int | int/long | uint32 | uint | integer | Bignum or Fixnum (as required) |
uint64 | Uses variable-length encoding. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum or Fixnum (as required) |
sint32 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) |
sint64 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. | int64 | long | int/long | int64 | long | integer/string | Bignum |
fixed32 | Always four bytes. More efficient than uint32 if values are often greater than 2^28. | uint32 | int | int | uint32 | uint | integer | Bignum or Fixnum (as required) |
fixed64 | Always eight bytes. More efficient than uint64 if values are often greater than 2^56. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum |
sfixed32 | Always four bytes. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) |
sfixed64 | Always eight bytes. | int64 | long | int/long | int64 | long | integer/string | Bignum |
bool | bool | boolean | boolean | bool | bool | boolean | TrueClass/FalseClass | |
string | A string must always contain UTF-8 encoded or 7-bit ASCII text. | string | String | str/unicode | string | string | string | String (UTF-8) |
bytes | May contain any arbitrary sequence of bytes. | string | ByteString | str | []byte | ByteString | string | String (ASCII-8BIT) |
The Tetragon Helm chart source is available under github.io/cilium/tetragon/install/kubernetes and is distributed from the Cilium helm charts repository helm.cilium.io.
To deploy Tetragon using this Helm chart you can run the following commands:
helm repo add cilium https://helm.cilium.io
helm repo update
helm install tetragon cilium/tetragon -n kube-system
To use the values available, with helm install
or helm upgrade
, use --set key=value
.
Key | Type | Default | Description |
---|---|---|---|
affinity | object | {} | |
daemonSetAnnotations | object | {} | |
daemonSetLabelsOverride | object | {} | |
dnsPolicy | string | "Default" | |
enabled | bool | true | Global settings |
export | object | {"filenames":["tetragon.log"],"mode":"stdout","resources":{},"securityContext":{},"stdout":{"argsOverride":[],"commandOverride":[],"enabledArgs":true,"enabledCommand":true,"extraEnv":[],"extraVolumeMounts":[],"image":{"override":null,"repository":"quay.io/cilium/hubble-export-stdout","tag":"v1.0.3"}}} | Tetragon event settings |
exportDirectory | string | "/var/run/cilium/tetragon" | |
exportFileCreationInterval | string | "120s" | |
extraConfigmapMounts | list | [] | |
extraHostPathMounts | list | [] | |
extraVolumes | list | [] | |
hostNetwork | bool | true | |
imagePullPolicy | string | "IfNotPresent" | |
imagePullSecrets | list | [] | |
nodeSelector | object | {} | |
podAnnotations | object | {} | |
podLabels | object | {} | |
podLabelsOverride | object | {} | |
podSecurityContext | object | {} | |
priorityClassName | string | "" | Tetragon agent settings |
selectorLabelsOverride | object | {} | |
serviceAccount.annotations | object | {} | |
serviceAccount.create | bool | true | |
serviceAccount.name | string | "" | |
serviceLabelsOverride | object | {} | |
tetragon.argsOverride | list | [] | |
tetragon.btf | string | "" | |
tetragon.commandOverride | list | [] | |
tetragon.enableK8sAPI | bool | true | |
tetragon.enableMsgHandlingLatency | bool | false | Enable latency monitoring in message handling |
tetragon.enablePolicyFilter | bool | true | Enable policy filter. This is required for K8s namespace and pod-label filtering. |
tetragon.enablePolicyFilterDebug | bool | false | Enable policy filter debug messages. |
tetragon.enableProcessCred | bool | false | |
tetragon.enableProcessNs | bool | false | |
tetragon.enabled | bool | true | |
tetragon.exportAllowList | string | "{\"event_set\":[\"PROCESS_EXEC\", \"PROCESS_EXIT\", \"PROCESS_KPROBE\", \"PROCESS_UPROBE\", \"PROCESS_TRACEPOINT\"]}" | |
tetragon.exportDenyList | string | "{\"health_check\":true}\n{\"namespace\":[\"\", \"cilium\", \"kube-system\"]}" | |
tetragon.exportFileCompress | bool | false | |
tetragon.exportFileMaxBackups | int | 5 | |
tetragon.exportFileMaxSizeMB | int | 10 | |
tetragon.exportFilePerm | string | "600" | |
tetragon.exportFilename | string | "tetragon.log" | |
tetragon.exportRateLimit | int | -1 | |
tetragon.extraArgs | object | {} | |
tetragon.extraEnv | list | [] | |
tetragon.extraVolumeMounts | list | [] | |
tetragon.fieldFilters | string | "{}" | |
tetragon.gops.address | string | "localhost" | The address at which to expose gops. |
tetragon.gops.port | int | 8118 | The port at which to expose gops. |
tetragon.grpc.address | string | "localhost:54321" | The address at which to expose gRPC. Examples: localhost:54321, unix:///var/run/tetragon/tetragon.sock |
tetragon.grpc.enabled | bool | true | Whether to enable exposing Tetragon gRPC. |
tetragon.hostProcPath | string | "/proc" | Location of the host proc filesystem in the runtime environment. If the runtime runs in the host, the path is /proc. Exceptions to this are environments like kind, where the runtime itself does not run on the host. |
tetragon.image.override | string | nil | |
tetragon.image.repository | string | "quay.io/cilium/tetragon" | |
tetragon.image.tag | string | "v1.0.0" | |
tetragon.processCacheSize | int | 65536 | |
tetragon.prometheus.address | string | "" | The address at which to expose metrics. Set it to "" to expose on all available interfaces. |
tetragon.prometheus.enabled | bool | true | Whether to enable exposing Tetragon metrics. |
tetragon.prometheus.metricsLabelFilter | string | "namespace,workload,pod,binary" | The labels to include with supporting metrics. The possible values are “namespace”, “workload”, “pod” and “binary”. |
tetragon.prometheus.port | int | 2112 | The port at which to expose metrics. |
tetragon.prometheus.serviceMonitor.enabled | bool | false | Whether to create a ‘ServiceMonitor’ resource targeting the tetragon pods. |
tetragon.prometheus.serviceMonitor.labelsOverride | object | {} | The set of labels to place on the ‘ServiceMonitor’ resource. |
tetragon.prometheus.serviceMonitor.scrapeInterval | string | "10s" | Interval at which metrics should be scraped. If not specified, Prometheus’ global scrape interval is used. |
tetragon.resources | object | {} | |
tetragon.securityContext.privileged | bool | true | |
tetragonOperator | object | {"affinity":{},"annotations":{},"extraLabels":{},"extraPodLabels":{},"extraVolumeMounts":[],"extraVolumes":[],"image":{"override":null,"pullPolicy":"IfNotPresent","repository":"quay.io/cilium/tetragon-operator","tag":"v1.0.0"},"nodeSelector":{},"podAnnotations":{},"podInfo":{"enabled":false},"podSecurityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}},"priorityClassName":"","resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"10m","memory":"64Mi"}},"securityContext":{},"serviceAccount":{"annotations":{},"create":true,"name":""},"skipCRDCreation":false,"strategy":{},"tolerations":[{"operator":"Exists"}]} | Tetragon Operator settings |
tetragonOperator.annotations | object | {} | Annotations for the Tetragon Operator Deployment. |
tetragonOperator.extraLabels | object | {} | Extra labels to be added on the Tetragon Operator Deployment. |
tetragonOperator.extraPodLabels | object | {} | Extra labels to be added on the Tetragon Operator Deployment Pods. |
tetragonOperator.extraVolumes | list | [] | Extra volumes for the Tetragon Operator Deployment. |
tetragonOperator.image | object | {"override":null,"pullPolicy":"IfNotPresent","repository":"quay.io/cilium/tetragon-operator","tag":"v1.0.0"} | tetragon-operator image. |
tetragonOperator.nodeSelector | object | {} | Steer the Tetragon Operator Deployment Pod placement via nodeSelector, tolerations and affinity rules. |
tetragonOperator.podAnnotations | object | {} | Annotations for the Tetragon Operator Deployment Pods. |
tetragonOperator.podInfo.enabled | bool | false | Enables the PodInfo CRD and the controller that reconciles PodInfo custom resources. |
tetragonOperator.podSecurityContext | object | {"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}} | securityContext for the Tetragon Operator Deployment Pod container. |
tetragonOperator.priorityClassName | string | "" | priorityClassName for the Tetragon Operator Deployment Pods. |
tetragonOperator.resources | object | {"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"10m","memory":"64Mi"}} | resources for the Tetragon Operator Deployment Pod container. |
tetragonOperator.securityContext | object | {} | securityContext for the Tetragon Operator Deployment Pods. |
tetragonOperator.serviceAccount | object | {"annotations":{},"create":true,"name":""} | tetragon-operator service account. |
tetragonOperator.strategy | object | {} | resources for the Tetragon Operator Deployment update strategy |
tolerations[0].operator | string | "Exists" | |
updateStrategy | object | {} |
Tetragon default controlling settings are set during compilation, so configuration is only needed when it is necessary to deviate from those defaults. This document lists those controlling settings and how they can be set as a CLI arguments or as configuration options from YAML files.
Tetragon CLI arguments:
Tetragon - eBPF-based Security Observability and Runtime Enforcement
Usage:
tetragon [flags]
Flags:
--bpf-lib string Location of Tetragon libs (btf and bpf files) (default "/var/lib/tetragon/")
--btf string Location of btf
--config-dir string Configuration directory that contains a file for each option
--data-cache-size int Size of the data events cache (default 1024)
-d, --debug Enable debug messages. Equivalent to '--log-level=debug'
--disable-kprobe-multi Allow to disable kprobe multi interface
--enable-export-aggregation Enable JSON export aggregation
--enable-k8s-api Access Kubernetes API to associate Tetragon events with Kubernetes pods
--enable-msg-handling-latency Enable metrics for message handling latency
--enable-pid-set-filter Enable pidSet export filters. Not recommended for production use
--enable-pod-info Enable PodInfo custom resource
--enable-policy-filter Enable policy filter code (beta)
--enable-policy-filter-debug Enable policy filter debug messages
--enable-process-ancestors Include ancestors in process exec events (default true)
--enable-process-cred Enable process_cred events
--enable-process-ns Enable namespace information in process_exec and process_kprobe events
--event-queue-size uint Set the size of the internal event queue. (default 10000)
--export-aggregation-buffer-size uint Aggregator channel buffer size (default 10000)
--export-aggregation-window-size duration JSON export aggregation time window (default 15s)
--export-allowlist string JSON export allowlist
--export-denylist string JSON export denylist
--export-file-compress Compress rotated JSON export files
--export-file-max-backups int Number of rotated JSON export files to retain (default 5)
--export-file-max-size-mb int Size in MB for rotating JSON export files (default 10)
--export-file-perm string Access permissions on JSON export files (default "600")
--export-file-rotation-interval duration Interval at which to rotate JSON export files in addition to rotating them by size
--export-filename string Filename for JSON export. Disabled by default
--export-rate-limit int Rate limit (per minute) for event export. Set to -1 to disable (default -1)
--expose-kernel-addresses Expose real kernel addresses in events stack traces
--field-filters string Field filters for event exports
--force-large-progs Force loading large programs, even in kernels with < 5.3 versions
--force-small-progs Force loading small programs, even in kernels with >= 5.3 versions
--gops-address string gops server address (e.g. 'localhost:8118'). Disabled by default
-h, --help help for tetragon
--k8s-kubeconfig-path string Absolute path of the kubernetes kubeconfig file
--kernel string Kernel version
--kmods strings List of kernel modules to load symbols from
--log-format string Set log format (default "text")
--log-level string Set log level (default "info")
--metrics-label-filter string Comma-separated list of enabled metric labels. (e.g. "namespace,workload,pod,binary") By default all labels are enabled.
--metrics-server string Metrics server address (e.g. ':2112'). Disabled by default
--netns-dir string Network namespace dir (default "/var/run/docker/netns/")
--process-cache-size int Size of the process cache (default 65536)
--procfs string Location of procfs to consume existing PIDs (default "/proc/")
--rb-queue-size string Set size of channel between ring buffer and sensor go routines (default 65k, allows K/M/G suffix) (default "65535")
--rb-size string Set perf ring buffer size for single cpu (default 65k, allows K/M/G suffix) (default "0")
--rb-size-total string Set perf ring buffer size in total for all cpus (default 65k per cpu, allows K/M/G suffix) (default "0")
--release-pinned-bpf Release all pinned BPF programs and maps in Tetragon BPF directory. Enabled by default. Set to false to disable (default true)
--server-address string gRPC server address (e.g. 'localhost:54321' or 'unix:///var/run/tetragon/tetragon.sock' (default "localhost:54321")
--tracing-policy string Tracing policy file to load at startup
--tracing-policy-dir string Directory from where to load Tracing Policies (default "/etc/tetragon/tetragon.tp.d")
--verbose int set verbosity level for eBPF verifier dumps. Pass 0 for silent, 1 for truncated logs, 2 for a full dump
Tetragon controlling settings can also be loaded from YAML configuration files according to this order:
From the drop-in configuration snippets inside the following directories where each filename maps to one controlling setting and the content of the file to its corresponding value:
/usr/lib/tetragon/tetragon.conf.d/*
/usr/local/lib/tetragon/tetragon.conf.d/*
From the configuration file /etc/tetragon/tetragon.yaml
if available,
overriding previous settings.
From the drop-in configuration snippets inside
/etc/tetragon/tetragon.conf.d/*
, similarly overriding previous settings.
If the config-dir
setting is set, Tetragon loads its settings from the
files inside the directory pointed by this option, overriding previous
controlling settings. The config-dir
is also part of Kubernetes
ConfigMap.
When reading configuration from directories, each filename maps to one controlling setting. If the same controlling setting is set multiple times, then the last value or content of that file overrides the previous ones.
To summarize the configuration precedence:
Drop-in directory pointed by --config-dir
.
Drop-in directory /etc/tetragon/tetragon.conf.d/*
.
Configuration file /etc/tetragon/tetragon.yaml
.
Drop-in directories:
/usr/local/lib/tetragon/tetragon.conf.d/*
/usr/lib/tetragon/tetragon.conf.d/*
To clear a controlling setting that was set before, set it again to an empty value.
Package managers can customize the configuration by installing drop-ins under
/usr/
. Configurations in /etc/tetragon/
are strictly reserved for the local
administrator, who may use this logic to override package managers or the
default installed configuration.
The examples/configuration/tetragon.yaml
file contains example entries showing the defaults as a guide to the
administrator. Local overrides can be created by editing and copying this file
into /etc/tetragon/tetragon.yaml
, or by editing and copying “drop-ins” from
the examples/configuration/tetragon.conf.d
directory into the /etc/tetragon/tetragon.conf.d/
subdirectory. The latter is
generally recommended.
Each filename maps to a one controlling setting and the content of the file to its corresponding value. This is the recommended way.
Changing configuration example:
/etc/tetragon/tetragon.conf.d/bpf-lib
with a corresponding value of:
/var/lib/tetragon/
/etc/tetragon/tetragon.conf.d/log-format
with a corresponding value of:
text
/etc/tetragon/tetragon.conf.d/export-filename
with a corresponding value of:
/var/log/tetragon/tetragon.log
/etc/tetragon/tetragon.yaml
and all drop-ins under /etc/tetragon/tetragon.conf.d/
The gRPC API supports unix sockets, it can be set using one of the following methods:
Use the --server-address
flag:
--server-address unix:///var/run/tetragon/tetragon.sock
Or use the drop-in configuration file /etc/tetragon/tetragon.conf.d/server-address
containing:
unix:///var/run/tetragon/tetragon.sock
Then to access the gRPC API with tetra
client, set --server-address
to point to the corresponding address:
sudo tetra --server-address unix:///var/run/tetragon/tetragon.sock getevents
tetra
client, if --server-address
is not specified,
it will try to detect if Tetragon daemon is running on the same host and use its
server-address
configuration.Tetragon daemon automatically loads Tracing policies from the default /etc/tetragon/tetragon.tp.d/
directory. Tracing policies can be organized in directories such: /etc/tetragon/tetragon.tp.d/file-access
, /etc/tetragon/tetragon.tp.d/network-access
, etc.
The --tracing-policy-dir
controlling setting can be used to change the default directory from where Tracing policies are loaded.
The --tracing-policy
controlling setting can be used to specify the path of one tracing policy to load.
Tetragon needs Linux kernel version 4.19 or greater.
We currently run tests on stable long-term support kernels 4.19, 5.4, 5.10, 5.15 and bpf-next, see this test workflow for up to date information. Not all Tetragon features work with older kernel versions. BPF evolves rapidly and we recommend you use the most recent stable kernel possible to get the most out of Tetragon’s features.
Note that Tetragon also needs BTF support which might take some work on older kernels.
# CORE BPF
CONFIG_BPF
CONFIG_BPF_JIT
CONFIG_BPF_JIT_DEFAULT_ON
CONFIG_BPF_EVENTS
CONFIG_BPF_SYSCALL
CONFIG_HAVE_BPF_JIT
CONFIG_HAVE_EBPF_JIT
CONFIG_FTRACE_SYSCALLS
# BTF
CONFIG_DEBUG_INFO_BTF
CONFIG_DEBUG_INFO_BTF_MODULES
# Enforcement
CONFIG_BPF_KPROBE_OVERRIDE
# CGROUP and Process tracking
CONFIG_CGROUPS=y Control Group support
CONFIG_MEMCG=y Memory Control group
CONFIG_BLK_CGROUP=y Generic block IO controller
CONFIG_CGROUP_SCHED=y
CONFIG_CGROUP_PIDS=y Process Control group
CONFIG_CGROUP_FREEZER=y Freeze and unfreeze tasks controller
CONFIG_CPUSETS=y Manage CPUSETs
CONFIG_PROC_PID_CPUSET=y
CONFIG_CGROUP_DEVICE=Y Devices Control group
CONFIG_CGROUP_CPUACCT=y CPU accouting controller
CONFIG_CGROUP_PERF=y
CONFIG_CGROUP_BPF=y Attach eBPF programs to a cgroup
CGROUP_FAVOR_DYNMODS=y (optional) >= 6.0
Reduces the latencies of dynamic cgroup modifications at the
cost of making hot path operations such as forks and exits
more expensive.
Platforms with frequent cgroup migrations could enable this
option as a potential alleviation for pod and containers
association issues.
You might have encountered the following issues:
level=info msg="BTF discovery: default kernel btf file does not exist" btf-file=/sys/kernel/btf/vmlinux
level=info msg="BTF discovery: candidate btf file does not exist" btf-file=/var/lib/tetragon/metadata/vmlinux-5.15.49-linuxkit
level=info msg="BTF discovery: candidate btf file does not exist" btf-file=/var/lib/tetragon/btf
[...]
level=fatal msg="Failed to start tetragon" error="tetragon, aborting kernel autodiscovery failed: Kernel version \"5.15.49-linuxkit\" BTF search failed kernel is not included in supported list. Please check Tetragon requirements documentation, then use --btf option to specify BTF path and/or '--kernel' to specify kernel version"
Tetragon needs BTF (BPF Type Format) support to load its BPF programs using CO-RE (Compile Once - Run Everywhere). In brief, CO-RE is useful to load BPF programs that have been compiled on a different kernel version than the target kernel. Indeed, kernel structures change between versions and BPF programs need to access fields in them. So CO-RE uses the BTF file of the kernel in which you are loading the BPF program to know the differences between the struct and patch the fields offset in the accessed structures. CO-RE allows portability of the BPF programs but requires a kernel with BTF enabled.
Most of the common Linux distributions
now ship with BTF enabled and do not require any extra work, this is kernel
option CONFIG_DEBUG_INFO_BTF=y
. To check if BTF is enabled on your Linux
system and see the BTF data file of your kernel, the standard location is
/sys/kernel/btf/vmlinux
. By default, Tetragon will look for this file (this
is the first line in the log output above).
If your kernel does not support BTF you can:
CONFIG_DEBUG_INFO_BTF
to y
.Tetragon will also look into /var/lib/tetragon/btf
for the vmlinux
file
(this is the third line in the log output above). Or you can use the --btf
flag to directly indicate Tetragon where to locate the file.
If you encounter this issue while using Docker Desktop on macOS, please refer to can I run Tetragon on Mac computers.
Yes! TBD docs
Otherwise you can build Tetragon from source by running make
to generate standalone
binaries.
Make sure to take a look at the Development Setup
guide for the build requirements. Then use sudo ./tetragon --bpf-lib bpf/objs
to run Tetragon.
You can run make vendor
then add and commit your changes.
You need to add a signed-off-by line to your commit messages. The easiest way
to do this is with git fetch origin/main && git rebase --signoff origin/main
.
Then push your changes.
Yes! You can run Tetragon locally by running a Linux virtual machine on your Mac.
On macOS running on amd64 (also known as Intel Mac) and arm64 (also know as Apple Silicon Mac), open source and commercial solutions exists to run virtual machines, here is a list of popular open source projects that you can use:
You can use these solutions to run a recent Linux distribution that ships with BTF debug information support.
Please note that you need to use a recent Docker Desktop version on macOS (for example 24.0.6
with Kernel 6.4.16-linuxkit
), because the Linux
virtual machine provided by older Docker Desktop versions lacked support for the BTF debug information.
The BTF debug information file is needed for CO-RE
in order to load sensors of Tetragon. Run the following commands to see if Tetragon can be used on your Docker Desktop version:
# The Kernel needs to be compiled with CONFIG_DEBUG_INFO_BTF and
# CONFIG_DEBUG_INFO_BTF_MODULES support:
$ docker run -it --rm --privileged --pid=host ubuntu \
nsenter -t 1 -m -u -n -i sh -c \
'cat /proc/config.gz | gunzip | grep CONFIG_DEBUG_INFO_BTF'
CONFIG_DEBUG_INFO_BTF=y
CONFIG_DEBUG_INFO_BTF_MODULES=y
# "/sys/kernel/btf/vmlinux" should be present:
$ docker run -it --rm --privileged --pid=host ubuntu \
nsenter -t 1 -m -u -n -i sh -c 'ls -la /sys/kernel/btf/vmlinux'
-r--r--r-- 1 root root 4988627 Nov 21 20:33 /sys/kernel/btf/vmlinux
Before you report a problem, make sure to retrieve the necessary information from your cluster.
You collect information in a Kubernetes cluster using the Cilium CLI:
cilium-cli sysdump
More details can be found in the Cilium docs.
If you are not running Kubernetes, it is also possible to run the bug collection tool manually with the scope of a single node using:
tetra bugtool
Tetragon’s bugtool captures potentially useful information about your environment for debugging. The tool is meant to be used for debugging a single Tetragon agent node. Note in Kubernetes, the command needs to be run from inside the Tetragon pod/container.
Security Observability with eBPF - Jed Salazar & Natália Réka Ivánkó, OReilly, 2022
Tetragon 1.0: Kubernetes Security Observability & Runtime Enforcement with eBPF - Thomas Graf, 2023
Tutorial: Setting Up a Cybersecurity Honeypot with Tetragon to Trigger Canary Tokens - Dean Lewis, 2023
Can I use Tetragon without Cilium? - Dean Lewis, 2023
Detecting a Container Escape with Cilium and eBPF - Natália Réka Ivánkó, 2021
Detecting and Blocking log4shell with Isovalent Cilium Enterprise - Jed Salazar, 2021
Security Observability with eBPF and Tetragon - Natália Réka Ivánkó, Roland Wolters, Raphaël Pinson