This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Process lifecycle

Tetragon observes by default the process lifecycle via exec and exit

1: Process execution
2: Advanced Process execution
3: Privileged execution
4: Namespace access monitoring

Tetragon observes process creation and termination with default configuration and generates process_exec and process_exit events:

The process_exec events include useful information about the execution of binaries and related process information. This includes the binary image that was executed, command-line arguments, the UID context the process was executed with, the process parent information, the capabilities that a process had while executed, the process start time, the Kubernetes Pod, labels and more.
The process_exit events, as the process_exec event shows how and when a process started, indicate how and when a process is removed. The information in the event includes the binary image that was executed, command-line arguments, the UID context the process was executed with, process parent information, process start time, the status codes and signals on process exit. Understanding why a process exited and with what status code helps understand the specifics of that exit.

Both these events include Linux-level metadata (UID, parents, capabilities, start time, etc.) but also Kubernetes-level metadata (Kubernetes namespace, labels, name, etc.). This data make the connection between node-level concepts, the processes, and Kubernetes or container environments.

These events enable a full lifecycle view into a process that can aid an incident investigation, for example, we can determine if a suspicious process is still running in a particular environment. For concrete examples of such events, see the next use case on process execution.

1 - Process execution

Monitor process lifecycle with process_exec and process_exit

This first use case is monitoring process execution, which can be observed with the Tetragon process_exec and process_exit JSON events. These events contain the full lifecycle of processes, from fork/exec to exit, including metadata such as:

Binary name: Defines the name of an executable file
Parent process: Helps to identify process execution anomalies (e.g., if a nodejs app forks a shell, this is suspicious)
Command-line argument: Defines the program runtime behavior
Current working directory: Helps to identify hidden malware execution from a temporary folder, which is a common pattern used in malwares
Kubernetes metadata: Contains pods, labels, and Kubernetes namespaces, which are critical to identify service owners, particularly in a multitenant environments
exec_id: A unique process identifier that correlates all recorded activity of a process

As a first step, let’s start monitoring the events from the xwing pod:

kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | tetra getevents -o compact --namespace default --pod xwing

Then in another terminal, let’s kubectl exec into the xwing pod and execute some example commands:

kubectl exec -it xwing -- /bin/bash
whoami

If you observe, the output in the first terminal should be:

🚀 process default/xwing /bin/bash
🚀 process default/xwing /usr/bin/whoami
💥 exit    default/xwing /usr/bin/whoami 0

Here you can see the binary names along with its arguments, the pod info, and return codes in a compact one-line view of the events.

For more details use the raw JSON events to get detailed information, you can stop the Tetragon CLI by Crl-C and parse the tetragon.log file by executing:

kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | jq 'select(.process_exec.process.pod.name=="xwing" or .process_exit.process.pod.name=="xwing")'

Example process_exec and process_exit events can be:

Process Exec Event

style=color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4>

{ "process_exec": { "process": { "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjExNDI4NjE1NjM2OTAxOjUxNTgz", "pid": 51583, "uid": 0, "cwd": "/", "binary": "/usr/bin/whoami", "arguments": "--version", "flags": "execve rootcwd clone", "start_time": "2022-05-11T12:54:45.615Z", "auid": 4294967295, "pod": { "namespace": "default", "name": "xwing", "container": { "id": "containerd://1fb931d2f6e5e4cfdbaf30fdb8e2fdd81320bdb3047ded50120a4f82838209ce", "name": "spaceship", "image": { "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6", "name": "docker.io/tgraf/netperf:latest" }, "start_time": "2022-05-11T10:07:33Z", "pid": 50 } }, "docker": "1fb931d2f6e5e4cfdbaf30fdb8e2fdd", "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjkwNzkyMjU2MjMyNjk6NDM4NzI=", "refcnt": 1 }, "parent": { "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjkwNzkyMjU2MjMyNjk6NDM4NzI=", "pid": 43872, "uid": 0, "cwd": "/", "binary": "/bin/bash", "flags": "execve rootcwd clone", "start_time": "2022-05-11T12:15:36.225Z", "auid": 4294967295, "pod": { "namespace": "default", "name": "xwing", "container": { "id": "containerd://1fb931d2f6e5e4cfdbaf30fdb8e2fdd81320bdb3047ded50120a4f82838209ce", "name": "spaceship", "image": { "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6", "name": "docker.io/tgraf/netperf:latest" }, "start_time": "2022-05-11T10:07:33Z", "pid": 43 } }, "docker": "1fb931d2f6e5e4cfdbaf30fdb8e2fdd", "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjkwNzkxODU5NTMzOTk6NDM4NjE=", "refcnt": 1 } }, "node_name": "kind-control-plane", "time": "2022-05-11T12:54:45.615Z" style=color:#111>}

Process Exit Event

{
  "process_exit": {
    "process": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjExNDI4NjE1NjM2OTAxOjUxNTgz",
      "pid": 51583,
      "uid": 0,
      "cwd": "/",
      "binary": "/usr/bin/whoami",
      "arguments": "--version",
      "flags": "execve rootcwd clone",
      "start_time": "2022-05-11T12:54:45.615Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://1fb931d2f6e5e4cfdbaf30fdb8e2fdd81320bdb3047ded50120a4f82838209ce",
          "name": "spaceship",
          "image": {
            "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
            "name": "docker.io/tgraf/netperf:latest"
          },
          "start_time": "2022-05-11T10:07:33Z",
          "pid": 50
        }
      },
      "docker": "1fb931d2f6e5e4cfdbaf30fdb8e2fdd",
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjkwNzkyMjU2MjMyNjk6NDM4NzI="
    },
    "parent": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjkwNzkyMjU2MjMyNjk6NDM4NzI=",
      "pid": 43872,
      "uid": 0,
      "cwd": "/",
      "binary": "/bin/bash",
      "flags": "execve rootcwd clone",
      "start_time": "2022-05-11T12:15:36.225Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://1fb931d2f6e5e4cfdbaf30fdb8e2fdd81320bdb3047ded50120a4f82838209ce",
          "name": "spaceship",
          "image": {
            "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
            "name": "docker.io/tgraf/netperf:latest"
          },
          "start_time": "2022-05-11T10:07:33Z",
          "pid": 43
        }
      },
      "docker": "1fb931d2f6e5e4cfdbaf30fdb8e2fdd",
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjkwNzkxODU5NTMzOTk6NDM4NjE="
    }
  },
  "node_name": "kind-control-plane",
  "time": "2022-05-11T12:54:45.616Z"
}

`2 - Advanced Process execution`

Advanced Process Execution monitoring using Tracing Policies

`Monitor ELF or Flat binaries execution`

Advanced process execution can be performed by using Tracing Policies to monitor the execve system call path.

If we want to monitor execution of Executable and Linkable Format (ELF) or flat binaries before they are actually executed. Then the process-exec-elf-begin tracing policy is a good first choice.

NoteThe process-exec-elf-begin tracing policy, will not report the
different binary format handlers or scripts being executed, but will report
the final ELF or flat binary, like the shebang handler.
To report those another tracing policy can be used.

Before going forward, verify that all pods are up and running, ensure you deploy our Demo Application to explore the Security Observability Events:

kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.15.3/examples/minikube/http-sw-app.yaml

It might take several seconds for some pods until they satisfy all the dependencies:

kubectl get pods -A

The output should be similar to:

NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
default              deathstar-54bb8475cc-6c6lc                   1/1     Running   0          2m54s
default              deathstar-54bb8475cc-zmfkr                   1/1     Running   0          2m54s
default              tiefighter                                   1/1     Running   0          2m54s
default              xwing                                        1/1     Running   0          2m54s
kube-system          tetragon-sdwv6                               2/2     Running   0          27m

Let’s apply the process-exec-elf-begin Tracing Policy.

kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-exec/process-exec-elf-begin.yaml

Then start monitoring events with the tetra CLI:

kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents

In another terminal, kubectl exec into the xwing Pod:

kubectl exec -it xwing -- /bin/bash

And execute some commands:

id

The tetra CLI will generate the following ProcessKprobe events:

{
  "process_kprobe": {
    "process": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE2NjY0MDI4MTA4MzcxOjM2NDk5",
      "pid": 36499,
      "uid": 0,
      "cwd": "/",
      "binary": "/bin/bash",
      "flags": "execve",
      "start_time": "2023-08-02T11:58:53.618461573Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://775beeb1a25a95e10dc149d6eb166bf45dd5e6039e8af3b64e8fb4d29669f349",
          "name": "spaceship",
          "image": {
            "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
            "name": "docker.io/tgraf/netperf:latest"
          },
          "start_time": "2023-08-02T07:24:54Z",
          "pid": 13
        },
        "pod_labels": {
          "app.kubernetes.io/name": "xwing",
          "class": "xwing",
          "org": "alliance"
        }
      },
      "docker": "775beeb1a25a95e10dc149d6eb166bf",
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE2NjYyNzg3ODI1MTQ4OjM2NDkz",
      "refcnt": 1,
      "tid": 36499
    },
      "parent": {
      "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE2NjYyNzg3ODI1MTQ4OjM2NDkz",
      "pid": 36493,
      "uid": 0,
      "cwd": "/",
      "binary": "/bin/bash",
      "flags": "execve rootcwd clone",
      "start_time": "2023-08-02T11:58:52.378178852Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://775beeb1a25a95e10dc149d6eb166bf45dd5e6039e8af3b64e8fb4d29669f349",
          "name": "spaceship",
          "image": {
            "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
            "name": "docker.io/tgraf/netperf:latest"
          },
          "start_time": "2023-08-02T07:24:54Z",
          "pid": 13
        },
        "pod_labels": {
          "app.kubernetes.io/name": "xwing",
          "class": "xwing",
          "org": "alliance"
        }
      },
      "docker": "775beeb1a25a95e10dc149d6eb166bf",
      "parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE2NjYyNzE2OTU0MjgzOjM2NDg0",
      "tid": 36493
    },
    "function_name": "security_bprm_creds_from_file",
    "args": [
      {
        "file_arg": {
          "path": "/bin/busybox"
        }
      }
    ],
    "action": "KPROBE_ACTION_POST"
  },
  "node_name": "kind-control-plane",
  "time": "2023-08-02T11:58:53.624096751Z"
}

In addition to the Kubernetes Identity and process metadata, ProcessKprobe events contain the binary being executed. In the above case they are:

function_name: where we are hooking into the kernel to read the binary that is being executed.
file_arg: that includes the path being executed, and here it is /bin/busybox that is the real binary being executed, since on the xwing pod the container is running busybox. The binary /usr/bin/id -> /bin/busybox points to busybox.

To disable the process-exec-elf-being Tracing Policy run:

kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-exec/process-exec-elf-begin.yaml

`3 - Privileged execution`

Monitor process capabilities and kernel namespace access

Tetragon also provides the ability to check process capabilities and kernel namespaces access.

This information would help us determine which process or Kubernetes pod has started or gained access to privileges or host namespaces that it should not have. This would help us answer questions like:

Which Kubernetes pods are running with CAP_SYS_ADMIN in my cluster?

Which Kubernetes pods have host network or pid namespace access in my cluster?

`Step 1: Enabling Process Credential and Namespace Monitoring`

Edit the Tetragon configmap:

kubectl edit cm -n kube-system tetragon-config

Set the following flags from “false” to “true”:

# enable-process-cred: true
# enable-process-ns: true

Save your changes and exit.

Restart the Tetragon daemonset:

kubectl rollout restart -n kube-system ds/tetragon

`Step 2: Deploying a Privileged Nginx Pod`

Create a YAML file (e.g., privileged-nginx.yaml) with the following PodSpec:

apiVersion: v1
kind: Pod
metadata:
  name: privileged-the-pod
spec:
  hostPID: true
  hostNetwork: true
  containers:
  - name: privileged-the-pod
    image: nginx:latest
    ports:
    - containerPort: 80
    securityContext:
      privileged: true

Apply the configuration:
```
kubectl apply -f privileged-nginx.yaml
```

`Step 3: Monitoring with Tetragon`

Start monitoring events from the privileged Nginx pod:

kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | tetra getevents --namespace default --pod privileged-the-pod

You should observe Tetragon generating events similar to these, indicating the privileged container start:
```
🚀 process default/privileged-nginx /nginx -g daemon off;  🛑 CAP_SYS_ADMIN
```

`4 - Namespace access monitoring`

Monitor namespace changes and detect container escapes

Tetragon can monitor Linux namespace operations to detect when processes change their namespaces. This is particularly useful for detecting container escapes where a process attempts to access host namespaces using tools like nsenter.

`Use Case: Detecting Container Escapes via Namespace Changes`

A common container escape technique involves using nsenter to switch into the host’s namespaces. Even if a container is running with restricted privileges, monitoring setns syscalls helps detect attempts to break out of the container’s isolation.

This answers questions like:

Which processes are attempting to change their Linux namespaces?

Is there a container escape attempt happening in my cluster?

`Deploying the Demo Application`

Before starting, deploy the Demo Application:

kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.15.3/examples/minikube/http-sw-app.yaml

Verify the pods are running:

kubectl get pods -A

The output should be similar to:

NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
default              deathstar-54bb8475cc-6c6lc                   1/1     Running   0          2m54s
default              deathstar-54bb8475cc-zmfkr                   1/1     Running   0          2m54s
default              tiefighter                                   1/1     Running   0          2m54s
default              xwing                                        1/1     Running   0          2m54s
kube-system          tetragon-sdwv6                               2/2     Running   0          27m

`Creating the TracingPolicy`

To monitor namespace access via the setns syscall, we use a TracingPolicy that hooks into sys_setns (this will be transparently resolved to the architecture specific symbol, __x64_sys_setns for x86_64 or __arm64_sys_setns for arm64). This syscall is invoked when a process attempts to change its namespace.

Create a file named namespace-access.yaml:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "namespace-access"
spec:
  kprobes:
  - call: "sys_setns"
    syscall: true
    args:
    - index: 0
      type: "int"
    - index: 1
      type: "int"

Note The first argument (index 0) is the file descriptor referring to a namespace,
and the second argument (index 1) is a flag specifying the type of namespace.
For more details, see the setns(2) man page.

Apply the TracingPolicy:

kubectl apply -f namespace-access.yaml

`Monitoring Namespace Access`

Start monitoring events with the tetra CLI:

kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents

In another terminal, simulate a container escape attempt by using nsenter to enter the host’s namespaces. First, kubectl exec into a pod:

kubectl exec -it xwing -- /bin/bash

Then attempt to enter the host namespaces:

nsenter -t 1 -m -u -n -i -p

Note This command attempts to enter the mount (-m), UTS (-u), network (-n), IPC (-i),
and PID (-p) namespaces of PID 1 (the host’s init process). In a properly secured
container, this command will fail, but Tetragon will still capture the attempt.

The tetra CLI will generate ProcessKprobe events similar to:

{
  "process_kprobe": {
    "process": {
      "exec_id": "dGV0cmFnb24tZGV2LWNvbnRyb2wtcGxhbmU6MTE2OTc3OTA1NDM0MjAxOjYzMDY2",
      "pid": 63066,
      "uid": 0,
      "cwd": "/",
      "binary": "/usr/bin/nsenter",
      "arguments": "-t 1 -m -u -n -i -p",
      "flags": "execve rootcwd clone",
      "start_time": "2026-01-30T18:15:19.985627895Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://057dc8d712abf96c628c82ea111d53a205a256b0dc39bb31284426aaf378acbc",
          "name": "spaceship",
          "image": {
            "id": "quay.io/cilium/json-mock@sha256:5aad04835eda9025fe4561ad31be77fd55309af8158ca8663a72f6abb78c2603",
            "name": "quay.io/cilium/json-mock:latest"
          },
          "start_time": "2026-01-28T08:51:07Z",
          "pid": 37
        },
        "pod_labels": {
          "app.kubernetes.io/name": "xwing",
          "class": "xwing",
          "org": "alliance"
        },
        "workload": "xwing",
        "workload_kind": "Pod"
      },
      "docker": "057dc8d712abf96c628c82ea111d53a",
      "parent_exec_id": "dGV0cmFnb24tZGV2LWNvbnRyb2wtcGxhbmU6MTE2OTc3NzI5OTA3MzUxOjYzMDUy",
      "refcnt": 1,
      "tid": 63066
    },
    "parent": {
      "exec_id": "dGV0cmFnb24tZGV2LWNvbnRyb2wtcGxhbmU6MTE2OTc3NzI5OTA3MzUxOjYzMDUy",
      "pid": 63052,
      "uid": 0,
      "cwd": "/",
      "binary": "/bin/bash",
      "flags": "execve rootcwd clone",
      "start_time": "2026-01-30T18:15:19.810100966Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://057dc8d712abf96c628c82ea111d53a205a256b0dc39bb31284426aaf378acbc",
          "name": "spaceship",
          "image": {
            "id": "quay.io/cilium/json-mock@sha256:5aad04835eda9025fe4561ad31be77fd55309af8158ca8663a72f6abb78c2603",
            "name": "quay.io/cilium/json-mock:latest"
          },
          "start_time": "2026-01-28T08:51:07Z",
          "pid": 31
        },
        "pod_labels": {
          "app.kubernetes.io/name": "xwing",
          "class": "xwing",
          "org": "alliance"
        },
        "workload": "xwing",
        "workload_kind": "Pod"
      },
      "docker": "057dc8d712abf96c628c82ea111d53a",
      "parent_exec_id": "dGV0cmFnb24tZGV2LWNvbnRyb2wtcGxhbmU6ODgzNjQ2NzQwMjg6NzIxMQ==",
      "tid": 63052
    },
    "function_name": "__x64_sys_setns",
    "args": [
      {
        "int_arg": 3
      },
      {
        "int_arg": 134217728
      }
    ],
    "action": "KPROBE_ACTION_POST",
    "policy_name": "namespace-access",
    "return_action": "KPROBE_ACTION_POST"
  },
  "node_name": "tetragon-dev-control-plane",
  "time": "2026-01-30T18:15:20.001042828Z"
}

Note The second argument (int_arg: 134217728) corresponds to CLONE_NEWIPC (0x08000000), indicating
an attempt to enter the IPC namespace. Other common values include CLONE_NEWNS (0x20000) for
mount namespace, CLONE_NEWPID (0x20000000) for PID namespace, and CLONE_NEWNET (0x40000000)
for network namespace.

`Enforcement: Blocking Namespace Changes`

To actively prevent namespace escape attempts, you can extend the policy with a Sigkill action. The following policy will terminate any process that attempts to call setns from within a container:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "deny-namespace-access"
spec:
  kprobes:
  - call: "sys_setns"
    syscall: true
    args:
    - index: 0
      type: "int"
    - index: 1
      type: "int"
    selectors:
    - matchPIDs:
      - operator: NotIn
        followForks: true
        isNamespacePID: true
        values:
        - 0
        - 1
      matchActions:
      - action: Sigkill

The matchPIDs selector ensures the policy only applies to container processes:

isNamespacePID: true: Uses the PID namespace ID rather than the host PID
operator: NotIn with values 0, 1: Excludes the init process (PID 1) and invalid PIDs, effectively targeting only non-init container processes
followForks: true: Applies the rule to forked child processes as well

Caution Please consult the Enforcement section
before using enforcement actions in production environments.

`Cleanup`

To remove the TracingPolicy:

kubectl delete -f namespace-access.yaml