Tetragon Observability Policies
Index
System Activity
Security Sensitive Events
Networking
Observability Policies
eBPF Subsystem Interactions
Description
Audit BPF program loads and BPFFS interactions
Use Case
Understanding BPF programs loaded in a cluster and interactions between applications and programs can identify bugs and malicious or unexpected BPF activity.
Policy
Example jq Filter
jq 'select(.process_kprobe != null) | select(.process_kprobe.function_name | test("bpf_check")) | "\(.time) \(.process_kprobe.process.binary) \(.process_kprobe.process.arguments) programType:\(.process_kprobe.args[0].bpf_attr_arg.ProgType) programInsn:\(.process_kprobe.args[0].bpf_attr_arg.InsnCnt)"
Example Output
"2023-11-01T02:56:54.926403604Z /usr/bin/bpftool prog list programType:BPF_PROG_TYPE_SOCKET_FILTER programInsn:2"
Kernel Module Audit Trail
Description
Audit loading of kernel modules
Use Case
Understanding exactly what kernel modules are running in the cluster is crucial to understand attack surface and any malicious actors loading unexpected modules.
Policy
Example jq Filter
jq 'select(.process_kprobe != null) | select(.process_kprobe.function_name | test("security_kernel_module_request")) | "\(.time) \(.process_kprobe.process.binary) \(.process_kprobe.process.arguments) module:\(.process_kprobe.args[0].string_arg)"'
Example Output
"2023-11-01T04:11:38.390880528Z /sbin/iptables -A OUTPUT -m cgroup --cgroup 1 -j LOG module:ipt_LOG"
Shared Library Loading
Description
Monitor loading of libraries
Use Case
Understanding the exact versions of shared libraries that binaries load and use is crucial to understand use of vulnerable or deprecated library versions or attacks such as shared library hijacking.
Policy
Example jq Filter
jq 'select(.process_loader != null) | "\(.time) \(.process_loader.process.pod.namespace) \(.process_loader.process.binary) \(.process_loader.process.arguments) \(.process_loader.path)"
Example Output
"2023-10-31T19:42:33.065233159Z default/xwing /usr/bin/curl https://ebpf.io /usr/lib/x86_64-linux-gnu/libssl.so.3"
Binary Execution in /tmp
Description
Monitor execution of a binary in the /tmp directory.
Use Case
Preventing execution of executables in /tmp
is a common best-practice as several canned exploits rely on writing and then executing malicious binaries in the /tmp
directory. A common best-practice to enforce this is to mount the /tmp
filesystem with the noexec
flag. This observability policy is used to monitor for violations of this best practice.
Policy
No policy needs to be loaded, standard process execution observability is sufficient.
Example jq Filter
jq 'select(.process_exec != null) | select(.process_exec.process.binary | contains("/tmp/")) | "\(.time) \(.process_exec.process.pod.namespace) \(.process_exec.process.pod.name) \(.process_exec.process.binary) \(.process_exec.process.arguments)"'
Example Output
"2023-10-31T18:44:22.777962637Z default/xwing /tmp/nc ebpf.io 1234"
sudo Invocation Monitoring
Description
Monitor sudo invocations
Use Case
sudo is used to run executables with particular privileges. Creating a audit log of sudo invocations is a common best-practice.
Policy
No policy needs to be loaded, standard process execution observability is sufficient.
Example jq Filter
jq 'select(.process_exec != null) | select(.process_exec.process.binary | contains("sudo")) | "\(.time) \(.process_exec.process.pod.namespace) \(.process_exec.process.binary) \(.process_exec.process.arguments)"'
Example Output
"2023-10-31T19:03:35.273111185Z null /usr/bin/sudo -i"
SUID Binary Execution
Description
Monitor execution of SUID “Set User ID” binaries.
Use Case
The “Set User Identity” and “Set Group Identity” are permission flags. When set on a binary file, the binary will execute with the permissions of the owner or group associated with the executable file, rather than the user executing it. Usually it is used to run programs with elevated privileges to perform specific tasks.
Detecting the execution of setuid
and setgid
binaries is a common
best-practice as attackers may abuse such binaries, or even create them
during an exploit for subsequent execution.
Requirement
Run Tetragon with enable-process-creds
setting set to enable visibility
into process_credentials and binary_properties.
kubectl edit cm -n kube-system tetragon-config
# Change "enable-process-cred" from "false" to "true", then save and exit
# Restart Tetragon daemonset
kubectl rollout restart -n kube-system ds/tetragon
docker run --name tetragon --rm -d \
--pid=host --cgroupns=host --privileged \
-v /sys/kernel:/sys/kernel \
-v /var/log/tetragon:/var/log/tetragon \
quay.io/cilium/tetragon:v1.0.0 \
/usr/bin/tetragon --enable-process-cred \
--export-filename /var/log/tetragon/tetragon.log
# Write to the drop-in file /etc/tetragon/tetragon.conf.d/enable-process-cred true
# Run the following as a privileged user then restart tetragon service
echo "true" > /etc/tetragon/tetragon.conf.d/enable-process-cred
systemctl restart tetragon
Policy
No policy needs to be loaded, standard process execution observability is sufficient.
Example jq Filter
jq 'select(.process_exec != null) | select(.process_exec.process.binary_properties != null) | select(.process_exec.process.binary_properties.setuid != null or .process_exec.process.binary_properties.setgid != null) | "\(.time) \(.process_exec.process.pod.namespace) \(.process_exec.process.pod.name) \(.process_exec.process.binary) \(.process_exec.process.arguments) uid=\(.process_exec.process.process_credentials.uid) euid=\(.process_exec.process.process_credentials.euid) gid=\(.process_exec.process.process_credentials.gid) egid=\(.process_exec.process.process_credentials.egid)"'
Example Output
"2023-11-13T08:20:43.615672640Z null null /usr/bin/sudo id uid=1000 euid=0 gid=1000 egid=1000"
"2023-11-13T08:20:45.591560940Z null null /usr/bin/wall hello uid=1000 euid=1000 gid=1000 egid=5"
"2023-11-13T08:20:47.036760043Z null null /usr/bin/su - uid=1000 euid=0 gid=1000 egid=1000"
Setuid System Calls
Description
Monitor execution of the setuid() system calls family.
Use Case
The setuid() and setgid() system calls family allow to change the effective user ID and group ID of the calling process.
Detecting setuid() and setgid() calls that set the user ID or group ID to root is a common best-practice to identify when privileges are raised.
Policy
The privileges-setuid-root.yaml is a catch all to the various interfaces of setuid()
and setgid()
to root.
Example jq Filter
jq 'select(.process_kprobe != null) | select(.process_kprobe.policy_name | test("privileges-setuid-root")) | "\(.time) \(.process_kprobe.process.pod.namespace) \(.process_kprobe.process.pod.name) \(.process_kprobe.process.binary) \(.process_kprobe.process.arguments) \(.process_kprobe.function_name) \(.process_kprobe.args)"'
Example Output
"2023-11-12T22:17:40.754680857Z null null /usr/bin/sudo id __sys_setresgid [{\"int_arg\":-1},{\"int_arg\":0},{\"int_arg\":-1}]"
"2023-11-12T22:17:40.754730285Z null null /usr/bin/sudo id __sys_setresuid [{\"int_arg\":-1},{\"int_arg\":0},{\"int_arg\":-1}]"
"2023-11-12T22:17:40.758125709Z null null /usr/bin/sudo id __sys_setgid [{\"int_arg\":0}]"
"2023-11-12T22:17:40.758747395Z null null /usr/bin/sudo id __sys_setresuid [{\"int_arg\":0},{\"int_arg\":0},{\"int_arg\":0}]"
SSHd connection monitoring
Description
Monitor sessions to SSHd
Use Case
It is best practice to audit remote connections into a shell server.
Policy
Example jq Filter
jq 'select(.process_kprobe != null) | select(.process_kprobe.function_name | test("tcp_close")) | "\(.time) \(.process_kprobe.process.binary) \(.process_kprobe.process.arguments) \(.process_kprobe.args[0].sock_arg.family) \(.process_kprobe.args[0].sock_arg.type) \(.process_kprobe.args[0].sock_arg.protocol) \(.process_kprobe.args[0].sock_arg.saddr):\(.process_kprobe.args[0].sock_arg.sport)"'
Example Output
"2023-11-01T04:51:20.109146920Z /usr/sbin/sshd default/xwing AF_INET SOCK_STREAM IPPROTO_TCP 127.0.0.1:22"
Outbound connections
Description
Monitor all cluster egress connections
Use Case
Connections made outside a Kubernetes cluster can be audited to provide insights into any unexpected or malicious reverse shells.
Environment Variables
PODCIDR=`kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'`
SERVICECIDR=$(gcloud container clusters describe ${NAME} --zone ${ZONE} | awk '/servicesIpv4CidrBlock/ { print $2; }')
SERVICECIDR=$(kubectl describe pod -n kube-system kube-apiserver-kind-control-plane | awk -F= '/--service-cluster-ip-range/ {print $2; }')
Policy
Example jq Filter
jq 'select(.process_kprobe != null) | select(.process_kprobe.function_name | test("tcp_connect")) | "\(.time) \(.process_kprobe.process.binary) \(.process_kprobe.process.arguments) \(.process_kprobe.args[0].sock_arg.saddr):\(.process_kprobe.args[0].sock_arg.sport) -> \(.process_kprobe.args[0].sock_arg.daddr):\(.process_kprobe.args[0].sock_arg.dport)"'
Example Output
"2023-11-01T05:25:14.837745007Z /usr/bin/curl http://ebpf.io 10.168.0.45:48272 -> 104.198.14.52:80"