eBPF - Linux/Windows Kernel "God-Mode"
Security Forum 2024
Carsten Strotmann
Created: 2024-05-22 Wed 09:20
Who is this?
Carsten Strotmann DNS(SEC)/DANE/DHCP/IPv6/Linux/xBSD/Security Trainer and Consultant
eBPF - Why this talk?
What is eBPF, XDP and BCC?
- eBPF is the extended Berkeley Packet Filter virtual machine in the Linux kernel
- XDP (eXpress Data Path) is an eBPF interface to the network stack
- BCC is the BPF Compiler Collection, a collection of tools and eBPF programs
- eBPF is a further development of the original Berkeley Packet Filter technology https://en.wikipedia.org/wiki/Berkeley_Packet_Filter
The eBPF idea
- eBPF allows the user to run programs in the operating system kernel
within a sandbox
- eBPF makes it possible to safely and efficiently extend the functions of the operating system kernel without having to modify the kernel source code or load modules
- eBPF programs can monitor and modify network packets (and other data structures) within the Linux kernel
- eBPF programs are not kernel modules, it is not necessary to be
a kernel developer to use eBPF
- knowledge of programming in the "C" language is however an advantage
eBPF
eBPF use cases
- Application areas for eBPF
- Network security (extended firewall functions)
- Host / container security
- Forensic analysis of running processes
- Error diagnosis
- Speed measurements
- Rootkits? Backdoors in the kernel or in network cards?
eBPF availability
- eBPF is available in modern Linux systems (from kernel 3.18+, December 2014)
- Microsoft started to port eBPF to the Windows kernel in 2021
eBPF on Windows systems
- eBPF is currently (as of May 2024) not yet part of the Windows
operating system
- eBPF can optionally be installed in a Windows system
- Since the eBPF drivers for Windows are currently not yet signed by Microsoft, Windows must be started under the control of a kernel debugger or run in "test-signing" mode in order to use eBPF
- Installing eBPF for Windows: https://github.com/microsoft/ebpf-for-windows/blob/main/docs/GettingStarted.md#installing-ebpf-for-windows
The roots of BPF
- The original BSD Packet Filter (BPF) was developed by Steven
McCanne and Van Jacobson at the Lawrence Berkeley Laboratory
(https://www.tcpdump.org/papers/bpf-usenix93.pdf)
- BPF has been ported to almost all Unix/Linux systems and many non-Unix operating systems (e.g. Windows, BeOS/Haiku, OS/2 …)
- BPF is the basic technology behind well-known network sniffing
tools like
tcpdumpand Wireshark
Classic BPF using the example of tcpdump
- A BPF filter (e.g. for tcpdump) is translated into a bytecode for
the BPF virtual machine in the Linux kernel and loaded into the
kernel
- The operating system calls the BPF program for each network packet that passes through the network stack
- Only packets that match the filter expression are sent to the
program in userspace (the
tcpdumpprocess in this example) - BPF reduces the amount of data that needs to be exchanged between the kernel and userspace
Classic BPF using the example of tcpdump
tcpdump can be instructed to output the BPF source code of the tcpdump filter:
# tcpdump -d port 53 and host 1.1.1.1 Warning: assuming Ethernet (000) ldh [12] (001) jeq #0x86dd jt 19 jf 2 (002) jeq #0x800 jt 3 jf 19 (003) ldb [23] (004) jeq #0x84 jt 7 jf 5 (005) jeq #0x6 jt 7 jf 6 (006) jeq #0x11 jt 7 jf 19 (007) ldh [20] (008) jset #0x1fff jt 19 jf 9 (009) ldxb 4*([14]&0xf) (010) ldh [x + 14] (011) jeq #0x35 jt 14 jf 12 (012) ldh [x + 16] (013) jeq #0x35 jt 14 jf 19 (014) ld [26] (015) jeq #0x1010101 jt 18 jf 16 (016) ld [30] (017) jeq #0x1010101 jt 18 jf 19 (018) ret #262144 (019) ret #0
eBPF vs. BPF
- While BPF (today also called cBPF = classic BPF) filters network
packets in the operating system kernel, eBPF applies filters to
other kernel data structures and can also execute program code:
- Kernel system calls
- Kernel tracepoints
- Kernel functions
- Userspace tracepoints
- Userspace functions
eBPF and the Linux kernel
- The first version of eBPF was introduced in Linux kernel 3.18
- New kernel versions bring further, new eBPF functions
- Linux distributions (Red Hat/Canonical/Suse) have partly backported eBPF functions to older LTS/EL kernel versions
- An overview of eBPF functions broken down by Linux kernel version: https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md
The eBPF architecture
The eBPF virtual machine
- eBPF programs are compiled for a virtual CPU architecture
- The program code is loaded into the Linux kernel and checked there
- On popular CPU architectures (amd64, AARCH64) the eBPF bytecode is recompiled into native machine code (Just in Time Compiler = JIT)
XDP / eBPF execution levels
XDP - Express Data Path
- The express data path (XDP) within the Linux kernel is an
infrastructure for exercising control over network packets at the
lowest level
- The normal data flow in the Linux network stack can be bypassed via XDP
- eBPF programs can be loaded into the eXpress Data Path (XDP)
XDP / eBPF Hardware Offloading
- XDP eBPF programs can be loaded into the Linux kernel at various
levels
- Offload XDP: directly into the network hardware (ASIC/FPGA, requires support for XDP in the hardware, e.g. available in the Netronome network adapters)
- Native XDP: Into the Linux kernel network driver of the network interface (requires support by the driver)
- Generic XDP: Into the Linux kernel network stack (less performance, but possible without special hardware or driver support)
XDP Functions
- XDP programs can
- read: Collect network packets and statistics
- modify: Change the content of network packets
- drop: Discard selected network packets (firewall)
- Reroute: Network packets can be redirected to the same or different network interfaces (switching/routing)
- pass: The network packet is passed to the Linux TCP/IP stack for normal processing
XDP vs DDoS attacks
XDP can discard unwanted network traffic very early in the network stack (e.g. within the network hardware). This can be used to protect against DDoS attacks
eBPF "real-world" applications
Linux eBPF tools
- The eBPF programming tools
bpftraceandbcc(eBPF Compiler Collection) contain a number of example programs, which can be used without eBPF programming knowledge - Brandon Gregg offers a collection of very useful eBPF programs on the website Linux Extended BPF (eBPF) Tracing Tools https://brendangregg.com/ebpf.html
eBPF programs for analyzing the runtime behavior of Linux systems
Example: eBPF Syscall Evaluation
- Evaluate the syscalls of a BIND 9 process with the program
syscount
# syscount-bpfcc -p `pgrep named` -i 10 Tracing syscalls, printing top 10... Ctrl+C to quit. [07:34:19] SYSCALL COUNT futex 547 getpid 121 sendto 113 read 56 write 31 epoll_wait 31 openat 23 close 20 epoll_ctl 20 recvmsg 20
Example: eBPF capabilities used by a process
- Display the Linux capabilities of running processes
# capable-bpfcc | grep named 07:36:17 0 29378 (named) 24 CAP_SYS_RESOURCE 1 07:36:17 0 29378 (named) 24 CAP_SYS_RESOURCE 1 07:36:17 0 29378 (named) 12 CAP_NET_ADMIN 1 07:36:17 0 29378 (named) 21 CAP_SYS_ADMIN 1 07:36:17 0 29378 named 6 CAP_SETGID 1 07:36:17 0 29378 named 6 CAP_SETGID 1 07:36:17 0 29378 named 7 CAP_SETUID 1 07:36:17 109 29378 named 24 CAP_SYS_RESOURCE 1
Example: gethostlatency
- The BCC program
gethostlatencymeasures the latency of client-side DNS name resolution through system calls such asgetaddrinfoorgethostbyname
# gethostlatency-bpfcc TIME PID COMM LATms HOST 10:21:58 19183 ping 143.22 example.org 10:22:18 19184 ssh 0.03 host.example.de 10:22:18 19184 ssh 60.59 host.example.de 10:22:35 19185 ping 23.44 isc.org 10:22:49 19186 ping 4459.72 yahoo.co.kr
eBPF/XDP support in DNS software
- The open source DNS load balancer DNSdist from PowerDNS can filter DNS packets via eBPF and XDP or limit them via rate limiting
- The Knot Resolver (since version 5.2.0) can bypass the Linux TCP/IP stack for DNS packets via eBPF and XDP and forward the DNS packets directly to the Knot DNS Resolver process in userspace (https://knot-resolver.readthedocs.io/en/stable/daemon-bindings-net_xdpsrv.html). This results in an enormous increase in the speed of the DNS response rate.
Security applications
- Audit frameworks for Linux use eBPF to monitor processes on a host or in a container entirely:
- File system access
- Network traffic
- Process execution
- System calls
Security applications
- Each process is assigned an execution-id and all event log entries (network, filesystem, syscalls etc) can be assigned to a process by means of the ID
- eBPF can directly adapt policy decisions of Linux security modules (LSMs such as SELinux, AppArmor, Tomoya)
- eBPF can directly prevent file access, network communication or syscalls
Security applications: Tetragon
Security applications: Falco
Security applications: tracee
Quis custodiet ipsos custodes?
Backdoors
- Using eBPF, potent backdoors can be hidden in the operating system kernel
- Persistent?
- In the network card?
CVE-2021-3490
boopkit
TripeCross
BPFdoor
Symbiote
Bvp47
Explore eBPF
bpftool
- The program
bpftoolis a universal tool to work with eBPF programsbpftoolis part of the Linux kernel source code and is maintained by the kernel developers- Under Linux it can be installed with the package
bpftool(it is usually installed automatically)
List eBPF programs
- The command
bpftool proglists all ePBF programs currently active in the Linux kernel:
# bpftool prog
2: tracing name hid_tail_call tag 7cc47bbf07148bfe gpl
loaded_at 2024-05-15T06:25:33+0200 uid 0
xlated 56B jited 115B memlock 4096B map_ids 2
btf_id 2
47: lsm name restrict_filesystems tag 713a545fe0530ce7 gpl
loaded_at 2024-05-15T06:25:46+0200 uid 0
xlated 560B jited 305B memlock 4096B map_ids 24
btf_id 62
pids systemd(1)
[...]
List eBPF program
- Using
bpftoolthe eBPF assembler code (eBPF bytecode) of a running program can be printed- With newer Linux versions the original source code (if available, here in the "C" language) is inserted as comments inside the assembler code
# bpftool prog dump xlated name restrict_filesystems int restrict_filesystems(unsigned long long * ctx): ; int BPF_PROG(restrict_filesystems, struct file *file, int ret) 0: (79) r3 = *(u64 *)(r1 +0) 1: (79) r0 = *(u64 *)(r1 +8) 2: (b7) r1 = 0 ; uint32_t *value, *magic_map, magic_number, zero = 0, *is_allow; 3: (63) *(u32 *)(r10 -24) = r1 ; int BPF_PROG(restrict_filesystems, struct file *file, int ret) 4: (bf) r1 = r0 5: (67) r1 <<= 32 [...]
list eBPF program
bpftoolcan also print an active program as assembler source code of the native CPU architecture (here x86_64):
# bpftool prog dump jited name restrict_filesystems int restrict_filesystems(unsigned long long * ctx): bpf_prog_713a545fe0530ce7_restrict_filesystems: ; int BPF_PROG(restrict_filesystems, struct file *file, int ret) 0: endbr64 4: nopl 0x0(%rax,%rax,1) 9: xchg %ax,%ax b: push %rbp c: mov %rsp,%rbp f: endbr64 13: sub $0x18,%rsp [...]
list eBPF maps
bpftoolcan list the eBPF maps present in the Linux kernel- eBPF maps are used for data exchange between eBPF programs in the kernel and programs in user space
# bpftool map
2: prog_array name hid_jmp_table flags 0x0
key 4B value 4B max_entries 1024 memlock 8512B
owner_prog_type tracing owner jited
24: hash_of_maps name cgroup_hash flags 0x0
key 8B value 4B max_entries 2048 memlock 165152B
pids systemd(1)
38: array name libbpf_global flags 0x0
key 4B value 32B max_entries 1 memlock 352B
39: array name pid_iter.rodata flags 0x480
key 4B value 4B max_entries 1 memlock 8192B
btf_id 211 frozen
pids bpftool(23682)
40: array name libbpf_det_bind flags 0x0
key 4B value 32B max_entries 1 memlock 352B
eBPF map create/fill/read/delete
# bpftool map create /sys/fs/bpf/mymap type hash key 4 value 4 entries 10 name mymap # ls -l /sys/fs/bpf/ total 0 -rw-------. 1 root root 0 May 16 19:43 mymap # bpftool map update name mymap key 0x00 0x00 0x10 0x20 value 10 10 10 10 # bpftool map dump name mymap key: 00 00 10 20 value: 0a 0a 0a 0a Found 1 element # rm /sys/fs/bpf/mymap rm: remove regular empty file '/sys/fs/bpf/mymap'? y
bpftool
bpftoolsupports working with other eBPF data structures- The output of
bpftoolcan be in JSON (JavaScript Object Notation) for further processing in scripts and programs
# bpftool help
Usage: bpftool [OPTIONS] OBJECT { COMMAND | help }
bpftool batch file FILE
bpftool version
OBJECT := { prog | map | link | cgroup | perf | net | feature | btf | gen | struct_ops | iter }
OPTIONS := { {-j|--json} [{-p|--pretty}] | {-d|--debug} |
{-V|--version} }
Restrict eBPF
Disable "Unprivileged eBPF"
- Until 2022, some (popular) Linux distributions had unprivileged eBPF enabled
- any user could load eBPF code into the kernel!
- Ubuntu (and other Linux distributions) closed this loophole in spring 2022
- Check that the Linux kernel variable (sysctl)
kernel.unprivileged_bpf_disabled> 0
KPROBE_OVERRIDE in the kernel
- If the switch
CONFIG_BPF_KPROBE_OVERRIDEis active in the kernel configuration (at kernel compile time), eBPF programs can overwrite the return values of (kernel) functions - This configuration should not be set in kernels in production environments (check kernel "config")
Switch off KProbes
- By writing the value
0in the pseudo-file/sys/kernel/debug/kprobes/enabledthe kernel probes (KPROBES) are switched off - Attention: a user with write access to this file (e. g.B.
root) can switch the KPROBES on again
Kernel without eBPF
- In security-critical areas, kernels without eBPF function
(eBPF) should be used if necessary. Kernels without eBPF function
(kprobes, XDP, eBPF TC filter) should be used
- For this, the kernel must be recompiled (this is often not practical)
- There is (currently, 2024) no kernel command line switch to switch off eBPF
Using eBPF
Create eBPF programs
- eBPF programs can be created in various ways
- As low level eBPF assembler code
- With a high level compiler (with the help of LLVM): C / GO / Rust / Lua / Python …
- Specialized eBPF script languages: e.g.
bpftrace
BCC
- BCC is the BPF compiler collection
- Website https://github.com/iovisor/bcc
- BCC translates C or Python source code into eBPF programs and can load them into the Linux kernel
Existing (example) BCC eBPF programs
Example of using an eBPF tool created with BCC
- Display the Linux capabilities of running processes
# capable-bpfcc | grep named 07:36:17 0 29378 (named) 24 CAP_SYS_RESOURCE 1 07:36:17 0 29378 (named) 24 CAP_SYS_RESOURCE 1 07:36:17 0 29378 (named) 12 CAP_NET_ADMIN 1 07:36:17 0 29378 (named) 21 CAP_SYS_ADMIN 1 07:36:17 0 29378 named 6 CAP_SETGID 1 07:36:17 0 29378 named 6 CAP_SETGID 1 07:36:17 0 29378 named 7 CAP_SETUID 1 07:36:17 109 29378 named 24 CAP_SYS_RESOURCE 1
The eBPF scripting language "bpftrace"
bpftraceis a small scripting language similar toawkordtrace- Website https://bpftrace.org
bpftraceprograms bind to eBPF probes and execute functions whenever a system event is reported (systemcall, function-call)bpftracehas built-in helper structures to work directly with eBPF data structuresbpftraceallows to write eBPF programs more compact compared to BCC
Instrumenting the Linux TCP/IP stack using eBPF
BCC and bpftrace programs
- There are hundreds of small eBPF programs that allow the user to look deep into the Linux network stack the Linux network stack
- The BCC example programs
- The
bpftraceexample programs - The programs from the eBPF books and related websites
Example: netqtop
netqtop- Outputs statistics about the queues of a network interface. This program can be used to collect information when a network interface is congested
# netqtop-bpfcc -n eth0 -i 10 Mon Nov 15 07:43:29 2021 TX QueueID avg_size [0, 64) [64, 512) [512, 2K) [2K, 16K) [16K, 64K) 0 297.82 2 48 1 4 0 Total 297.82 2 48 1 4 0 RX QueueID avg_size [0, 64) [64, 512) [512, 2K) [2K, 16K) [16K, 64K) 0 70.95 43 34 0 0 0 Total 70.95 43 34 0 0 0 -----------------------------------------------------------------------------
Example: tcptracer
- This program shows the status of TCP connections in the system (A = accept, C = connected, X = closed) as well as the source and destination IP addresses and ports
# tcptracer-bpfcc -p $(pgrep named) Tracing TCP established connections. Ctrl-C to end. T PID COMM IP SADDR DADDR SPORT DPORT C 29404 isc-net-0000 4 127.0.0.1 127.0.0.1 41555 953 A 29378 isc-socket-0 4 127.0.0.1 127.0.0.1 953 41555 X 29404 isc-socket-0 4 127.0.0.1 127.0.0.1 41555 953 X 29378 isc-socket-0 4 127.0.0.1 127.0.0.1 953 41555 C 29378 isc-net-0000 4 46.101.109.138 192.33.4.12 43555 53 C 29378 isc-net-0000 4 46.101.109.138 192.33.4.12 33751 53 X 29378 isc-socket-0 4 46.101.109.138 192.33.4.12 43555 53 X 29378 isc-socket-0 4 46.101.109.138 192.33.4.12 33751 53 C 29378 isc-net-0000 4 46.101.109.138 193.0.14.129 38145 53 C 29378 isc-net-0000 4 46.101.109.138 192.33.14.30 40905 53 X 29378 isc-socket-0 4 46.101.109.138 193.0.14.129 38145 53 X 29378 isc-socket-0 4 46.101.109.138 192.33.14.30 40905 53
Example: tcpconnlat
tcpconnlatoutputs the latency of a TCP-based connection, here an outgoing DNS query via TCP from a BIND 9 resolver (in the example a query frommicrosoft.com txt, where the response is too large for a 1232 byte UDP packet)isc-net-0000is the internal name of the BIND 9 thread
# tcpconnlat-bpfcc PID COMM IP SADDR DADDR DPORT LAT(ms) 29378 isc-net-0000 4 46.101.109.138 193.0.14.129 53 37.50 29378 isc-net-0000 4 46.101.109.138 192.52.178.30 53 14.01 29378 isc-net-0000 4 46.101.109.138 199.9.14.201 53 8.48 29378 isc-net-0000 4 46.101.109.138 192.42.93.30 53 1.90 29378 isc-net-0000 4 46.101.109.138 40.90.4.205 53 14.27 29378 isc-net-0000 4 46.101.109.138 199.254.48.1 53 19.21 29378 isc-net-0000 4 46.101.109.138 192.48.79.30 53 7.66 29378 isc-net-0000 4 46.101.109.138 192.41.162.30 53 7.97 29396 isc-net-0000 4 127.0.0.1 127.0.0.1 53 0.06
Example: udplife
udplifeis abpftracescript to print the UDP round trip time (here DNS round trip time) of a UDP communication (program by Brendan Gregg, see links at the end of the slide deck)
# udplife.bt Attaching 8 probes... PID COMM LADDR LPORT RADDR RPORT TX_B RX_B MS 29378 isc-net-00 46.101.109.138 0 199.19.57.1 16503 48 420 268 29378 isc-net-00 46.101.109.138 0 51.75.79.143 81 49 43 13 29378 isc-net-00 46.101.109.138 0 199.6.1.52 16452 48 408 24 29378 isc-net-00 46.101.109.138 0 199.249.120.1 81 44 10 9 29378 isc-net-00 46.101.109.138 0 199.254.31.1 32891 64 30 273 29378 isc-net-00 46.101.109.138 0 65.22.6.1 32891 64 46 266
Examples for the use of eBPF: "Server agnostic DNS augmentation using eBPF"
- A master thesis by Tom Carpay (with support from NLnet Labs)
- eBPF functions tested by Tom Carpay - Rewriting DNS query names via eBPF
- DNS response rate limiting in the Linux kernel independent of DNS server product
- https://www.nlnetlabs.nl/downloads/publications/DNS-augmentation-with-eBPF.pdf
BIND 9 Instrumentation
Example: Logging DNS Forwarding Decisions
- A BIND 9 DNS resolver configured with a forward zone
zone "dnslab.org" {
type forward;
forwarders { 1.1.1.1; 8.8.8.8; };
};
- The BIND 9 logging system is very powerful, but has no function to log DNS forwarding decisions
- Goal: Create a
bpftracescript to print BIND 9 DNS forwarding decisions
Step 1 - Use the force source
- The BIND 9 source code is openly available and searchable on the ISC Github server: https://gitlab.isc.org
- A search in the BIND 9 source code for forwarding finds the
function
dns_fwdtable_findin/lib/dns/forward.c. This looks promising:
Step 2 - A proof-of-concept test
- The function
dns_fwdtable_findtakes a domain name as input parameter and returns the value0if the name has to be resolved via forwarding, and a value > 0 if forwarding is not used- A
bpftraceone-liner gives us the information whether this function can be used for this task:
- A
bpftrace -e 'uretprobe:/lib/x86_64-linux-gnu/libdns-9.16.22-Debian.so:dns_fwdtable_find { print(retval) }'
Step 2 - A proof-of-concept test
Step 3 - Plan the bpftrace script
- Now that we have found and verified the function for the task we can write a bpftrace script
- The script will
- Save the domain name passed to the
dns_fwdtable_findfunction - Store the domain name passed to the function
dns_fwdtable_findon entry to the function - Check the return value of the function (
retval) for the value zero (0) and output the domain name if forwarding is used
- Save the domain name passed to the
Challenge - Fighting with structs
- The domain name to be checked for forwarding is passed to the function as a data structure (struct) of type
dns_name_t- Unfortunately, it is not a simple pointer to a string that we can print
- A search through the Documentation of the ISC BIND 9 source code
finds the data structure
dns_name_t. The 2nd field is anunsigned char * ndata, this seems to be the domain name
Challenge - fighting with structs
The definition of the data structure
dns_name_tis in the filelib/dns/include/dns/name.h
Challenge - Fighting with structs
bpftraceuses a syntax similar to the C programming language, so we can import the definition of the data structure from the BIND 9 source code directly into thebpftracescript- The linked list and the
isc_buffer_tfield are not needed for our script and since these fields are not based on built-in data types we comment them out:
- The linked list and the
#!/usr/bin/bpftrace
struct dns_name {
unsigned int magic;
unsigned char *ndata;
unsigned int length;
unsigned int labels;
unsigned int attributes;
unsigned char *offsets;
// isc_buffer_t *buffer;
// ISC_LINK(dns_name_t) link;
// ISC_LIST(dns_rdataset_t) list;
};
[...]
Output a text at the start of the script
- The
BEGINpseudo probe becomes active at the start of the script and outputs a message to the terminal to inform the user that the script has been started successfully
[...]
BEGIN {
print("Waiting for forward decision...\n");
}
[...]
Monitor the function call
- This probe becomes active when the function is called in BIND 9
- It is an
uprobe(user-space entry probe) - The probe instrumentalizes the function
dns_fwdtable_findin the dynamic library file/lib/x86_64-linux-gnu/libdns-9.16.22-Debian.so - The 2nd argument of the function call (
arg1) is cast to astruct dns_nameand the fieldndatais referenced - The content of the field is stored in the variable
@dns_name[tid](indexed with the thread ID (tid) of the running BIND 9 thread in the process)
- It is an
[...]
uprobe:/lib/x86_64-linux-gnu/libdns-9.16.22-Debian.so:dns_fwdtable_find
{
@dns_name[tid] = ((struct dns_name *)arg1)->ndata
}
[...]
Monitor the return from the function
- The 3rd probe becomes active when the function is exited
(
uretprobe- User-space function Return Probe)- Same library file and function as before
- If the return value of the function is zero
0(domain name must be resolved via forwarding), the value of the variable@dns_name[tid]is converted into a character string and printed on the terminal - The variable
@dns_name[tid]is no longer required and is deleted
uretprobe:/lib/x86_64-linux-gnu/libdns-9.16.22-Debian.so:dns_fwdtable_find
{
if (retval == 0) {
printf("Forwarded domain name: %s\n", str(@dns_name[tid]));
}
delete(@dns_name[tid]);
}
The complete script
#!/usr/bin/bpftrace
struct dns_name {
unsigned int magic;
unsigned char *ndata;
unsigned int length;
unsigned int labels;
unsigned int attributes;
unsigned char *offsets;
// isc_buffer_t *buffer;
// ISC_LINK(dns_name_t) link;
// ISC_LIST(dns_rdataset_t) list;
};
BEGIN
{
print("Waiting for forward decision...\n");
}
uprobe:/lib/x86_64-linux-gnu/libdns-9.16.22-Debian.so:dns_fwdtable_find
{
@dns_name[tid] = ((struct dns_name *)arg1)->ndata
}
uretprobe:/lib/x86_64-linux-gnu/libdns-9.16.22-Debian.so:dns_fwdtable_find
{
if (retval == 0) {
printf("Forwarded domain name: %s\n", str(@dns_name[tid]));
}
delete(@dns_name[tid]);
}
The script in use
- Whenever a domain name is resolved in the BIND 9 resolver, the
bpftracescript also becomes active- In this example, all requests to the domain
dnslab.orgdomain are forwarded via forwarding, but not the requests toietf.org
- In this example, all requests to the domain
Packet filter with eBPF
eBPF as a network firewall
- eBPF can be used to build very efficient firewall systems
- eBPF can stop network traffic before it reaches the Linux TCP/IP stack or the application
- Since eBPF executes a full program for each network packet, complex filters can be defined
- Filter based on DNS query names
- DNSSEC data available in the response?
- Source IP of the responding authoritative DNS server (block known "malicious" DNS servers)
- EDNS data (prefer DNS packets with DNS cookies)
- …
eBPF Firewall example: Block non-DNS
- Our hands-on example today shows a simple eBPF network filter
- The filter blocks all UDP traffic to an interface (here the loopback interface) with the exception of UDP DNS packets (port 53)
- This helps to defend against non-DNS DDoS attacks on an authoritative DNS resolver
Literature
Book: Linux Observability with BPF By David Calavera, Lorenzo Fontana (November 2019)
Book: Systems Performance (2nd ed.) By Brendan Gregg (December 2020)
Book: BPF Performance Tools By Brendan Gregg (December 2019)
Report: What is eBPF? (O'Reilly)
Report: Security Observability with eBPF (O'Reilly)
Workshop "eBPF"
- Impromptu workshop 18:00-18:45 CEST in HS3 (after closing words)
- Instructions / Discussions in German/English language
- "Hands-On" eBPF - Using eBPF in real-world problems on a VM
- BCC and bpftrace
- Writing your own eBPF programs
- Using
bpftoolto find and analyze eBPF programs on a Linux system
Links
eBPF
- eBPF support and Linux kernel versions https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md
- Awesome BPF https://github.com/zoidbergwill/awesome-ebpf
- Using user-space tracepoints with BPF https://lwn.net/Articles/753601/
- Extending systemd Security Features with eBPF https://kinvolk.io/blog/2021/04/extending-systemd-security-features-with-ebpf/
- Absolute Beginner's Guide to BCC, XDP, and eBPF https://dev.to/satrobit/absolute-beginner-s-guide-to-bcc-xdp-and-ebpf-47oi
eBPF
- Linux Extended BPF (eBPF) Tracing Tools https://www.brendangregg.com/ebpf.html
- Performance Implications of Packet Filtering with Linux eBPF https://www.net.in.tum.de/fileadmin/bibtex/publications/papers/ITC30-Packet-Filtering-eBPF-XDP.pdf
- eBPF for perfomance analysis and networking
https://marioskogias.github.io/students/debeule.pdf
- BPF and XDP Reference Guide https://docs.cilium.io/en/v1.10/bpf/
BCC
- Intro to Kernel and Userspace Tracing Using BCC, Part 1 of 3 https://blogs.oracle.com/linux/post/intro-to-bcc-1
bpftrace
- bpftrace Reference Guide https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md
- Kernel analysis with bpftrace https://lwn.net/Articles/793749/
- The bpftrace One-Liner Tutorial https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.md
- Full-system dynamic tracing on Linux using eBPF and bpftrace https://www.joyfulbikeshedding.com/blog/2019-01-31-full-system-dynamic-tracing-on-linux-using-ebpf-and-bpftrace.html
- bpftrace Cheat Sheet https://www.brendangregg.com/BPF/bpftrace-cheat-sheet.html
Network Scripts
eBPF Prometheus exporter
- eBPF exporter https://blog.cloudflare.com/introducing-ebpf_exporter/
eXpress Data Path (XDP)
- Introduction to: XDP and BPF building blocks https://people.netfilter.org/hawk/presentations/ebplane2019/xdp-bpf-building-blocks.pdf
- A practical introduction to XDP https://www.linuxplumbersconf.org/event/2/contributions/71/attachments/17/9/presentation-lpc2018-xdp-tutorial.pdf
- eBPF/XDP https://www.slideshare.net/Netronome/ebpfxdp-sigcomm-2018
- XDP Packet filter and UDP https://fly.io/blog/bpf-xdp-packet-filters-and-udp/
- XDP Firewall https://github.com/gamemann/XDP-Firewall
eXpress Data Path (XDP)
- How to filter packets super fast: XDP & eBPF! https://jvns.ca/blog/2017/04/07/xdp-bpf-tutorial/
- Load XDP programs using the ip (iproute2) command https://medium.com/@fntlnz/load-xdp-programs-using-the-ip-iproute2-command-502043898263
- L4Drop: XDP DDoS Mitigations https://blog.cloudflare.com/l4drop-xdp-ebpf-based-ddos-mitigations/
- How to drop a packet in Linux in more ways than one https://codilime.com/blog/how-to-drop-a-packet-in-linux-in-more-ways-than-one/
- eBPFsnitch https://github.com/harporoeder/ebpfsnitch
- XDP minimal example https://ruderich.org/simon/notes/xdp-minimal-example
- Why is the kernel community replacing iptables with BPF? https://cilium.io/blog/2018/04/17/why-is-the-kernel-community-replacing-iptables
BPF Backdoor
- Linux eBPF backdoor over TCP https://github.com/kris-nova/boopkit
- A Linux eBPF rootkit with a backdoor, C2, library injection, execution hijacking, persistence and stealth capabilities https://github.com/h3xduck/TripleCross
- Offensive BPF: Using bpftrace to host backdoors https://embracethered.com/blog/posts/2021/offensive-bpf-bpftrace-message-based/
- Offensive BPF: Malicious bpftrace https://embracethered.com/blog/posts/2021/offensive-bpf-bpftrace/
BPF malware/attacks
- BPFDoor: Chinese tool almost undetected for FIVE years is second BPF-based attack uncovered this year https://linuxsecurity.com/news/hackscracks/bpfdoor-chinese-tool-almost-undetected-for-five-years-is-second-bpf-based-attack-uncovered-this-year
- Anatomy of suspected top-tier decade-hidden NSA backdoor https://www.theregister.com/2022/02/23/chinese_nsa_linux/
BPF malware/attacks
- Hackers using stealthy Linux backdoor Symbiote to steal credentials https://www.csoonline.com/article/3663510/hackers-using-stealthy-linux-backdoor-symbiote-to-steal-credentials.html
- Bvp47 - Top-tier Backdoor of US NSA Equation Group https://www.pangulab.cn/files/The_Bvp47_a_top-tier_backdoor_of_us_nsa_equation_group.en.pdf
eBPF security
- Mapping It Out: Analyzing the Security of eBPF Maps https://www.crowdstrike.com/blog/analyzing-the-security-of-ebpf-maps/
- With Friends like eBPF, who needs enemies ? https://i.blackhat.com/USA21/Wednesday-Handouts/us-21-With-Friends-Like-EBPF-Who-Needs-Enemies.pdf
- Kernel Pwning with eBPF: a Love Story https://www.graplsecurity.com/post/kernel-pwning-with-ebpf-a-love-story
eBPF security
- Ubuntu is releasing updated kernels that disable unprivileged eBPF by default https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/BHI
- Toward signed BPF programs https://lwn.net/Articles/853489/
- CONFIG_BPF_KPROBE_OVERRIDE: Enable BPF programs to override a kprobed function https://cateee.net/lkddb/web-lkddb/BPF_KPROBE_OVERRIDE.html
- The kprobes debugfs interface https://www.kernel.org/doc/Documentation/kprobes.txt
eBPF security issues
- CVE-2021-33624 kernel: Linux kernel BPF protection against speculative execution attacks can be bypassed to read arbitrary kernel memory
eBPF security tools
- A flow-based IDS using Machine Learning in eBPF https://arxiv.org/abs/2102.09980
- ebpfkit-monitor is a tool that detects and protects against eBPF powered rootkits https://github.com/Gui774ume/ebpfkit-monitor
- MAC and Audit policy using eBPF (KRSI) https://lwn.net/Articles/813057/
- KRSI — the other BPF security module https://lwn.net/Articles/808048/
eBPF Audit-Tools
- ptcpdump - Process-aware
tcpdumphttps://github.com/mozillazg/ptcpdump - bpfcontain-rs - https://github.com/willfindlay/bpfcontain-rs/
- Tracee - Linux Runtime Security and Forensics using eBPF https://github.com/aquasecurity/tracee
- falco - Cloud Native Runtime Security https://github.com/falcosecurity/falco
- Cilium Tetragon - eBPF-based Security Observability and Runtime Enforcement https://github.com/cilium/tetragon
Thank you!
Contact:
cs@sys4.de