leodido.dev - eBPFLeo's take on security, eBPF, Linux, kernel, and whatever tech he meetsZola2022-02-07T00:00:00+00:00https://leodido.dev/tags/ebpf/atom.xmlCode coverage for eBPF programs2022-02-07T00:00:00+00:002022-02-07T00:00:00+00:00https://leodido.dev/code-coverage-ebpf/<p>I bet we all have heard so much about <abbr title="extended Berkeley Packet Filter">eBPF</abbr> in recent years. Every day we hear about a new project or application using some eBPF black magic underneath. <a rel="noopener noreferrer" target="_blank" href="https://www.google.com/url?q=https://thenewstack.io/this-week-in-programming-ebpf-coming-to-a-windows-near-you">Data</a> shows that eBPF is quickly becoming the first choice for implementing tracing and security applications.</p>
<p>However, one major challenge is that the eBPF ecosystem lacks tooling to make developers’ lives easier. eBPF programs are written in C but compiled for a specific ISA later executed by the eBPF Virtual Machine. LLVM has a specific backend allowing us to write C and get eBPF ELF objects out. There are no tools helping developers to clearly <mark>understand</mark> which <mark>path</mark> their code took while running in the Linux kernel, which <mark>code regions</mark>, or, even better, <mark>code branches</mark> are uncovered, and maybe why.</p>
<p>You may already be familiar with code coverage, which shows you which lines of code execute. Code coverage is usually applied to tests to discover which line gets run and which does not. But even testing the eBPF programs is a pain, given that <code>BPF_PROG_TEST_RUN</code> does not support all the types of eBPF programs living in the Linux kernel.</p>
<p><a rel="noopener noreferrer" target="_blank" href="https://github.com/elastic/bpfcov">That’s why I sat down and wrote bpfcov</a>: a tool to gather <mark>source-based coverage</mark> info for our eBPF programs running in the Linux kernel. Whether they are getting loaded via <code>BPF_PROG_TEST_RUN</code> or by other ordinary means. Until today, there was no simple way to visualize how the flow of your eBPF program running in the kernel was. Hopefully, there is one starting today.</p>
<p>So, let’s jump straight into the topic.</p>
<h2 id="source-based-code-coverage-for-ebpf">Source-based code coverage for eBPF<a class="zola-anchor" href="#source-based-code-coverage-for-ebpf" aria-label="Anchor link for: source-based-code-coverage-for-ebpf">§</a>
</h2>
<p>Everyone reading this post probably knows what the code coverage is. Common line-level coverage gives us a sense of what line is executed. In some cases, it even tells us how many times a line got executed.</p>
<p>When building this tool, driven by my experience fighting with BPF, I knew I wanted something more. Line-level granularity is often too coarse. We do not want an approximation of what code actually executed.
We need something more precise to understand the execution path of our eBPF programs in the Linux kernel. We even want to know what part of an <code>if</code> conditional executed<label for="sn-1" class="sidenote-toggle sidenote-number"></label><input type="checkbox" id="sn-1" class="sidenote-toggle"/><span class="sidenote">I suggest you to watch <a rel="noopener noreferrer" target="_blank" href="https://www.youtube.com/watch?app=desktop&v=H1hvtJPGWNQ">this talk</a> by Alan Phipps to know more about branch coverage.</span>. Source-based code coverage is what I wanted for this reason.</p>
<p>Since it starts at code generation time in LLVM, it has the notion of <mark>regions</mark> of the code, <mark>branches</mark>, and so on. It even precisely counts things like <mark>short-circuited conditionals</mark>, thanks to the <mark>counter expression</mark> and arithmetics between them. It generates coverage summaries with very fined-grained code regions, helping us find grasps in the code and its execution flow.</p>
<p>So, given eBPF programs are usually written in C, couldn’t we just instrument them for <a rel="noopener noreferrer" target="_blank" href="https://clang.llvm.org/docs/SourceBasedCodeCoverage.html">source-based code coverage</a> as we would commonly do with Clang for our C programs? I bet this is the first argument pumping into the head of many readers. It also was one of my thoughts when approaching this problem. Turns out that we definitely can, and we will do it. But it won’t work as is.</p>
<p>To understand why it won’t work, we need to keep in mind that BPF programs are compiled via Clang<label for="sn-2" class="sidenote-toggle sidenote-number"></label><input type="checkbox" id="sn-2" class="sidenote-toggle"/><span class="sidenote">Thanks to the <a rel="noopener noreferrer" target="_blank" href="https://github.com/llvm/llvm-project/tree/main/llvm/lib/Target/BPF">LLVM BPF target</a>.</span> to BPF ELF object files, containing instructions specific to the chosen BPF instruction set, which need to be later loaded in the Linux kernel via the <a rel="noopener noreferrer" target="_blank" href="https://www.kernel.org/doc/html/latest/userspace-api/ebpf/syscall.html">bpf()</a> syscall. Furthermore, it’s paramount to mention that the BPF programs will be verified by an in-kernel verifier and then executed by the BPF VM.</p>
<p>Such a lifecycle and environment imposes a set of constraints that make it infeasible to get them working with the plain instrumentation that LLVM applies to get source-based code coverage. In fact, when compiling a C program for source-based coverage with the <code>-fprofile-instr-generate</code> and <code>-fcoverage-mapping</code> Clang flags, LLVM instruments it with a bunch of global variables and, in some cases, functions.</p>
<p>Observing the LLVM IR of a C program after compiling it with the <code>-fprofile-instr-generate</code> flag, we can notice that, LLVM…</p>
<ul>
<li>defines the counters, in the form of <code>__profc_<function_name></code> private global arrays whose size is the number of counters needed to cover all the regions and branches of such a function</li>
<li>marks those counters to be into the <code>__llvm_prf_cnts</code> section of the ELF</li>
<li>defines (and initializes) <code>__profd_<function_name></code> private global struct instances that contains an identifier of the function, the pointer to its counters, the pointer to the function itself, the number of the counters for the target function, and a bunch of other info needed to tie together the counters and the coverage mappings</li>
<li>marks the <code>__profd_</code> globals to end up into the <code>__llvm_prf_data</code> section of the ELF</li>
<li>defines (and initializes) a private constant (<code>__llvm_prf_nm</code>) containing the names of the target functions</li>
<li>marks the names constant to end up in the <code>__llvm_prf_names</code> section of the ELF</li>
</ul>
<p>The first issue to dismount here is that we can’t have random ELF sections (like<code>__llvm_prf_cnts</code>, etc.) into valid BPF ELF files.</p>
<p>We know that we can have eBPF global constants with a scalar type like <code>__profc_</code> ones, at least on recent Linux kernels. At the same time, we know it’s not wise to have global structs like <code>__profd_</code> or whatever loader for our eBPF programs we will write, as we’ll need the struct definitions. Too clunky.</p>
<p>The <code>-fprofile-instr-generate</code> flag also generates a global constructor and a set of global functions, all prefixed with <code>__llvm_profile</code>, intended to set up the profiling runtime in the resulting binary so that when it dies or exits it will automatically generate a <mark>profraw</mark> file.</p>
<p>Here we meet another stumbling block to overcome. The BPF verifier will refuse our BPF ELF containing strange global functions, not to mention “constructors”…</p>
<p>We need to get rid of them and replace their functionality with a feasible approach for BPF. Clearly, LLVM also patches our instructions in the right spots, by placing the counters and incrementing them. As you can see in the following annotated screenshot:</p>
<p><a href="profile-instr-generate2.png"><img src="profile-instr-generate2.png" alt="LLVM IR incrementing profiling counters at the correct spots" /></a></p>
<p>If you write an eBPF program incrementing a <mark>counter</mark> (defined as an <mark>eBPF global variable</mark>) you will notice that the resulting instructions will be like those here in the screenshot. It means we can keep these the way they are generated by LLVM and let it do all its magic without interfering at the instructions level. Finally, some good news!</p>
<p>Instead, focusing on the <code>-fcoverage-mapping</code> Clang flag and inspecting the LLVM IR it outputs, we notice that it…</p>
<ul>
<li>defines (and initializes) <code>__covrec_</code> global (constant) structs, most notably containing the same ID that the <code>__profd_</code> variables contain, and a LEB128<label for="sn-3" class="sidenote-toggle sidenote-number"></label><input type="checkbox" id="sn-3" class="sidenote-toggle"/><span class="sidenote">The encoding of the coverage mapping values is <a rel="noopener noreferrer" target="_blank" href="https://en.wikipedia.org/wiki/LEB128">LEB128</a>.</span> encoded string containing all the region, branches, and generally the coverage mapping info</li>
<li>defines (and initializes) a <code>__llvm_coverage_mapping</code><label for="sn-4" class="sidenote-toggle sidenote-number"></label><input type="checkbox" id="sn-4" class="sidenote-toggle"/><span class="sidenote">The <a rel="noopener noreferrer" target="_blank" href="https://llvm.org/docs/CoverageMappingFormat.html">Coverage Mapping</a> format.</span> function containing meta info about the coverage mapping format (eg., the version), and the source file names</li>
<li>marks the <code>__llvm_coverage_mapping</code> variable to be put into the <code>__llvm_covmap</code> section of the ELF.</li>
</ul>
<p>In this case, we have the same category of issues mentioned before.</p>
<p>We need to keep at least the header of <code>__llvm_coverage_mapping</code> because it contains the coverage mappings version, which we need for generating a valid <mark>profraw</mark> file<label for="sn-5" class="sidenote-toggle sidenote-number"></label><input type="checkbox" id="sn-5" class="sidenote-toggle"/><span class="sidenote">More on the <mark>profraw</mark> format <a href="/demystifying-profraw">in a previous blog post</a>.</span>. Also, we need to put it in a <mark>custom BPF ELF section</mark>.</p>
<p>Luckily, we can remove the <code>__covrec_</code> structs from the BPF ELF meant to be loaded in the kernel. We can keep them in a second BPF ELF that would be intended to be given to tools needing the coverage mappings for generating the coverage reports, just like <mark>llvm-cov</mark> does.</p>
<h2 id="how-it-s-done">How it’s done<a class="zola-anchor" href="#how-it-s-done" aria-label="Anchor link for: how-it-s-done">§</a>
</h2>
<h3 id="libbpfcov-so-a-llvm-pass">libBPFCov.so - A LLVM pass<a class="zola-anchor" href="#libbpfcov-so-a-llvm-pass" aria-label="Anchor link for: libbpfcov-so-a-llvm-pass">§</a>
</h3>
<p>By analyzing the resulting LLVM IR for source-based coverage instrumented programs, we should now have a better understanding of what we need to do for having it on eBPF programs running in the Linux kernel. We now know what to completely get rid of and do differently, and what we need to patch to make it loadable and usable by the BPF VM in the kernel.</p>
<p>The plan is simple: get Clang to instrument a BPF LLVM intermediate representation for source-based code coverage, then patch it to model it into a valid representation for BPF ELF.</p>
<p>How do we need to transform it?</p>
<p>First of all, we are so lucky we don’t have to mess with the actual BPF instructions — namely the counters increments. We can keep them the way they are. This is a huge win because we let LLVM keep track of the global state of the registers and we avoid a lot of work this way.</p>
<p>But for sure we have to strip any profile initialization stuff that Clang creates, things like <code>__llvm_profile_runtime</code> and <code>__llvm_profile_init</code> - when present - are no good for the BPF VM in the kernel.</p>
<p>We also want to ensure the global variables, whether constants or not, have the right visibility (ie., <code>dso_local</code>) and linkage, to have them in the <mark>libbpf</mark> skeletons if we plan to use them.</p>
<p>For the global structs that we need for generating the <mark>profraw</mark> files, namely the <code>__profd_</code> variables, we just transform them into different and single global variables, one for each field.</p>
<p>For example, this is what I did for the <code>__profd_*</code> variables which originally are a struct with 7 fields. For other global structs like the <code>__covrec_</code> ones, we can just strip them from the BPF ELF that is meant to be loaded in the kernel.</p>
<p>Anyway, the report generation phase (ie., <mark>llvm-cov</mark> or <mark>bpfcov out</mark>) will need them for knowing at which line and column a code region or a branch starts. For this reason, I decided to give the LLVM pass an option (enabled with the <code>strip-initializers-only</code> flag) that keeps them, so we can later create a BPF ELF that is only meant for this phase and not for loading.</p>
<p>This BPF ELF will have <code>.bpf.obj</code> as an extension, rather than <code>.bpf.o</code>.</p>
<p>Finally, we know that <a rel="noopener noreferrer" target="_blank" href="https://github.com/libbpf/libbpf">libbpf</a> supports (on recent Linux kernels) <mark>eBPF global variables</mark><label for="sn-6" class="sidenote-toggle sidenote-number"></label><input type="checkbox" id="sn-6" class="sidenote-toggle"/><span class="sidenote">Kernel patch: <a rel="noopener noreferrer" target="_blank" href="https://patchwork.ozlabs.org/project/netdev/cover/20190409212018.32423-1-daniel@iogearbox.net/">eBPF support for global data</a>.</span>, which are simply eBPF maps with one single value, and we are planning to use them. But, as already mentioned, it does not accept or recognize the ELF sections that the Clang instrumentation injects in the intermediate representation.</p>
<p>So we need our LLVM pass to change them to <mark>custom eBPF sections</mark><label for="sn-7" class="sidenote-toggle sidenote-number"></label><input type="checkbox" id="sn-7" class="sidenote-toggle"/><span class="sidenote">Kernel patch: <a rel="noopener noreferrer" target="_blank" href="https://lore.kernel.org/bpf/20211021014404.2635234-8-andrii@kernel.org/">libbpf: support global data/bss/rodata sections</a>.</span>. The eBPF custom sections are in the form of <code>.rodata.*</code> or <code>.data.*</code> made to contain static and/or global data. We can change the section of the counters to be <code>.data.profc</code>. The section of the <code>__llvm_prf_nm</code> from <code>_llvm_prf_names</code> to <code>.rodata.profn</code>, and so on. You can find all this logic summarized in these <a rel="noopener noreferrer" target="_blank" href="https://github.com/elastic/bpfcov/blob/5607ee6b5419cd72dfd9c17e00455188865f8766/lib/BPFCov.cpp#L660-L694">bits of code</a>.</p>
<p>So, assuming the following dummy eBPF program:</p>
<p><a href="code-coverage-ebpf-programs-dummy-ebpf.png"><img src="code-coverage-ebpf-programs-dummy-ebpf.png" alt="Raw tracepoint on sys_enter with eBPF" /></a></p>
<p>I think that the following snippet tells everything about what we obtain from the <mark>libBPFCov.so</mark> LLVM pass:</p>
<p><a href="libbpfcov-llvm-ir.png"><img src="libbpfcov-llvm-ir.png" alt="Resulting LLVM IR after running bpfcov libBPFCov.so LLVM pass" /></a></p>
<p>For example, you may notice that we have 2 <code>__profc_</code> counters.</p>
<p>The first is for the <code>BPF_PROG</code> macro that expands to a function, the second for the actual BPF raw tracepoint program <code>hook_sys_enter</code>. This one has size 3. That’s because the <code>hook_sys_enter</code> function has 3 main regions: the entry of the function, the <code>if</code> conditional, and the <code>for</code> cycle.</p>
<p>You may also notice that the LLVM pass, for each one of the 2 functions we have, split the <code>__profd_</code> global structs into 7 different global variables in the <code>.rodata.profd</code> section.</p>
<p>Someone who has an eye for it may also have noticed that the third field of <code>__profd_</code> — now <code>__profd_something.2</code> — does not contain anymore the address of its counters. I didn’t want (nor I could) to expose kernel addresses, so I put here the offset of the counters in their section (<code>.data.profc</code>).</p>
<p>Finally, you can also see that, as anticipated before, we completely deleted the <code>__covrec_</code> global constant structs from this IR that’s meant to generate a valid and loadable BPF ELF. While the instructions incrementing the counters in the correct spots are not touched at all. So we don’t need another screenshot to show them!</p>
<p>The only missing moving part is how to generate a valid <mark>profraw</mark> file. We stripped any logic for doing it. We know that for generating it we need all the globals we left in this LLVM intermediate representation. But we have no sane way to hook the exit or the stop of an eBPF program in the Linux kernel.</p>
<p>Suddenly, inspiration came: let’s <mark>pin</mark> the globals to the <mark>BPF file system</mark> so that we can decouple the process of generating the <mark>profraw</mark> file from the running (and exiting) of the instrumented eBPF application!</p>
<p>And that’s what the <mark>bpfcov</mark> <strong>CLI</strong> does.</p>
<p>Before moving to the next section, I suggest you go to the <a rel="noopener noreferrer" target="_blank" href="https://github.com/elastic/bpfcov">bpfcov repository</a> and start building the pass to obtain <mark>libBPFCov.so</mark>. You can find the instructions on <a rel="noopener noreferrer" target="_blank" href="https://github.com/elastic/bpfcov/blob/main/README.md#building">how to build it here</a>.</p>
<h2 id="instrument-your-ebpf-program">Instrument your eBPF program<a class="zola-anchor" href="#instrument-your-ebpf-program" aria-label="Anchor link for: instrument-your-ebpf-program">§</a>
</h2>
<p>Now that we have built <mark>libBPFCov.so</mark> we can finally take action!</p>
<p>Instrumenting an eBPF program for source-based coverage is not that different than compiling it normally with Clang and the BPF target.</p>
<p>The only difference is that we ask Clang to output LLVM IR (either in textual or binary form), run the <mark>libBPFCov.so</mark> pass on it (with <code>opt</code>), and finally compile it (with <code>llc</code>) to a BPF ELF.</p>
<pre data-lang="sh" class="language-sh "><code class="language-sh" data-lang="sh">clang -g -O2 \
-target bpf \
-D__TARGET_ARCH_x86 -I$(YOUR_INCLUDES) \
-fprofile-instr-generate -fcoverage-mapping \
-emit-llvm -S \
-c raw_enter.bpf.c -o raw_enter.bpf.ll
opt -load-pass-plugin $(BUILD_DIR)/lib/libBPFCov.so -passes="bpf-cov" \
-S raw_enter.bpf.ll -o raw_enter.bpf.cov.ll
llc -march=bpf -filetype=obj -o cov/raw_enter.bpf.o raw_enter.bpf.cov.ll
</code></pre>
<p>You can see in the Makefile inside the <a rel="noopener noreferrer" target="_blank" href="https://github.com/elastic/bpfcov/tree/main/examples/src">examples/src</a> directory of the <a rel="noopener noreferrer" target="_blank" href="https://github.com/elastic/bpfcov">bpfcov GitHub repository</a> how to automate those steps.</p>
<p>We now have a valid and coverage instrumented BPF ELF: <code>cov/raw_enter.bpf.o</code>.</p>
<p>From now on, you can instruct your loader and userspace code to use it, so to obtain a binary (eg., <code>/cov/raw_enter</code>) that is your eBPF application.</p>
<h3 id="use-it">Use it<a class="zola-anchor" href="#use-it" aria-label="Anchor link for: use-it">§</a>
</h3>
<h4 id="bpfcov-run-bpfcov-gen">bpfcov run + bpfcov gen<a class="zola-anchor" href="#bpfcov-run-bpfcov-gen" aria-label="Anchor link for: bpfcov-run-bpfcov-gen">§</a>
</h4>
<p>What’s left to do? Just three steps:</p>
<ol>
<li>Run our eBPF application with the <mark>bpfcov</mark> <strong>CLI</strong> run command</li>
<li>Generate its <mark>profraw</mark> file</li>
<li>Generate beautiful source-based coverage reports</li>
</ol>
<p>So, let’s run our eBPF application with:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">sudo ./bpfcov -v2 run cov/raw_enter
</code></pre>
<p>This command acts similar to strace. It will detect the <code>bpf()</code> syscalls with the <code>BPF_MAP_CREATE</code> command.</p>
<p>Meaning that it will detect the eBPF globals in the <code>.profc</code>, <code>.profd</code>, <code>.profn</code>, and <code>.covmap</code> custom eBPF sections and pin them to the BPF file system, as you can see in the following screenshot.</p>
<p><a href="bpfcov-run.png"><img src="bpfcov-run.png" alt="Output of bpfcov run" /></a></p>
<p>You may also notice that - since the LLVM pass annotated the counters correctly - <mark>libbpf</mark> can collect relocations for them…</p>
<p>At this point, whether we stopped our eBPF application or it exited… We have eBPF maps pinned to our BPF file system. Let’s check it:</p>
<figure class="center">
<a href="pinned-maps.png">
<img src="pinned-maps.png" alt="eBPF maps pinned by bpfcov" />
</a>
</figure>
<p>Wonderful, we already know that the <code>hook_sys_enter</code> function executed 1 time, the <code>if</code> condition did not evaluate to true, while the <code>for</code> iterated 9 times!</p>
<p>It’s time to put the counters, the function names, the functions data, in a <mark>profraw</mark> file now.</p>
<p>This is why I created the <mark>bpfcov gen</mark> command exists: to dump the pinned maps in a <mark>profraw</mark> file.</p>
<pre data-lang="shell" class="language-shell "><code class="language-shell" data-lang="shell">sudo ./bpfcov -v2 gen –unpin cov/raw_enter
</code></pre>
<figure class="center">
<a href="bpfcov-gen.png">
<img src="bpfcov-gen.png" alt="Generating a profraw file from eBPF pinned maps with bpfcov" />
</a>
</figure>
<p>And this is the resulting <mark>profraw</mark> file for our instrumented eBPF program!</p>
<p>You can see it’s made of four parts: a header, <code>.rodata.profd</code>, <code>.data.profc</code> (ie., the counters!), and the names (<code>.rodata.profn</code>), plus some padding for alignment…</p>
<figure class="center">
<a href="bpf-profraw.png">
<img src="bpf-profraw.png" alt="A profraw file from an eBPF program running in the Linux kernel" />
</a>
</figure>
<p>We can now either use the existing LLVM tools (<mark>llvm-profdata</mark> and <mark>llvm-cov</mark>) with it or simply use the <mark>bpfcov out</mark> subcommand.</p>
<p>The <mark>bpfcov out</mark> command is an opinionated shortcut to generate HTML, JSON, or LCOV coverage reports even from multiple eBPF programs and their <mark>profraw</mark> files.</p>
<p>It is very convenient because it avoids us having to generate <mark>profdata</mark> from the <mark>profraw</mark>, calling <mark>llvm-cov</mark> with a bunch of long and different options. And it even works with multiple <mark>profraw</mark> files coming from different eBPF applications…</p>
<pre data-lang="shell" class="language-shell "><code class="language-shell" data-lang="shell">./bpfcov out -o yey –f html cov/raw_enter.profraw ...
</code></pre>
<p>It outputs a very nice HTML directory whose index file gives us summaries not only about function and line coverage but also and notably about region and branch coverages for our eBPF applications.</p>
<figure class="center">
<a href="mult1.png">
<img src="mult1.png" alt="HTML report of source-base code coverage of a few eBPF programs" />
</a>
</figure>
<p>By clicking on any item in the table we end up visualizing very fine-grained, source-based coverage. A good example is the one in the following image:</p>
<figure class="center">
<a href="html1.png">
<img src="html1.png" alt="HTML report of a raw tracepoint eBPF program on sys_enter" />
</a>
</figure>
<p>Furthermore, it does work also on very complicated and real-life eBPF programs. For example, the following screenshot is a part of the coverage report obtained for a BPF LSM test (<a rel="noopener noreferrer" target="_blank" href="https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/progs/lsm.c">progs/lsm.c</a>, loaded by <a rel="noopener noreferrer" target="_blank" href="https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/prog_tests/test_lsm.c">prog_tests/test_lsm.c</a>) living in the Linux kernel.</p>
<p><a href="html2.png"><img src="html2.png" alt="HTML report of a few BPF LSM programs from the Linux kernel" /></a></p>
<p>Thanks to this tool I can finally understand that over a total of eight executions of the <code>lsm/file_mprotect</code> BPF LSM program on my kernel, its <code>is_stack</code> variable was true two times out of eight, because six times the <code>vma->vm_end >= vma->vm_mm->start_stack</code> branch condition (98:58) evaluated to false.</p>
<p>Line 100, using <code>is_stack</code> in another condition, confirms that it was indeed true two times out of six. And that, for this reason (first operand - 100:6 - <code>is_stack</code> being false six times), the following check (100:18) on <code>monitored_pid</code> was short-circuited and evaluated (to true, by the way) only two times.</p>
<p>We finally have a tool helping us write and understand the way our eBPF programs run in the Linux kernel. I can’t stress enough how this is something I dreamt of so many times during the past few years I’ve been working with BPF…</p>
<p>Hope that the eBPF community and ecosystem will find bpfcov useful and cool the same way I do.</p>
<h2 id="bits-of-the-future">Bits of the future<a class="zola-anchor" href="#bits-of-the-future" aria-label="Anchor link for: bits-of-the-future">§</a>
</h2>
<p>The <a rel="noopener noreferrer" target="_blank" href="https://github.com/elastic/bpfcov">bpfcov</a> tool is open-source, and it will stay that way. It is still in its early days so it will probably need a bit more tests, examples, and fixes. Just like any new software out there.</p>
<p>I will soon publish another project showcasing the coverage of the kernel BPF selftests, using this tool. This means there are a lot of contribution opportunities. I’d invite you to take a look at <a rel="noopener noreferrer" target="_blank" href="https://github.com/elastic/bpfcov">its source code</a> and send patches! 😊</p>
<p>Also, in case you want to start using it on your eBPF applications and repositories, feel free to contact me for support. I’d love to help you to use it.</p>
<p>From a technical perspective these are the topics that are on top of my mind for its future:</p>
<ul>
<li>A project like <a rel="noopener noreferrer" target="_blank" href="https://github.com/elastic/bpfcov">bpfcov</a> <strong>must</strong> have a logo!</li>
<li>Write <mark>llvm-lit</mark> tests for the <mark>libBPFCov.so</mark> LLVM pass</li>
<li>Support newer LLVM versions</li>
<li>Workaround solutions to have it working on Linux kernels where BPF does not support custom eBPF sections and eBPF globals</li>
<li>Create a versioning and release process</li>
<li>Publish artifacts on GitHub (<mark>libBPFCov.so</mark>, <mark>bpfcov</mark> <strong>CLI</strong> binary)</li>
<li>Add more examples</li>
<li>Publish HTML reports of the example eBPF applications via GitHub pages</li>
<li>Maybe, contribute it <strong>upstream</strong> to the LLVM BPF target!</li>
</ul>
<h2 id="fosdem-2022">FOSDEM 2022<a class="zola-anchor" href="#fosdem-2022" aria-label="Anchor link for: fosdem-2022">§</a>
</h2>
<p>In case you made it to the end, and you still want to hear more details on building source-based coverage for eBPF, I want to bring to your attention that I gave a <a rel="noopener noreferrer" target="_blank" href="https://fosdem.org/2022/schedule/event/llvm_ebpf/">talk</a> on <a rel="noopener noreferrer" target="_blank" href="https://github.com/elastic/bpfcov">bpfcov</a> at <a rel="noopener noreferrer" target="_blank" href="https://fosdem.org/2022/schedule/event/llvm_ebpf/">FOSDEM 2022</a>.</p>
<p>I will update my <a href="/talks">speaking</a> section with the video recording as soon it’ll be ready.</p>
<p>In the meantime, you can take a look at the <a rel="noopener noreferrer" target="_blank" href="https://speakerdeck.com/leodido/coverage-for-ebpf-programs">talk’s deck</a><label for="sn-8" class="sidenote-toggle sidenote-number"></label><input type="checkbox" id="sn-8" class="sidenote-toggle"/><span class="sidenote">While you can find the <a rel="noopener noreferrer" target="_blank" href="https://github.com/leodido/presentations/tree/master/2022/02/05/fosdem-llvm-room/coverage-for-ebpf-programs">slides in PDF and markdown format here, if you prefer</a>.</span>.</p>
<p>While at <a rel="noopener noreferrer" target="_blank" href="https://github.com/elastic/bpfcov">elastic/bpfcov</a> you can take a look at the project. Don’t forget to star it, if you don’t mind! ⭐</p>
bpfcov2022-01-10T00:00:00+00:002022-01-10T00:00:00+00:00https://leodido.dev/projects/bpfcov/<p>This project provides 2 main components:</p>
<ol>
<li><code>libBPFCov.so</code> - an <strong>out-of-tree LLVM pass</strong> to <strong>instrument</strong> your <strong>eBPF programs</strong> for coverage.</li>
<li><code>bpfcov</code> - a <strong>CLI</strong> to <strong>collect source-based coverage</strong> from your eBPF programs.</li>
</ol>
<table><thead><tr><th align="center"></th><th align="center"></th><th align="center"></th></tr></thead><tbody>
<tr><td align="center"><a href="https://github.com/elastic/bpfcov/tree/main/docs/assets/stdo1.png" target="_blank" rel="noopener noreferrer"><img width="1604" alt="Source-based code coverage for BPF raw tracepoints" src="https://raw.githubusercontent.com/elastic/bpfcov/main/docs/assets/stdo1.png"></a></td><td align="center"><a href="https://github.com/elastic/bpfcov/tree/main/docs/assets/stdo2.png" target="_blank" rel="noopener noreferrer"><img width="1604" alt="Source-based code coverage for BPF LSM programs" src="https://raw.githubusercontent.com/elastic/bpfcov/main/docs/assets/stdo2.png"></a></td><td align="center"><a href="https://github.com/elastic/bpfcov/tree/main/docs/assets/mult1.png" target="_blank" rel="noopener noreferrer"><img width="1604" alt="HTML coverage index for multiple eBPF programs" src="https://raw.githubusercontent.com/elastic/bpfcov/main/docs/assets/mult1.png"></a></td></tr>
<tr><td align="center"><a href="https://github.com/elastic/bpfcov/tree/main/docs/assets/html2.png" target="_blank" rel="noopener noreferrer"><img width="1604" alt="HTML coverage report for eBPF programs" src="https://raw.githubusercontent.com/elastic/bpfcov/main/docs/assets/html2.png"></a></td><td align="center"><a href="https://github.com/elastic/bpfcov/tree/main/docs/assets/html1.png"><img width="1604" alt="HTML coverage report for eBPF programs" src="https://raw.githubusercontent.com/elastic/bpfcov/main/docs/assets/html1.png"></a></td><td align="center"><a href="https://github.com/elastic/bpfcov/tree/main/docs/assets/json1.png" target="_blank" rel="noopener noreferrer"><img width="1604" alt="JSON report for multiple eBPF programs" src="https://raw.githubusercontent.com/elastic/bpfcov/main/docs/assets/json1.png"></a></td></tr>
<tr><td align="center"><a href="https://github.com/elastic/bpfcov/tree/main/docs/assets/lcov1.png" target="_blank" rel="noopener noreferrer"><img width="1604" alt="LCOV info file from multiple eBPF programs" src="https://raw.githubusercontent.com/elastic/bpfcov/main/docs/assets/lcov1.png"></a></td><td align="center"><a href="https://github.com/elastic/bpfcov/tree/main/docs/assets/html3.png" target="_blank" rel="noopener noreferrer"><img width="1604" alt="HTML line coverage report for eBPF programs" src="https://raw.githubusercontent.com/elastic/bpfcov/main/docs/assets/html3.png"></a></td><td align="center"><a href="https://github.com/elastic/bpfcov/tree/main/docs/assets/html4.png" target="_blank" rel="noopener noreferrer"><img width="1604" alt="HTML line coverage report for eBPF programs" src="https://raw.githubusercontent.com/elastic/bpfcov/main/docs/assets/html4.png"></a></td></tr>
</tbody></table>
<h2 id="overview">Overview<a class="zola-anchor" href="#overview" aria-label="Anchor link for: overview">§</a>
</h2>
<p>This section aims to provide a high-level overiew of the steps you need to get started with <strong>bpfcov</strong>.</p>
<ol>
<li><a href="https://leodido.dev/projects/bpfcov/#building">Compile the LLVM pass</a> obtaining <code>libBPFCov.so</code></li>
<li>Instrument your eBPF program by compiling it and by running the LLVM pass (<code>libBPFCov.so</code>) on it</li>
<li>Build the userspace code of your eBPF application</li>
<li>Execute your eBPF application in the kernel through the <code>bpfcov run ...</code> command</li>
<li>Generate the <code>.profraw</code> file from the run through the <code>bpfcov gen ...</code> command
<ol>
<li>Having a <code>.profraw</code> makes this tool fully interoperable</li>
<li>Having a <code>.profraw</code> allows you to generate a variety of coverage reports in different formats</li>
</ol>
</li>
<li>Use the LLVM toolchain to create coverage reports as documented in the <a rel="noopener noreferrer" target="_blank" href="https://clang.llvm.org/docs/SourceBasedCodeCoverage.html#creating-coverage-reports">LLVM docs</a></li>
</ol>
<p>In case you are impatient and want to jump straight into getting your hands dirty, then the <a rel="noopener noreferrer" target="_blank" href="https://github.com/elastic/bpfcov/tree/main/examples/">examples</a> directory contains a few dummy eBPF programs to showcase what <strong>bpfcov</strong> does.</p>
<p>It basically automates steps 2 and 3. Its <a rel="noopener noreferrer" target="_blank" href="https://github.com/elastic/bpfcov/tree/main/examples/README.md">README</a> contains more details.</p>
<p>While the <a rel="noopener noreferrer" target="_blank" href="https://github.com/elastic/bpfcov/tree/main/cli/README.md">README of the cli directory</a> gives you more details about the steps 4 and 5 (and also 6).</p>
<h2 id="usage">Usage<a class="zola-anchor" href="#usage" aria-label="Anchor link for: usage">§</a>
</h2>
<p>Here I will highlight the <em>manual</em> steps to use it.</p>
<p>I suggest you to automate most of them like I did in the <a rel="noopener noreferrer" target="_blank" href="https://github.com/elastic/bpfcov/tree/main/examples/src/Makefile">examples Makefile</a>.</p>
<p>Anyway, assuming you have <a href="https://leodido.dev/projects/bpfcov/#building">built</a> the LLVM pass, you can then use your fresh <code>libBPFCov.so</code> to instrument your eBPF programs for coverage (steps 2 and 3 above).</p>
<p>How to do it?</p>
<p>First, you need to compile your eBPF program almost as usual but to LLVM IR…</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">clang -g -O2 \
-target bpf -D__TARGET_ARCH_x86 -I$(YOUR_INCLUDES) \
-fprofile-instr-generate -fcoverage-mapping \
-emit-llvm -S \
-c program.bpf.c \
-o program.bpf.ll
</code></pre>
<p>Notice it doesn’t matter if you use the textual (<code>*.ll</code>) or the binary form (<code>*.bc</code>).
Obviously, the former is more readable.</p>
<p>The same logic applies to <code>opt</code>: by default it generates <code>*.bc</code>.
Using the <code>-S</code> flag you can obtain the output in textual form (<code>*.ll</code>).</p>
<p>Anyhow, it’s time to run the LLVM pass on the LLVM IR we obtained.</p>
<p>Let’s do it:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">opt -load-pass-plugin $(BUILD_DIR)/lib/libBPFCov.so -passes="bpf-cov" \
-S program.bpf.ll \
-o program.bpf.cov.ll
</code></pre>
<p>We should have obtained a new LLVM IR that’s now valid and loadable from the BPF VM in the Linux kernel. Almost there, YaY!</p>
<p>From it, we can obtain a valid BPF ELF now:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">llc -march=bpf -filetype=obj -o cov/program.bpf.o program.bpf.cov.ll
</code></pre>
<p>While we are at it, it is also worth running the LLVM pass again (with a flag) to obtain another BPF ELF containing all the <strong>profiling</strong> and <strong>coverage mapping</strong> info.
It will come in handy later with <code>llvm-cov</code>.</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">opt -load $(BUILD_DIR)/lib/libBPFCov.so -strip-initializers-only -bpf-cov \
program.bpf.ll | \
llc -march=bpf -filetype=obj -o cov/program.bpf.obj
</code></pre>
<p>At this point, we can compile our userspace application loading the eBPF instrumented program (<code>cov/program.bpf.o</code>).</p>
<p>Doing this when using <code>libbpf</code> and skeletons is very easy. Nothing different from the common steps: <code>bpftool</code>, <code>cc</code>, etc.</p>
<p>In the <a rel="noopener noreferrer" target="_blank" href="https://github.com/elastic/bpfcov/tree/main/examples/">examples</a> directory, you can find further explainations.</p>
<p>So assuming we got our instrumented binary ready (<code>cov/program</code>), we can run it via the <code>bpfcov</code> CLI.</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">sudo ./bpfcov run cov/program
# Wait for it to exit, or stop it with CTRL+C
sudo ./bpfcov gen --unpin cov/program
</code></pre>
<p>Again, in case you wanna know more about these 2 steps, refer this time to the <a rel="noopener noreferrer" target="_blank" href="https://github.com/elastic/bpfcov/tree/main/cli/README.md">CLI README</a>.</p>
<p>Now we have a magic <code>cov/program.profraw</code> file…</p>
<p>And we can use the LLVM toolchain to generate very fine-grained coverage reports like those in the screenshots!</p>
<p>Refer to the <a rel="noopener noreferrer" target="_blank" href="https://clang.llvm.org/docs/SourceBasedCodeCoverage.html#creating-coverage-reports">LLVM docs</a> to learn how to do it.</p>
<p>But no worries, it’s just about invoking <code>llvm-profdata</code> and <code>llvm-cov</code>:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">lvm-profdata merge -sparse cov/program.profraw -o cov/program.profdata
llvm-cov show \
--format=html \
--show-line-counts-or-regions --show-region-summary --show-branch-summary \
--instr-profile=cov/profdata.profdata \
-object cov/program.bpf.obj \
--output-dir=cov/html_report
</code></pre>
<p>Anyayws, <strong>bpfcov</strong> also provides you an opinionated shortcut command to generate HTML, JSON, and LCOV coverage reports:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">./bpfcov out --format=html cov/program.profraw
</code></pre>
<h2 id="development-environment">Development Environment<a class="zola-anchor" href="#development-environment" aria-label="Anchor link for: development-environment">§</a>
</h2>
<p>In order to <strong>build</strong> the BPFCov library (<code>libBPFCov.so</code>) you will need:</p>
<ul>
<li>LLVM 12+</li>
<li>CMake 3.13.4+</li>
<li>C++ compiler that supports C++14</li>
</ul>
<p>In order to <strong>use</strong> it, you will need:</p>
<ul>
<li>clang 12 (to generate the input LLVM files)</li>
<li>its <a rel="noopener noreferrer" target="_blank" href="http://llvm.org/docs/CommandGuide/opt.html">opt</a> binary to run the LLVM pass</li>
</ul>
<p>This project has been tested on 5.15 Linux kernels.</p>
<h2 id="building">Building<a class="zola-anchor" href="#building" aria-label="Anchor link for: building">§</a>
</h2>
<p>Build as follows:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">mkdir -p build && cd build
cmake -DLT_LLVM_INSTALL_DIR=/path/to/llvm/installation ..
make
</code></pre>
<p>Notice that the <code>LT_LLVM_INSTALL_DIR</code> variable should be set to the root of either the installation (usually <code>/usr</code>) or the build directory of LLVM.</p>
<p>It is used to locate the corresponding <code>LLVMConfig.cmake</code> script that is used to set the include and the
library paths.</p>
<h2 id="testing">Testing<a class="zola-anchor" href="#testing" aria-label="Anchor link for: testing">§</a>
</h2>
<p>To run the tests you will need to install <strong>llvm-lit</strong>.</p>
<p>Usually, you can install it with <strong>pip</strong>:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">pip install lit
</code></pre>
<p>Running the tests is as simple as:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">lit build/test
</code></pre>
falco2016-01-19T00:00:00+00:002016-01-19T00:00:00+00:00https://leodido.dev/projects/falco/
<figure class="center">
<img src="https://raw.githubusercontent.com/falcosecurity/community/master/logo/primary-logo.png" style="width:360px" />
<figcaption class="center" >Cloud Native Runtime Security.</figcaption>
</figure>
<p>Want to talk? Join us on the <a rel="noopener noreferrer" target="_blank" href="https://kubernetes.slack.com/archives/CMWH3EH32">#falco</a> channel in the <a rel="noopener noreferrer" target="_blank" href="https://slack.k8s.io">Kubernetes Slack</a>.</p>
<h3 id="latest-releases">Latest releases<a class="zola-anchor" href="#latest-releases" aria-label="Anchor link for: latest-releases">§</a>
</h3>
<p>Read the <a rel="noopener noreferrer" target="_blank" href="https://github.com/falcosecurity/falco/tree/master/CHANGELOG.md">change log</a>.</p>
<!--
Badges in the following table are constructed by using the
https://img.shields.io/badge/dynamic/xml endpoint.
Parameters are configured for fetching packages from S3 before
(filtered by prefix, sorted in ascending order) and for picking
the latest package by using an XPath selector after.
- Common query parameters:
color=#300aec7
style=flat-square
label=Falco
- DEB packages parameters:
url=https://falco-distribution.s3-eu-west-1.amazonaws.com/?prefix=packages/deb/stable/falco-
query=substring-before(substring-after((/*[name()='ListBucketResult']/*[name()='Contents'])[last()]/*[name()='Key'],"falco-"),".asc")
- RPM packages parameters:
url=https://falco-distribution.s3-eu-west-1.amazonaws.com/?prefix=packages/rpm/falco-
query=substring-before(substring-after((/*[name()='ListBucketResult']/*[name()='Contents'])[last()]/*[name()='Key'],"falco-"),".asc")
- BIN packages parameters:
url=https://falco-distribution.s3-eu-west-1.amazonaws.com/?prefix=packages/bin/x86_64/falco-
query=substring-after((/*[name()='ListBucketResult']/*[name()='Contents'])[last()]/*[name()='Key'], "falco-")
Notes:
- if more than 1000 items are present under as S3 prefix,
the actual latest package will be not picked;
see https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html
- for `-dev` packages, the S3 prefix is modified accordingly
- finally, all parameters are URL encoded and appended to the badge endpoint
-->
<table><thead><tr><th></th><th>development</th><th>stable</th></tr></thead><tbody>
<tr><td>rpm</td><td><a rel="noopener noreferrer" target="_blank" href="https://download.falco.org/?prefix=packages/rpm-dev/"><img src="https://img.shields.io/badge/dynamic/xml?color=%2300aec7&style=flat-square&label=Falco&query=substring-before%28substring-after%28%28%2F%2A%5Bname%28%29%3D%27ListBucketResult%27%5D%2F%2A%5Bname%28%29%3D%27Contents%27%5D%29%5Blast%28%29%5D%2F%2A%5Bname%28%29%3D%27Key%27%5D%2C%22falco-%22%29%2C%22.asc%22%29&url=https%3A%2F%2Ffalco-distribution.s3-eu-west-1.amazonaws.com%2F%3Fprefix%3Dpackages%2Frpm-dev%2Ffalco-" alt="rpm-dev" /></a></td><td><a rel="noopener noreferrer" target="_blank" href="https://download.falco.org/?prefix=packages/rpm/"><img src="https://img.shields.io/badge/dynamic/xml?color=%2300aec7&style=flat-square&label=Falco&query=substring-before%28substring-after%28%28%2F%2A%5Bname%28%29%3D%27ListBucketResult%27%5D%2F%2A%5Bname%28%29%3D%27Contents%27%5D%29%5Blast%28%29%5D%2F%2A%5Bname%28%29%3D%27Key%27%5D%2C%22falco-%22%29%2C%22.asc%22%29&url=https%3A%2F%2Ffalco-distribution.s3-eu-west-1.amazonaws.com%2F%3Fprefix%3Dpackages%2Frpm%2Ffalco-" alt="rpm" /></a></td></tr>
<tr><td>deb</td><td><a rel="noopener noreferrer" target="_blank" href="https://download.falco.org/?prefix=packages/deb-dev/stable/"><img src="https://img.shields.io/badge/dynamic/xml?color=%2300aec7&style=flat-square&label=Falco&query=substring-before%28substring-after%28%28%2F%2A%5Bname%28%29%3D%27ListBucketResult%27%5D%2F%2A%5Bname%28%29%3D%27Contents%27%5D%29%5Blast%28%29%5D%2F%2A%5Bname%28%29%3D%27Key%27%5D%2C%22falco-%22%29%2C%22.asc%22%29&url=https%3A%2F%2Ffalco-distribution.s3-eu-west-1.amazonaws.com%2F%3Fprefix%3Dpackages%2Fdeb-dev%2Fstable%2Ffalco-" alt="deb-dev" /></a></td><td><a rel="noopener noreferrer" target="_blank" href="https://download.falco.org/?prefix=packages/deb/stable/"><img src="https://img.shields.io/badge/dynamic/xml?color=%2300aec7&style=flat-square&label=Falco&query=substring-before%28substring-after%28%28%2F%2A%5Bname%28%29%3D%27ListBucketResult%27%5D%2F%2A%5Bname%28%29%3D%27Contents%27%5D%29%5Blast%28%29%5D%2F%2A%5Bname%28%29%3D%27Key%27%5D%2C%22falco-%22%29%2C%22.asc%22%29&url=https%3A%2F%2Ffalco-distribution.s3-eu-west-1.amazonaws.com%2F%3Fprefix%3Dpackages%2Fdeb%2Fstable%2Ffalco-" alt="deb" /></a></td></tr>
<tr><td>binary</td><td><a rel="noopener noreferrer" target="_blank" href="https://download.falco.org/?prefix=packages/bin-dev/x86_64/"><img src="https://img.shields.io/badge/dynamic/xml?color=%2300aec7&style=flat-square&label=Falco&query=substring-after%28%28%2F%2A%5Bname%28%29%3D%27ListBucketResult%27%5D%2F%2A%5Bname%28%29%3D%27Contents%27%5D%29%5Blast%28%29%5D%2F%2A%5Bname%28%29%3D%27Key%27%5D%2C%20%22falco-%22%29&url=https%3A%2F%2Ffalco-distribution.s3-eu-west-1.amazonaws.com%2F%3Fprefix%3Dpackages%2Fbin-dev%2Fx86_64%2Ffalco-" alt="bin-dev" /></a></td><td><a rel="noopener noreferrer" target="_blank" href="https://download.falco.org/?prefix=packages/bin/x86_64/"><img src="https://img.shields.io/badge/dynamic/xml?color=%2300aec7&style=flat-square&label=Falco&query=substring-after%28%28%2F%2A%5Bname%28%29%3D%27ListBucketResult%27%5D%2F%2A%5Bname%28%29%3D%27Contents%27%5D%29%5Blast%28%29%5D%2F%2A%5Bname%28%29%3D%27Key%27%5D%2C%20%22falco-%22%29&url=https%3A%2F%2Ffalco-distribution.s3-eu-west-1.amazonaws.com%2F%3Fprefix%3Dpackages%2Fbin%2Fx86_64%2Ffalco-" alt="bin" /></a></td></tr>
</tbody></table>
<p>The Falco Project, originally created by <a rel="noopener noreferrer" target="_blank" href="https://sysdig.com">Sysdig</a>, is an incubating <a rel="noopener noreferrer" target="_blank" href="https://cncf.io">CNCF</a> open source cloud native runtime security tool.
Falco makes it easy to consume kernel events, and enrich those events with information from Kubernetes and the rest of the cloud native stack.
Falco has a rich set of security rules specifically built for Kubernetes, Linux, and cloud-native.
If a rule is violated in a system, Falco will send an alert notifying the user of the violation and its severity.</p>
<h3 id="installing-falco">Installing Falco<a class="zola-anchor" href="#installing-falco" aria-label="Anchor link for: installing-falco">§</a>
</h3>
<p>If you would like to run Falco in <strong>production</strong> please adhere to the <a rel="noopener noreferrer" target="_blank" href="https://falco.org/docs/getting-started/installation/">official installation guide</a>.</p>
<h5 id="kubernetes">Kubernetes<a class="zola-anchor" href="#kubernetes" aria-label="Anchor link for: kubernetes">§</a>
</h5>
<table><thead><tr><th>Tool</th><th>Link</th><th>Note</th></tr></thead><tbody>
<tr><td>Helm</td><td><a rel="noopener noreferrer" target="_blank" href="https://github.com/falcosecurity/charts/tree/master/falco#introduction">Chart Repository</a></td><td>The Falco community offers regular helm chart releases.</td></tr>
<tr><td>Minikube</td><td><a rel="noopener noreferrer" target="_blank" href="https://falco.org/docs/getting-started/third-party/#minikube">Tutorial</a></td><td>The Falco driver has been baked into minikube for easy deployment.</td></tr>
<tr><td>Kind</td><td><a rel="noopener noreferrer" target="_blank" href="https://falco.org/docs/getting-started/third-party/#kind">Tutorial</a></td><td>Running Falco with kind requires a driver on the host system.</td></tr>
<tr><td>GKE</td><td><a rel="noopener noreferrer" target="_blank" href="https://falco.org/docs/getting-started/third-party/#gke">Tutorial</a></td><td>We suggest using the eBPF driver for running Falco on GKE.</td></tr>
</tbody></table>
<h3 id="developing">Developing<a class="zola-anchor" href="#developing" aria-label="Anchor link for: developing">§</a>
</h3>
<p>Falco is designed to be extensible such that it can be built into cloud-native applications and infrastructure.</p>
<p>Falco has a <a rel="noopener noreferrer" target="_blank" href="https://falco.org/docs/grpc/">gRPC</a> endpoint and an API defined in <a rel="noopener noreferrer" target="_blank" href="https://github.com/falcosecurity/falco/blob/master/userspace/falco/outputs.proto">protobuf</a>.
The Falco Project supports various SDKs for this endpoint.</p>
<h5 id="sdks">SDKs<a class="zola-anchor" href="#sdks" aria-label="Anchor link for: sdks">§</a>
</h5>
<table><thead><tr><th>Language</th><th>Repository</th></tr></thead><tbody>
<tr><td>Go</td><td><a rel="noopener noreferrer" target="_blank" href="https://github.com/falcosecurity/client-go">client-go</a></td></tr>
<tr><td>Rust</td><td><a rel="noopener noreferrer" target="_blank" href="https://github.com/falcosecurity/client-rs">client-rs</a></td></tr>
<tr><td>Python</td><td><a rel="noopener noreferrer" target="_blank" href="https://github.com/falcosecurity/client-py">client-py</a></td></tr>
</tbody></table>
<h3 id="what-can-falco-detect">What can Falco detect?<a class="zola-anchor" href="#what-can-falco-detect" aria-label="Anchor link for: what-can-falco-detect">§</a>
</h3>
<p>Falco can detect and alert on any behavior that involves making Linux system calls.
Falco alerts can be triggered by the use of specific system calls, their arguments, and by properties of the calling process.
For example, Falco can easily detect incidents including but not limited to:</p>
<ul>
<li>A shell is running inside a container or pod in Kubernetes.</li>
<li>A container is running in privileged mode, or is mounting a sensitive path, such as <code>/proc</code>, from the host.</li>
<li>A server process is spawning a child process of an unexpected type.</li>
<li>Unexpected read of a sensitive file, such as <code>/etc/shadow</code>.</li>
<li>A non-device file is written to <code>/dev</code>.</li>
<li>A standard system binary, such as <code>ls</code>, is making an outbound network connection.</li>
<li>A privileged pod is started in a Kubernetes cluster.</li>
</ul>
<h3 id="documentation">Documentation<a class="zola-anchor" href="#documentation" aria-label="Anchor link for: documentation">§</a>
</h3>
<p>The <a rel="noopener noreferrer" target="_blank" href="https://falco.org/docs/">Official Documentation</a> is the best resource to learn about Falco.</p>
<h3 id="join-the-community">Join the Community<a class="zola-anchor" href="#join-the-community" aria-label="Anchor link for: join-the-community">§</a>
</h3>
<p>To get involved with The Falco Project please visit <a rel="noopener noreferrer" target="_blank" href="https://github.com/falcosecurity/community">the community repository</a> to find more.</p>
<p>How to reach out?</p>
<ul>
<li>Join the #falco channel on the <a rel="noopener noreferrer" target="_blank" href="https://slack.k8s.io">Kubernetes Slack</a></li>
<li><a rel="noopener noreferrer" target="_blank" href="https://lists.cncf.io/g/cncf-falco-dev">Join the Falco mailing list</a></li>
<li><a rel="noopener noreferrer" target="_blank" href="https://falco.org/docs/">Read the Falco documentation</a></li>
</ul>
<h3 id="contributing">Contributing<a class="zola-anchor" href="#contributing" aria-label="Anchor link for: contributing">§</a>
</h3>
<p>See the <a rel="noopener noreferrer" target="_blank" href="https://github.com/falcosecurity/.github/blob/master/CONTRIBUTING.md">CONTRIBUTING.md</a>.</p>
<h3 id="security-audit">Security Audit<a class="zola-anchor" href="#security-audit" aria-label="Anchor link for: security-audit">§</a>
</h3>
<p>A third party security audit was performed by Cure53, you can see the full report <a rel="noopener noreferrer" target="_blank" href="https://github.com/falcosecurity/falco/tree/master/audits/SECURITY_AUDIT_2019_07.pdf">here</a>.</p>
<h3 id="reporting-security-vulnerabilities">Reporting security vulnerabilities<a class="zola-anchor" href="#reporting-security-vulnerabilities" aria-label="Anchor link for: reporting-security-vulnerabilities">§</a>
</h3>
<p>Please report security vulnerabilities following the community process documented <a rel="noopener noreferrer" target="_blank" href="https://github.com/falcosecurity/.github/blob/master/SECURITY.md">here</a>.</p>
<h3 id="license-terms">License Terms<a class="zola-anchor" href="#license-terms" aria-label="Anchor link for: license-terms">§</a>
</h3>
<p>Falco is licensed to you under the <a rel="noopener noreferrer" target="_blank" href="https://github.com/falcosecurity/falco/tree/master/COPYING">Apache 2.0</a> open source license.</p>