00 01 02 03 04 05 06 07</th>	08 09 0A 0B 0C 0D 0E 0F</th></tr></thead>
magic</td>	version</td></tr>
data size</td>	padding before counters</td></tr>
counters size</td>	padding after counters</td></tr>
names size</td>	counters delta</td></tr>
names begin</td>	value kind last</td></tr> </tbody></table> But probably an image is worth a thousand words.</p> This is the `default.profraw</code> file after executing ./hello yay</code>:</p>` `</a></p>` `We can see that its header is composed of:</p>` the magic (in black)</li> the version (in blue)</li> the number of functions (in red) it equals to the number of __profd_<function_name></code> variables too: 3</strong></li> </ul> </li> the padding bytes before the counters (in light green)</li> the total number of the counters (in fluo green) meaning, the total size of __profc_<function_name></code> variables: 5</strong></li> </ul> </li> the padding bytes after the counters (in dark green)</li> the length of the __llvm_prf_nm</code> variable (in magenta): 0F in hex, 15 decimal</li> the address of the first counter (in brown)</li> the address of __llvm_prf_nm</code> (in pink)</li> IPVK_Last</code> (1) in yellow</li> </ul> Exactly the way we were now expecting it.</p> Data§</a> </h2> Focusing on the data part, I've highlighted with (orange) rectangles the 3 __profd_<function_name></code> global variables.</p> </a></p> Each of the __profd_<function_name></code> contains 6 parts:</p> 1 is the ID (hex, reverse order) of the function to which it refers</li> 2 is the function control flow hash</li> 3 points to the counter it relates to</li> 4 points to the function it refers to</li> 5 points to the value expressions, if any</li> 6 contains the number of the counters (2 bytes) and the initialization array (number of value sites, 2 elements, 2 bytes each, often zero)</li> </ul> Counters§</a> </h2> After the data part comes the counters part.</p> </p> It just is a serialization of what the variables in the __llvm_prf_cnts</code> ELF section contains.</p> In fact, in this screenshot you can see underlined (with waves, in shades of cyan/blue):</p> __profc_ciao</code>: [1 x i64] 1 counter, with value 16</strong></li> it means the ciao()</code> function was executed 22 (decimal) times</li> </ul> </li> __profc_foo</code>: [1 x i64] 1 counter, with value 1</strong></li> it means the foo()</code> function only was executed 1 time</li> </ul> </li> __profc_main</code>: [3 x i64] 3 counters, first two with value 1</strong>, last one with value 16</strong></li> it means the function main()</code> was executed 1 time, its if</code> conditional evaluated to true, and the for</code> inside it got executed 22</strong> (decimal) times</li> </ul> </li> </ul> Names§</a> </h2> Finally, the __llvm_prf_nm</code> variable populates the last part of the profraw file.</p> </a></p> Here (in grey) you can see its content</label>Function names are uncompressed because I compiled the binary with -mllvm -enable-name-compression=false</code> flags.</span>, padded with a \0</code> byte (the one circled) at the end to align the profraw file size to be multiple of 8.</p> Usage§</a> </h2> It's time to use the default.profraw</code> file now:</p> llvm-profdata</span> show</span> --all-functions --counts</span> default.profraw</span></span> # Counters:</span></span> # ciao:</span></span> # Hash: 0x0000000000000000</span></span> # Counters: 1</span></span> # Function count: 22</span></span> # Block counts: []</span></span> # foo:</span></span> # Hash: 0x0000000000000000</span></span> # Counters: 1</span></span> # Function count: 1</span></span> # Block counts: []</span></span> # main:</span></span> # Hash: 0x000000a71211b451</span></span> # Counters: 3</span></span> # Function count: 1</span></span> # Block counts: [1, 22]</span></span> # Instrumentation level: Front-end</span></span> # Functions shown: 3</span></span> # Total functions: 3</span></span> # Maximum function count: 22</span></span> # Maximum internal block count: 22</span></span></code></pre> Isn't it cool?</p> There's more you can do with profraw files and the LLVM toolchain, indeed. But showing that was not the goal of this post.</p> I hope you now have a better understanding of what profraw files are.</p>

leodido.dev - Clang

Code coverage for eBPF programs

2022-02-07T00:00:00+00:00

I bet we all have heard so much about eBPF</abbr> in recent years. Every day we hear about a new project or application using some eBPF black magic underneath. Data</a> shows that eBPF is quickly becoming the first choice for implementing tracing and security applications.

However, one major challenge is that the eBPF ecosystem lacks tooling to make developers' lives easier. eBPF programs are written in C but compiled for a specific ISA later executed by the eBPF Virtual Machine. LLVM has a specific backend allowing us to write C and get eBPF ELF objects out. There are no tools helping developers to clearly understand which path their code took while running in the Linux kernel, which code regions, or, even better, code branches are uncovered, and maybe why.

You may already be familiar with code coverage, which shows you which lines of code execute. Code coverage is usually applied to tests to discover which line gets run and which does not. But even testing the eBPF programs is a pain, given that BPF_PROG_TEST_RUN</code> does not support all the types of eBPF programs living in the Linux kernel.

That's why I sat down and wrote bpfcov</a>: a tool to gather source-based coverage info for our eBPF programs running in the Linux kernel. Whether they are getting loaded via BPF_PROG_TEST_RUN</code> or by other ordinary means. Until today, there was no simple way to visualize how the flow of your eBPF program running in the kernel was. Hopefully, there is one starting today.

So, let's jump straight into the topic.

Source-based code coverage for eBPF§</a>
</h2>
Everyone reading this post probably knows what the code coverage is. Common line-level coverage gives us a sense of what line is executed. In some cases, it even tells us how many times a line got executed.</p>
When building this tool, driven by my experience fighting with BPF, I knew I wanted something more. Line-level granularity is often too coarse. We do not want an approximation of what code actually executed.
We need something more precise to understand the execution path of our eBPF programs in the Linux kernel. We even want to know what part of an if</code> conditional executed</label>I suggest you watch this talk</a> by Alan Phipps to know more about branch coverage.</span>. Source-based code coverage is what I wanted for this reason.</p>
Since it starts at code generation time in LLVM, it has the notion of regions</mark> of the code, branches</mark>, and so on. It even precisely counts things like short-circuited conditionals</mark>, thanks to the counter expression</mark> and arithmetics between them. It generates coverage summaries with very fine-grained code regions, helping us find gaps in the code and its execution flow.</p>
So, given eBPF programs are usually written in C, couldn't we just instrument them for source-based code coverage</a> as we would commonly do with Clang for our C programs? I bet this is the first argument popping into the head of many readers. It also was one of my thoughts when approaching this problem. Turns out that we definitely can, and we will do it. But it won't work as is.</p>
To understand why it won't work, we need to keep in mind that BPF programs are compiled via Clang</label>Thanks to the LLVM BPF target</a>.</span> to BPF ELF object files, containing instructions specific to the chosen BPF instruction set, which need to be later loaded in the Linux kernel via the bpf()</a> syscall. Furthermore, it's paramount to mention that the BPF programs will be verified by an in-kernel verifier and then executed by the BPF VM.</p>
Such a lifecycle and environment imposes a set of constraints that make it infeasible to get them working with the plain instrumentation that LLVM applies to get source-based code coverage. In fact, when compiling a C program for source-based coverage with the -fprofile-instr-generate</code> and -fcoverage-mapping</code> Clang flags, LLVM instruments it with a bunch of global variables and, in some cases, functions.</p>
Observing the LLVM IR of a C program after compiling it with the -fprofile-instr-generate</code> flag, we can notice that, LLVM...</p>

defines the counters, in the form of __profc_<function_name></code> private global arrays whose size is the number of counters needed to cover all the regions and branches of such a function</li>
marks those counters to be into the __llvm_prf_cnts</code> section of the ELF</li>
defines (and initializes) __profd_<function_name></code> private global struct instances that contains an identifier of the function, the pointer to its counters, the pointer to the function itself, the number of the counters for the target function, and a bunch of other info needed to tie together the counters and the coverage mappings</li>
marks the __profd_</code> globals to end up into the __llvm_prf_data</code> section of the ELF</li>
defines (and initializes) a private constant (__llvm_prf_nm</code>) containing the names of the target functions</li>
marks the names constant to end up in the __llvm_prf_names</code> section of the ELF</li>
</ul>
The first issue to address here is that we can't have random ELF sections (like__llvm_prf_cnts</code>, etc.) into valid BPF ELF files.</p>
We know that we can have eBPF global constants with a scalar type like __profc_</code> ones, at least on recent Linux kernels. At the same time, we know it's not wise to have global structs like __profd_</code> or whatever loader for our eBPF programs we will write, as we'll need the struct definitions. Too clunky.</p>
The -fprofile-instr-generate</code> flag also generates a global constructor and a set of global functions, all prefixed with __llvm_profile</code>, intended to set up the profiling runtime in the resulting binary so that when it dies or exits it will automatically generate a profraw</mark> file.</p>
Here we meet another stumbling block to overcome. The BPF verifier will refuse our BPF ELF containing strange global functions, not to mention "constructors"...</p>
We need to get rid of them and replace their functionality with a feasible approach for BPF. Clearly, LLVM also patches our instructions in the right spots, by placing the counters and incrementing them. As you can see in the following annotated screenshot:</p>
</a></p>
If you write an eBPF program incrementing a counter</mark> (defined as an eBPF global variable</mark>) you will notice that the resulting instructions will be like those here in the screenshot. It means we can keep these the way they are generated by LLVM and let it do all its magic without interfering at the instructions level. Finally, some good news!</p>
Instead, focusing on the -fcoverage-mapping</code> Clang flag and inspecting the LLVM IR it outputs, we notice that it...</p>

defines (and initializes) __covrec_</code> global (constant) structs, most notably containing the same ID that the __profd_</code> variables contain, and a LEB128</label>The encoding of the coverage mapping values is LEB128</a>.</span> encoded string containing all the region, branches, and generally the coverage mapping info</li>
defines (and initializes) a __llvm_coverage_mapping</code></label>The Coverage Mapping</a> format.</span> function containing meta info about the coverage mapping format (eg., the version), and the source file names</li>
marks the __llvm_coverage_mapping</code> variable to be put into the __llvm_covmap</code> section of the ELF.</li>
</ul>
In this case, we have the same category of issues mentioned before.</p>
We need to keep at least the header of __llvm_coverage_mapping</code> because it contains the coverage mappings version, which we need for generating a valid profraw</mark> file</label>More on the profraw</mark> format in a previous blog post</a>.</span>. Also, we need to put it in a custom BPF ELF section</mark>.</p>
Luckily, we can remove the __covrec_</code> structs from the BPF ELF meant to be loaded in the kernel. We can keep them in a second BPF ELF that would be intended to be given to tools needing the coverage mappings for generating the coverage reports, just like llvm-cov</mark> does.</p>
How it's done§</a>
</h2>
libBPFCov.so - A LLVM pass§</a>
</h3>
By analyzing the resulting LLVM IR for source-based coverage instrumented programs, we should now have a better understanding of what we need to do for having it on eBPF programs running in the Linux kernel. We now know what to completely get rid of and do differently, and what we need to patch to make it loadable and usable by the BPF VM in the kernel.</p>
The plan is simple: get Clang to instrument a BPF LLVM intermediate representation for source-based code coverage, then patch it to model it into a valid representation for BPF ELF.</p>
How do we need to transform it?</p>
First of all, we are so lucky we don't have to mess with the actual BPF instructions, namely the counters increments. We can keep them the way they are. This is a huge win because we let LLVM keep track of the global state of the registers and we avoid a lot of work this way.</p>
But for sure we have to strip any profile initialization stuff that Clang creates, things like __llvm_profile_runtime</code> and __llvm_profile_init</code> - when present - are no good for the BPF VM in the kernel.</p>
We also want to ensure the global variables, whether constants or not, have the right visibility (ie., dso_local</code>) and linkage, to have them in the libbpf</mark> skeletons if we plan to use them.</p>
For the global structs that we need for generating the profraw</mark> files, namely the __profd_</code> variables, we just transform them into different and single global variables, one for each field.</p>
For example, this is what I did for the __profd_*</code> variables which originally are a struct with 7 fields. For other global structs like the __covrec_</code> ones, we can just strip them from the BPF ELF that is meant to be loaded in the kernel.</p>
Anyway, the report generation phase (ie., llvm-cov</mark> or bpfcov out</mark>) will need them for knowing at which line and column a code region or a branch starts. For this reason, I decided to give the LLVM pass an option (enabled with the strip-initializers-only</code> flag) that keeps them, so we can later create a BPF ELF that is only meant for this phase and not for loading.</p>
This BPF ELF will have .bpf.obj</code> as an extension, rather than .bpf.o</code>.</p>
Finally, we know that libbpf</a> supports (on recent Linux kernels) eBPF global variables</mark></label>Kernel patch: eBPF support for global data</a>.</span>, which are simply eBPF maps with one single value, and we are planning to use them. But, as already mentioned, it does not accept or recognize the ELF sections that the Clang instrumentation injects in the intermediate representation.</p>
So we need our LLVM pass to change them to custom eBPF sections</mark></label>Kernel patch: libbpf: support global data/bss/rodata sections</a>.</span>. The eBPF custom sections are in the form of .rodata.*</code> or .data.*</code> made to contain static and/or global data. We can change the section of the counters to be .data.profc</code>. The section of the __llvm_prf_nm</code> from _llvm_prf_names</code> to .rodata.profn</code>, and so on. You can find all this logic summarized in these bits of code</a>.</p>
So, assuming the following dummy eBPF program:</p>
</a></p>
I think that the following snippet tells everything about what we obtain from the libBPFCov.so</mark> LLVM pass:</p>
</a></p>
For example, you may notice that we have 2 __profc_</code> counters.</p>
The first is for the BPF_PROG</code> macro that expands to a function, the second for the actual BPF raw tracepoint program hook_sys_enter</code>. This one has size 3. That's because the hook_sys_enter</code> function has 3 main regions: the entry of the function, the if</code> conditional, and the for</code> cycle.</p>
You may also notice that the LLVM pass, for each one of the 2 functions we have, split the __profd_</code> global structs into 7 different global variables in the .rodata.profd</code> section.</p>
Someone who has an eye for it may also have noticed that the third field of __profd_</code> (now __profd_something.2</code>) does not contain anymore the address of its counters. I didn't want (nor I could) to expose kernel addresses, so I put here the offset of the counters in their section (.data.profc</code>).</p>
Finally, you can also see that, as anticipated before, we completely deleted the __covrec_</code> global constant structs from this IR that's meant to generate a valid and loadable BPF ELF. While the instructions incrementing the counters in the correct spots are not touched at all. So we don't need another screenshot to show them!</p>
The only missing moving part is how to generate a valid profraw</mark> file. We stripped any logic for doing it. We know that for generating it we need all the globals we left in this LLVM intermediate representation. But we have no sane way to hook the exit or the stop of an eBPF program in the Linux kernel.</p>
Suddenly, inspiration came: let's pin</mark> the globals to the BPF file system</mark> so that we can decouple the process of generating the profraw</mark> file from the running (and exiting) of the instrumented eBPF application!</p>
And that's what the bpfcov</mark> CLI</strong> does.</p>
Before moving to the next section, I suggest you go to the bpfcov repository</a> and start building the pass to obtain libBPFCov.so</mark>. You can find the instructions on how to build it here</a>.</p>
Instrument your eBPF program§</a>
</h2>
Now that we have built libBPFCov.so</mark> we can finally take action!</p>
Instrumenting an eBPF program for source-based coverage is not that different than compiling it normally with Clang and the BPF target.</p>
The only difference is that we ask Clang to output LLVM IR (either in textual or binary form), run the libBPFCov.so</mark> pass on it (with opt</code>), and finally compile it (with llc</code>) to a BPF ELF.</p>
clang</span> -g -O2</span> \</span></span>
  -target</span> bpf</span> \</span></span>
  -D__TARGET_ARCH_x86 -I$(</span>YOUR_INCLUDES</span>)</span> \</span></span>
  -fprofile-instr-generate -fcoverage-mapping</span> \</span></span>
  -emit-llvm -S</span> \</span></span>
  -c</span> raw_enter.bpf.c</span> -o</span> raw_enter.bpf.ll</span></span>
</span>
opt</span> -load-pass-plugin</span> $(</span>BUILD_DIR</span>)</span>/lib/libBPFCov.so</span> -passes=</span>"bpf-cov"</span> \</span></span>
    -S</span> raw_enter.bpf.ll</span> -o</span> raw_enter.bpf.cov.ll</span></span>
</span>
llc</span> -march=bpf -filetype=obj -o</span> cov/raw_enter.bpf.o raw_enter.bpf.cov.ll</span></span></code></pre>
You can see in the Makefile inside the examples/src</a> directory of the bpfcov GitHub repository</a> how to automate those steps.</p>
We now have a valid and coverage instrumented BPF ELF: cov/raw_enter.bpf.o</code>.</p>
From now on, you can instruct your loader and userspace code to use it, so to obtain a binary (eg., /cov/raw_enter</code>) that is your eBPF application.</p>
Use it§</a>
</h3>
bpfcov run + bpfcov gen§</a>
</h4>
What's left to do? Just three steps:</p>

Run our eBPF application with the bpfcov</mark> CLI</strong> run command</li>
Generate its profraw</mark> file</li>
Generate beautiful source-based coverage reports</li>
</ol>
So, let's run our eBPF application with:</p>
sudo</span> ./bpfcov</span> -v2</span> run cov/raw_enter</span></span></code></pre>
This command acts similar to strace. It will detect the bpf()</code> syscalls with the BPF_MAP_CREATE</code> command.</p>
Meaning that it will detect the eBPF globals in the .profc</code>, .profd</code>, .profn</code>, and .covmap</code> custom eBPF sections and pin them to the BPF file system, as you can see in the following screenshot.</p>
</a></p>
You may also notice that - since the LLVM pass annotated the counters correctly - libbpf</mark> can collect relocations for them...</p>
At this point, whether we stopped our eBPF application or it exited... We have eBPF maps pinned to our BPF file system. Let's check it:</p>


    
    
    </a>
    
</figure>
Wonderful, we already know that the hook_sys_enter</code> function executed 1 time, the if</code> condition did not evaluate to true, while the for</code> iterated 9 times!</p>
It's time to put the counters, the function names, the functions data, in a profraw</mark> file now.</p>
This is why the bpfcov gen</mark> command exists: to dump the pinned maps in a profraw</mark> file.</p>
sudo</span> ./bpfcov</span> -v2</span> gen –unpin cov/raw_enter</span></span></code></pre>

    
    
    </a>
    
</figure>
And this is the resulting profraw</mark> file for our instrumented eBPF program!</p>
You can see it's made of four parts: a header, .rodata.profd</code>, .data.profc</code> (ie., the counters!), and the names (.rodata.profn</code>), plus some padding for alignment...</p>


    
    
    </a>
    
</figure>
We can now either use the existing LLVM tools (llvm-profdata</mark> and llvm-cov</mark>) with it or simply use the bpfcov out</mark> subcommand.</p>
The bpfcov out</mark> command is an opinionated shortcut to generate HTML, JSON, or LCOV coverage reports even from multiple eBPF programs and their profraw</mark> files.</p>
It is very convenient because it avoids us having to generate profdata</mark> from the profraw</mark>, calling llvm-cov</mark> with a bunch of long and different options. And it even works with multiple profraw</mark> files coming from different eBPF applications...</p>
./bpfcov</span> out</span> -o</span> yey –f html cov/raw_enter.profraw ...</span></span></code></pre>
It outputs a very nice HTML directory whose index file gives us summaries not only about function and line coverage but also and notably about region and branch coverages for our eBPF applications.</p>


    
    
    </a>
    
</figure>
By clicking on any item in the table we end up visualizing very fine-grained, source-based coverage. A good example is the one in the following image:</p>


    
    
    </a>
    
</figure>
Furthermore, it does work also on very complicated and real-life eBPF programs. For example, the following screenshot is a part of the coverage report obtained for a BPF LSM test (progs/lsm.c</a>, loaded by prog_tests/test_lsm.c</a>) living in the Linux kernel.</p>
</a></p>
Thanks to this tool I can finally understand that over a total of eight executions of the lsm/file_mprotect</code> BPF LSM program on my kernel, its is_stack</code> variable was true two times out of eight, because six times the vma->vm_end >= vma->vm_mm->start_stack</code> branch condition (98:58) evaluated to false.</p>
Line 100, using is_stack</code> in another condition, confirms that it was indeed true two times out of six. And that, for this reason (first operand - 100:6 - is_stack</code> being false six times), the following check (100:18) on monitored_pid</code> was short-circuited and evaluated (to true, by the way) only two times.</p>
We finally have a tool helping us write and understand the way our eBPF programs run in the Linux kernel. I can't stress enough how this is something I dreamt of so many times during the past few years I've been working with BPF...</p>
Hope that the eBPF community and ecosystem will find bpfcov useful and cool the same way I do.</p>
Bits of the future§</a>
</h2>
The bpfcov</a> tool is open-source, and it will stay that way. It is still in its early days so it will probably need a bit more tests, examples, and fixes. Just like any new software out there.</p>
I will soon publish another project showcasing the coverage of the kernel BPF selftests, using this tool. This means there are a lot of contribution opportunities. I'd invite you to take a look at its source code</a> and send patches! :blush:</p>
Also, in case you want to start using it on your eBPF applications and repositories, feel free to contact me for support. I'd love to help you to use it.</p>
From a technical perspective these are the topics that are on top of my mind for its future:</p>

A project like bpfcov</a> must</strong> have a logo!</li>
Write llvm-lit</mark> tests for the libBPFCov.so</mark> LLVM pass</li>
Support newer LLVM versions</li>
Workaround solutions to have it working on Linux kernels where BPF does not support custom eBPF sections and eBPF globals</li>
Create a versioning and release process</li>
Publish artifacts on GitHub (libBPFCov.so</mark>, bpfcov</mark> CLI</strong> binary)</li>
Add more examples</li>
Publish HTML reports of the example eBPF applications via GitHub pages</li>
Maybe, contribute it upstream</strong> to the LLVM BPF target!</li>
</ul>
FOSDEM 2022§</a>
</h2>
In case you made it to the end, and you still want to hear more details on building source-based coverage for eBPF, I want to bring to your attention that I gave a talk</a> on bpfcov</a> at FOSDEM 2022</a>.</p>
I will update my speaking</a> section with the video recording as soon it'll be ready.</p>
In the meantime, you can take a look at the talk's deck</a></label>While you can find the slides in PDF and markdown format here, if you prefer</a>.</span>.</p>
While at elastic/bpfcov</a> you can take a look at the project. Don't forget to star it, if you don't mind! :star:</p>



Demystifying the profraw format
2022-01-28T00:00:00+00:00
Three months ago, I was looking into building coverage for eBPF programs.</p>
That's how bpfcov</a> was born.</p>
I did not want to reinvent the wheel, so I was looking into the existing coverage instrumentation features of LLVM.</p>
Among the different options LLVM provides, source-based code coverage</a> suddenly appeared very appealing to me.</p>
It's the most precise type of code coverage. In my opinion, the only one that gives its user a damn clue on what code region of the program is executing. It also helps understand why a particular code branch gets executed.</p>
Using it is pretty straightforward. All resolve to the following steps:</p>

compile your C/C++ code with the -fprofile-instr-generate -fcoverage-mapping</code> Clang options</li>
run the program</li>
the execution of the program automatically creates a default.profraw</code> file</label>You can set it to something else via the LLVM_PROFILE_FILE</code> environment variable.</span></li>
feed the default.profraw</code> file to LLVM tools like llvm-profdata</code> and llvm-cov</code></li>
</ol>
You now have a beautifully fine-grained coverage report for your C/C++ code.</p>
But what caught my eye was feeling that this kind of coverage was not using debug info (like gcov</code> does) at all, but it was using profiling data, coming directly from the darkest parts of LLVM.</p>
My curiosity took me to some extraordinary places in the LLVM codebase...</p>
I wanted to understand what the -fprofile-instr-generate</code> was causing.</p>
Also, I wanted to know how I got a binary file (default.profraw</code>) without my code writing anything to the disk.</p>
And what it contained.</p>
Spoiler</em>. It contains the globals that the -fprofile-instr-generate</code> instruments into your code.</p>
But we probably need to move one step at a time to learn something new.</p>
-fprofile-instr-generate§</a>
</h2>
So, this is the dummy C code - hello.c</code> - that I used to walk myself through this study.</p>
#include</span> <stdio.h></span></span>
#include</span> <stdint.h></span></span>
</span>
void</span> ciao</span>()</span></span>
{</span></span>
    printf</span>(</span>"ciao</span>\n</span>"</span>);</span></span>
}</span></span>
</span>
void</span> foo</span>()</span></span>
{</span></span>
    printf</span>(</span>"foo</span>\n</span>"</span>);</span></span>
}</span></span>
</span>
int</span> main</span>(</span>int</span> argc</span>,</span> char</span> **</span>argv</span>)</span></span>
{</span></span>
    if</span> (argc ></span> 1</span>)</span></span>
    {</span></span>
        foo</span>();</span></span>
        for</span> (</span>int</span> i =</span> 0</span>; i <</span> 22</span>; i++) {</span></span>
            ciao</span>();</span></span>
        }</span></span>
    }</span></span>
    printf</span>(</span>"main</span>\n</span>"</span>);</span></span>
}</span></span></code></pre>
I compiled it to textual LLVM IR:</p>
clang</span> -g -fprofile-instr-generate -emit-llvm -S</span> $<</span> -o</span> hello.ll</span></span></code></pre>
In a matter of milliseconds, I was able to look at its intermediate representation:</p>

    
    LLVM IR / PROFILING DATA</span>hello.ll</span></span>
    </label>@__profc_ciao</span> = </span>private global</span> [</span>1</span> x </span>i64</span>] </span>zeroinitializer</span>, </span>section</span> "__llvm_prf_cnts"</span>, </span>align</span> 8</span></span>
@__profd_ciao</span> = </span>private global</span> { </span>i64</span>, </span>i64</span>, </span>i64*</span>, </span>i8*</span>, </span>i8*</span>, </span>i32</span>, [</span>2</span> x </span>i16</span>] } { </span>i64</span> -1479480177954886802</span>, </span>i64</span> 0</span>, </span>i64* getelementptr inbounds</span> ([</span>1</span> x </span>i64</span>], [</span>1</span> x </span>i64</span>]* </span>@__profc_ciao</span>, </span>i32</span> 0</span>, </span>i32</span> 0</span>), </span>i8* bitcast</span> (</span>void</span> ()* </span>@ciao</span> to i8*</span>), </span>i8* null</span>, </span>i32</span> 1</span>, [</span>2</span> x </span>i16</span>] </span>zeroinitializer</span> }, </span>section</span> "__llvm_prf_data"</span>, </span>align</span> 8</span></span>
@__profc_foo</span> = </span>private global</span> [</span>1</span> x </span>i64</span>] </span>zeroinitializer</span>, </span>section</span> "__llvm_prf_cnts"</span>, </span>align</span> 8</span></span>
@__profd_foo</span> = </span>private global</span> { </span>i64</span>, </span>i64</span>, </span>i64*</span>, </span>i8*</span>, </span>i8*</span>, </span>i32</span>, [</span>2</span> x </span>i16</span>] } { </span>i64</span> 6699318081062747564</span>, </span>i64</span> 0</span>, </span>i64* getelementptr inbounds</span> ([</span>1</span> x </span>i64</span>], [</span>1</span> x </span>i64</span>]* </span>@__profc_foo</span>, </span>i32</span> 0</span>, </span>i32</span> 0</span>), </span>i8* bitcast</span> (</span>void</span> ()* </span>@foo</span> to i8*</span>), </span>i8* null</span>, </span>i32</span> 1</span>, [</span>2</span> x </span>i16</span>] </span>zeroinitializer</span> }, </span>section</span> "__llvm_prf_data"</span>, </span>align</span> 8</span></span>
@__profc_main</span> = </span>private global</span> [</span>3</span> x </span>i64</span>] </span>zeroinitializer</span>, </span>section</span> "__llvm_prf_cnts"</span>, </span>align</span> 8</span></span>
@__profd_main</span> = </span>private global</span> { </span>i64</span>, </span>i64</span>, </span>i64*</span>, </span>i8*</span>, </span>i8*</span>, </span>i32</span>, [</span>2</span> x </span>i16</span>] } { </span>i64</span> -2624081020897602054</span>, </span>i64</span> 717562688593</span>, </span>i64* getelementptr inbounds</span> ([</span>3</span> x </span>i64</span>], [</span>3</span> x </span>i64</span>]* </span>@__profc_main</span>, </span>i32</span> 0</span>, </span>i32</span> 0</span>), </span>i8* bitcast</span> (</span>i32</span> (</span>i32</span>, </span>i8**</span>)* </span>@main</span> to i8*</span>), </span>i8* null</span>, </span>i32</span> 3</span>, [</span>2</span> x </span>i16</span>] </span>zeroinitializer</span> }, </span>section</span> "__llvm_prf_data"</span>, </span>align</span> 8</span></span>
@__llvm_prf_nm</span> = </span>private constant</span> [</span>23</span> x </span>i8</span>] </span>c</span>"\0D\15x\DAK\CEL\CCgL\CB\CFg\CCM\CC\CC\03\00\1Fy\04\88"</span>, </span>section</span> "__llvm_prf_names"</span>, </span>align</span> 1</span></span></code></pre>
</div>
It stands out from what we see that it contains:</p>

3 __profc_<function_name></code> global arrays

one for every function existing in the source code</li>
</ul>
</li>
3 __profd_<function_name></code> global structs

one for every function existing in the source code</li>
</ul>
</li>
1 constant - __llvm_prf_nm</code> - encoding the function names</strong></label>Recompiling with the -mllvm -enable-name-compression=false</code> flags makes this variable easier to read.</span></li>
</ul>
Also, all these variables have ELF sections.</p>
The __profc_<function_name></code> variables end up into the __llvm_prf_cnts</code> section.
The __profd_<function_name></code> variables end up into the __llvm_prf_data</code> section.
The __llvm_prf_nm</code> variable ends up into the __llvm_prf_names</code> section.</p>
But what __profc_<function_name></code> variables are?</p>
The section names helped me here: they are the counters</mark> used by LLVM for instrumenting our code for profiling</mark>!</p>
Scrolling down into the LLVM IR, we see they're getting incremented (load</code>, add</code>, store</code> instructions sequence) at the correct spots:</p>

    
    LLVM IR / Main function</span>hello.ll</span></span>
    </label>define dso_local i32</span> @main</span>(</span>i32</span> %0</span>, </span>i8**</span> %1</span>) #</span>0</span> !dbg</span> !</span>19</span> {</span></span>
  ...</span></span>
  %7</span> = </span>load i64</span>, </span>i64* getelementptr inbounds</span> ([</span>3</span> x </span>i64</span>], [</span>3</span> x </span>i64</span>]* </span>@__profc_main</span>, </span>i64</span> 0</span>, </span>i64</span> 0</span>), </span>align</span> 8</span>, </span>!dbg</span> !</span>30</span></span>
  %8</span> = </span>add i64</span> %7</span>, </span>1</span>, </span>!dbg</span> !</span>30</span></span>
  store i64</span> %8</span>, </span>i64* getelementptr inbounds</span> ([</span>3</span> x </span>i64</span>], [</span>3</span> x </span>i64</span>]* </span>@__profc_main</span>, </span>i64</span> 0</span>, </span>i64</span> 0</span>), </span>align</span> 8</span>, </span>!dbg</span> !</span>30</span></span>
  ...</span></span>
  br i1</span> %10</span>, </span>label</span> %11</span>, </span>label</span> %24</span>, </span>!dbg</span> !</span>34</span></span>
</span>
11</span>:                                               </span>; preds = %2</span></span>
  %12</span> = </span>load i64</span>, </span>i64* getelementptr inbounds</span> ([</span>3</span> x </span>i64</span>], [</span>3</span> x </span>i64</span>]* </span>@__profc_main</span>, </span>i64</span> 0</span>, </span>i64</span> 1</span>), </span>align</span> 8</span>, </span>!dbg</span> !</span>34</span></span>
  %13</span> = </span>add i64</span> %12</span>, </span>1</span>, </span>!dbg</span> !</span>34</span></span>
  store i64</span> %13</span>, </span>i64* getelementptr inbounds</span> ([</span>3</span> x </span>i64</span>], [</span>3</span> x </span>i64</span>]* </span>@__profc_main</span>, </span>i64</span> 0</span>, </span>i64</span> 1</span>), </span>align</span> 8</span>, </span>!dbg</span> !</span>34</span></span>
  call void</span> @foo</span>(), </span>!dbg</span> !</span>35</span></span>
  ...</span></span>
  br label</span> %14</span>, </span>!dbg</span> !</span>40</span></span>
</span>
14</span>:                                               </span>; preds = %20, %11</span></span>
  %15</span> = </span>load i32</span>, </span>i32*</span> %6</span>, </span>align</span> 4</span>, </span>!dbg</span> !</span>41</span></span>
  %16</span> = </span>icmp slt i32</span> %15</span>, </span>22</span>, </span>!dbg</span> !</span>43</span></span>
  br i1</span> %16</span>, </span>label</span> %17</span>, </span>label</span> %23</span>, </span>!dbg</span> !</span>44</span></span>
</span>
17</span>:                                               </span>; preds = %14</span></span>
  %18</span> = </span>load i64</span>, </span>i64* getelementptr inbounds</span> ([</span>3</span> x </span>i64</span>], [</span>3</span> x </span>i64</span>]* </span>@__profc_main</span>, </span>i64</span> 0</span>, </span>i64</span> 2</span>), </span>align</span> 8</span>, </span>!dbg</span> !</span>44</span></span>
  %19</span> = </span>add i64</span> %18</span>, </span>1</span>, </span>!dbg</span> !</span>44</span></span>
  store i64</span> %19</span>, </span>i64* getelementptr inbounds</span> ([</span>3</span> x </span>i64</span>], [</span>3</span> x </span>i64</span>]* </span>@__profc_main</span>, </span>i64</span> 0</span>, </span>i64</span> 2</span>), </span>align</span> 8</span>, </span>!dbg</span> !</span>44</span></span>
  call void</span> @ciao</span>(), </span>!dbg</span> !</span>45</span></span>
  br label</span> %20</span>, </span>!dbg</span> !</span>47</span></span>
</span>
20</span>:                                               </span>; preds = %17</span></span>
  %21</span> = </span>load i32</span>, </span>i32*</span> %6</span>, </span>align</span> 4</span>, </span>!dbg</span> !</span>48</span></span>
  %22</span> = </span>add nsw i32</span> %21</span>, </span>1</span>, </span>!dbg</span> !</span>48</span></span>
  store i32</span> %22</span>, </span>i32*</span> %6</span>, </span>align</span> 4</span>, </span>!dbg</span> !</span>48</span></span>
  br label</span> %14</span>, </span>!dbg</span> !</span>49</span>, </span>!llvm.loop</span> !</span>50</span></span>
</span>
23</span>:                                               </span>; preds = %14</span></span>
  br label</span> %24</span>, </span>!dbg</span> !</span>53</span></span>
</span>
24</span>:                                               </span>; preds = %23, %2</span></span>
  %25</span> = </span>call i32</span> (</span>i8*</span>, ...) </span>@printf</span>(</span>i8* getelementptr inbounds</span> ([</span>6</span> x </span>i8</span>], [</span>6</span> x </span>i8</span>]* </span>@.str.2</span>, </span>i64</span> 0</span>, </span>i64</span> 0</span>)), </span>!dbg</span> !</span>54</span></span>
  %26</span> = </span>load i32</span>, </span>i32*</span> %3</span>, </span>align</span> 4</span>, </span>!dbg</span> !</span>55</span></span>
  ret i32</span> %26</span>, </span>!dbg</span> !</span>55</span></span>
}</span></span></code></pre>
</div>
We now know why the __profc_main</code> array has size 3 (see [3 x i64]</code>): LLVM instrumented one counter for the main()</code> function's entry, one for the if</code> block, while the last one regards the for</code> block.</p>
We also know that we have a total of 5 counters, given the __profc_ciao</code> and __profc_foo</code> have both size 1.</p>
Verifying this is just a matter of obtaining our program binary...</p>
clang</span> -g -fprofile-instr-generate</span> hello.c</span> -o</span> hello</span></span></code></pre>
And playing with llvm-objdump</code>:</p>
llvm-objdump</span> --section-headers</span> hello</span> |</span> grep</span> __llvm</span></span>
# 17 __llvm_prf_names   00000017 0000000000008b89 DATA</span></span>
# 26 __llvm_prf_cnts    00000028 000000000000c110 DATA</span></span>
# 27 __llvm_prf_data    00000090 000000000000c138 DATA</span></span></code></pre>llvm-objdump</span> --syms</span> hello</span> |</span> grep</span> __llvm_prf_cnts</span></span>
# 000000000000c110 g       __llvm_prf_cnts        0000000000000000 .hidden __start___llvm_prf_cnts</span></span>
# 000000000000c138 g       __llvm_prf_cnts        0000000000000000 .hidden __stop___llvm_prf_cnts</span></span></code></pre>
The output here helps us confirm that we have 5 total counters of 8 bytes each, starting at address c110</code> and ending at c138</code>.</p>
In fact, $c138 - c110 = 28$, which means 40 bytes in decimal base (ie., $8 * 5$).</p>
What about the __profd_<function_name></code> variables instead?</p>
llvm-objdump</span> --syms</span> hello</span> |</span> grep</span> __llvm_prf_data</span></span>
# 000000000000c1c8 g       __llvm_prf_data        0000000000000000 .hidden __stop___llvm_prf_data</span></span>
# 000000000000c138 g       __llvm_prf_data        0000000000000000 .hidden __start___llvm_prf_data</span></span></code></pre>
At the moment, all you need to know is that those are structs containing data about the function they refer to and the instrumented counters for it.</p>
I hope the following image clarifies the relationship. Anyways, we'll dig deeper into the __profd_<function_name></code> variables later</a>.</p>
</a></p>
Generating the profraw§</a>
</h2>
At this point, I still didn't know how</strong> the default.profraw</code> file got created every time I executed my binary.</p>
I came back to the LLVM source code. It's incredible the superpowers that reading the source gives you...</p>
I found InstrProfilingFile.c</code></label>Source</a>.</span>.</p>
Long story short: the code in this LLVM source file implements the profiling runtime initialization</mark>, which means it is responsible for gathering all the data from the variables (counters, data, etc.) we've seen and flushing them out in the default.profraw</code> file.</p>
While looking at it, I suddenly noticed the __llvm_profile_initialize</code></label>Source</a>.</span> function.</p>
It gets invoked by the profiling runtime initialization hook to get the filename of the profraw to create. Either from an environment variable (LLVM_PROFILE_FILE</code>) or using the default one (default.profraw</code>). Finally, it calls the C library function atexit(__llvm_profile_write_file)</code>. Which will cause the __llvm_profile_write_file</code> function to be invoked when the program terminates.</p>
Thus, I also noticed the __llvm_profile_write_file</code></label>Source</a>.</span> function.</p>
The main thing this function does is to invoke writeFile(Filename)</code></label>Source</a>.</span>. Which, in turn, calls lprofWriteData</code> </label>Source</a>.</span>.</p>
It's again time to verify with llvm-objdump</code>.</p>
llvm-objdump</span> --syms</span> hello</span> |</span> grep</span> __llvm_profile</span></span>
# 0000000000004180 l     F .text  00000000000001a1 __llvm_profile_write_file</span></span>
# 0000000000004610 g     F .text  0000000000000037 .hidden __llvm_profile_register_write_file_atexit</span></span>
# 00000000000121e4 g     O .bss   0000000000000004 .hidden __llvm_profile_runtime</span></span>
# 000000000000c108  w    O .data  0000000000000008 __llvm_profile_raw_version</span></span>
# 00000000000122e8  w    O .bss   0000000000000001 __llvm_profile_filename</span></span>
# 00000000000040a0 g     F .text  00000000000000ae .hidden __llvm_profile_initialize</span></span>
# ...</span></span></code></pre>
Our ELF got flooded by __llvm_profile_*</code> functions.</p>
Seeing all those symbols in the ELF made me 100% sure this is how LLVM makes our binary open the default.profraw</code> file as soon it executes. In this way, it also makes our program automatically dump the profile data when it terminates.</p>
Coming back to the lprofWriteData</code> function implementation (in the InstrProfilingWriter.c</code> source file), we instead discover what</strong> it dumps into default.profraw</code>. It's while doing so that we understand what a profraw file contains.</p>
Such a function</label>Source</a>.</span>:</p>

calculates the size of the data, counters, and names</li>
calculates the paddings needed before and after the counters section (__llvm_prf_cnts</code>) and after the names section (__llvm_prf_names</code>)</li>
initializes the header structure (more on it below</a>)</li>
writes the header</li>
writes the binary IDs

if NT_GNU_BUILD_ID</code> is defined, otherwise skips them</li>
</ol>
</li>
writes the data

the __profd_<function_name></code> globals</li>
</ol>
</li>
writes the padding bytes before the counters for page alignment</li>
writes the counters

the content of the __profc_<function_name></code> globals at program termination</li>
</ol>
</li>
writes the padding bytes after the counters for page alignment</li>
writes the names

the __llvm_prf_nm</code> variable</li>
</ol>
</li>
writes the padding bytes after the names part

the profraw format want its size to be a multiple of 8</li>
</ol>
</li>
writes the value profiling data

only in case it was enabled</li>
</ol>
</li>
</ol>
All these moving parts are written using the ProfDataIOVec</code> data structure</label>Source</a>.</span>.</p>
So, we can finally assert what a profraw binary file is composed by:</p>

a header</a></li>
the data</a> variables</li>
the counters</a> variables</li>
the names</a> variable</li>
</ul>
Header§</a>
</h2>
The INSTR_PROF_RAW_HEADER</code> macro at InstrProfData.inc</a> defines the parts composing the header of the profraw file. Which are:</p>

the magic</li>
the version</li>
the size of binary IDs</li>
the size of the data section:

it is equal to the number of __profd_<function_name></code> global variables</li>
the data part starts where __start___llvm_prf_data</code> is</li>
the data part ends where __stop___llvm_prf_data</code> is</li>
assuming 3 __profd_<function_name></code> global struct exists</label>{i64, i64, i64*, i8*, i8*, i32, [2 x i16]}</code> (48 bytes)</span>, the data section size is the difference between the __stop_</code> (c1c8</code>) and __start_</code> addresses (c138</code>), thus 90 in hexadecimal</label>0x90 is 144 decimal, so $144 / 48 = 3$.</span></li>
</ol>
</li>
the padding bytes before the counters section</label>It can also be 0 bytes.</span></li>
the size of the counters section

it is equal to the number of __profc_<function_name></code> global variables</li>
the counters part starts where the (__start_</code>) __llvm_prf_cnts</code> is</li>
the counters part ends where the (__stop_</code>) __llvm_prf_cnts</code> is</li>
assuming 3 __profc_*</code> global variables exists</label>For a total of 5 counters, 8 bytes each: [1 x i64]</code>, [1 x i64]</code>, [3 x i64]</code></span>, the counters section size is the difference between the __stop_</code> (c138</code>) and the __start_</code> addresses (c110</code>), thus 28 in hexadecimal</label>0x28 is 40 decimal, so $40 / 8 = 5$ which is the total number of counters.</span></li>
</ol>
</li>
the padding bytes after the counters section</label>It can also be 0 bytes.</span></li>
the size of the names section

it is equal to the size of __llvm_prf_nm</code> global variable</li>
the names part starts where the (__start_</code>) __llvm_prf_names</code> is</li>
the names part ends where the (__stop_</code>) __llvm_prf_names</code> is</li>
</ol>
</li>
the counters delta</li>
it is the difference between the address at which the counter section begins and the address at which the data section begins</li>
it is equal to the address of the first __profc_<function_name></code> counter</li>
the names delta

the address at which the names section begins</li>
</ol>
</li>
IPVK_Last</code></label>It seems to be always 1.</span></li>
</ol>
With the following table I tried to summarize the header's content:</p>
00 01 02 03 04 05 06 07</th> 08 09 0A 0B 0C 0D 0E 0F</th></tr></thead>

magic</td> version</td></tr>
data size</td> padding before counters</td></tr>
counters size</td> padding after counters</td></tr>
names size</td> counters delta</td></tr>
names begin</td> value kind last</td></tr>
</tbody></table>
But probably an image is worth a thousand words.</p>
This is the default.profraw</code> file after executing ./hello yay</code>:</p>
</a></p>
We can see that its header is composed of:</p>

the magic (in black)</li>
the version (in blue)</li>
the number of functions (in red)

it equals to the number of __profd_<function_name></code> variables too: 3</strong></li>
</ul>
</li>
the padding bytes before the counters (in light green)</li>
the total number of the counters (in fluo green)

meaning, the total size of __profc_<function_name></code> variables: 5</strong></li>
</ul>
</li>
the padding bytes after the counters (in dark green)</li>
the length of the __llvm_prf_nm</code> variable (in magenta): 0F in hex, 15 decimal</li>
the address of the first counter (in brown)</li>
the address of __llvm_prf_nm</code> (in pink)</li>
IPVK_Last</code> (1) in yellow</li>
</ul>
Exactly the way we were now expecting it.</p>
Data§</a>
</h2>
Focusing on the data part, I've highlighted with (orange) rectangles the 3 __profd_<function_name></code> global variables.</p>
</a></p>
Each of the  __profd_<function_name></code> contains 6 parts:</p>

1 is the ID (hex, reverse order) of the function to which it refers</li>
2 is the function control flow hash</li>
3 points to the counter it relates to</li>
4 points to the function it refers to</li>
5 points to the value expressions, if any</li>
6 contains the number of the counters (2 bytes) and the initialization array (number of value sites, 2 elements, 2 bytes each, often zero)</li>
</ul>
Counters§</a>
</h2>
After the data part comes the counters part.</p>
</p>
It just is a serialization of what the variables in the __llvm_prf_cnts</code> ELF section contains.</p>
In fact, in this screenshot you can see underlined (with waves, in shades of cyan/blue):</p>

__profc_ciao</code>: [1 x i64]

1 counter, with value 16</strong></li>
it means the ciao()</code> function was executed 22 (decimal) times</li>
</ul>
</li>
__profc_foo</code>: [1 x i64]

1 counter, with value 1</strong></li>
it means the foo()</code> function only was executed 1 time</li>
</ul>
</li>
__profc_main</code>: [3 x i64]

3 counters, first two with value 1</strong>, last one with value 16</strong></li>
it means the function main()</code> was executed 1 time, its if</code> conditional evaluated to true, and the for</code> inside it got executed 22</strong> (decimal) times</li>
</ul>
</li>
</ul>
Names§</a>
</h2>
Finally, the __llvm_prf_nm</code> variable populates the last part of the profraw file.</p>
</a></p>
Here (in grey) you can see its content</label>Function names are uncompressed because I compiled the binary with -mllvm -enable-name-compression=false</code> flags.</span>, padded with a \0</code> byte (the one circled) at the end to align the profraw file size to be multiple of 8.</p>
Usage§</a>
</h2>
It's time to use the default.profraw</code> file now:</p>
llvm-profdata</span> show</span> --all-functions --counts</span> default.profraw</span></span>
# Counters:</span></span>
#   ciao:</span></span>
#     Hash: 0x0000000000000000</span></span>
#     Counters: 1</span></span>
#     Function count: 22</span></span>
#     Block counts: []</span></span>
#   foo:</span></span>
#     Hash: 0x0000000000000000</span></span>
#     Counters: 1</span></span>
#     Function count: 1</span></span>
#     Block counts: []</span></span>
#   main:</span></span>
#     Hash: 0x000000a71211b451</span></span>
#     Counters: 3</span></span>
#     Function count: 1</span></span>
#     Block counts: [1, 22]</span></span>
# Instrumentation level: Front-end</span></span>
# Functions shown: 3</span></span>
# Total functions: 3</span></span>
# Maximum function count: 22</span></span>
# Maximum internal block count: 22</span></span></code></pre>
Isn't it cool?</p>
There's more you can do with profraw files and the LLVM toolchain, indeed. But showing that was not the goal of this post.</p>
I hope you now have a better understanding of what profraw files are.</p>

leodido.dev - Clang

Code coverage for eBPF programs

Use it§</a> </h3> bpfcov run + bpfcov gen§</a> </h4> What's left to do? Just three steps:</p> Run our eBPF application with the bpfcov</mark> CLI</strong> run command</li> Generate its profraw</mark> file</li>

bpfcov run + bpfcov gen§</a> </h4> What's left to do? Just three steps:</p> Run our eBPF application with the bpfcov</mark> CLI</strong> run command</li> Generate its profraw</mark> file</li>

Demystifying the profraw format