Linux is the de facto foundation for today’s computing infrastructure, and Linux developers have never eliminated memory and concurrency bugs [35, 39, 49, 56], which have been plaguing systems software for years. Bugs keep emerging [9, 16, 17, 18] despite years of security hardening and engineering efforts from the Linux community [1, 12]. Rust seems to be a promising solution, which may finally resolve the aforementioned problems [8]. As an emerging, statically and strongly-typed systems programming language, Rust claims to deliver both safety and performance without runtime overhead [34]. Backing its claim is the ownership mechanism [31] for eliminating memory and concurrency bugs. Rust eliminates a heavyweight and costly memory checker [51, 55] as well as a garbage collector [48, 52], eschewing being interrupted and having unpredictable delays at run time.
These intriguing properties of Rust led to the advent of Rust-For-Linux (RFL), which began in 2013 as a hobbyist project [2]. As the first attempt to bring Rust into Linux, the project built a Rust object file against kernel headers and invoked one Rust function from the file in a loadable kernel module. In 2019, a proposal to write kernel modules fully in Rust emerged [4, 6] to lead Rust further into Linux. To achieve this goal, the proposal took a bold move by directly adding a thin Rust wrapping layer to kernel interfaces and data structures in upstream Linux. Only a year after the RFC, RFL was officially merged into upstream Linux v6.1 as an experimental kernel feature [13]. As RFL gradually improved in the journey of building a more robust Linux, there emerged numerous attempts to use RFL to write drivers in various areas, such as network [40], block device [45, 46], file system [20, 22], android [38], and GPU [44]. Among them, one network driver [24] first made it into the Linux mainline in v6.8 after 11 rounds’ co-review by the RFL and network community. This means RFL can receive feedback from users, which is a sign of RFL stepping into the real use cases from the experimental states. Despite still being in an early stage, RFL has become one of the most active kernel subsystems [30], on par with ebpf [41], and io_uring [43].
This article takes the first in-depth look into RFL and how a new programming language blends into a giant, old-school codebase which has been already shipped on billions of computing devices. We report our findings on two key research questions (RQs). First, what is the status quo of RFL? Second, does RFL live up to the hype?
For full details, please refer to our USENIX ATC’24 paper [58].
RFL development status
(1) Development progress. We first analyze the development progress of RFL. Overall, RFL is still at a very early stage in blending with the Linux kernel: in terms of LoC, the merged code (7.1%) only constitutes 0.125% of the kernel code, while the rest (92.9%) is still pending review or is staged for merging. We further break down the merged code by their respective kernel subsystems to understand individual status and show the results in Figure 1.
Insight 1: drivers, netdev, and file systems are the long tail of RFL code.
We have observed a clear long tail: most Rust code resides in scheduling, memory management, and IRQ infrastructure. By contrast, drivers, file systems, netdev, and security subsystems which account for most kernel code (i.e., 78% in Linux v6.2) only have received little RFL code, constituting the “tail”. The results are sensible. As scheduling, memory management, and IRQ subsystems are most commonly used by all drivers, optimizing RFL development for them has a high value and priority. In comparison, drivers and file systems have more specific use cases (e.g., for a particular model of a device), which require more programming and reviewing effort. For instance, reviewers from netdev communities spent 6 months on 11 versions of draft patches, before settling one the final merged network PHY driver [29].
(2) Patch distribution. To study how individual RFL components develop, we categorize the code into three types depending on their use cases, i.e. for building safe abstraction, the Kbuild system, and the Rust compiler. We show the results in Figure 2 and conclude the following insight:
Insight 2: RFL infrastructure has matured, with safe abstraction and drivers being the next focus.
We base the insight on two key pieces of evidence: 1) as time passes, Kbuild undergoes a clear recession in its portion of the RFL cake, indicating the foundation of RFL has been laid; 2) in the meantime, abstraction takes up more portion, e.g., from 20% to 60% in 18 months. Interestingly, a surge in the number of Rust commits appeared half a year after RFL started, as seen by the 23/4 timestamp of Figure 2. It belongs to a patch that directly modifies the RFL lib code for supporting a safe implementation of the Rust initializer for pinned objects; prior to the fix, early RFL has been using an unsafe initializer in gpio_pl061 and bcm2835_rng drivers.
(3) The trend. To understand how RFL development progresses with time, we project the commits and the email exchanges onto the timeline and highlight the reviewing time length of the PRs, as shown in Figure 3.
Insight 3: RFL is bottlenecked by code review but not by code development.
From the slope changes in the lefthand graph of Figure 3, we are witnessing the committed/staged RFL code start to plateau after the initial steep slope when RFL first started. Yet, the number of email exchanges shows RFL is an increasingly active community. Besides, the PR reviewing becomes significantly slower as time goes by, e.g., the PRs between Jan. 2023 and Jul. 2023 take 280 hours to be reviewed on average, which is 200× as long as 3 years ago. This suggests the speed of producing RFL code is much faster than that of consumption, i.e., reviewing and eventually merging RFL code into the upstream kernel. This can also be confirmed by the huge imbalance between merged code versus the rest of the code, where the latter is 13× larger than the former and represents a huge bulk of the area.
An inspiring observation is that RFL is gradually being embraced by the kernel community, seeing the increasing engagement of traditional kernel developers. For example, the recently developed NVME, NULL block, V4L2, and e1000 drivers using RFL are all driven by the Linux community.
Rustify Linux with safe abstraction
Safe abstraction is the key ingredient towards rustifying Linux kernel and among the largest portions of RFL code. As the name itself implies, the layer extends C kernel features safely into Rust drivers: it abstracts kernel data structures and interfaces, so that upon invocation they may still ensure memory and thread safety.
Converting kernel data structures RFL leverages bindgen to automatically generate Rust bindings of C kernel struct prior to use. The generation is rule-based and syntax-directed, which translates the C types and symbols into their Rust counterparts. The translation is mechanical, following the rules in Table 1. For instance, uint32_t of C translates into c_uint in Rust namespace core::ffi, which aliases to the Rust primitive u32. However, not every C type translates into a corresponding Rust primitive. We have found such incompatibility exists especially in language features which kernel developers exploit for manually consolidating memory layout. We detail them as follows.
(1) Emulated bitfields and unions. The kernel extensively uses bitfields and unions for improving memory efficiency, e.g., the e1000 driver uses a single word to store 4 flags to indicate link states. Bit operations on struct members contradict Rust memory safety principle and hence do not have native Rust support. As a workaround, RFL emulates bitfields with a byte array, which implements bit operations as accessors to the array. Although Rust has a union primitive, it cannot provide ABI compatibility with C union. Thus RFL implements a struct called __BindgenUnionField with the same memory layout as the C interface. Both workarounds are based on the transmute operation, which reinterprets memory at run time, hence are implemented in unsafe blocks. The major overhead of the emulation code is the increase of binary size, which we will discuss in the "Does Rust incur any overhead?" section.
(2) Incomplete attribute support. The kernel often relies on packed and aligned attributes for better locality and memory efficiency, e.g., task_struct groups most frequently access scheduling data in one cache line. Despite the attributes that are supported by Rust repr(C), Rust may still mishandle them and cause bindgen to generate the wrong code [3, 5]. For attributes less commonly used, RFL does not support them, e.g., BTF tags [14]. Notably, RFL ignores the randomize_layout attribute, which kernel utilizes to mitigate memory bugs [37]. This is reasonable because Rust has already mitigated such vulnerabilities through ownership and boundary checks.
Despite the generated bindings having the identical data layout as their C counterpart, the safe abstraction layer still cannot directly expose them to drivers. This is because the bindings involve numerous raw pointers (i.e., *mut), which are prevalent in the kernel but are unsafe to use in Rust. To safely use them, Rust needs to manually reason about the pointer validity (i.e., not via borrow checker at compile time) and specify their ownership. To this end, RFL uses helper types to embed generated kernel data structure bindings and bakes Rust flavors into them.
Insight 4: The kernel’s initiative to control memory in fine granularity conflicts Rust philosophy, which incurs overhead for RFL.
| Type | C | Rust |
|---|---|---|
| Primitive types | foo | core::ffi:c_foo |
| Typed pointers | foo * | *mut foo |
| Attributes | aligned | #repr(c)(with caveats [3, 5]) |
| unused | ignored | |
| weak | ignored | |
| randomize_layout | ignored | |
| Function pointer | fn | option<fn> |
Incorporating kernel functions
Following similar translation rules in converting the kernel data structures (Table 1), RFL generates FFI bindings of kernel functions. Then, RFL extensively leverages Rust traits to massage Rust features into them in the safe abstraction layer. We summarize three major measures.
(1) Functions as members of structs. RFL groups kernel functions related to a type and incorporates them as members of the type struct, following an OOP paradigm. For instance, it groups work queue related functions such as queue_work_on, __INIT_WORK_WITH_KEY under Rust struct Queue, the RFL helper type for kernel struct workqueue_struct. Doing so improves code readability and ensures the caller of these functions is never null, avoiding pointer validity checks inside the function body.
(2) Functions pointers as traits. Many kernel functions are dangling and used only as callbacks of kernel structs at run time. RFL incorporates them as traits of the helper type and specify bounds on them. The trait bounds specify the callback types and owner struct for the dangling kernel functions, preventing vulnerabilities caused by incorrect type casting [10].
(3) Wrapping inlined functions and macros. Inlined functions and macros provide important and handy utilities to kernel drivers, e.g., for_each_online_cpu to enumerate available CPUs. RFL wraps static inline functions with non-inlined Rust functions. This is because current Rust lacks a convenient mechanism for inlining Foreign Function Interfaces (FFIs) of C functions; while it is still possible to inline them, it takes lots of efforts and hence is not encouraged by the community [27]. For function-like macros, RFL prefers wrapping them with helper functions instead of rewriting them with Rust macros. The main reason is RFL inclines not to maintain two sets of kernel interfaces known to be unstable [50].
Insight 5: RFL uses helper types to delegate management of kernel data to Rust while leaving the operation to the kernel itself.
RFL makes Linux more “securable”
We first collect all bug reports and safety-related code reviews among the RFL abstractions and drivers. In total, we have found 25 bugs from merged and staged RFL code. Among them, 15 of them are in the Linux mainline and 10 of them are in the stage Rust branch. We list them in Table 2. Of the bugs within merged code, 11 are compilation bugs and the other 4 are related to safe abstraction. The compilation bugs do not introduce safety vulnerabilities; they are mostly caused by the misconfig of the kernel, incompatibility of various Clang toolchain versions and the mismatch between the Kbuild and rustc compiler [11]. Of the soundness bugs, 6 are in the safe abstraction layer and break memory safety and 3 break thread safety.
RFL safety hinges on the safety assurance of Rust language, which concretely relies on the elimination of all unsafe blocks in drivers and the safe abstraction APIs. Therefore, they are the entry point to RFL safety vulnerabilities, if any. We examine all existing RFL drivers and Rust kernel crates in the upstream repo [15] and analyze the usage of unsafe code blocks. We have not found any unsafe usage of RFL drivers in the Linux mainline, because no serious driver has made into the mainline except for one with around 130 lines of code [33]. However, we have found unsafe cases in the drivers that are proposed to the RFL mailing list. We show the results in Table 3.
| Source | Compilation bug | Soundness bug |
|---|---|---|
| GitHub [15] | 4(1/3) | 7(3/4) |
| Intel LKP [26] | 8(6/2) | 0 |
| Mailing List [28] | 4(4/0) | 2(1/1) |
| Driver | Number of Unsafe usage | |
|---|---|---|
| Driver logic | Safety abstractions | |
| GPU [44] | 107 | 7 |
| NVME [46] | 44 | 16 |
| Null block [45] | 0 | 0 |
| E1000 [40] | 4 | 2 |
| Binder [38] | 45 | 9 |
| Gpio_pl061 [42] | 0 | 3 |
| Semaphore [37] | 0 | 4 |
Post analysis We audit the bug reports and unsafe code to summarize our findings on RFL safety as follows.
Our verdict is reminiscent of that of the Multics security audit [54] and is based on the following facts.
(1) Rust safety mechanism constructs the pillar of kernel safety. The language-level support helps kernel drivers fix existing bugs and eschew potential memory/concurrency bugs. As a modern language with rich type specifiers, it facilitates more canonical safety checkers such as klint [19], and RustBelt [53] to further harden the kernel. Compared with C, RFL greatly reduces the vast attack space of kernel software caused by memory bugs. As a result, the developer has much less to reason about in terms of kernel security.
(2) unsafe is inevitable, though vulnerabilities are optional. In our audit, we have found unsafe code exist commonly in all major drivers and it is hard, if not impossible, to fully eliminate them. The reason is twofold. First, as the kernel asserts full control over memory and hardware, its operations need to bypass Rust ownership checks. For instance, kernel developers exploit inline assembly for managing TLB and issuing memory barrier [44], raw pointers for de-referencing MMIO registers, and unions/bitfields. Second, the community sometimes has to compromise on unsafe functions. This is because ownership sometimes introduces twisted implementation, which often requires long reviewing cycles. A notable example we have found is a memory initialization interface pin-init. After rounds of debates by seasoned developers [7, 36], has the interface been finally fixed in a recent patch [32].
(3) Bugs do not disappear; they only hide deeper. On the one hand, kernel functions invoked by the safe abstraction and Rust drivers may still contain bugs and be exploited. On the other hand, although Rust compile-time borrow checker detects memory bugs on the spot, it does not detect semantic bugs, which are often caused by subtle differences in Rust and kernel memory allocation methods. Such bugs may take a long to fix and can only be detected by experts who are familiar with both Rust and kernel. For example, the C binder driver has a use-after-free [21] bug. When being re-implemented in Rust, it will not cause the same symptom as in C. Instead, it incurs a mapping bug putting the memory into the wrong place which passes all the checks by the Rust compiler.
Insight 6: with RFL, Linux becomes more “securable” but still cannot be fully secure.
Does rust incur any overhead?
Setup We collect 4 drivers with serious use cases, which span diverse IO functions (e.g., network, storage). Notably, NVME and binder are considered the first batch of drivers to be merged in the Linux mainline. In addition, we include 2 more toy drivers (i.e., gpio and sem) from RFL Rust branch which are often used to explore the difference between Rust and C implementation. Among the 6 drivers tested, only binder and the 2 toy example drivers have faithfully implemented full features as C drivers do; while the rest of them have only implemented a subset of the features.
Binary size As illustrated in Figure 4, Rust drivers with fully implemented features are significantly larger than C: Since the text section is to be blamed for most of the increased binary size, we looked into it further and find that Rust generates extra code (99%) to support its unique features that C does not have: generic programming, boundary checks, lifecycle management, etc. Even for the gpio driver which simply wraps around five kernel APIs (i.e., probe, mask/unmask, irq_type, suspend and resume), Rust expands its size by 33%.
Performance Mostly, Rust drivers show on par performance with C drivers within a 20% gap. Occasionally, Rust underperforms C significantly; in a few cases, Rust surprisingly outperforms C. Looking into individual drivers, we have found:
- For e1000, Rust driver is 11× slower than C driver for ping latency as presented in Figure 5. The reason we dug out is that the Rust driver has not yet implemented as many features that can accelerate data transmission as the C driver does, e.g., prefetch.
- For binder, the Rust driver shows similar performance with C, with only a 10% gap in ping latency.
- For storage devices (NVME and NULL block), Rust drivers lead to overall similar performance with C, with up to 61% degradation and 67% improvement in throughput, depending on the specific settings (e.g., job number and batch sizes) as shown in Figure 6 and Figure 7. We observe that Rust drivers favor smaller job numbers and blocksize, possibly because Rust often has smaller structs (reasons explained later) which are more likely to fit in the cache line.
Why Rust drivers may perform poorly
- Locks in Rust drivers are coarse-grained. Despite Rust off-loading the duty of ensuring thread-safety to the language itself (i.e., via rules), it does not lift the burden of high-performance concurrency programming of a developer.
- Rust runtime checks in accessing arrays (e.g., boundary checks) introduce extra performance costs. The results are consistent with the prior study which reports Rust program overhead can be 2.49× larger than C program [57]. Rust performs poorly in memory-intensive workloads.
- Rust uses the emulated bit fields. As introduced earlier in "Rustify Linux with safe abstraction", bit field accesses are emulated via array accesses. It further gets exacerbated by the runtime boundary checks.
- Rust massively use pointers to share the ownership of objects, which results in a higher cache/TLB/branch miss rate.
Why Rust drivers may perform better
- The Rust struct has a smaller size compared with the C due to the usage of smart pointers instead of allocating item memory inside the struct. We use pahole to identify that Rust structs use fewer cache lines than their C counterpart.
- The Rust driver does not implement full features compared with C, thus some code paths may be omitted.
Insight 7: There is no free lunch for performance – it is the programmer that counts!
In this article, we thoroughly investigate RFL, the very first and increasingly popular project that aims to use Rust to enhance the Linux kernel. We first study the status quo of RFL by diving into the construction of the safe abstraction layer and Rust drivers, which reveals the tension between the language features of Rust and kernel programming. We then look into whether and to what extent RFL has delivered its promise in building a safer kernel with zero overhead. The results show RFL brings better safety but still has leaks which are stealthier and undetectable by the compiler, and that the overhead brought by the tension between Rust and Linux can not be totally erased. Last, we summarize key lessons learned in this study, hoping it may guide future development of RFL.
