At NSDI'25, Ben Pfaff accepted the Test-of-Time award for the 2015 paper about Open vSwitch [1], the software implementation of a network switch that distributed computing in today's clouds rely upon. Open vSwitch is also open source, another important aspect to OvS' success. Clouds include racks of servers, with each server running many virtual machines, and implementing software-based network switches are crucial for the use and management of the vast numbers of VMs.
The paper on the Design and Implementation of Open vSwitch won the Best Paper award at NSDI'15, and I had actually asked the paper's authors before
the conference to write an article about OvS. I think I was intrigued with the notion of software defined networks (SDN), as well as being impressed that this was an open source project.
Rik Farrow: Based on bios in the 2015 article in ;login: [2] about OvS, it sounds like the software was started by a group out of Stanford that formed a company called Nicira. And that's how software later taken over by VMware remained open source. Could you explain a bit of the history
of OvS?
Ben Pfaff: That's right, Nicira was founded by Nick McKeown and Martin Casado from Stanford, and Scott Shenker from Berkeley, as an extension of their research into flow-based networking, the SANE [3] and Ethane [4] papers in particular.
OvS being open source was essential. Nicira's mission was to sell better, software-based networking to companies making use of virtualization software. That could only work if Nicira's controller products had something to control, and Open vSwitch enabled that control. To allow Nicira's software to be as useful as possible, and therefore to maximize sales potential, it made sense for Open vSwitch to run in as many environments as possible. Open vSwitch as open source allowed it to be ported and extended widely, and since it was useful for many things beyond running Nicira's controller, that actually happened.
The userspace and kernel portions of Open vSwitch were fairly well separated. While the kernel portion obviously must be open source, it would have been practical to make the userspace portion closed source. We made it open and kept it open because it was to our advantage to do so. And I am a long-time personal proponent of free and open source software as well.
There are, and were, other virtual switches. Open vSwitch filled a niche that the others did not: it was fast, it was portable, it was flexible, and it was designed to be controlled remotely. We wrote about some of this at the time in a document titled "Why Open vSwitch?" that is still in the Open vSwitch repository:
https://github.com/openvswitch/ovs/blob/main/Documentation/intro/why-ovs.... For all these reasons, and because we worked hard to integrate outside feedback and contributions, it fit well into many other projects.
RF: Speaking of your commitment to open source, you have worked on a number of other open source projects over the years. Can you speak about your experience working on these? Were there ones you particularly liked? What stuck in my mind was the small graphics module.
BP: Free and open source software is where I started. As a teenager, I got involved in contributing to the GNU project and to Debian GNU/Linux. I was a Debian developer for over 20 years, with my biggest contributions around packaging GNU Autoconf. Besides Open vSwitch, my longest-running project is GNU PSPP (see https://www.gnu.org/software/pspp/), a clone of the SPSS statistical software package from IBM, which I started writing in 1996 and still work on. Currently I'm rewriting it in Rust. I think that the graphics library you're talking about is BOGL, which was a quick hack I built in the late 1990s to help with the Debian installer. You might also be thinking of one of my Linux kernel contributions that happened at the same time: I contributed the 16-color VGA framebuffer driver, which allowed ordinary PCs at the time to support simple graphics on text consoles.
RF: That you have created and continue to work on PSPP is impressive. The duty of continuing to support an open source package just seems daunting to me. I interviewed Arnold Robbins [5], that author and maintainer of gawk, and his main complaint was about getting requests for features that if he added them, he would need to continue to support them, even if that feature would rarely be used.
BP: Supporting open source is hard. I have had some of the same experiences as you say Arnold did. We often had users asking for particular features. Since VMware, our employer, made a controller (NVP, later renamed NSX) that used Open vSwitch, the loudest requests often came from those users inside VMware, and those were often the users that were hardest to refuse. On the Open vSwitch team, we knew that we would have to explain and justify those features to everyone, not just VMware employees, and document and maintain them. This led us to push back against the ones that wouldn't be useful outside of whatever NSX wanted at the time. I'm really glad that we did that most of the time, because in some of the cases where we didn't, Open vSwitch ended up with orphan features that no one used—maybe not even NSX, since it turns out that more generally useful features were also more generally useful for NSX, too.
Sometime, contributors outside VMware would contribute not just a feature but all of the code for it, too. That's really good from one point of view, but it doesn't solve the problem of maintenance. You don't want to get thousands of lines of code you don't know well dumped on you to maintain. So we had some high reviewing standards for big contributions like that, and we also tried to get an idea of whether the contributors were likely to stick around to maintain their own code. There was also the question of how entangled the new contribution was with the existing code. Some contributions change existing code in all kinds of intimate ways, so that they would be hard to remove or maintain later; we were pickier about accepting those.
I'm convinced that Rust is the future of a lot of programming; certainly, of a lot of my programming. It is a little harder to write code in Rust, but once you've done it, that code is going to be a lot more likely to work properly, and the kinds of failures you get are less catastrophic. It's simply beautiful that I don't see segmentation faults or bus errors anymore, since Rust eliminates those. And Rust makes it safe to write multithreaded code, which was always a minefield before.
Writing PSPP has been gratifying. I know that PSPP is important to a number of users, although I don't know how many users that is. A few times I've run into people in real life for whom it's been a big deal. Most notably, when I was sitting in a cafe in Sarajevo many years ago, someone bicycled through heavy rain just to say "thank you" to me for PSPP, which I found moving.
PSPP is almost the opposite of Open vSwitch in terms of contributors. Open vSwitch has very technical users, mostly using Linux, a fair fraction of whom can contribute back or at least do a good job of diagnosing and reporting problems. It seems that most PSPP users, though, are running on Windows (an operating system that I do not use) and sometimes can't say much more than "it crashed". Since PSPP is complicated and written in C and includes a graphical user interface, it does crash more than I'd like, and I often can't find out why. One of the motivations for the rewrite in Rust is to eliminate this class of error. I want my users to be able to rely on my software.
RF: I've often wondered: was it better to have an expert statistician write a statistics program, or an expert programmer who knew little about statistics. When you wrote PSPP, how much did you know about statistics?
BP: My knowledge of statistics was and is still somewhat basic. There is more than enough work of both kinds in PSPP for statisticians and programmers to share. When statisticians have contributed to PSPP, they've needed help from programmers; sometimes when I contribute to PSPP, I've needed help from statisticians. I feel like programmers do have an advantage in your comparison, because they can gain confidence by comparing PSPP results against SPSS results for the same task, whereas statisticians must simply learn programming.
RF: Let's tackle your current project, Feldera. Could you explain the concept behind this? Is this done via static analysis or dynamically?
BP: Feldera is about incremental computing, which comes up when you've got a SQL query that you want to compute over and over again over a lot of data that changes in small ways. It makes sense that it would be cheaper to update the result of your query based on the changes, instead of recalculating the whole query. This is called incremental view maintenance and some databases have been able to do it for some queries for a long time. What Feldera brings in is that we can update the results of any SQL query for any change to a database. That's what the paper that my co-founders published in VLDB in 2023 [6] explained. That paper described a static, mechanical way to transform a database query into an incremental form, producing a new query that works in terms of changes: it takes a stream of insertions and deletions as its input and produces a stream of insertions and deletions as its output as well.
RF: In ML, particularly LLMs, there are lots of matrix operations, and one of the things I've been trying to understand is what happens during pre-training of an LLM when a token is input: does ingesting this token affect every cell in every matrix in the model, or just related ones? If the former, I can imagine that Feldera would save immense amounts of work.
BP: Feldera does not directly help with the matrix algebra parts of machine learning. Incremental computation makes sense when, each time the input to a computation changes, only a small part of the output changes. I think that, in machine learning training, each step changes all of the cells in the matrices (even if only a little), so it's not obvious to me that that part of ML is an application for incremental computing.
But Feldera is a good fit for feature engineering, that is, for figuring out what data to feed into a ML model for training and execution. In real-time ML pipelines, it's necessary to calculate the features to feed into the model in real time as well. Before Feldera, the main choices for computing them that quickly were either to precompute them periodically, which limits how up-to-date they can be and sacrifices accuracy, or to use a streaming system to compute them. The latter limits the kinds of SQL that one can use. Feldera, on the other hand, supports all kinds of computations with SQL, which frees up ML teams to use more sophisticated features that can make their models more accurate.
My co-founder Leonid wrote blog entries on this topic last year: https://www.feldera.com/blog/feature-engineering-part1 and https://www.feldera.com/blog/feature-engineering-part2.
