When I read that USENIX was ending the LISA conference after 35 years I felt a huge sense of pride for what it had accomplished.
LISA made LISA obsolete. It set out to change the world, and did so. The radical ideas that LISA promoted since its creation are now conventional wisdom. They are now so commonplace that new system administrators and technology workers would be surprised to know that things hadn't always been as they are today.
In short: we won!
Having achieved its goals, LISA created the space and pre-conditions for the SRE/DevOps era. I am glad to see SRECon picking up the baton and, dare I say, I expect SREcon to make itself obsolete in less than half the time. The world moves so much more quickly now.
LISA had more impact on my career and personal life than any other single thing I can think of. I do not say that lightly, nor in jest.
Therefore, I wish to thank everyone who has ever been involved with LISA. The people that helped start it, every chair and co-chair, every program committee, every presenter, every attendee, and last but not least the USENIX board and staff.
LISA's radical ideas
Some movements start with a manifesto. LISA did not. However, if I were to construct one based on my (imperfect) memory of history, I would say it was:
- System administration is important
- Computers should be actively managed
- Automation is important
- System administration is a human process
- Open systems like TCP/IP and POSIX (Unix) are the future
Readers may be surprised that any of these were considered radical ideas in the late 1980s and early 1990s, so let me explain.
Radical idea #1: System administration is important
When LISA was starting in 1987, system administration was not highly regarded work. Producing software, not running it, was highly valued.
There was an assumption that computers mostly ran themselves. There was no "sysadmin community" and why should there be? “If we need support, shouldn't vendors provide it?” In those days most shops got all their technology from a single vendor, usually IBM, Digital, or Sun. This was often the result of an organization only owning one or two large computers. One vendor could provide all the support you needed. The inevitable evolution to multi-vendor environments ended that.
System administration was considered clerical work or just specialized customer support. A common misperception was that only failed developers became system administrators. You were the "system janitor". We were underpaid. Grossly so.
To illustrate how little respect system administration had, here are a few personal anecdotes I have not shared publicly until now:
When I was graduating from university in 1991, I proudly announced to the chair of my computer science department that I had accepted a job offer as a system administrator. “Oh Tom,” he said, “we had much higher hopes for you." I was deflated.
At that job a senior engineer spontaneously told me to have faith that my career would get better. He explained that he used to do system administration, "but look where I am today!" I hadn’t asked for advice. He simply blurted this out one day, unprompted.
When I first heard about the LISA conference in 1992, I asked a prominent USENIX and Usenet personality (one I considered a friend and mentor) if he attended. He literally turned up his nose and said, "Oh, I haven't done operaaaaaaations in years!"
His nearly allergic reaction lead me to ignoring the LISA conference for many years. I attended other USENIX conferences but it was a while before I decided to attend LISA and associate with people that did operaaaaaaations for a living.
Yes, dear reader, even I internalized the disrespect for system administration!
The lack of respect for system administration nearly prevented the LISA conference from starting at all. The founders had to leverage the USENIX board's negative experiences with system administrators to Tom Sawyer their way into getting approval!
While grossly oversimplified, the origins of LISA went something like this:
First attempt:
Sysadmins: We'd like to do a conference about system administration.
USENIX board: No. We're academics and there's nothing science-y or research-y or interesting about system administration.
Second attempt: (months later)
Sysadmins: Gosh, we thought about what you said last time. We agree that system administration isn’t interesting [not really, but we're buttering them up]. However sites with a large number of computers have interesting research-y challenges. How about we focus on Large Installation System Administration?
USENIX board: Now you have our ear... but we're still not convinced.
Sysadmins: Is it because you don't respect the sysadmins where you work?
USENIX board: We're not answering that in public.
Sysadmins: Well, maybe you'll get better service if they have conferences to attend.
USENIX board: Good point. We approve the creation of LISA.
The rest is history.
Fast forward to 2022. We won! System administration is important. System administration is a respected career option.
This change is not by accident. USENIX provided leadership that paved the way for this change:
- USENIX commissioned the SAGE Salary Survey, which for many years was the only resource for system administration salary information. It influenced policies at the corporate and government level. HR departments used it as a guide. It established nomenclature that is still in use today. The visibility and transparency that this survey created drove salaries upward.
- The LISA Code of Ethics, originally called the "SAGE Code of Ethics", established a basis for ethics and professionalism which was widely copied.
Radical idea #2: Computers should be actively managed
More specifically, computers and their associated operating systems, should be actively managed.
This idea has been so widely adopted that I feel like I’m explaining water to a fish. To understand, let's travel back to when LISA was starting.
At the time, "serious" computing was done on big mainframes or minicomputers that cost 6 or 7 digits. To amortize the cost, users would share such computers and access them via terminals. As a footnote to those of you fortunate enough not to have used a terminal: A terminal is a physical device with a screen and keyboard with a serial connection to a larger computer. You'd visit "the terminal room" at LISA which would have dozens of terminals. The "Terminal" program you run on your laptop is emulating one of those devices.
It would be years until a graphical, multi-tasking, multi-user, operating system like Unix or Linux would run on a computer one person could afford.
Such large computers were installed by the vendor and you didn't make OS-level system changes. Vendors expected their computers to be used as delivered. I remember when we added networking to my university's three big computers: the vendor arrived, installed the Ethernet cards (which cost thousands of dollars each), and configured the network addresses for us.
This made sense when an entire campus would share a few large computers. Back then the idea of having a personal Unix system like we have today was science fiction.
Two trends were about to wreak havoc with the paradigm of vendor-managed computers. First was the rise of Unix-based workstations. Soon organizations would have dozens or hundreds of such machines. Second was the rise of multi-vendor environments. A vendor-managed paradigm makes sense when all your computing needs are met by one company. A diverse computing infrastructure required a higher degree of local involvement.
The paradigm of vendor-managed systems would not scale. By design or by force, they had to forego that model.
Meanwhile, LISA attendees were dealing with new challenges that were best met by the customer actively managing their own machines.
We took this well beyond simply taking responsibility for powering up the machine and configuring its network settings.
Managing the configuration of hundreds of machines required automation. This evolved from simple post-install scripts that were run manually, to automated OS install systems, to configuration management (cfEngine, Puppet, Chef, and so on), to what we now call Infrastructure as Code (IaC). When vendors didn’t provide a way to automate OS installation, we reverse-engineered the process and did it anyway (example: AutoInstall for NT: Complete NT Installation Over the Network”).
Active management also meant that if a vendor wasn't fixing something, we'd replace it. If we didn't like the vendor's "grep", we'd add a better one in /usr/local/bin.
We would also out right replace broken OS components when vendors weren't moving fast enough. Some examples:
- In the early days of DNS, BIND was the de facto standard. Sadly it was riddled with security holes. It was common to rip out the vendor version and replace it rather than wait a year or more for the vendor to ship an update.
- I recall at least two open source projects for replacing Sun's shared libraries related to name service. This was very risky, but this was before pluggable name and authentication systems. It was our only choice!
- The original NFS automounter was unreliable. LISA was where I learned about open-source replacements that worked better. In those days I managed large clusters that heavily depended on NFS and automounter; replacing such a key element of the operating system seemed shocking!
It was at LISA that I learned about radical ideas such as having one username and password that worked on many machines and applications. Now we talk about LDAP/Kerberos, ActiveDirectory, SSO, SAML, and OAUTH as if they've been around forever. They weren't.
Fast forward to 2022. We won! Vendors now expect their customers to manage their own machines, every OS has package repos full of add-on and replacement software, and I can’t imagine an OS that doesn’t have some kind of automated, scriptable, installer. Vendors still don't like it when we flat out replace their broken binaries, but they acknowledge our need to go beyond their default offerings by offering plug-in mechanisms.
Radical idea #3: Automation is important
Well, duh. But this was not so obvious back then.
Before networks, each computer was an island and there was a lot of "system administration by walking around" (i.e. physically touching each machine to roll out a new software release or fix a configuration issue). Networks enabled us to "ssh in a loop" which was only slightly better (or worse if you liked the exercise). Actually, it was "rsh in a loop" until the late 1990s. Yes, for many decades remotely logging into another machine wasn't cryptographically authenticated nor private!
LISA was the first place I heard someone say that their goal was to automate themselves out of a job. This is not a request to be unemployed, but an acknowledgement that there is an infinite amount of IT work to be done in any organization and that a good system administrator can move to more interesting projects by eliminating their current assignment through automation.
When LISA was new, automation was typically done in sh, csh, sed and awk, technologies that did not scale to large projects or large numbers of hosts. Imagine writing a disk backup system entirely in awk? A co-worker of mine did this once. It even generated PostScript labels for the tapes. Not all heroes wear capes!
LISA helped popularize Perl: the first language for sysadmins. Tom Christiansen taught day-long classes about this revolutionary language that let you manipulate strings, talk to networks, and do system calls just like C! All that and no buffer overflows!
More importantly LISA had a huge impact on what we now call Infrastructure as Code (IaC).
Colocated with LISA for many years was the "configuration management workshop". These workshops birthed what we now call Infrastructure as Code (IaC). Attendees conducted famously heated debates about important issues of the day: should the language be imperative or declarative? Should systems modify a file or should they exclusively generate them from scratch? Stateful or stateless? How to distribute changes securely? These debates informed or inspired early configuration management systems such as cfEngine, BCFG2, and Puppet; which lead to Chef, Ansible, and others.
While I didn't attend most of these workshops, the few I could attend were memorable. One included a debate that got so heated that I was concerned a fistfight would break out! It was a classic "theory vs. practice" debate where one side was demanding that you could not get theoretically provable results if you mutated files; they argued that a CM system should never mutate a file, only generate it from scratch. The other side felt that this was impractical for real-world systems. Imagine regenerating /etc/passwd (from what source of truth?) just to add a single user!
The other CM workshop I remember featured a snarky young gentleman named Tom Limoncelli giving a presentation that scolded everyone for gloating about creating perfect systems that were too computer science-y to be adopted by typical sysadmins; he subsequently presented mock-ups of what a usable ecosystem might look like. I was proud to see my ideas eventually made it into real products.
Fast forward to 2022. We won! Modern system administrators are expected to be coders. Configuration Management evolved into Infrastructure as Code and begot the GitOps paradigm. Even sysadmins that claim "they don't code" use Ansible to automate processes.
Radical idea #4: System administration is a human process
Computers may be all ones and zeros, but the administration of them is a human process.
In the early days of LISA, the concept of teaching "soft skills" like communication, time management and meeting facilitation was a foreign concept in the industry.
Evi Nemeth drove this home by selling t-shirts that pointed out that on top of the seven-layer protocol stack were two more layers: financial and political. You can purchase them at https://evi-shirts.myspreadshop.com.
I can't take credit for many innovations in this industry; I've always considered my work akin to a journalist spreading the word of other people's good works. However, I was the first person to publish a "soft skills" paper at LISA (cite: "Upgrading Yourself from System Clerk to System Advocate", LISA '97) and the first to teach a purely soft skills class as part of the tutorial track. I was told there was much debate at the program committee regarding whether accepting a soft-skills paper might be the camel's nose at best, or set a bad precedent at worst.
LISA talks also advocated for better ways to manage our work. A clear example of this is the early advocacy for the use of ticket systems to track user requests. In the years since LISA started we've gone from "What's a ticket system?" to "We need a ticket system!" to "What do you mean you don't have a ticket system?" We've even seen presentations about the perils of over-using ticket systems, and new work-management systems such as Kanban.
Fast forward to 2022. We won! Soft skills talks and training are now commonplace at all technology conferences. One of the main principles of DevOps is "process over tools" because tools don't fix problems; people fix problems.
Radical idea #5: Open systems like TCP/IP and POSIX (Unix) are the future
Well, duh… again. However this was not so obvious at the time.
Most sysadmins today don’t even know that there are non-IP protocols. Digital was pushing DECNET, Novell had IPX, Apple was pushing AppleTalk, and the ISO was pushing the OSI protocol. None of these scaled, interoperated, or were as simple as TCP/IP. Likewise, the concept of Unix and Unix-like systems being able to share software with minimal porting effort was a threat to vendors whose sales strategy depended on lock-in.
Fast forward to 2022. We won! All those proprietary vendors are either no longer in business or ship Linux/Unix systems. More than 5 billion people use TCP/IP every day, whether they know it or not.
LISA: The good and the bad
LISA was like a family to me. Like a good family, we can talk about the good and bad aspects honestly.
On the good side, LISA was a conference where you could be you.
It was the first professional conference I attended that didn't have a dress code. Some people wore suits in the early days, but most dressed in t-shirts and jeans.
LISA was LGBT-friendly when other conferences most certainly were not. When LISA started it was legal to fire someone for being LGBT, and being out of the closet was a rarity. LISA was a welcoming community and people were more "out" with their LISA family than their biological family. The night time "birds of a feather" (BoF) sessions included one for LGBT attendees. That was unusual considering that most BoFs were focused on particular technologies or products.
The LGBT BoF session was always on the first night so "you'd recognize people's faces for the remainder of the conference". The agenda was simple: everyone would introduce themselves and describe their work environment. Over the years we watched people's answers evolve from identifying which employers did or didn't actively fire people for being LGBT, to which had health benefits for same-sex partners, to who was innovative in their benefits and diversity policies. The USENIX organization itself is always welcoming and friendly. Often the USENIX board president would make a point to stop by the LGBT BoF to say “hello” and make it clear that USENIX conferences welcomed and celebrated diversity. USENIX was ahead of the curve not just in technology.
Likewise there was a Woman in Technology BoF which was always very well attended and provided community, networking opportunities, and support.
Speaking of "you can be you", in the late 1990s and early 2000s LISA always had a large Goth contingent. Some years the so-called hallway track looked more like fans camping out for a Siouxsie and the Banshees concert. I'm not sure why LISA attracted Goths but I always felt it was a testament to the welcoming nature of the conference.
However, not everything about LISA was good.
We turned down some awesome papers. Specifically I regret that we rejected a paper about ZFS that, in hindsight, would have been the first mention of ZFS in public. What a scoop that would have been! The paper was submitted with the caveat that we had to keep it hush-hush until a certain date. This made the program committee uncomfortable. The official reason we rejected it was that it sounded too good to be true! We didn't want to accept a paper about a product that turned out to be vapor-ware. A file system that was powerful and could be managed entirely from the command line? Inconceivable! Boy were we wrong!
There was the failed plan where USENIX would split out all sysadmin-related activities (SAGE and the LISA conference itself) to a new and separate organization. I honestly don't know what happened and it isn't worth trying to reconstruct what did. Suffice to say that it divided the community, distracted the organization for many years, and left everyone with hurt feelings. It nearly killed the conference. One friend was so discouraged by the entire affair that they left the tech industry entirely. I felt sad and upset that this organization that I loved was in so much pain.
There was also the "LISA chair curse". I don't have any hard numbers here but it did seem like a statistically relevant number of conference chairs/co-chairs had hardships unrelated to the conference. This included car accidents, deaths in the family, unexpected job changes, and more. The curse skipped my year as co-chair (with the most excellent Doug Hughes) for which I am thankful. The lesson I learned from this was that LISA is the kind of community where we can get support about our non-LISA problems and the value of being open about such things.
I was dismayed and disappointed that over the years LISA attendees developed a bad attitude about new technology. When LISA was new it seemed as if we eagerly greeted all new technologies with excitement. Over time, however, it seemed to me that LISA attendees mocked and distrusted anything new. Many initially poo-pooed Linux, web apps, "the cloud", and DevOps.
At first I thought this negative reaction was similar to people's music tastes. Every generation loves the music from their youth and thinks the next generation's music is junk. LISA attendees loved what was cutting edge when they started but doubted anything that strayed too far from what had been “normal” to them.
Looking back, however, I see a different dynamic. It wasn't a resistance to new technology, it was resentment that something they had been doing for years was being repackaged or rebranded as something new. When you’re on the cutting edge of technology, you forget how rarified the air is. When your niche becomes popular, it’s like seeing your favorite indy band make it big.
For example, the push-back against cloud computing wasn't about the content, but the presentation. Complaints were typically: "This isn't new! For years all my systems were in a datacenter I've never physically visited!" Similarly DevOps didn't feel all that new if you'd been attending LISA for years. “Of course release engineering should be fully automated, didn't y'all know that already?” Seeing it packaged and promoted as something new felt disingenuous and off-putting.
We should enjoy that something we love is now packaged in a way that more people can enjoy it. Like the XKCD comic "Ten Thousand", we should be excited when someone is unaware of something we love because it is an opportunity to share that joy with them.
Fan or fanatic?
LISA was my obsession for many years.
I spent my evenings reading and re-reading LISA papers. Originally LISA was the kind of conference where people would write 5-15 page papers which would be judged by the program committee. Talks were summaries of these papers. Presentations without a paper were a rarity reserved for the "invited talks" track. All attendees would receive a book (later, optionally with a CD, and still later on a USB stick) with all the papers finely typeset. Sometimes it would take me months to read them all.
LISA helped my career. In my day job I would strive to pick projects that would be innovative and audacious enough that it might generate a LISA paper.
I served on a few conference program committees. The program committees decided on which papers were accepted or rejected. The committee met in person at exotic places like a tiny hotel near O'Hare Airport and Park City, Utah. I enjoyed these so much that for a few years if I wasn’t selected for the committee, I would ask to volunteer anyway. (So naive!)
LISA is one of the few tech conferences where people would take vacation time and attend on their own dime if their employer wasn't willing to send them. I’m proud to say that I was part of that crowd. LISA was full of many once-a-year friends and family. How could I not attend a family reunion? These are friendships that have lasted to this day.
In 2011 I took my obsession to a new level by co-chairing the conference. Special shout-out to my co-chair Doug Hughes for being the sane co-chair that counter-balanced my madness. I have a confession. The theme for our conference was DevOps but at the time DevOps was so new we didn't really understand what it meant. I did know, however, that it was going to be the next big thing. As a result, the theme applied more to the keynote than the rest of the conference. Oh well. At least we tried!
Was this an unhealthy obsession or just plain fun? I don't know for sure. However I do know that LISA was the one place I could have deep technical conversations that propelled my career, both technically and non-technically.
For that I am grateful.
Writing books
I owe my book-writing to LISA.
Being an author is rarely profitable. But, being an author boosted my credibility and led to many opportunities including jobs, speaking gigs, and opportunities to travel around the world to speak at conferences. For all that I am grateful.
My co-author Christine Hogan and I had a simple vision for The Practice of System and Network Administration (TPOSANA): capture all the conventional wisdom of LISA.
William Gibson famously noted "The future is already here, it's just not evenly distributed." We realized that what is "accepted truth" at LISA would be considered lightning in a bottle at most organizations. If we could write it all down, the book would be a winner.
Much of the book's outline was planned during LISA 2000. Christine and I didn't live near each other, but we would both be at LISA. It was a rare opportunity for in-person collaboration.
We used a stack of index cards to record every truism, every cool technique, every best practice, and every fun anecdote we wanted to appear in the book.
We spread out the cards on the floor and kept re-arranging them until we had the first draft of our table of contents.
Between then and September 2001 those cards became the book.
The book-signing event we did at LISA was one of my favorite memories. Seeing a line extend out of the room and down the hallway made me verklempt.
In 2005 we were honored to receive the LISA Outstanding Achievement Award for publishing the book. And thus it all came full circle.
When I returned to my university to give a presentation about TPOSANA, the department chair I mentioned earlier was in the audience. After the talk he approached me, shook my hand, and told me that I had redeemed myself.
Gratitude
I am indebted to LISA for my career and much of the good things that happened in my life as a result.
I thank LISA for giving my early career a boost. Early in my sysadmin career I would return from LISA with a list of new technologies to try. I thought I was just playing with the cool new toys. My employer saw me as an innovator worthy of promotions and raises.
I thank LISA for developing my public speaking skills. My first couple presentations were not exactly smooth but LISA gave me an opportunity to get better.
My first presentation was a disaster. Back then we printed our slides onto plastic transparencies and presented them via an overhead projector. One person would present and another would turn the slides. Laptops were rare, and video projectors were nearly unheard of, but I was lucky enough to work at Bell Labs which had excellent printing facilities. We could even print in color! A few minutes into my presentation the box of transparencies was knocked over by accident. The transparencies spread out over the floor like a card dealer doing a ribbon spread. My co-worker, Tom Reingold, crawled around the floor trying to reassemble the slide deck while I vamped. Fun times!
It was after one presentation (literally while I was still at the podium) when I was approached by an editor from Pearson/Addison Wesley. She liked my talk and recruited me to write a book. That’s how I became a book author.
I thank LISA for teaching me how to ask questions. It was while watching Q&A sessions at LISA that I realized how I could ask questions in a more constructive and giving way. I apologize to any presenter previous to my change.
I thank my LISA family. Over the years we got to know each other very well. In some ways, y'all knew me better than I knew myself. I didn't realize what a toll my divorce was taking on me, but you all saw it. I was holding in my feelings and didn’t know it. At the following LISA a number of friends commented, unprompted, how much better I seemed to be doing. “Why would people say that to me unless …? Oh!” That’s when all the tears I had been holding inside finally came out.
Special thanks
Thank you to my LISA family. Thank you to Adam, Adele, Aeleen, Alice, Alva, Amy, Andrew, Anne, Arnold, Ben, Bill, Branson, Brendan, Brent, Brett, Carol, Carolyn, Casey, Caskey, Cat, Celeste, Chris, Christine, Connie-Lynne, Cory, Curtis, Daniel, David, Deeann, Derek, Dinah, Doug, Duncan, Eliot, Elizabeth, Eric, Esther, Evi, Gerald, Greg, Gretchen, Hal, Helen, Henry, Jeff, Jeffrey, Jennifer, Joe, John, Jon, Josh, Karl, Kent, Kurt, Kyrre, Laura, Lee, Leslie, Luke, Lynda, Mandi, Marc, Marcus, Mario, Mark, Matt, Matthew, Melanie, Michael, Mike, Narayan, Natalie, Niall, Nicole, Pat, Patrick, Paul, Peg, Peter, Phil, Philip, Rik, Rémy, Rob, Robert, Rudi, Ruth, Simon, Skaar, Ski, Snoopy, Steve, Steven, Strata, Tim, Tom, Tony, Travis, Trent, Trey, William, Xev, and OMG I’m sure I’ve forgotten someone and I apologize to them in advance!
I also have to thank all the USENIX staff and board of directors, past and present. They are top notch at what they do and never get as much recognition they deserve.
The future
The end of LISA is an important milestone in our industry. We leave the world in a better place. Our radical ideas are now the conventional wisdom and we've left an excellent foundation for future generations to build on.
I'm not saying LISA did this all by itself or that some of these changes weren't the result of industry trends that would have happened anyway, but LISA was at the center of it all and it was an excellent ride.
If pre-LISA was the industry's infant years, LISA was the teen years. We were growing fast, exploring, and certainly awkward. We learned a lot, however, and have a lot of accomplishments to be proud of.
I look forward to DevOps/SRE being the adult phase... or maybe a second childhood.
See you at SREcon!
