Musings, July 2022

We need better programmers

July 18, 2022

Musings

Authors:

Article shepherded by:

Rik Farrow

I just returned from OSDI/ATC 2022, the collocated systems conferences, and still find myself disturbed by something one of the keynote speakers had to say. During questions after his talk about improving the security of Open Source, Eric Brewer was asked by a professor if he thought that graduate students should curate their software. Brewer answered that they should focus on finishing their graduate studies instead. I disagree.

Before I get into a riff about programming, here’s the lineup of articles since I last wrote a column.

David Lion, Adrian Chiu, Michael Stumm, and Ding Yuan wrote an article based on a paper presented at ATC’22 where the authors researched the performance differences between Python, JavaScript, Java, Go, and C/C++. As Yuan writes, people will argue passionately about the performance of their preferred language, but what the authors did was to craft benchmarks designed to run fast in each of these five languages so they could definitively compare differences in performance. They also instrumented Python and JS’ interpreting loops so they could better understand why these managed programming languages are so different in performance, besides JS using JIT.

I spent a lot of time learning about CHERI, a new system designed to end buffer overflows, end pointer misuse and use-after-free buggs, as well as supporting the compartmentalization of code. The researchers spent years designing and proving that their new system works securely and efficiently. At OSDI’21, Mothy Roscoe argued that researchers should move beyond the systems we currently use, systems based on designs from the 1970s, and the CHERI researchers have done this. The work is supported by Microsoft, Google, and Arm, and Arm has implemented CHERI in hardware and currently offers a development system, named Morello. I explain where this research begins, how it works, its limitations and potential future applications.

Lexiang Huang et al wrote about their research into a new form of system failure. Metastable failure occurs when a distributed system enters a vulnerable state where a period of extra load results in the system entering a state of reduced performance that is difficult to recover from. That’s quite a mouthful, but the researchers both uncovered numerous examples in published incident reports as well as building their own system that they could induce into entering a metastable failure.

Niall Murphy mused about how SRE could be different than it is today. He proposes that SLOs are not simply the right way to measure performance, and that using them can actually harm the way that SRE is done today.

Danny Huang has been working on a system for monitoring the network connections made by home devices. Smart devices, such as doorbells, televisions, cameras, speakers, and thermostats, may all be busy reporting your activities to companies who want to monetize your information, but uncovering what they are doing can be difficult. Huang built IoT Inspector to uncover this network traffic, collect anonymized reports, and report the traffic to the users of his tool. Huang explains the difficulties he encountered in getting correctly labeled information about smart devices and their network connections, as well as explaining how you can gain access to the data he has collected as well as running an IoT Inspector for yourself.

Programmers

Strangely enough, what got me started on this thread was code.org. A non-profit organization that wants to bring “CS into the classroom”, does seem like a good idea. I have friends who I introduced to computing through the game of Adventure that went on to have good careers in IT.

Another site is much more blunt. To answer the question about why teach kids to code, they said:

According to the Bureau of Labor Statistics, computer science occupations are growing, and programming knowledge can equip kids with marketable skills for the field later on. Even without looking ahead 18 years, kids should learn to code for these fundamental reasons.

Sounds good, but I think of this more like teaching some basic gymnastics skills: most kids will manage to climb ropes or do shoulder rolls onto a pad, but few will manage to do backflips. The number who go on to become professional acrobats, that is, compete in national or international competitions, is miniscule. When kids learn to program, that may indeed turn into a skill they will use in later life, but few will become good programmers.

I wondered if my experience with programmers working in IT was terribly skewed in the direction of seeing poor coding. So I asked the people I like to talk to, professors, about just how many of their CS graduate students were really good coders. Guess what? Apparently, CS graduate students churn out pretty terrible code too.

I like to think that I am an above average programmer, but that may be the Dunning-Kruger effect in action. I did get to work with people who were much better programmers than I, and that has affected my own opinion of my programming skills. Those folks also affected how I thought about other programmers in general, until I started going off into the commercial world teaching classes to people in IT.

There I saw terrible examples of code, configuration, and a general inability to solve what seemed to me to be simple coding problems. I twice wrote simple shell scripts that solved problems that an IT group had spent weeks on, spending about 15 minutes each time. Was all the world like this?

One of the professors I asked, someone who had switched into academia after working in industry, answered this way.

What I remember from the software industry was that our best coders were not only better than our weak coders by a factor of 10, but that the worst coders were in fact a terrible liability, and might end up costing (on an ongoing basis) the time of 2-3 other coders just to fix their mistakes.

Another related a story of being asked by a CEO who had 200 programmers working for him, how to improve code quality. The professor’s answer was to fire the 200 programmers and replace them with two excellent programmers who you pay really well.

You can find examples of the failures of programmers without much searching online. One person’s blog post that went viral describes how the programmers he hired hit a plateau. During their job interviews, they appeared capable of solving problems while sitting in front of a computer. But that was because they were copying solutions they found online. Their speciality was search, not programming.

Another blogger complained of incredible code bloat: 237 megabytes of executables to copy some files from client to server.

I realize now that the problems I quickly solved for IT folks were because I could think algorithmically. I could convert a statement of the problem into an algorithm that would solve that problem. And it seems likely that most people are not very good at creating good algorithms.

Open Source Programmers

What about open source? Aren’t there millions of folks working together to create the software that we all use? I took a look at a recent study by the Linux Foundation and Harvard into who produces the most popularly used open source software. The Open Source Software – Application Libraries report is 162 pages long, and I did not read the entire report. I focused on some summaries, one of which I want to share with you now. This is from page 18, point 3, in the Lessons Learned section:

Reviewing 49 of the top 50 non-npm projects from our lists, for commits in the year 2021, it was found that 23% of projects had one developer accounting for more than 80% of the lines of code (LOC) added. Further, 94% of projects had fewer than ten developers accounting for more than 90% of the LOC added. These findings are counter to the typically held belief that thousands or millions of developers are responsible for developing and maintaining FOSS projects. At a higher level, it was found that 136 developers were responsible for more than 80% of the LOC added to these 50 FOSS projects.

So much for Eric Raymond’s “millions of eyes” that would uncover and repair bugs in code. The reality is much starker: a relatively small number of developers are responsible for writing and maintaining the code that millions of developers are using. Note that the one project not included in this excerpt is missing because it is an outlier, but not in a good way. That project hides the identities of committers.

Eric Brewer’s keynote was about the trustworthiness of the supply chain for open source software. I had, mistakenly, thought that most internally developed software used by large corporations would have been developed in house. Instead, the norm has become using open source as the basis for internal projects, while not contributing to those projects. Someone at the conference, perhaps Brewer, showed a letter sent by lawyers to the maintainer of an open source project demanding that he fix a problem they had found “at once” or face legal consequences. Imagine, taking someone’s software, using it for free, then demanding support. Immediately, yet, when very large software companies often take over two months to produce a patch. Perhaps that company, and others like them, should consider paying open source developers, at least those who are key committers, instead of paying lawyers for demand letters.

Brewer talked about efforts at Google to improve the software supply chain. Some of that includes actually assigning programmers to curate open source software. Brewer also showed us deps.dev, a website backed by a Big Table database of open source software and each project’s list of dependencies, and the dependencies of the dependencies, and so on. Quite fascinating, although I learned from a senior programmer who I told about this that his company has been using a commercial service to do the same thing for a while now for JavaScript libraries.

Where I parted ways with Brewer had to do with his suggestion that graduate students not be expected to curate the code they produce. Granted, many research projects resulting in published papers rely on code that will never be reused. But certainly not all. At OSDI’21, 84% of the accepted papers included artifacts. At OSDI’22 and ATC’22, more people were involved in examining artifacts, including both code and data, than were on the program committees. This seems to suggest that there has been an increase of attention on code quality, at least to the point of being able to reproduce the experimental results reported in a paper submission.

I’ve talked with lots of people who have switched to working at the big tech companies after already having worked long enough to be considered senior. These folks spoke of spending well over six months learning how things were done in their new environment, from using accepted tools, working only from internal libraries, as well as following specific programming practices. One of those practices is code review, a key part of what Brewer had called curating.

To me, suggesting that graduate students not be involved in curating their own code is ignoring the tens of thousands of graduates who will not go to work at Alphabet, Meta, AWS, and so on, where there are very stringent rules for programming and the process for accepting code that will be used in production. For those programmers who will not be working in large tech corporations, how will they learn this discipline if it has been ignored while they were at university?

And perhaps this doesn’t matter. My experience, and the experience of everyone I have asked about the general quality of programmers suggest that excellent programmers are very rare. This really shouldn’t be a surprise. While it is common for school children to join a sports team, a tiny minority of them will continue playing sports as adults, and even fewer will become pros.

Why do we expect anything different of our programmers? Perhaps the uptick in use of Copilot means that things will get better, if programmers rely more on AI to do their programming for them.

Article Categories:

Programming

Last updated February 8, 2023

Authors:

Rik Farrow has been a consultant for 40 years. He has written two books, as well as worked as the technical editor for a UNIX magazine and for two editions of a popular operating system book. He also taught UNIX system administration and Internet security during the 90s internationally, and worked as a volunteer for USENIX program and steering committees. Rik has been the editor of ;login: since 2005.

rik@rikfarrow.com