An Evening With Berferd

Hey there -

welcome to my newsletter! I love learning from great whitepapers, so every month-ish I write about one.

I explain small details, get them out of the way so that the big ideas are approachable, and I highlight the human elements to remind us that these are real papers by real people. By focusing on details we get confidence in the bigger picture of what was going on when the paper was written.

If you like what you read here, consider my teaching and mentoring. I love helping people master the details, so the big picture clarifies itself.

What's This Berferd About?

On 7 January 1991 a cracker, believing he had discovered the famous sendmail DEBUG hole in our Internet gateway machine, attempted to obtain a copy of our password file. I sent him one.

An Evening With Berferd, Bill Cheswick, 1991

This first whitepaper is called An Evening with Berferd, written in 1991 by one of the inventors of the packet filtering firewall, Bill Cheswick. It's the story of Bill trapping a cracker who got past his recently invened Internet Gateway. Berferd, it is explained, is a name that Bill assigns to a cracker who is trying to break in.

There is richness in the details in this paper, showing us what the tech world was like in 1991. For example, the "famous sendmail DEBUG hole" is something familiar to Bill and his colleagues in 1991; it was a bug exploited by The Morris Worm, which was a cultural event in the late 80s. It was a self-reproducing piece of code that tried half a dozen debug features or code bugs to gain unauthorised access to systems.

The DEBUG hole was simple to exploit, because it wasn't a secret. A contemporary mailing list post shows this:

The programmer who designed the network's electronic mail program, instructions controlling the flow of electronic messages among thousands of computers around the country, deliberately left a secret 'back door' so that he close the 'door' he originally put in place to allow him to make adjustments to the program.

It remained open for several years, until Robert T. Morris Jr., a graduate computer student at Cornell University, discovered it and used it to let loose the 'virus' program that ultimately paralyzed more than 6,000 computers last Wednesday and Thursday.

John Markoff, Computer Invasion: 'Back Door' Ajar, 1988

I reproduced the sendmail debug hole in a containerised environment; you can too using Ye Olde BSD, a project I'm currently reimplementing.

 Putting /etc/passwd in an FTP directory?

"Sometimes system administrators put their real password file in the FTP directory."

- Bill Cheswick, An Evening with Berferd, 1991

Bill was a systems administrator just as the CSNet and the ARPANET were being combined and talked about as this new thing, the "internet". He worked at Bell Labs, famous among other things for Unix, as a researcher who wanted to know what it meant that he was connecting systems to a large network.

In the paper, Bill tells us he wrote a listener script that sent breakins to a log file. He noticed Berferd coming in through FTP, and then tells us that sys admins put their real password file into their FTP directory. What on earth was that about?

How FTPd Was Set Up in 1990

A contemporary linux journal article from 1995 about how to set up FTPd sheds light; it's a consequence of how FTPd was configured using a lightweight file-system based isolation tool called chroot, which is the same tool Cheswick uses to build file system level isolation from his primary OS in his honeypot. Let me give you a quick chroot primer.

Quick Chroot Primer

Legend has it, Bill Joy invented chroot to make compiling new OS patches more reliable. chroot changes the root directory of your filesystem for a process launched within chroot. chroot only changes your file system root, so you have a new file system tree. In this example, you see my shell is in my home directory, but when the command chroot is executed against the directory chroot_dir then the shell thinks that /home/gwyn/chroot_dir is actually the root directory:

    % pwd 
/home/gwyn 
% chroot chroot_dir 
..some output redacted...
$ pwd /
    
  

I removed a lot of OS messaging above for clarity; you see, inside this new file system structure, there's no shell, no cli tools, none of that unless you copy them in somehow. You solve this in containers by using a base image and a Dockerfile. With chroot, I wrote a shell script to copy dependencies into the chroot. Here's the example you need to see:

Outside the chroot, my OS knows how to look up my UID and GID:

$ ls -l chroot_dir 
total 10 
drwxr-xr-x 2 gwyn gwyn 4 May 4 03:51 bin 
drwxr-xr-x 2 gwyn gwyn 7 May 4 03:51 lib 
drwxr-xr-x 2 gwyn gwyn 3 May 4 03:51 libexec

Inside the chroot, I have to copy in the files that contain that info, otherwise UID and GID lookups fail:

$ chroot chroot_dir 
$ ls -l 
total 10 
drwxr-xr-x 2 1001 1001 4 May 4 09:51 bin 
drwxr-xr-x 2 1001 1001 7 May 4 09:51 lib 
drwxr-xr-x 2 1001 1001 3 May 4 09:51 libexec

Back to FTP

Because unix users in the early 90s were used to UID and GID lookups succeeding, and because FTPd was run using chroot, unix systems administrators would copy the UID and GID lookup files into the chroot. Those files were /etc/group and /etc/passwd, and this is why systems administrators were copying their /etc/passwd file into their FTP directories : it was to work around the limitations of chroot.

Now, every article I have found about setting up FTP mentions clearly "don't leave your encrypted password strings in the passwd file", but not many of them notice something more unpleasant: it's convenient to have your UID and GID map to something you know, but if someone else breaks in to your FTP server then you're giving them the usernames of people on your network.

The internet was small and trusting, composed of academics and researchers, not a place where commercial ventures were taking root yet. There was a big wakeup call in the late 1980s when The Morris Worm blasted through thousands of systems, but by 1991 no major second or third worm was released.

Cheswick is conscientious, actively building a logging system to see what more invisible threats might be happening. For the rest of the nascent network, though, the researchers and teachers get hit by a human behavioural problem, namely that research shows that humans tend to downplay ambiguous threats, we don't focus on problems that might or might not recur.

The remote login tools that were in use in 1990 were incredibly insecure (Kirk McKusick claimed that Bill Joy wrote them in an afternoon to debug something). Knowing someone's username was a big lift for getting around an internal network. It's easy today to see enabling that kind of security leak as naive; but remember, the world back then was not the world of today. For example, one of the smtpd holes that the attacker used against Bill Cheswick was sending an email with a debug string in the subject line, which dumped the sender into a root prompt. The debug string was intentionally there, but nobody thought of networks as a hostile environment.

If you're interested in the smtpd bug from the 1990s, I reproduced it Ye Olde BSD. Not confident with containers? Book a mentoring session.

Diligence, Habit, and Understanding

Focusing on small details, a debug entry in sendmail, or the idea that people put their passwords into their download folders to make usernames appear in a system lookup, the overall paper is easier to read. These details show us that not understanding consequences, not being thoughtful about how the details hold together, is a pernicious problem that's been with us for decades.

We're in a new tooling revolution, large language models are powerful and may eventually be better than humans at writing code and doing a good, diligent job. That day is not today, not this year, probably not in the next couple of years. Right now, the problem is that they're just smart enough to give us footguns but not smart enough to tell us. The solution today is for us to understand our fundamentals, to have an idea when a system design smell or a code smell is happening, so we can course-correct the tool.

The focus in my career has been fundamentals, details, diligence, and habit. We want systems reliability to be the outcome of something outside our control, but truthfully reliability is the result of habits, good habits that are informed by a clear understanding of the principles of our tools. If you find yourself not confident that you can build reliable systems, book a mentoring session and we'll work on that together.