20 technology horror stories about learning the hard way

Halloween will be here before you know it! This fun, over-the-top holiday is a great time to ponder the mortal fears of the developer in each of us. What haunts you the most, in the quiet moments just before your code starts to run?

Getting into the spirit of Halloween, I asked some Opensource.com writers: What’s the scariest code you’ve seen or written?

Bad permissions

I was responsible for a server, and I FTP’d something up. There were some funky things displaying, so I thought some permissions needed to be changed.

Needless to say, I foolishly turned off read mode and took down the site. (A website is not much good when nobody can access it.)

It took me hours to fix. This was at an agency years ago when I was the sole web developer.

—Miriam Goldman

Shambling HTML

I took down a client’s website, who was an author on the Wall Street Journal bestseller list at the time, because the original WordPress default theme had an update available.

His developer had hardcoded HTML into the theme instead of creating a child theme. I ran the update.

This was in the days when folks didn’t have nightly backups easily, so I spent hours on the phone with the hosting provider. Things like staging, child themes, nightly backups, or manual backups, are all now normal things, as well as the ability to auto-update and manually roll back. Not so in that era.

—Courtney Robertson

Not-so-secret key

I think many of us have seen a secret key in public code before. Or another favorite: A friend of mine sending emails to 100,000 users from the dev server.

—John E. Picozzi

Unix mix-up

This is a Unix story. It’s fixed in Linux today.

A day before I was going to give an important demo of a new component to management, I had to refresh my code (this was way before Git existed.) I went to my home directory, found the project directory, and deleted everything. Unfortunately, in that flavor of Unix, this command followed symbolic links, and I had a link to the latest version of the code (not all was on the source code system as it was still in the testing phase).

A day later, there was a network problem in the building, so the demo was delayed for a day, and we managed to recover. It was more than three decades ago. Even now I have no clue whether the network problem was a coincidence or an attempt of our sysadmin to save us (if so, it worked!)

—Josh Salomon

Imperative

Seeing !important; all over a CSS file instead of proper use of specificity.

I once had to override and customize almost all of a WordPress theme’s CSS because the owner of the site wouldn’t budge on getting a new theme that was closer to the design he wanted.

That same theme was last updated by the developer in 2018, and the website is still using it.

—Christi Nickerson

Misquoted

In a previous role, my predecessor misquoted the lyrics to Journey’s “Any Way You Want It” in a code comment.

—Ben Cotton

The ghost of Algol68

Algol68’s complexity, back in the late 1960s and early 1970s, frightened away many influential people, including Niklaus Wirth. The most common complaint I can recall back then was along the lines of “who could write a compiler for such a complicated beast?” And yet many people did. Moreover, many of the concepts developed or at least formalized as Algol68 appeared in later languages, notably in C and the Bourne shell (thanks to Steve Bourne).

Some of Algol68’s concepts have not aged well. The concept of I/O dealing with “books” and “chapters,” and so on, is a bit weird today. Leaving things like character sets to the implementation seems pretty old-fashioned.

But some are, or should be, tremendously relevant today, such as expressions that yield a value, strong typing (types in Algol68 are called “modes”), heap memory and garbage collection, definition and overloading of operators, and more.

Sticking with the Hallowe’en theme, both tricks and treats.

Algol68 is a language that merits study, if for no other reason than to see where so many of modern computing’s ideas came from, and to see how many have been lost along the way.

—Chris Hermansen

Passwords exposed

I was doing a tech audit for an incoming support client, and the previous developer put passwords in plain text throughout the full theme, and used horrible ways to connect to a remote database. Their composer file was also ghoulishly bloated. It took five minutes every time I tried to get the site up and running locally. Outdated dependencies, repos I could not access, the list goes on.

—Miriam Goldman

The maze

The scariest code I ever saw was a piece of PDP-11 assembly language in the kernel of an operating system named RSTS, which nobody remembers today. Source code was on microfiche in those days, and I had followed this code path through a few twists and turns, trying to figure out what was going on. And then, I ran into this instruction:

MOV R5,PC

I threw up my hands and wailed. I really did wail. People in the office thought I’d hit my head, or had a heart attack.

In those days, memory was precious and a MOV instruction used a teeny tiny bit less memory than a BR (for “branch”) instruction. Copying the contents of register 5 into the program counter was really a cheap unconditional branch to the address stored in register 5. Except, I had no clue what was stored in register 5, or how to find it.

To this day, almost 40 years later, I wonder who would write code like that and how anyone could debug it.

—Greg Scott

Off by one

I work in the automation industry, where the PLCs are programmed in some pretty weird languages.

An example that haunts me is the fact that in the language ST, you can define arrays to begin at index 1. It means that the first element is at position 1, not 0. It drives me nuts when I see it.

—Stephan Avenwedde

Divergence

I took a MongoDB instance down for 40 minutes once during a stage-to-prod launch. Our staging environment had diverged from production. It was just a database configuration difference—not very exciting. But it’s a good lesson to make sure your staging and prod environments are in sync!

—Em Nouveau

Unearthly whispers

This is from a project that’s still alive and kicking, but I’ve changed the code to hide the source.

for (int c = 0; y  yyy && c  ccc; y++, c++) {

 // some code here
}

It seems like an innocent loop at first. But maybe you’re asking why there are two variables with two stop conditions and two increments. And then you realize there’s only one initializer and the second variable (y) is initialized before this loop in a different code block.

When I realized this, it took me about an hour to understand why the code was written in this way, and how it’s supposed to work. Obviously, there were no c comments and the variable names are meaningless (c is called c in the code and y has a bit more meaningful name, but not meaningful enough to explain to me its meaning, not even today when I understand what it does).

—Josh Salomon

Critical data

Around 1980, I got my first job after college. I was the Assistant Computing Center Director at an engineering college in Indiana. That’s a fancy title for the second-in-command of a two-person IT shop. I handled administrative computing on a PDP-11/40, with RK05 removable “pizza platter” disk drives (2.5 MB each.) Each admin office had one drive, and part of my job was to back them up, disk to disk, every week. But I got busy over that summer and skipped the Registrar’s Office four weeks in a row. And then I realized the risk, so I made sure to start my monthly disk-to-tape backup.

I dismounted the Registrar’s pizza platter from the 11/40 and mounted it on the 11/70, which had a 9-track tape drive, and started my backup. A few minutes later, I heard a scraping noise inside that disk drive. Yep, the heads crashed. In a few short minutes, I’d destroyed all the Registrar’s data, and the then-most-recent backup, which was a four-week-old 9 track tape.

It was a, well, uncomfortable moment when I had to look the Registrar department head in the eye and tell him I had destroyed all his data.

Today, I tell new IT people you’re not a pro until you’ve destroyed somebody’s critical data, and there’s no way to recover it. Remember that feeling in the pit of your stomach forever.

—Greg Scott

Angry mob

A client hacked WordPress core to add features that later came out in a routine update and couldn’t understand why the site kept crashing every time they attempted to update LearnDash. (They also didn’t like our report that called out their poor development practices.) They basically showed us the door calling us liars and incompetents. To this day, I still have delegate access to their domains and wp-admin access to production and development of two domains.

They also, despite us sharing a link to an encrypted location for sharing access credentials, sent our logins over emails.

—Laura Byrne

Don’t forget to backup

I’ve not worked much on corporate networks, so I haven’t downed any servers. However, as I young person, I tried to help a person with an IT problem and somehow caused Windows 95 to crash, and had to reinstall it for free.

Another of my saddest moments as a very young Amiga user was when my save disk, containing all my files, broke due to some mechanical failure. Nowadays, I’ve gotten better at backing up more of my important personal files.

—Rikard Grossman-Nielsen

Root of all evil

I was new to Linux, and I’d just come from DOS where I used Norton Commander. Then Midnight Commander got released and I was very happy about it. It wasn’t packaged for the Linux distro I used at the time (Jurix), so I compiled it myself from source, just like other software I used at that time. It worked perfectly well, and suddenly I felt more at home on Linux.

That’s not the horror story.

My colleagues told me not to run Midnight Commander as root, regardless of how comforting it was. But root was easy, and it felt more like DOS, so I ignored their advice. Long story short: I accidentally removed the content of the entire /etc directory. Until that time, I’d never had to use backups, but that day I learned that backups are actually useful.

27 years later, I still remember this story, and I do regular backups.

—Peter Czanik

Illusion

The worst project one agency had me “make” was a one-pager that seemed straightforward at first. I said I’d be able to hash it together with some HTML and CSS, maybe a little Javascript. But they specifically asked me not to do that. They wanted me to cut out the design and literally use CSS to position those pieces around the page. They also had me add all CSS inline, directly into the HTML file, because they literally wanted one page.

None of the text was real text.

There were no real HTML elements aside from the ones needed to position those images.

I told them that the design was simple enough that I could throw it together with actual code, but they didn’t want that. They only wanted me to spend the time to cobble the pieces together and then move on to a different project. They had me make two little one-page sites like that.

It hurt my front-end soul. It was physically painful for me to do that project. It was a temp-to-perm gig, and when they offered me full-time, I politely declined.

—Rachel Vasquez

Corruption

The scariest things to me are memory corruptions that can occur in ANSI C99. During a screencast, I captured this (not quite) paranormal occurrence in this YouTube clip.

Image by:

(Joël Krähemann, CC BY-SA 4.0)

The GtkEntry labeled file shows some random glyphs. I’ve double checked the code, but didn’t find any issues.

The ags_export_soundcard_open_response_callback() function is a callback to the “response” event of GtkFileChooserDialog. (For the record, the tool to target this problem is valgrind.)

Image by:

(Joël Krähemann, CC BY-SA 4.0)

—Joël Krähemann

Python fears

The most horrific programming feature I ever saw is the access Python gives to its dict. Changing the type of an object at runtime is against my programming code of conduct.

—Josh Salomon

Franken-net

In 2006, I built firewalls based on Fedora and a bunch of scripting, and persuaded a customer with a large website inside a colo center to replace a proprietary firewall with one of mine. I built it and showed up to install it at 4AM one morning. That was when I learned (the hard way) that he had a load balancer behind his firewall, but with a public IP address. The customer endured a 5-minute outage, but I reconnected everything to the original, and it all came back online.

I found a way to handle his franken-net configuration by using proxy ARP. The idea was whenever anyone from the outside world did an ARP request for the load balancer, I would answer. A few days later, I showed up at 4AM again and installed my system. This time, I knocked everything in the entire colo center offline. I had set up my proxy ARP to respond to everything, and so all traffic on the LAN eventually found me and disappeared into a black hole.

Once I realized what I’d done, I put it all back the way it was. But the damage was done. If you tried to browse your favorite website around 4AM US Central time one morning in 2006 and it didn’t respond, it might have been my fault. I knocked an entire colo site offline by installing one system in a rack and turning it on.

The website operator screamed and I slunk out the door. They never invited me back to try again. That was a shame, because bridging probably would have worked.

—Greg Scott

Your horror story

What’s your favorite technology-related horror story? Tell us in the comments (but be nice, and change project names to protect the innocent!)