Pythonistas, Beware the Ghost Bus

If you misuse mutation in your code, you’re gonna have a bad time

Ski Teacher in South Park

Here at Fenris, we have a lean development team. We move quickly, ensuring we can deliver features as fast as possible for customers, without breaking things. Based on my experience at larger companies, we do this far faster than they are able to. Achieving this balance between speed and reliability requires a fanatical devotion to automated testing. For us, we achieve this using Test Driven Development (TDD). TDD means that every change made to the code triggers a large number of tests across ALL OF THE CODE. If they don’t all pass, the code will not be considered for review, let alone put into our User Acceptance Testing environment, where it becomes eligible for production.

Recently, an extremely talented young engineer on our team encountered an odd issue. She had made a small change to piece of code that wasn’t touched by anything else in the project. She also wrote a small test that tested this new piece of code. Suddenly, tests unrelated to this snippet of code started breaking. As we discovered, she’d been hit by the Ghost Bus. And I had unwittingly set it into motion. Before I talk about the specific issue she encountered, let’s back up (pun totally intended) and talk about the Ghost Bus.

The term “Ghost Bus” originated for me at a previous software company, where many of the developers had read the amazing book Fluent Python by the prolific Brazilian engineer Luciano Ramalho. There was a section of the book that became a canonical reference, mentioned when engineers would see nasty bugs with a certain root cause. In Chapter 8 of his book, Luciano depicts a specific kind of bug, and demonstrated it as a “Haunted Bus”. As it was repeated and retold amongst my large development team, it gradually evolved into “ghost bus”. Originally, it referred to a very specific kind of bug, but gradually grew to encompass bugs created by snippets of code that unintentionally change things used by other snippets of code.

NOTE: Luciano's second edition of Fluent Python is due in September, 2021, but an early release edition can be accessed via O'Reilly's subscription website.  I'm already reading it and loving the new additions and updates.

Let’s start by showing an example of Luciano’s original HauntedBus:

class HauntedBus:
    """A bus model haunted by ghost passengers"""

    def __init__(self, passengers=[]):  # <1>
        self.passengers = passengers  # <2>

    def pick(self, name):
        self.passengers.append(name)  # <3>

    def drop(self, name):
        self.passengers.remove(name)

So far, so good, right? Let’s put this code into a Jupyter Notebook and see what happens:

Now let’s get the A-Team involved, because the fools of the world don’t get enough pity, and Mr. T is here to help. And while we’re at it, for every A-Team there must be a B-Team, because life.fair = False.

At this point, you may be wondering what just happened? Didn’t that YouTube video you watched on Object Oriented Programming say that objects encapsulate state? How is this happening?

Notice that both the A-Team and B-Team vans were initialized with no passengers. This means that when they were created, the self.passengers class attribute was set equal to the default value specified in the __init__ method, which is just an empty list, right? The code snippet says passengers = [], so what’s the big deal here? Why is Python behaving like this? Let’s dig under the covers a bit, and the explanation we’ll find, I should note, is not unique to Python. Java, as well as many other OO languages, is subject to the same behavior. Function parameters in Java that are reference types are passed by reference, primitive types by value. In Python, all function parameters are passed by reference (or as Luciano says, “call by sharing”) as well. Keep this in mind.

Call by sharing means that each formal parameter of the function gets a copy of each reference in the arguments. In other words, the parameters inside the function become aliases of the actual arguments.

Ramalho, Luciano. Fluent Python. O’Reilly Media.

For reasons of efficiency, when a piece of Python code is evaluated, the default parameters for a given function are evaluated and attached to that function. If this function happens to be a method in a class, those values are attached to the method at the class level, not its instances.

So far so good. Each instance’s methods have the same default values, but they aren’t actually references to the same location in memory, right? Right?

Wrong.

When the snippet of code in the HauntedBus class is evaluated, this:

def __init__(self, passengers=[]):

evaluates like this:

HauntedBus.__init__.__defaults__[0] = []

The passengers default parameter is assigned to an empty list that resides in memory, attached to the HauntedBus class. Any and all instances of this class that initialize with the default parameter, rather than a list passed into the constructor, end up sharing this reference to the empty list.

Ok, so if you are using immutable default parameters, this isn’t a big deal. An immutable default parameter could be shared by hundreds of functions, but since none of them can modify it, no issue can arise. But here, we aren’t. Any given default argument for a method will reference the same exact location in memory, ACROSS ALL INSTANCES OF SAID CLASS. Meaning that any instance initialized with a default MUTABLE value has the ability to change that value for all other instances initialized with the same default value. A mutable value as a default argument to a function is, like it or not, a kind of accidental global variable. See below:

I can do the whole “id” trick in Python like above to show they point to the same value, but let’s do something more colorful instead:

Ok, by now I think you get the picture. Default arguments in Python are evaluated once, when a function or class is first evaluated. If you use mutable values for default params, you have a mutable variable that is also shared across instances and function invocations. Let’s hammer this point home with our now fixed HauntedBus class:

A Good Rule of Thumb

Mutable values are often useful, and required. But they can be dangerous. Avoid using them as default parameters, always. If you CAN use immutable types, do it. Mutable types can be dangerous in other ways that are a little more obvious than the specific issue above, but this “obvious” nature can be far less obvious as a code base grows. Which brings us back to the ghost bus I left parked at the top of a metaphorical hill, waiting to roll onto one of my colleagues.

I did a bad thing

We use pytest for our automated tests for the specific project where I adeptly parked the ghost bus. A key part of pytest are reusable items, shared across tests, that make writing tests more efficient. These items are called fixtures.

Below is a fixture which returns a pandas DataFrame object:

The problem?

I had written a test which would remove certain columns from the dataframe to simulate bad data. As long as that test happened to be the last to run, there was no issue. However, when my unsuspecting coworker followed my lead, and did similar things in another test, it caused numerous other tests using the same fixture to suddenly find themselves consuming a dataframe that was altered by another test.

Why? The dataframe being returned was a reference, shared by all of the tests consuming it. We could have changed the scope of the fixture to read in a fresh dataframe once for each test, but that would have been wasteful. We only want to read in the dataframe once per run of all tests, to keep our automated tests as snappy as possible.

The solution was simple:

Here, instead of returning a shared reference to a single dataframe, we return a function which itself returns a deep copy of the dataframe. (NOTE: Pandas Dataframe’s copy method defaults to returning deep copies. Beware of python’s normal copy.copy function, which performs shallow copies. If you are concerned about accidental shared state, default to using Python’s copy.deepcopy functionality.)

By subtly changing the fixture to ensure it returns a function, which itself returns a copy of the original dataframe, we ensure that team members can write tests the way they want to, without having to think about potential side effects. This is a lesson that applies to all code, not just tests. When you minimize shared state and mutation, the very expensive humans tasked with writing the code can focus more attention on the piece of code in front of them, without having to burden themselves with all of the code that isn’t in front of them.

Software development is a constant exercise in battling the complexities and chaos inevitably produced by a system interfacing with reality. Don’t add needlessly to the externally provided chaos by introducing ghost buses into your code. While I can’t give my colleague her wasted time back, I’m hoping someone reading this can be prevented from adding yet another ghost bus in THEIR project.

Author JP Kabler | find me here