Stratus3D

A blog on software engineering by Trevor Brown

The Let It Crash Philosophy Outside Erlang

Abstract

Erlang is known for its use in systems that are fault tolerant and reliable. One of the ideas at the core of the Erlang runtime system’s design is the Let It Crash error handling philosophy. The Let It Crash philosophy is an approach to error handling that seeks preserve the integrity and reliability of a system by intentionally allowing certain faults to go unhandled. In this talk I will cover the principles of Let It Crash and then talk about how I think the principles can be applied to other software systems. I will conclude the talk by presenting a couple real world scripts written in various languages and then showing how they can be improved using the Let It Crash principles.

What is Let It Crash?

Let it crash is a fault tolerant design pattern. Like every design pattern, there are some variations on what it actually means. I looked around online and didn’t find a simple and authoritative definition of the term. The closest thing I found was a quote from Joe Armstrong.

only program the happy case, what the specification says the task is supposed to do

That’s a good, terse, description of what Let It Crash is. It’s about coding for the happy path and acknowledging the fact there are faults that will occur that we aren’t going to be able to handle properly because we don’t know how the program ought to handle them.

So how does approach improve our programs? And how do we apply this in practice?

Let It Crash principles

While I didn’t find an authoritative definition of what the Let It Crash philosophy was I have several years of experience applying it while developing software in Erlang. The core tenet of Let It Crash is code for the happy path. There are two other guiding principles that I follow as well, so here are my three principles of Let It Crash.

  • Code for the happy path

    Focus on what your code should do in the successful case. Code that first and only if necessary deal with some of the unhappy paths. You write software to DO something. Most of the code you write should be for doing that thing, not handling faults when they occur.

  • Don’t catch exceptions you can’t handle properly

    Never catch exceptions when you don’t know how to remedy them in the current context. Trying to work around something you can’t remedy can often lead to other problems and make things worse at inopportune times. Catching exceptions at higher levels is always possible. It is harder to re-throw an exception without losing information. Catch-all expressions that trap exceptions you didn’t anticipate are usually bad.

  • Software should fail noisily

    The worst thing software can do when encountering a fault is to continue silently. Unexpected faults should result in verbose exceptions that get logged and reported to an error tracking system.

Benefits

  • Less code

    Less code means less work and fewer places for bugs to hide. Error handling code is seldom used and often contains bugs of its own due to being poorly tested.

  • Less work

    Faster development in the short term. More predictable program behavior in the long term. Programs that simply crash when a fault is encountered are predictable. Simple restart mechanisms are easier to understand than specialized exception handling logic.

  • Reduced coupling

    Catching a specific exceptions is connascence of name. When you have a lot of exceptions that you are catching by name, your exception handling code is coupled to the code that raises the exceptions. If the code raising the exceptions changes you may have to change your exception handling code as well. Being judicious about handling exceptions reduces coupling.

  • Clarity

    Smaller programs are easier to understand, and the code more clearly indicate the programmer intent - unexpected faults are not handled by the code. Catch all expressions make the developers intent unclear to anyone reading the code by obscuring their assumptions about what could happen.

  • Better performance

    An uncaught exception is going to terminate the program or process almost immediately. Custom exception handling logic will use additional CPU cycles and may not be able to recover from the fault it encountered. Retry logic can also use a lot of resources and doesn’t always succeed.

Applying Let It Crash

Rather than try to explain more about the Let It Crash principles, I will show how they can be applied in other programming languages. While the application of these principles may vary from language to language. Let It Crash is applicable everywhere.

Let It Crash in Bash

I’m going to use Bash scripting as an example, because it’s something that many developers vaguely familiar with, and it’s radically different from Erlang as far as language semantics go.

Original Code

Here is a simple Bash script that reads from STDIN on the command line and uploads the input to Hastebin. I took this script from my own dotfiles.

#!/usr/bin/env bash

input_string=$(cat);
curl -X POST -s -d "$input_string" $HASTEBIN_URL/documents \
       | awk -v HASTEBIN_URL=$HASTEBIN_URL -F '"' '{print HASTEBIN_URL"/"$4}';

While this may seem like simple code there are a lot of potential faults that would cause unexpected behavior. There are at least three possible faults in this script that would result in unexpected behavior:

  1. If the cat command that reads data into the input_string variable fails the script will continue and will upload an empty snippet to Hastebin.

  2. If the curl command fails the awk command to parse out the Hastebin URL will still be executed. The awk command will not print out the snippet URL since there is no response body, and it might print out some other URL instead.

  3. If the HASTEBIN_URL environment variable is not set all of these commands will still be executed without it and will fail due to the empty variable.

Improved Code

This simple script can be greatly improved by setting some Bash flags. The improved code is shown below.

# Unoffical Bash "strict mode"
# http://redsymbol.net/articles/unofficial-bash-strict-mode/
set -euo pipefail
#ORIGINAL_IFS=$IFS
IFS=$'\t\n' # Stricter IFS settings

input_string=$(cat);
curl -X POST -s -d "$input_string" $HASTEBIN_URL/documents -v \
  | awk -v HASTEBIN_URL=$HASTEBIN_URL -F '"' '{print HASTEBIN_URL"/"$4}';

Before executing the commands we set -e / errexit to tell the script to exit when an non-zero (unsuccessful) exit code is returned by any command. This will the first and second issues pointed about above. We also set -u flag so any time an expression tries to use an unset variable the script exits with an error. The third flag we use is -o pipefail which tells Bash to use the exit code of the first unsuccessful command as the exit code for the whole pipeline, instead of using the exit code of the last command. This prevents a pipeline from returning a successful status code when a command in the pipeline fails. The script now exits if HASTEBIN_URL is not set and print a readable error message for the user. So the user knows what the script expects to be set by the environment.

Note About the Name

I don’t think Let It Crash is a good name for the philosophy Erlang has about handling faults. Yes, letting things crash is often the right thing to do, but it is a form of defensive programming (offensive programming actually) that allows software to recover to a known good state when faults are encountered. I also think defensive programming is a poor name for a design philosophy intended to improve software reliability as it seems to indicate the programmer should be defensive and proactively guard against unexpected faults. This is a topic for another blog post.