Rake Routes

by Stephen Ball

The Fastest Possible Tests

I love tests as a fast programming feedback loop! In the last twenty or so years I’ve seen them go from an occasional critical verification to a foundational component of modern systems design.

Why care about fast tests?

When I’m writing code, especially when doing test driven development, I want the fastest feedback possible. That means fast tests.

I find 100ms is my upper limit for considering any individual test to be fast enough. That is still painfully slow but only just.

Any slower than 100ms and I find the delay between calling for the test and noting its result is just enough of a stuttering friction that I — even unconsciously — start to shy away from running the tests. I’ll spend more time writing the code I think is right vs simply allowing the tests to give me that sweet green confirmation. I’ll go longer between writing new tests and start to spike off code without any guardrails or guidance.

If I’m in a codebase whose unit test suite take seconds or (gasp) minutes to complete I soon advocate for deleting such painfully useless tests.

To be clear I am talking about unit tests and not integration tests. Integration tests can require an actual database to actually take the data and actually store it by actually writing it to some structure either in memory, on disk, or (worse of all) on a remote machine. Those kind of tests have real limits on their speed depending on how much you are willing to isolate them and still consider them useful.

But unit tests: no network, no services. Only the purity of potentially production code and testing code calling the production code and making assertions about the results.

How fast can they go?

How fast should they go?

Ideally unit tests should take less than 10ms to run and at most 100ms.

If your language or testing framework does not allow running individual unit tests then the entire unit test suite should run in less than 1000ms (1 second).

If you’re using any widespread general programming language this is entirely possible even for test suites with thousands of tests. Computers are FAST!

How is that possible?

That’s a topic for a future blog post! But start with A Set of Unit Testing Rules by Michael Feathers.

A test is not a unit test if:

  • It talks to the database
  • It communicates across the network
  • It touches the file system
  • It can’t run at the same time as any of your other unit tests
  • You have to do special things to your environment (such as editing config files) to run it.

How fast is the fastest possible test? Theoretically?

There is an absolutely fastest possible testing speed. If we free ourselves from all the bounds of hardware and software there is a fundamental speed limit on how fast our fastest tests can possibly be.

That speed limit is — of course — the speed of light.

No information can travel faster than the speed of light. No matter how much we optimize, improve, tweak, and streamline we cannot break free of the limitation that we have a roundtrip of information from human to machine and back. Information fundamentally has a speed limit.

  1. We need to tell our system to run a test
  2. Our system needs to transmit the results of that test

How fast is that allowed to be by the speed of light? Obviously VERY fast unless our computer is very far away.

Let’s assume our computer is about two feet away. How fast could our test feedback be?

Two feet of distance means our signal and response needs to travel four feet. (Of course light has to travel within the machine and our own heads but let’s set those distances aside.)

To WolframAlpha! Lightspeed over four feet

lightspeed / 4 feet = 4.067ns (nanoseconds)

It’s a fun mathematical quirk that it takes light just about one nanosecond to travel one foot which makes gauging the speed of light to travel human scale distances using antiquated imperial units relatively easy.

How fast is 4ns? Let’s dive down the units of time!

1 second             = 1 second      (1 s)
1 second      / 1000 = 1 millisecond (1 ms)
1 millisecond / 1000 = 1 microsecond (1 µs)
1 microsecond / 1000 = 1 nanosecond  (1 ns)

Only a few orders of magnitude, no problem!

Compared to our reasonable case of 10ms for one unit test? In the time it would take to serially run a thousand of our 10ms unit tests we’d be able to run 2.5 million of our fastest possible 4ns tests!

It’s fair to say we’ll never reach that speed and have any meaningful computation for an individual test. But it’s good to have goals.

Up next Shrink your data into bitfields (and out again) Some applications want to save every byte. Every last byte! Maybe they have memory constraints, maybe they’re sending data over the network, maybe A simple language spec isn’t a feature when you’re building applications Go famously gushes about its simple and readable language specification. One one hand kudos to Go for having a written specification and kudos for
Latest posts Clojure Functions in Four Ways See Some Clojure A simple language spec isn’t a feature when you’re building applications The Fastest Possible Tests Shrink your data into bitfields (and out again) Every “if” statement is an object waiting to be extracted Choose Generic Tools Hyperlinks you might find interesting — #4 Running bundle install on rails master Use tldr for command line examples Friday Lunch Links — #3 Friday Lunch Links — #2 Logical Solver: Turn facts into conclusions Programming with jq Command line tools - jq Friday Lunch Links — #1 Why diversity matters Music for coding - October 2019 Code puzzles are a poor way to gauge technical candidates Add vim to a pipeline with vipe Connecting Objects with Observable Let’s write a shell script What’s a $PATH anyway? Let’s Use Hwacha to Scan URLs Deliberate Git Customize Your IRB Program Like a Videogamer Gem Spotlight: interactive_editor Things Most Interviewees Fail to Discover Rails isn’t for beginners How to use bundler instead of rvm gemsets