Building on Quicksand

Quicksand is a mixture of sand and water that looks like it is solid but will slowly consume anyone that tries to walk over it. This talk will argue that for many years our systems have built their reliability by layering on top of unreliable components - effectively, they are built on top of quicksand. Furthermore, the granularity of the unreliable components has been steadily increasing through the years and is gradually showing new and challenging semantics to the application developer. Fault tolerant algorithms are built by creating pieces that are idempotent or repeatable without harm. Ensuring these idempotent operations can be performed by components that are unreliable, a reliable design must capture the state ACROSS the failure units and keep going if something fails. This has been done with circuitry, mirrored disks, multiprocessors, and more. In the past, failures were usually masked from the running application. Now, the size of the unreliable component is inexorably getting larger as the economics of our systems change. This is making it impossible for the underlying hardware and software platforms to hide the failures. Increasingly, applications must design to cope with the failures of these unreliable components. This talk will sketch the theory of fault tolerance and show a number of examples of how systems have been designed through the years and how their evolution has gradually impacted application design. Hopefully, this discussion will help application designers understand the new responsibilities they face in building reliable distributed systems. It is possible to stay afloat on top of quicksand with sufficient care!