The Abstraction Trap

Abstractions exist so that we can take shortcuts when building stuff.  They readily implement some of the messy necessities behind things like, for example, network access in distributed systems – but too often they’re used as an alternative to in-depth understanding rather than a reusable way to render a system.

Staying with our distributed system example; understanding the latency between nodes, synchronisation and blocking (and their impact on concurrent work), transport overhead and reliability, and what the behaviour will be if you don’t hear back from the remote service are all critical to the success of a complex project.  Without having to explicitly program these things it’s too easy to just drop something into a pipe or throw it at a web service and merrily continue on with life having never even considered those neatly masked complexities.  Until go live.

It’s the difference between something that functions on the bench and something that will work reliably in real life’s volatile conditions.

On the infrastructure side I see the same pattern emerging with virtualisation – we’ve now got this nifty and easy to use platform which takes us another step further away from the metal.  On the plus side we can stamp out more nodes really quickly and move instances of a server around from hardware to hardware to accommodate growth and failure, but on the minus side correlating what’s happening on pretend CPUs with a real CPU and oversubscribing hosts makes capacity planning a degree harder.

All these environments are designed to abstract things away from us so that we don’t have to worry about them or recreate them every time – but that’s not the same as saying we no longer need to understand the basics of what happens under the hood.

You’ll always have better product when your engineers have the low level knowledge (primitives and patterns) needed to design and build systems and understand how they will behave.  That’s different to simply knowing how to drive the tools.