Just a little more on SLAs…

Yesterday I posted a little something about SLAs and I’m always happier with things when I can wrap them up with a handful of guidelines.  Not always possible in the complicated world we live in, but here goes anyway:

1. Discover the things that are meaningful for the business.  I risk stating the obvious but there is always a temptation to approach this ‘backwards’ by starting off with what can be measured rather than what is significant (and then working out how to measure it).  You don’t want to end up with a bunch of metrics that are easy to count but don’t describe desired system performance.
2. Strike a balance between persistence and change.  Unless doing lots of projects isn’t important to you, be careful not to base all your KPIs on availability/stability metrics – or if you do, at least be aware of how that can drive reluctance to push changes through the system.
3. Make appropriate interpretations for each product or system.  In most organisations different systems, or parts of each system, are subject to different uptime, capacity, latency etc demands.  And assuming you pick some basics like performance they should be specific to each product; for a website that might be a number of page impressions, and for an analytic system that might be a time to render when a data set is updated.
4. Include time as a dimension.  Most businesses – particularly on the web – have a number of 24x7 products, but there are also a lot of systems that only get used during business hours or at certain intervals (e.g. payroll is usually a monthly thing).
5. Disregard #1.  Kind of.  Now that you’ve gotten this far, you will need to consider some feasibility, because signing up to unachievable SLAs doesn’t help anyone.  Have a look at what devices and services underpin the business functionality you are measuring.  Trees of dependencies, composites in SOA for example, tend to live up to the least strict SLA rather than the aggregate of the set.

Rules of thumb – apply in conjunction with local knowledge!

More Meaningful SLAs

Establishing internal service levels with the rest of the business is a difficult process - there are so many variables that can be measured and, as we all know, you change what you measure by measuring it. For example, if you express your SLA exclusively in terms of system uptime, then you improve all the activities around keeping your system available. The flipside of this is that you often discourage the activities around effecting change in the system - after all, any releases or upgrades or new features always carry some risk to availability and that's what they're measured on...

The place to start is to work out what's important to the organisation. Performance and availability are critical to us (a latency sensitive transactional platform with variable usage patterns) but so is change (a content driven web application correlated with events in the real world). We decided that performance, availability, change, and support response were the key metrics for us - nothing unique so far, and next we had to make an interpretation of each of these that was relevant to our various systems.

A basic principle here is recognising that it isn't just the raw numbers that should be appropriate to each individual product, but what is being measured too. Throwing an overall value at the problem (for example 99% availability across the board) makes the job of putting together your SLA easier, but is it a true reflection of your infrastructure? Whenever I've seen this coarse-grained approach used it has always led to less than acceptable uptime for the most critical applications and wasted investment propping up others that are realistically less important.

Another way to make sure your SLAs really closely matches business need is to introduce the dimension of time. In many systems and many organisations demand - and the cost of downtime - varies over time. For example, how many accounts and payroll systems are used around the clock? If you can trade off to 'best endeavors' over weekends and evenings then you shouldn't have too much trouble meeting a five nines commitment during business hours between Monday and Friday.

For our website we have a flat availability target (such is the nature of a 24x7 site) and performance we interpreted in a latency metric for price publishing and order placement. For reporting systems - which do not experience the same round-the-clock demands - we have different availability targets during business and after hours. Performance in the context of those systems is interpreted as a certain set of daily reports delivered by a fixed time each morning and a message delivery SLA on alerts on certain events. SLA's around change and product delivery are much more complicated and fraught with subjective measures. We've gone with measuring development projects iteration-by-iteration; what got delivered vs. what was committed during that sprint's planning. It's objective and encourages good estimation and strict control of scope creep during a sprint.

Making SLAs commensurate with what the business genuinely demands from a given piece of technology is important. Setting your sights high can seem like a good idea on the surface but, when you consider the frightening magnitude of difference in cost between 99.5% and 99.9% uptime, that couple of points can only ever be described as waste if they are not intimately linked to the organisations success.

Development Environment SLAs

When we use the word 'production' to label our customer-facing systems we create certain expectations around how we support and manage them - careful change control, detailed monitoring, rapid response to incidents, and administration that works around user requirements.

But when we label our software delivery lifecycle environments 'development' and 'test' systems, we likewise heap upon them certain expectations. Unfortunately for the development process, those expectations are usually secondary priorities and open season for any sort of experimentation or ad-hoc change.

Perhaps this is why so many projects come across environmental issues from time to time? Development environments are in production too, they just produce something a little different to what your customer facing systems produce - more customer facing systems.

Sure, when faced with a direct choice between doing something for a lifecycle environment and something for live products the decision is easy, however handling development and test systems with a little more production-like behaviors doesn't need to impact your live system activities and will make all your projects a whole lot smoother.

Root Cause Analysis

To help me kill some time at an airport (which seems to be my second job these days), let me reach into my wardrobe of soap-box issues and pick something out. Ah, root cause analysis, here we go.

In my opinion, proper root cause analysis is the most important part of any operational support process.

Having a professional, predictable response and the skills to restore service quickly are critical - but you have to ensure that your support processes don't stop there. If they do, then you're simply doomed to let history repeat itself, and this means more downtime, more reactive firefighting, and less satisfied customers.

Good root cause analysis takes into account the entire event - systemwide conditions, the teams response, the available data, the policies applied - not just the technical issue which triggered the fault, and looks for ways to reduce the likelihood of recurrence.

Doing root cause analysis properly can be expensive, because you don't need to get to the bottom of why it happened this time, it's why it keeps happening, and why the system was susceptible to the issue in the first place that you need to uncover to really add future value. Think of the time spent on it as an investment in availability, freeing up your team to work more strategically (as well as enjoy their jobs more), and happier users (which oddly seems to make happier engineers).

But what you learn by doing this isn't really worth the time you spend on it without the organizational discipline to follow up with real changes. If you're truly tracing issues back to their root, you'd be surprised how many are the result of a chain of events that could stretch right back to the earliest phases in projects. This needs commitment.

If you make money out of responding to problems then you'll probably want to ignore my advice. There is a whole industry of IT suppliers whose core business lives here, and while it's an admirable pursuit, don't take the habit with you when you join an internal team!

Availability or Control?

In the web business, we usually consider availability to be paramount - and given that motivation, we're getting pretty good at things like graceful degradation and partial failure. But now that you've pulled your system apart and neatly isolated all the features, how do you cope with the situation where no service is preferable to partial service?

This can be true. Consider, if you will, a trading system operated by a team of risk managers. You have built the system to be fault tolerant and allow partial failures - and this usually works out great - but what happens if a failure in the infrastructure or application results in the risk managers no longer being able to administer the system? It's still running publicly (thanks to you awesome failure isolation) so customers are still buying and selling. You cant change your prices and respond to changing market conditions - uh oh - exposure. What do we do?

One answer is a word we don't like - especially if we just built a reasonably decoupled system - dependency. Yuck, but there is no shame in creating some intentional dependencies that support the business rules. If you never want to execute trades unless you can manage your position, then what is the advantage to running the trading system without the liability tool? Nothing - if anything it's an undesirable risk.

So draw up some service dependencies, or make the applications depend on their monitors at runtime. It might not appeal to how we'd like to run the system, but the truth is it accurately reflects how we'd like to run the business.

Scheduled Reboots and Natures Way

One of the basic aspects of a biological computing mindset is the appreciation that nothing lasts forever. Everything degrades, corrupts, and dies over time - and that is perfectly normal, because it's duly replaced by a fresh-faced youngster, eager to service the rest of the organism [system] from a nice, fresh cellular structure [empty memory space].

This applies to systems in exactly the same way as it does to organisms. How many issues can you recall where memory leaks, counter errors, and freaky edge conditions all occurred after servers have been running exactly X long, or when a service has processed more than Y connections. I'm sure we could swap tales of woe late into the evening.

This being the case, why do we feel this rottweiler-like dedication to keeping individual devices going for the longest possible duration? I think there is 2 sources; a kind of point scoring pride effect engendered by the output of the "uptime" command, and good old fashioned poor system design. Perhaps one even leads to the other...

So - we design systems poorly. If we want a product to be available, why do we build it in a way that it's availability depends upon a piece of tin that we accept is inherently unreliable? So now the application is the server. This means the only way we can increase its availability is by increasing the availability of the underlying hardware. Not only is this expensive, it's doomed to failure because, as we accepted, servers grow old. So we spend a lot of time and money trying to achieve something we already decided that we cannot. No wonder we're so excited when that uptime counter rolls over to a nice big number!

Do you know what would be better? Accepting that product availability - the uptime of the whole system overall - is what we're really reaching for, and besides, it's how our customers will measure us. Next we need to apply this philosophy to how we design systems, let go of our attachment to keeping individual servers on life support, and put together services that don't rely on any one node, network, or storage device in order to serve our customers.

If you can master that arcane art, then you'll be able to arbitrarily recycle resources, anytime, when there is absolutely nothing whatsoever wrong at all - because this helps keep it that way.

Oh and you'll never be that guy with the box thats been going so long he's scared to reboot it just in case it doesn't come back!

My Demotivational Poster

I just has to make one - and what better than my favorite topic (credit to whoever photoshopped tetris):

Helmuth von Moltke (a Prussian/German military strategist in the 1800's) once wisely observed that "no battle plan survives contact with the enemy".  I think you can say the same thing about software - no matter how elegant or simple your feature is, it will never survive contact with users.

That's why response and recovery plans are just as important as the code they are in place to support - you will need them sooner or later...

Teach Your Computer to Say No

As good engineers we always try to build systems that have enough capacity to service all our customers requests, all the time, anytime.  It's an admirable goal and very commercially sensible, but there is no such thing as unlimited capacity and sooner or later we'll get our scalability wrong, suffer partial failure or maybe it won't be cost effective to scale up to those rare usage peaks - what will the user experience be then?

By default it will be things like; page cannot be displayed, connection timed out or our staple favorites 404 and 500 - basically the middle finger of the internet.  You have to think about how your systems will behave under unexpected load and what you'd want your customers to see when it happens.  There are a few things you can keep in mind when building your products that, if properly addressed, will lead to a very different user experience when things get tough:

Customise your error messages

Those unexpected peaks or partial failures are always going to happen sooner or later, so why not serve up a nicely formatted (but lightweight) page with alternative links or customer service contact details rather than a nasty, cryptic error message?  It is so easy and cheap to do and makes such a huge difference to user perception that there really is no reason not to.

Queue or throttle excess traffic

What's better than a nicer error message?  Some kind of service - even if it doesn't match our ideals.  Queuing is a nice solution, it protects your in-flight users just like denial would and provides a better experience for those new arrivals who would otherwise push you over the edge and get an error.

A holding page, an alternative URL or maybe a nice countdown with an automatic refresh is all you need.  These things can be a little more time consuming to build because your application needs to have some awareness of it's environment if you expect it to make decisions about what to serve up based on remaining capacity.  There are some very easy ways to buy this back in hardware if you're using load balancers like F5, Netscaler, or Redline - a little experience will tell you how many users/connections/Mbps each node can tolerate, and you can configure an alternative page or redirect for anything above that threshold.  Depending on what you have installed this can even be served directly from the device cache, making it even lower impact for your over-busy system.  Queuing is just like it's real-life namesake; "line up here and wait for access to the system".

Throttling might suit certain applications better - particularly API's with a heavy non-human user population, as client applications might not understand queuing messages, holding pages or redirects returned from the oversubscribed feature but they'll be less sensitive to a general slowdown.  Throttling, as I use it here, is about reducing the maximum share of total system capacity any individual can consume in the hope that this will allow more total individuals to use the system concurrently.  The theory is that once we hit a certain threshold, if we flatten out everyone's usage, the big consumers will be held back a little (but still served) and the rest of us can still have a turn too.  Like most ideas here, this can be implemented in a variety of places in your stack; queries/sec or concurrent data sets searchable in the database, concurrent logons in the middle tier, TPS in the application, GETs/POSTs on the front end or even number of concurrent connections per unique client IP address at the network edge.

Where is the right place to impose planned limits?  I'd suggest multipoint coverage, but to start with think about your most critical constraint (what dies first as usage climbs?) and then do it at the layer above that one!

Fail gracefully

This is one of the toughest things to do with a distributed system; your features need to be aware of their environment both locally and on remote instances and they need a way to bring up/take down/recycle themselves and each other.  If your system suffers a partial failure, or load is climbing towards the point marked "instability", then you need to make a decision.  Do you bring up or repurpose more nodes to handle the same feature?  Redirect a percentage of requests to another cluster/DC or trigger any of the other techniques we talked about above?  Great if you can, but if you can't, it might be time to use those [refactored to be friendly] error messages.

Turning away users by conscious decision may rub us up the wrong way at first glance, but depending on how your system behaves under excessive load it might actually be for the best.  If you are at maximum capacity and serving all currently connected users, would new user connections result in degradation of service to those already in the system?  Maybe saying no to additional connections might be frustrating for those users being turned away, but you have to weigh that against kicking off users midway through transactions, or worse still, crashing the entire system!

Graceful failure is essentially the art of predicting the immediate future of your system and handling what are likely to be excess users/transactions/connections in a premeditated fashion.  First you need to know what your thresholds are; then you need very fine resolution instrumentation to tell you how close you are to them on a second-by-second basis, and finally, you need an automated way to respond to impending trouble - because humans are too slow to prevent poor customer service in busy systems; we're better at cleaning up once the trouble has been and gone.

For example let's say you're aware of a memory utilization threshold or a maximum number of users/web node - wouldn't it be better to actively restart a service or show a holding page before you lost control of the stack?  If you haven't managed to build in some kind of queuing or throttling then you might be showing an error or denying service but is that better than losing the whole system.

The last part of graceful failure somewhat overlaps with recovery-oriented computing; make sure that, upon death, the last thing you ever do is take a snapshot of what you were doing and what the environment was like.  If your processes do this, then you are able to have watchdog processes (or monitoring systems) that know whether or not it's safe to restart that failed instance (or take an alternative action based on the data), you'll have an easier time diagnosing faults, and the data generated will help you keep a rolling benchmark of the thresholds in a system with a high rate of change.

S**t Happens

I talk a lot about failure, how to build for it and recover from it.  Of all the things that will happen to your system during its lifetime failure of some sort is one of the few inevitable events.

A lot can go wrong with computers, but surely their best-known weakness has to be their fundamental incompatibility with water.

Focusing on building systems that survive individual node failure is an excellent discipline, but as you can see from that clip, you can't count on your datacenter to always be there.  That means distributing your system across servers in the same location will protect you from a number of (the most common) failure scenarios but if it's really, really important that you are always up then it needs to be in more than one place.

Think electricity.  Think connectivity.  Think geography.

Rack Mount 1, Technician 0

Failure is coming to get you, but we're getting better at predicting the scenarios and coding for them.  We think a lot about servers dying, losing network connectivity, power cuts, and how to respond to critical bugs.  These things are essentially unexpected technical events, but there is a whole other category at play in real life - human error.

Imagine this server has your data on it...

What will your customers see while that gets put back together?  How are you going to get the data back?

Despite the comedy value of that clip, this is exactly the sort of thing that happens in real life - people make mistakes.  But even when this kind of maintenance is less clownishly executed, it still needs to happen - and you need to decide what effect you're going to let planned maintenance events have on your revenue stream.

Isolating Failure

A few weeks ago we launched our Sportsbook, our flagship risk-taking product, into the Italian market.  It's a very strategic product both because of how it fits into our international expansion plans and because of how it's built.  We intended for this system to get big fast so we built it wide - distributed, message based and API driven.

Just days after launch the one thing we can always guarantee will happen happened - part of the system failed.  Our bet placement engine hung [bad news] but the rest of the system continued to work [good news] so while our customers couldn't place any bets, they could hit the site, view the markets, register, login, deposit, manage risk... everything else basically.  This kind of failure scenario would see a lot of more monolithic web systems firing out 404s or 500s in the blink of an eye.

This is exactly the benefits you look for when you build decoupled functionality and minimize dependencies, consciously seeking to isolate features from each other.  The next step for us is automatic detection and repair - working to minimize human intervention.

Epistle to the Vendors

Before doing what I do now I was always in system integrators/consultancies.  Now that I'm on the other side of the boardroom table I've seen the whole picture and I have to say - we're playing right into the hands of the resellers, integrators and consultancies.  If we're going to do this anyway we may as well be a bit more open about it.  Let's go in up front and do it properly in a more formal, planned manner rather than stumbling from one bad decision to the next.

Therefore I have written this open letter to vendors everywhere outlining exactly what they need to do to take us to the cleaners:

 

Dear Sir/Madam

We're a web enterprise just starting out and we're teetering on the brink of profitability - we have a popular product and the business is expanding at a fearsome rate.  We're running around like madmen trying to work out how to get the right balance between scaling our system, shipping more features and protecting our time to market.  You've come in to meet us disguised as help - nice touch - and you're trying to work out how you can squeeze the largest possible cross-section of what you sell into our requirements over the longest possible timeline.  Using these 3 big phases you can really stretch it out.

Phase 1 - hardware (AKA brute force)

Popular product, lots of customers, increasing load - get some bigger computers, we'll buy those.  Next sell us some faster network, we'll need gigabit everywhere, some bigger firewalls and then how about load balancers?  Let's go brand crazy and throw together some F5, Netscaler, Cisco and Foundry.  Load increasing a bit more?  Next size up computers please.  Now we're a little scared too (evil hackers!) so let's have some Top Layer, maybe Netscreen and Juniper too.  Uh-oh load is up again so more memory, more CPU please.  Wait a minute, we're now out of bigger boxes in the x86 space?  Can't address any more memory in a single node?  I guess we'd better move into something like SPARC, jumping a whole class of computer - ka-ching!

Phase 2 - middleware (AKA short sighted)

OK we're still hitting capacity ceilings but now we worry equally about uptime.  How can you help [yourself to our cash] here?  The best tactic is going to be some really expensively licensed software that contorts scale and availability from products that just weren't built for it.  Ideally we're looking for some second-rate results by forcing artificial distribution using things like GemStone, Tangosol, Veritas, CA, TimesTen.  It'll also get us some failover (yay) but without delivering too much more actual availability (boo) since our application is blissfully unaware of the state of it's infrastructure.  Tricky but lucrative times.

Phase 3 - consultancy (AKA told-you-so)

Now's the best bit - you get to sell us your most expensive, highest margin product - consultancy [meanwhile in the real world we finally agree on something; we're in a bind we can't buy our way out of].  So what's your advice?  Decouple and distribute, scale horizontally, version data and services, isolate features and automate the hell out of the whole thing.  Thanks for that (and ka-ching again).  The most important thing to remember is not to ever let on that you knew this all along.  Oh and if you're clever you can usually also work in some sort of licenses or something in this phase as you never really want to completely wean us off you.

So there you have it, think of this as your 12-step programme compressed into 3 easy stages - just do your bit (turn up with PDF's and nicely ironed shirts) and we'll do ours (make appalling decisions in our early product development) and together we'll milk us dry.

Yours insincerely

N. E. webscale business

 

Hang on this all sounds pretty sweet, why did I ever leave consultancy?  But it's not too late!  You don't have to put Capgemini, Dimension Data or Logica's children through college if you don't want to!  You just have to think differently now.  You're making a product you intend to be successful right?  You intend for it to be popular, right?  Well decouple, distribute, scale horizontally, version data and services, isolate features and automate the hell out of the whole thing now, today!  You will end up doing that (or die trying) anyway, it's just a matter of how much it costs you on the way.

The Great Divide

I'm into availability at the moment, so much so that I'm leading an organization-wide change initiative that targets cultural, technical and process habits (busting established, forming new) in order to deliver a more consistent experience.  Our goal is to always have something to offer our customers regardless of any maintenance we're performing or failures we're experiencing.

This is how I discovered The Great Divide.  The namesake of this post is that gap between the infrastructure and the software - because of which we offer our product to our users much less often than we could.

Here is how it works:

We've got a pairs of firewalls that can fail over while maintaining session state.  We've got tiers of load balancers that can reroute traffic around down network devices.  We've got clustered databases that can move active systems between nodes in a couple of minutes.

But guess what else we've got?

We've got applications that lose session information without contiguous sequence numbers.  We've got applications that cant match users to activity if their traffic suddenly comes from another IP address.  We've got applications that depend so heavily on their databases that death occurs within a few seconds of separation.

Damn.

You don't get any partial credit in product uptime - your customers will not award you a bonus point if your site is down but your servers are up.  If they cant log in they cant log in, if they cant place orders they cant place orders; they're quite a binary bunch.

For us product = infrastructure + software + operational know-how to run it.  We need to stop worrying about server/network availability and start worrying about product availability - because guess what, that's what our customers are measuring us on.

Close that gap and let your customers see the benefit of those cool devices.

A Good Quote

I talk a lot about failure in systems - it's inevitability and the things we can do to minimize it's impact and recover in a quick, automated fashion (note I never said "prevent" failure).

I recently came across a good quote on the topic that I think is really punchy:

"failure is not an option - it's bundled with your software"

I would credit it's author but I can't seem to work out where it came from - I've seen it tacked on the end of a few people's forum posts. I can't help but think they meant it a bit more cynically than I'm taking it but it's a good thinking point when you're delivering online services.