Availability or Control?

In the web business, we usually consider availability to be paramount - and given that motivation, we're getting pretty good at things like graceful degradation and partial failure. But now that you've pulled your system apart and neatly isolated all the features, how do you cope with the situation where no service is preferable to partial service?

This can be true. Consider, if you will, a trading system operated by a team of risk managers. You have built the system to be fault tolerant and allow partial failures - and this usually works out great - but what happens if a failure in the infrastructure or application results in the risk managers no longer being able to administer the system? It's still running publicly (thanks to you awesome failure isolation) so customers are still buying and selling. You cant change your prices and respond to changing market conditions - uh oh - exposure. What do we do?

One answer is a word we don't like - especially if we just built a reasonably decoupled system - dependency. Yuck, but there is no shame in creating some intentional dependencies that support the business rules. If you never want to execute trades unless you can manage your position, then what is the advantage to running the trading system without the liability tool? Nothing - if anything it's an undesirable risk.

So draw up some service dependencies, or make the applications depend on their monitors at runtime. It might not appeal to how we'd like to run the system, but the truth is it accurately reflects how we'd like to run the business.