NestEgg is hiring!

I founded NestEgg a little over a year ago, and it's been an INCREDIBLE first year.  We've created an AI that replaces the need for a human property manager, delivering happier tenants, less stressed landlords, and better deals on tradespeople.

As a small team of co-founders we have achieved a lot over the last 12 months; validated our idea, launched a successful beta in a test market, signed a bunch of awesome distribution deals, and - most recently - closed our first round of external funding.  That last one means we can now dial it up to 11, so we need to grow the team!

WE ARE HIRING

NestEgg is at the center of two things that are exploding with new ideas, innovation, and investment - cognitive computing and prop tech.  As an ex-CTO I find this incredibly stimulating, and as a current CEO I am energized by the powerful customer outcomes we can create with this platform.  It's not just technology to us, it's technology with a purpose.  A purpose that really matters to tens of millions of people across the US on a daily basis.

We're offering all the creative freedom and influence an early startup usually provides, as well as a life-changing stake in that future outcome.  We're a distributed team with tons of flexibility, so finding the absolute best people is more important to us than current location.

Our standard package includes

• Competitive salary for early companies

• Significant equity stake - you will really own what you do

• Bonus tied to clearly established business metrics

• Flexible healthcare package

• 401k integrated with payroll

• Tech allowance to upgrade phones and laptops

• A 'take what you need' vacation policy

In return we expect

• Strong sense of urgency and ownership

• Accomplished technical skills in more than 1 domain

• Experience in a consumer facing internet business

• Comfortable working in a distributed environment

• Experience collaborating with a multi-disciplinary team

• Demonstrated intuition and proactivity; can execute against broad goals

• Appetite for personal growth and taking on new things

We need a couple more product-minded full stack engineers and a great, customer-first UI developer.  The people we want to bring on will know what it really means to be entrepreneurial; to execute quickly across a broad set of contemporary technologies (without sacrificing quality!) and be passionate about creating a future outcome that transforms an entire industry.

If developing your career in these areas is something you're excited by, then get in touch so that I can tell you more of the story of a better future you could be part of making come true.

The power and the pitfalls of test and learn

My core philosophy on product is that product/market fit is a journey; it is a function of discovery and learning over time, and it is impractical to expect to be able to make a set of ideal decisions up front (i.e. in advance of any product development and real feedback from real customers).

We now live in the world where the fast eat the slow, and I am going to argue that learning is speed:

At Hotwire we’ve dramatically improved our business performance in the last year by focussing on learning as a primary goal.  It’s a first order metric, along with the ‘hard’ KPIs most businesses are familiar with (revenue, room nights, activations, etc) but - as any good teacher will tell you - measuring learning is extremely hard to do.  Fortunately for us, there’s another feature which is both easy to observe and highly correlated with learning: experimentation.

Whenever you see a high rate of improvement (in nature and in science) you will almost always see a high rate of experimentation.  There’s a great Thomas Edison quote which goes something like; “None of my inventions came by accident. I see a worthwhile need to be met and I make trial after trial until it comes.”  Most of our early learning as human beings is heavily experiment driven; it is interactions with the world that teach us how to behave effectively within it.  But my favorite story about the value of pursuing learning as a primary goal is the Kremer Prize:

In 1959 Henry Kremer, an industrialist and patron of early aviation, established a prize of £50,000 for the first human-powered aircraft to achieve a controlled flight.  Hundreds of attempts were made over 20 years with no success.  And most of these were well-informed experts; aviation companies and universities etc.  The prize went unclaimed until 1977 when Paul MacCready won it with his Gossamer Condor.  The secret to his success wasn’t being smarter than any of those who previously attempted flights or knowing something they didn’t, it was his approach to the problem being fundamentally different.

Most teams spent months designing and building their craft, then they took it out to a field or an airstrip to try it out, then they crashed, then they swept all the pieces up and returned to the hanger for another few months to rebuild.  MacCready focused not on the airframe that would work, but on a cheap construction which was easy to assemble and disassemble by hand in the field.  Using this he was able to try more designs out in one single day than every preceding team added together for the entire previous year.

His formula for success had 2 main features; variation and repetition.  You can see this in nature too; natural selection tries out variations of organisms over and over again (optimizing to the ultimate KPI - life itself!) as those organisms improve their suitability to their environments.  While not strictly mathematical, Fisher’s theorem is a useful formalization of this (and the other examples we’ve discussed here).  It goes something like this; “The rate of increase in fitness of any organism at any time is equal to its genetic variance in fitness at that time.”  Or, more simply:

The capability to try out more hypothesis at any given time is highly correlated with faster improvement.

So learning is speed, and number of experiments is a useful approximation of learning.  But before you just count all a/b tests etc, there are some subtleties to success here:

The first thing I like to watch out for is confirmation bias.  As you embrace test and learn and make experiments cheaper to run you lower the bar for organizational participation.  This is unquestionably another benefit - harnessing the innovation of a larger slice of your org - but not everyone is as academically disciplined as you’d hope, and there is an underlying human tendency to like our own ideas and see them in a less-critical light than the ideas of others.  A while ago I was fortunate enough to be able to spend some time with Alan Kay, and he told me that science is the process that stops people from falling in love with their own ideas.  This is more than just a principle; if you come up with a hypothesis which you’re trying to prove instead of disprove, you will tend to discount contradictory evidence (i.e. proof points that suggest customers do not like it) and waste a lot of time trying marginally different manifestations of the same core idea in the desperate hope that you can somehow make them love it.  You lose speed, not gain speed, this way.

A fun way to look at this is to examine the difference between scientific thinking and religious thinking.  Jack Cohen is another wonderful human being, with a vast amount to teach the world, with whom I had the pleasure of spending a little time.  He argues that; in religious thinking all that matters is how hard you believe, especially in the presence of contradictory evidence (it’s there to test your belief).  In scientific thinking, all that matters is how hard you doubt, especially in the presence of confirmatory evidence (easy answers there to trick you into overgeneralizing your observations).

People will naturally come up with ideas that they like, but good product hygiene is about coming up with ideas that your customer likes.  Rigor in testing is the ultimate arbiter which can make that distinction clear to you, if you let it.  If you’re not open to being wrong - in fact expecting it and actively seeking to make it so - then you cannot, by definition, learn.

Another big-picture mistake is confusing iteration with learning.  It’s a common problem in the whole ‘transition to agile’ world; break up a big-up-front-plan into a number of predetermined phases, label them ‘iterations’ and then cash in your huge cheque as a profound agile coach.  Starting with the same inflexible ideas and delivering them across a higher number of software releases has some benefits in terms of quality and system risk, but it does nothing to improve your product/market fit.  You had determined the level of product/market fit at the beginning of the project and you have not improved that fit over the whole, say, year you were working on it whether it’s one single release at end of the 12 months on a whole series of incremental drops every two weeks.  Learning is about regularly shipping something customers can touch, then watching their interactions with that thing.  Looking for where it enriches and where it detracts, and only then deciding on the final scope for that next iteration.  The point is that each iteration should reflect the learnings from the previous iteration and improve upon it with respect to the customer experience, and therefore cannot be rigidly determined in advance of that feedback.

To be inclusive and stimulate the organizational pathways for innovating, a pretty broad filter is useful in the beginning.  All ideas are valid, and anything can be reduced to a testable hypothesis.  But, as you execute on a test and learn backlog, selecting ideas which are both coherent with the existing product and carry a higher probability of resulting in an improved experience for the customer becomes important.  Think back to our Kremer Prize example; while MacCready focused on the ability to explore a large number of variants, he was not randomly experimenting with geometry in the hope that he got lucky and found something that would fly.  He was an engineer who ran an aerospace company and he had a detailed understanding of all the mechanics involved in lift and control etc.  In user experience terms that means figuring out what customers are sensitive to, and using those things as inspiration points for ideas.  On the internet these sensitivities are rooted in behavioral science (decision theory, nudge theory etc) and are often things like social signals (how many others have bought the same item or currently viewing this page), urgency messaging (this is a popular item or this deal expires soon), and recommendation (if you like x you might like y) etc.  Discovering what these sensitivities are for your particular product can get you more ideas aligned to the physics of your particular business.

When talking about ways to increase the likelihood of ‘winning’ tests I always like to reiterate the value of ‘losers’ too.  A losing test is essentially an idea someone had for a new product or interaction which, without the experiment disproving the hypothesis, would have resulted in a project that would have taken away valuable product development resources for no (or negative) return.  What did you save right there?  How much customer attrition did you avoid by staying away from that unpopular option?

There is much more to doing this well - a whole series of posts wouldn’t do it justice - but I like to focus on the quality of the thinking first.

So does learning == speed?  At least in the special case of how rapidly you can get to the right product/market fit and grow, it certainly does here.  Is the number of experiments a good heuristic for organizational learning?  If you chart the improvement in our real business performance and the number of concurrent tests over time, you see very similar growth curves, just as Fisher’s theorem predicted...

Steriods for Signups

Anyone who runs a business on the web should be intimately familiar with the registration funnel (sometimes called the recruitment or conversion funnel) so - except for the briefest review to frame the problem - I'm not going to rehash that here. Instead I'm going to cover some common and some not-so-common ways to boost registration numbers.

So, for the quick review, the registration funnel is basically the set of steps a potential customer traverses on their way to becoming an actual customer (converted) and typically starts with a site hit and ends with an active user account. What these steps consist of varies with the nature of each business and might include things like a shipping address or a credit check along with the basics such as username and password.

The problem every web business faces is that the registration funnel is essentially a very lossy process. For example; of every 100 site visitors, 50 will start the registration process, of that 50 25 will complete the process, and of that 25 10 might go on to place their first order. Even with the best husbandry many web businesses have single digit conversion rates. With marketing on the web becoming more sophisticated and SEO/semantics driving smarter access to content getting the numbers in the top of the funnel usually isn't the problem and - just like any other lossy process - there can be a big impact on the bottom line by improving the rate by just a couple of points.

So how can you juice this up?

The very first thing you need is data. Just like everything else you need to understand the flow and then you need to instrument the heck out of it before you should be fiddling with it. If you haven't already got this in place then stop reading now and get on with it - the rest of this is useless without insight.

Separate your registration process out in your head; this will help you understand and instrument it. Think through all the steps in your funnel make it as granular as possible - this gives you much more insight into where potential customers drop out and much more flexibility in how you modify the process in response. Crudely; if you're thinking visitor -> credentials -> personal details -> checkout then you should be thinking visitor -> credentials -> name and email address -> shipping address -> checkout. This doesn't necessarily mean more pages or more data for customers to enter (in fact the goal is the opposite) but it does mean than you can count these things individually to see exactly what bit of information your potential customers get reluctant about handing over and it allows you to stage things a little more (see below).

The general principle is to put the fewest barriers between your potential customers and your product as possible. This means looking at what information you're asking for, when you're asking for it, and how it's captured. In some businesses there can be constraints which have to be acknowledged; with a product which is regulated or controlled in some way there may be some external rules which can reduce your options - such as the requirement for identity verification or credit checking - but the keyword here is reduce. The registration pipeline in these types of business is often poorly managed for this reason but it doesn't always need to be; take the time to understand the details of your governance and you will usually find that you have more flexibility than you assumed. For example you might need to verify your customers identity somehow before allowing them to make a withdrawal but does that have to stop you from granting them access to the rest of your product entirely?

Nice segue into staging. One of my favorite techniques in businesses with more complex registration requirements is to focus on capturing the absolute bare minimum during an initial signup (usernames, email addresses, passwords, etc) and capturing the rest 'just in time' in line with access to features. For example; why not delay asking for a physical address until the first time a user reaches the checkout? Why not ask for payment information just before their first purchase? You don't need that data until those events and having to enter it all up front can be prohibitive especially when potential customers might not be 100% sure that you're the service they want. Also think carefully about when you want to trigger the 1st step of the registration process - there can be benefit to letting potential customers play with a little bit more of your product before hitting them up for some credentials (but you have to balance this with less complete early contact information and the risk of taking it too far and having fewer accounts because the majority of the value is available anonymously).

Usability is an important aspect here. Considering the number of screens in your signup process, the number of fields, and even the phraseology of the questions can improve your conversion rates.

Also consider the order that you're capturing information in; if you start with email addresses and you're automatically storing fields as they're completed - a highly recommended Ajax trick - then you have some actionable data to go with your telemetry. You might want to use it for a follow-up contact to see if you can claw a customer back from the edge of registration oblivion.

Another consideration when you're designing your forms is to keep in mind the sort of information users are likely to have close at hand when signing up for your site. The more esoteric (in day-to-day terms) the information you ask for is the higher the chances that a potential customer will need to go elsewhere to retrieve it during the signup process. Data always shows that anytime a customer leaves the page(s) for any reason the changes that they'll return and complete them drop through the floor. What might you not even need users to enter at all? You can pick up things like language and locale from the browser and tools like geo-IP; adjusting a probably-correct field from a list of common choices is far preferable to data entry.

While we're on forms; short and sweet is the way to go. Whenever you have to go 'below the fold' then you're better off splitting the form into multiple pages (otherwise it appears too daunting) and, when you're going across multiple pages, then label the progress. A simple 'page X of Y' can work wonders setting expectations.

And finally, if there is a relevant infrastructure for your line of business (meaning that you can trust it and that a portion of your users are likely to be members) you can consider using an identity service, such as OpenID, to implement a kind of 1-click signup and then just capture the additional information you need to operate the account.

Change it and see what happens - after all you're collecting all the data you need, right? If you're crafty enough then you can do some A/B testing with alternate pages or different text. As long as you're measuring the process you will really quickly find out what's better and what's worse.

The Paradox of Freemium

One of the businesses I advise recently flipped over to a freemium model and – as expected – saw a step change in paying subscribers. It does lead to the question; in a world where we typically make money by making things and then selling them to others, how did we decide that giving valuable things away made commercial sense?

Firstly, for those who may have spent the last few years in a cave, I broadly define freemium as giving away something of substantial value along with an associated upgrade/expansion path which returns benefit to the organisation.

This is already commonplace. So many of the everyday tools I use have an element of freemium to their business model; google apps, zen agile, bitbucket, linkedin, mockingbird, a number of podcasts (which lead to paid-for books or other products), and of course almost every iPhone app I see these days has a fairly comprehensive ‘lite’ version.

For a long time we’ve given away tightly limited trial versions, exposing a very small subset of the functionality (or usage time) in order to coax users into a purchase to unlock the rest, and freemium is almost a reversal of this. Give away quite a rich set of functionality with no expectations, and you’ll pick up a group of buyers for that additional stuff on top.

Making this happen comes down to product design; you have to make sure that you hold back something(s) compelling enough to encourage purchases while making sure that your base product is functional and complete enough to be of significant utility on its own. There are so many ways to slice this, and sometimes the differences don’t even need to be rendered – you might simply impose several distinct licensing terms (free for academic or home use for example).

In one of my all-time favourite Ted Talks, Dan Pink says:

"The mid 1990s, Microsoft started an encyclopedia called Encarta. They had deployed all the right incentives. All the right incentives. They paid professionals to write and edit thousands of articles. Well compensated managers oversaw the whole thing to make sure it came in on budget and on time. A few years later another encyclopedia got started. Different model, right? Do it for fun. No one gets paid a cent, or a Euro or a Yen. Do it because you like to do it.

Now if you had, just 10 years ago, if you had gone to an economist, anywhere, And said, "Hey, I've got these two different models for creating an encyclopedia. If they went head to head, who would win?" 10 years ago you could not have found a single sober economist anywhere on planet Earth, who would have predicted the Wikipedia model."

I think those same sober economists would probably have said much the same thing about freemium, and probably much more recently. Getting customers to pay for your product by first giving most of it away wasn’t common sense – and maybe it still isn’t – but you certainly can’t ignore the evidence that it works.

No problem too small, no reward too big

Do you ever sit around with friends, talking over a few beers, and solve all the worlds problems? As I get older this seems to be becoming my chief pastime, and we recently got ever so excited about power generation and distribution. We figured the whole thing could be completely re-architected in a P2P model, where everyone in the grid both consumed and generated/supplied power.

Rushing to the nearest browser in case a patent application was necessary, we became a slightly despondent upon seeing a bunch of articles like this one on microgrids and what the EPRI are getting up to. That said, it's great to see this kind of stuff getting mindshare.

So what, if anything, is still missing? To see this come true, it seems like we'd have to work our how the power companies would still make money in our community-spirited utopian electricity supplying future. So perhaps we can still contribute by bringing some economics into the solution.

Someone has to provide that infrastructure - the grid we'd all have to be connected to in order to exchange electrical capacity. That sounds like something someone should be paid to provide. So perhaps there is a subscription fee? Or maybe we can do one better - how about a variable charge based on each household's net balance at the end of every month/quarter/year? The closer you are to netting out (producing the same number of kilowatt hours you consume) the closer to zero the bill gets.

The beauty of that model is it incentivizes consumers to be as self-sufficient as possible and disincents (is that even a word?) waste. The expected impact of such a price control is for participating individuals to want to align their production with their consumption - probably by boosting the capabilities of the former while working out efficiencies in the latter.

Reducing overconsumption is probably a no-brainer, but you could look at reducing overproduction both ways. On the one hand, overproducers supply overconsumers and we'd need that, but on the other hand we wouldn't want participating individuals to profit from overproduction. Ultimately all overproduction is wasteful (by definition if nothing else) so there should still be a price for it, a lower price than overconsumption, but not as low as netting out.

Well that was all kind of off-topic, but I hope you enjoyed it nevertheless. There are some intriguing engineering challenges in the basic problems inherent in sustaining our society, and if we gave this kind of stuff a little more attention, we'd all be living in a much better world before you know it.

Gartner Economic Downturn Briefing - Part 2

Continuing on from yesterdays post; the sessions reinforced to me how little of the core Gartner content is aimed at web companies. Almost all yesterday's stuff was only really relevant to 10,000 seat corporate enterprises. I've worked in several big companies before, and the internal "business process support" technical challenges they face are very different to our external product-led technical challenges. We also don't tend to have such epic workforces and supply chains, we tend to be pretty lightweight and entrepreneurial by comparison.

Back on track - the economic downturn. It's kind of uncomfortable to think about the upside of something that is affecting so many so negatively, however we're well positioned to take advantage of the opportunities that are coming to the surface.

With a battening-down-the-hatches mentality, people are preparing for the worst. Sites like Fool and Lovemoney pick up as we try to get as many money-saving tips as we can. The first preventative action we're likely to take is reviewing utilities and other basic household expenses, and sites like Gocompare and Moneysupermarket see a lot more visitors.

On top of this, there are a whole slew of creative new ideas starting to come through, all predicated on our current need to squeeze a little more value out of things we otherwise tolerated waste in.

At the worst end of the scale, people are being made redundant. Again this is a terrible tragedy affecting people I know, and it feels unfair to be thinking about the opportunity at a time like this. Nonetheless, the internet is now the de facto tool of choice for finding your next role, and this means more actives on Jobserve and Monster. Whether you've been affected yet or not, it is now more important than ever to get an up-to-date profile out there and build up strength in your network (before you need it). Off we go to the professional social networking sites like Plaxo and Linkedin.

Lining my own pockets for a moment, online gaming and gambling is fairly recession-resilient, but even simple ad-supported business models can do OK. You're still turning page views and actives into revenue, and if you're providing the service or content that's currently in demand, then that's not insurmountable in any times.

The Good and the Bad of Google Product Management

I don't often pick up stuff from the news - I don't really consider what I write here as commentary - but this one I couldn't resist because it's such a good segue into something I wanted to talk about anyway.

In a way it's a little sad to think that these harsh economic times are starting to have an effect on even that most hallowed of engineer-driven organization. Still, there are those of us probably a little less melancholy - I can just imagine how much fun serial cloud hater, John C Dvorak, is having - "told you not to keep stuff in the cloud, one day they just switch it off and then what do you do?"

The decision to shelve a product is a difficult one, and one that inevitably leads to some degree of unpopularity both internally and externally. What I wanted to do here was take a look at what this article tells us about product management practices in general.

Good Stuff

• Willingness to take risk. Especially in these times, it can be tempting to stick to safe territory. That can be good advice, but what are you missing out on? Another way to look at it is when everybody else is playing it safe, it’s easier to stand out with something unique.
• Recognise the capability gain in products even if the product itself doesn’t work out in its current incarnation. Taking what they gained from Dodgeball and turning it into the Maps add-on Latitude is a great example of the sort of utility you can get from something that doesn’t work out in its initial form.
• Customer feedback. Accessing the community through techniques like product development blogs and using that knowledge to shape the features and user experience is a powerful way to make sure your system ends up meeting expectations as closely as possible.
• Attitude to failure. Viewing things that don’t work out as contributing to overall success rather than detracting from it helps you really internalise what you learn and encourages the innovation that fear of failure will crush.
• Internal testing. Eating your own dog food is always helpful, after all, if you find your product difficult to use then what are you asking of your customers?
• Opportunity cost. I liked the part where they consider the opportunity cost of a project as well as its real cost; i.e. what is the value of the things that the engineers are not working on because they’re working on this?

Bad Stuff

• Incentives. Paraphrasing, the article talks about difficulty attracting engineers to work on certain products. There are some reasonably well publicised shared reward schemes in place, and in this free market-like environment management needs to consider how it might need to provide external subsidies to make sure that less glamorous, but perhaps strategic, projects get attention too.
• Perpetual betas. I really couldn’t decide whether this one belonged in the good or the bad section, so I let the need for balanced paragraphs drive the decision. On one hand, I think there is no better way to refine user experience and focus the tail end of your development efforts on polishing the things users like the most, but on the other hand there are circumstances under which you need to show the commitment of a production-ready seal of approval.
• Communication. While not exclusively a product management issue, communication across geographical boundaries was a major issue here. This shows how important collocation is in a product development process.
• 20% time. I think we’ve all heard mixed reports about how this really works, but regardless of that, if you have a product that you’re committed to and have set targets for then you simply must formally assign it resources. Relying on product management to recruit people from their 20% time is setting it up for failure.

The article also mentions internal performance targets for new products, and in all businesses objectives and measures are the true meat of how product decisions are driven, so it's a shame we didn't learn more about that. Also some kudos due for how they ended Jaiku, the Twitter clone, it's good to see them pass that on to the community rather than just confine it to the shelf.

Welcome to 2009

It's been 2009 for a little under 2 weeks now, and so far I can confidently say it will continue to be for around another 50 weeks or so. OK - I accept that's not my A material - moving swiftly on...

At this time of year it's customary to make a few new year's resolutions - fresh commitments you somehow feel renewed fervor towards keeping just because we incremented the year counter by 1 - and they almost always involve gym memberships.

If I take a brief look back at last year's resolutions I think I can safely say I hit my top priorities, but just as predicted, the exercise part seems to have slipped through the cracks. No matter, that's what these annual counter-flips are for, getting invigorated about your health for 30 days or so.

I'm always interested in hearing from people out there in the technical community for any reason, and from time to time I am lucky enough to receive some feedback on my posts (thanks by the way!). Over the last 12 months the articles that have generated the most interest have been on management, dealing with people, and being effective in organizations. It seems like you want to hear more about getting things done around technology than about the technology itself. Well there's some resolution input right there...

This year I'm going to try changing the balance of the content I write, aiming for a higher ratio on project management and professional disciplines without completely dropping the things I like to bring up on architecture and computer science. After all, a hundred little things happen every day, and it's just a matter of being a bit more selective about what's captured and shared.

So yeah, welcome to 2009, you can make it whatever you what you want it to be.

Constraints and Creativity

Does the greatest creativity come from unlimited choice; a totally blank canvas with every possible option and no restrictions, or from difficult circumstances; where pressure is high, the problems are acute and all the easy avenues are closed?

History has proven necessity to be the mother of invention time and time again, and I've personally seen a lot of otherwise-excellent people become paralyzed when faced with too much opportunity.

The current economic circumstances are bringing this into somewhat sharp relief. From time to time I get approached by other entrepreneurs with an idea they want to develop - usually only a couple of times a year, but in the last couple of quarters I've seen half a dozen. There is something about this credit crunch...

Conditions are right for the sorts of things I've seen; internet based, self-service, person to person applications that look for efficiency by cutting out middlemen and using smarter fulfillment. They're all great ideas and I really wish I had the time to help them all out.

Returning to the point, would you get this kind thinking in times of plenty? Who knows, maybe something good will come out of this credit crunch after all.

Tradeoffs - You're Doing It Wrong

Our time is always our scarcest resource, and most of the web companies I know are perpetually bristling with more ideas than they could ever manage to ship. We always wish we could do them all, but at the same time we know deep down that it isn't realistic. Under these circumstances, tradeoffs are an inevitable and perfectly normal part of life.

The secret to doing this well is in what things you weigh against each other when deciding how to spend the resources you have.

It's so easy to start down the slippery slope of trading features off against the underlying technologies that support them. Fall into this trap and you'll get a small surge of customer-facing things out quickly, but stay with the pattern (easy to do - as it does reward you well in the short term) and you'll soon find yourself paralyzed by technical debt; struggling against continuously increasing time to market and building on top of a shaky, unreliable platform.

Instead, try trading off a feature and it's supporting technology against another feature and it's supporting technology. This isn't a carte blanche ticket for building hopelessly over-provisioned systems - it's simply about setting yourself up in the most appropriate way for future requirements and scalability, rather than foregoing whatever platform work that would have been diligent and starting off behind the curve.

Trading features off against their underlying technology is not just short sighted, it's also unrealistic. You see, the fact is that all customer-facing features have to be maintained and supported anyway, no matter how badly we wish we could spend the time on other things. And right up front, when we first build each feature, we have the most control we're ever going to have over how difficult and expensive that's going to be for us.

This is a philosophy that has to make it all the way back to the start of the decision making process. When considering options, knowing the true cost of ownership is as vital as the revenue opportunity and the launch cost. Remember - you build something once, but you maintain it for life.

Cloud Computing Isn't...

Thought for the day - when does a new idea gain enough structure to graduate from meme to tangible concept? Is there some quorum of 'experts' that need to agree on its shape? Perhaps we need a minimum number of books written on the topic, or a certain number of vendors packaging the idea up for sale? Or maybe it is as simple as all of us trying it our own way for long enough for observable patterns to emerge?

We might have already crossed this bridge with cloud computing thanks to the accelerated uptake of robust platforms such as EC2 and App Engine (and the adherence to theme of newer offerings like Azure), but there is still a lot of residual confusion that we might start to mop up if we were so inclined.

The first thing we might stop doing is retrospectively claiming any sort of non-local activity as cloud computing. What's that? You've been using Gmail or Hotmail for years? No. sorry. You are not an ahead-of-the-curve early adopter, you are just a guy who has been using a free web based email system for a while.

Before the inevitable torrent of angry emails rains down upon my inbox, let's pause to think about what we're trying to achieve here. Does classifying the likes of Hotmail and - well, insert your favorite SaaS here - as cloud computing help or hinder the adoption and development of cloud technology? I think we probably establish these analogies because we believe that the familiarity we create by associating a trusted old favorite with a radical new concept may add comfort to perceived risks. But what about the downside of such a broad classification?

These systems are typically associated with a very narrow band of functionality (for example, sending and receiving email or storing and displaying photos) and are freely available (supported by advertising or other 2nd order revenue). They tend to lack the flexibility, identity, and SLA that an enterprise demands. This analogy may well be restricting adoption in the popular mind. Besides, where do you draw the line? Reading this blog? Clicking 'submit' on a web form? Accessing a resource in another office on your corporate WAN? I'm not knocking anyone's SaaS, in fact the noble pursuits that are our traditional online hosted email and storage systems have been significant contributing forces in the development of the platforms that made the whole cloud computing idea possible.

So, if a lot of our common everyday garden variety SaaS != a good way to talk about cloud computing, then what is?

Let's consider cloud computing from the perspective of the paradigm shift we're trying to create. How about cloud computing as taking resources (compute power and data) typically executed and stored in the corporate-owned datacenter, and deploying them into a shared (but not necessarily public) platform which abstracts access to, and responsibility for, the lower layers of computing.

That may well be the winner of Most Cumbersome Sentence 2008, but I feel like it captures the essence to a certain degree. Let's test our monster sentence against some of the other attributes of cloud computing - again from the perspective of what you're actually doing in a commercial and operational sense when you build a system on a cloud platform:

• Outsourcing concern over cooling, power, bandwidth, and all the other computer room primitives.
• Outsourcing the basic maintenance of the underlying operating systems and hardware.
• Converting a fixed capital outlay into a variable operational expense.
• Moving to a designed ignorance of the infrastructure (from a software environment perspective).
• Leveraging someone else's existing investment in capacity, reach, availability, bandwidth, and CPU power.
• Running a system in which cost of ownership can grow and shrink inline with it's popularity.

I think talking about the cloud in this way not only tells us what it is, but also a little about what we can do with it and why we'd want to. If you read this far and still disagree, then enable your caps lock and fire away!

Don't be a developer, be an Engineer

I do part time lecturing engagements in local universities whenever I can make a break for it from my day job - dodging and weaving through a kind of webscale Logan's Run while being hunted down by product owners and various other stakeholders, each individually as determined and unyielding as a T-800 on steroids, yet all the more terrifying for their emergent pack behavior. It's nice to have friends.

Over the last year I've made a quite a few such regularly scheduled escapes, as a result of which, I've noticed a disturbing trend. It can be summed up in one simple sentence; people are being taught to be developers, not engineers. Let me tell you a little more about that, and why I think it's a bad thing.

Developers vs Engineers...

Firstly, we need to appreciate the difference between a developer and an engineer. Again simple to summarize; a developer writes code, an engineer solves problems. I think it helps to define a developer in relation to an engineer, as I think "developer" is a subset of "engineer", so:

Engineers work as part of the business, helping to define the problem, helping to flesh out the specification, and bringing the practicalities and constraints to the table. They are creative; coming up with a number of technical solutions which solve the business problem, yet commercially savvy enough to evaluate the candidate solutions against the organizations goals to determine the best one. Engineers also appreciate that when it comes time to descend upon the keyboard in anger, writing the actual code is one little part of the whole job; they need to keep themselves and the rest of the team on track, report on progress, manage stakeholders such as operations, external vendors, and IT support staff, and plan for the release, version control, maintenance and quality of the system they're building.... On top of all this, good engineers are also constantly looking at their tools and their environments; working out how to make their own lives easier by improving the build systems, repositories, and automation that supports their work.

Developers take a tightly specified piece of functionality and express said functionality in whatever programming language their proficiency is in. You'll recall that we mentioned that as part of being an engineer.

What's wrong with this...

Is this necessarily a bad thing? It doesn't have to be, but here is why I think it is; we're not increasing the intelligence of our environments at the same speed as we're reducing our appreciation of the complexity they mask. Let me expand a little more on what I mean by this; by the time you drag together a new class in your favorite java IDE a massive amount of stuff just got done for you - even assuming you haven't included any non-essential resources from the toolkit. Machines will interpret your instructions so that they're executable by a given CPU. They'll manage the memory allocation and reclamation. They'll keep track of state and they'll write data for you. they'll take care of IPC and RPC and even hook you up to a database if you want. They'll put things onto the network stack and pull them off, and when worst comes to worst they'll give you a whole bunch of debugging information to ponder. Easy. But this kind of ready-to-roll stack doesn't just occur naturally in the wild; people made these machine-helpers. Those people were engineers, most certainly not developers. So, until machines can make us continually more advanced machine-helpers, we'll need a healthy population of those people to keep us going. Sooner or later everything we do is gates, voltage, magnetism, and wavelengths of light.

If a long term gloomy vision of the future doesn't do it for you, here is something quite selfish to get upset about. Being a .net or java developer is pretty much guaranteed to get you a job somewhere pretty quickly after graduating - but what next? While you're immediately productive from the first day, a lack of appreciation of the science behind what you do (how computers store and retrieve data, how memory is referenced and logical operations performed) will hold you back. You'll always be up against a steeper learning curve than your compsci-based compatriots, forced into learning about new things as a delta from what you know (an implementation like .net), rather than as their own unique systems - and that's before we even mention the wider appreciation of technology as a whole required to make a decent architect (or even make good implementation decisions). That's a serious competitive disadvantage in the workplace.

How did we get here...

We'll I'm not entirely sure we're "here" yet, but I tell you what, we're not far off. This is a trend thats getting stronger, and I think there are 2 key causes:

Firstly, people value immediate employability over long term careers. We're in competitive times, and now more than ever people view their classmates as competition for the next big step - turning that degree into gainful employment. Courseware is changing to accommodate this. We also live in times where people change jobs every few years - and total career changes are becoming very common and happening later and later in people's careers. This means there is a lot less incentive for students to consider the longer term, bigger picture when planing their studies, after all, they might only be in software for a couple of years and then become inspired by writing, or groundskeeping, or zoology. So are students and professors right or wrong here? Perhaps individually they're doing the right thing, but that doesn't mean that the combined effects of their decisions isn't going to hurt our trade.

Secondly, there is an imbalance of power between academic and commercial concerns. We are seeing unprecedented levels of cooperation between the academic world and the commercial world - which I totally, wholeheartedly, endorse by the way - there are joint research projects, internship programmes, technical community workshops and speakers, and a whole bevy of activities which help keep technology relevant and give willing students the ability to see how the science (key word there) they study can be applied to solve real life business problems. Sounds awesome, so why is it going wrong? To put it bluntly, I think academia are bending over for companies. Organizations are obviously only interested in participating in (or sponsoring) the aforementioned events if there is something in it for them - I know, I do it (responsibly!) myself. The most immediately obvious benefits are recruitment (hey, we have to work to get employees these days) and the ability to influence study programmes to favor the needs of the company in question. Here is where I think the academic side of these partnerships has a duty to push back - a balance needs to be maintained between what companies need today and making engineering, as a profession, sustainable. If we load all our papers and study plans to suit next quarter's recruitment targets where will we be in 10 years? I can see that this is the path we're starting down.

Can we save ourselves...

Sure we can - all is not lost. We might not be able to do much about the social pressures that form inputs into people's study plans, but we can at least help them appreciate the value that sound logical thinking, reasoning, and research/presentation skills brings to any career choice.

The single biggest thing we can do today is get the message to tomorrow's engineers - focus on becoming an engineer; being a developer will come naturally after that. You might have to study a bit harder, and look more strategically for your first few roles, but believe me - you are in the strongest possible position after you and your non-engineer peers have had 5 years to average out in the real world.

Faculties need to take action too - don't forget what you're for. It's always nice to have companies interested in working with you but you have a duty to maintain that balance. It isn't in the best interests of any organization to do long term damage to the core capabilities of the engineering industry, so you have to make them understand that's what they're heading for. If they are even slightly responsibly technical citizens (and I can't think of any I know not to be) they'll appreciate this, know that it is in their own long term best interests, and support it. Don't sell papers.

Rise up, good engineers, for we are becoming few!

Would you do it in real life?

Caution: may contain traces of rant.

Guess what, the internet world is just like real world.  The dotcom bust was a harsh teacher, before then we believed anything would make money if it was done on the web; and we learned that this was not so.  We learned that even when you run a business online you still needed a solid business plan, you still needed to understand your market, know your customers and control your costs.  In other words, we learned that you still needed to run a proper business.

I am starting to wonder if that lesson stuck.  We received the message from an investment point of view, but how has that translated operationally?

Additional customer-facing channels for traditional operations is a high growth area of the web.  Banks, utilities, transport, retail, and a whole bunch more - all these things started almost exclusively face to face, some had a postal 'interface' with their customers, they added call centers as telephony matured, and now the web is the next way in.  Most companies did pretty well when adding call centers to their repertoire (heavily accented outsourced operators you can barely understand excluded), but in the jump online, a lot of them are much less successful in my experience.

My reasoning is as so; it looks to me like many organization decided, for some (presumably well-researched) reason, that their web booking/ordering/support/purchasing process should be different to their stores and call centers.  Why?  I'm the same guy, I've been to your branches for years, called you contact center, filled in your forms and met your staff.  I like the internet, it's the most convenient channel for me, but why can't I do the same things the same way?  Why isn't my experience of your operation consistent?

An example from yesterday.

I booked an overnight ferry trip through the web.  As part of said booking process, I was offered the option of booking dinner at one of the onboard restaurants, and receiving a small discount when paying for both ticket and meal together online.  The Scotsman in me was totally sucked in by this and I went for it, but alas, it was not to be.  I was offered a choice of 4 restaurants on the boat, and each time I selected one, I was informed by the site that it was fully booked.  Not being fed?  Yikes!  Nonetheless I girded up my loins and prepared myself for a hungry crossing.  As I wandered around the ship that evening (no wifi at sea...) I was greeted by the welcome sight of many empty restaurants.  I was pretty happy about that at the time so off I went for a meal.  Saved from the brink of starvation [OK people that know me, I admit this would have taken years], I began to wonder why I couldn't book online and lay rightful claim to my 10% off.  A brief investigation was in order and I swiftly conducted one...

Anyone who works in the industry knows that failures happen; mistakes, technical faults and data errors - these things I am much more than averagely forgiving about, in the hope that the great circle of BGP karma will ensure my users are with me.  But this was not the case; what I found was quite simply 2 easily-avoidable counts of a process that works well in real life but wasn't translated to the web particularly well.

Count 1 (the root cause); apparently it is not possible to make advance restaurant bookings less than 24 hours before the ferry departure time, as they cannot get the message to the boat in time, and I was booking for that very evening.  Cool - but why make me go through each individual restaurant one by one?  Why mislead me that they're fully booked, when the real reason is simply not enough notice?  In fact, why offer that step at all?  If I was being talked through my booking by contact center staff, would they offer to book a restaurant, then just tell me it's not possible when I accept?  No, so why do it online?

Count 2; when I called up, the helpful lady in the contact center said I should have called them during the online booking process and they could have clarified the reason (less than 24 hours notice) at the time.  Hang on, use the call center to check up on the truth of the website?  When I'm booking over the phone, should I be expected to use the website to check up on the validity of what I'm being told by the operator?  It sounds a little absurd when put that way around, so why engineer the system so that it's necessary this way?

OK machines are dumb, I'm cool with that, but you can do better than this.  It's not difficult to teach your system simple rules about these restrictions (if departure time < 24 hours from current time then do not show restaurant options page) or at least set the right expectations (display a 'not enough notice, please book in person on the ferry' message instead of a 'restaurant fully booked' message).  It's just good customer service.

So if you're considering taking a traditional process online and you're thinking about modifying it slightly for the web, first ask yourself if you'd do it in that way real life.  If it wouldn't make sense when done face to face or over the phone, then it probably isn't a smashing idea online either - don't put your customers through it!

Remember, if it's a dumb idea in real life, it's a dumb idea online.

The Law of Conservation of Complexity in the Business

Last month, I wrote about conservation of complexity and how it applies to how we design systems. It's also a factor in the organization too, for a company to achieve a certain result there is a minimum amount of effort that needs to be expended by someone. Just like with technology, you can make it a whole lot harder on top of this with excess bureaucracy and orthogonal activities, but there is a certain amount of minimum trouble people need to go to. If someone (or some department) does less, then someone else must do more, else the result will not occur.

A big picture example of this is how adopting agile software delivery, and sticking to the triangle, changes marketing and communications.

One of the fundamental tradeoffs you might have made if you're using agile is exchanging [perceived*] certainty about what will happen in the future for the flexibility to make it whatever you need it to be as you move forward.

This means it's much harder to make promises about exactly when exactly what features will be available. You can usually hardcode a scope and get it when it's done, and you can usually hardcode a shipping date and get what's finished by then - but both are a rare luxury, the exclusive playground of those with deep pockets, easy problems, and a lightweight attachment to reality.

This is greatly upsetting to marketing departments, because their role is to get the word out (ideally in advance), raise awareness, and generally get people into the site. When they can't have what they consider to be some pretty basic information, like an exact date when everything will be fully online with all bugs banished to the ether, that makes it hard for them to ensure cash is rolling in from day 1. They have to smarter, they have to be more creative in how they get the message out and they have to work more closely with engineering.

I agree that could be easier (read: complexity for marketing could be reduced), but guess what? It's hard (read: more complex) for me to predict exact shipping dates, ferociously defend scope, and cope with all unforeseen technology and human resource issues. It has to be hard for someone, and to me that's simply the law of conservation of complexity at work in the organization.

* that one's for you, Ewan.

There's something about Darwin

Biological computing isn't very well defined in the collective mind, and it means different things to different people. To me, it is more powerful as an analogy, a way of thinking about a systems overall behavior, than as any specific piece of technology.

I'm going to argue that there are 2 fundamental tenets upon which systems can be designed and built; mechanical and biological. Both start from a different mindset and both exhibit different basic capabilities that constrain what sort of things can be achieved within a given effort. I'm also going to postulate (triple word score!) that a system exhibiting biological characteristics is much more suitable to web-scale computing than it's mechanical counterpart.

Before we get into the meat of it, let's do some simple word association:

Forgetting completely about software for a minute (we'll deal with the analogies later), what comes to mind when you think about a mechanical system? Probably things like gears, cogs meshed together, reliability, tight integration, close supervision, a single contiguous chain, unthinking automation, dependency, predictability, and control.

Again leaving software out of the picture, what comes to mind when you think about a biological system? Probably things like change, iteration, flexibility, loose relationships between units, unpredictability, variance, the hive over the individual, response to environment, awareness, and cooperation.

The mechanical analogy is how we've all traditionally built computer systems. It kind of makes sense, we've been building mechanisms for a very long time and the logical, mathematical basis of computer science lends itself to this kind of thinking - representing in software a chain of cogs, each coupled to the next, powered by and dependent upon it's neighbor.  Look inside an old watch. Now take something small and seemingly insignificant out. Still working? Thought not.

A mechanical type of system has a number of 'moving parts', all of which must be functioning for any of them to function. The system is very easy to observe and very easy to predict - you'll pretty much be able to guarantee what state it will be in under any given circumstances. Information will flow through this 'production line' from one step to the next, operated on with monotonous exactitude. You'll have to keep a close eye on it though, because no steps in the chain will ever be aware of any others, and therefore can't make a decision about whether or not it's safe to pass their output on to the next step (will it be lost?) or whether they can trust the input of the previous step (is it corrupt?). Luckily it's pretty easy to observe, but unluckily it will more than likely need manual intervention to 'realign the teeth' when a gear goes bad. Scaling is tough too, you can always use bigger cogs but everything will only ever go as fast as the slowest wheel.

Recently we've recognized some patterns of behavior that occur in the natural world that would be of benefit in running a large scale computer system, and at the cost of increased complexity, we can replicate these in software just as we replicated a chain of hardwired moving parts.

The biological analogy is a way to build systems that are aware of their environments and their fellows, possess a degree of self-governance, and can respond 'intelligently' to changing circumstances. Look at a line of ants. Block their path, take some away, move their food source. Still working? You bet.

There are a couple of equally valid analogies to draw with the natural world; firstly observe how a single organism works on the cellular level. What matters is the whole animal functions competitively, individual cells are irrelevant, but they all work together to keep the 'system' alive. They have a way to replicate themselves, kill off corruption, heal 'faults' and modify their role as requirements change. They don't need externally monitored or controlled by a central point - they all have a little bit of this distributed amongst their number. Now imagine that animal is your system and the cells are servers, nodes, units of functionality - whatever it is that makes up your system. The other way to think about biological computing is to study social insects, cooperative hunters, and hives/swarms. You end up in pretty much the same place, a group of individuals which make up a whole with the whole being more valuable than the individuals, I think which one you prefer depends on where you went to school (or how much animal planet you've watched). It can be easier to think about bees making conscious (instinctive?) choices, communicating and reaching a consensus, than trying to divine meaning in the more mysterious chemical reactions that govern the cellular world.

A biological type of system is made up of a collection of disposable nodes, the sum of which is the whole system. Nothing is more important than the survival of the system, and these 'cells' must change purpose and even 'die' to keep the system going. There is no central government or master nodes, all the 'cells' are peers and the administrative tasks in the system are distributed amongst them. The system is capable of making simple decisions for itself, responding to environmental stimulus like load, failure conditions and available upgrades. Many of these decisions are reached by consensus, using protocols which create a 'hive mind' pulling together the opinion of every relevant individual and answering questions such as; is a feature down? Where am I? Am I the right version? What are my neighbors doing? Who will service the next request? Is this safe to eat?

Biological systems are more difficult to observe and their behavior becomes harder to predict the further into the future you look. Their decisions are only as good as the data they have and the rules you give them to assess it with - so you can end up worse off than if humans manually configured everything. A cellular system is composed of a number of small, totally independent pieces of functionality, and the upsides to this are scalability and partial failure. Scale comes through being able to arbitrarily add more 'cells' to your 'organism' as it needs to get bigger and bottlenecks are easier to solve as every individual component is able to operate as fast as it's legs will carry it (these designs are usually asynchronous). Partial failure means that even if all the nodes that make up a certain feature are down, your system as a whole will still work sans that one bit of functionality. Self healing is when a system is aware of what it should look like (as a gecko I should have a tail) and is able to recognize when it does not (ouch I don't anymore) and take some corrective action (better grow it back). This itself is a double edged sword, imagine you're intentionally taking something down because of a security flaw; you have to give yourself some way to prevent the page springing back up somewhere else. The 'split brain' problem becomes even more significant; usually in a split brain you face potential inconsistency with data being written in more than one place. With a system designed to repair itself, you might just end up with 2 complete copies working independently - that may not be all bad (depending on how your system works) but the ability to kill of duplication once connectivity is returned is something that needs addressed.

So now the previously promised postulation (I am good at scrabble). We know we can build systems whose behavior is easy to predict, they are easy to observe but they are rigid and require a lot of manual oversight and intervention. The mechanical way. We know we can build systems that are flexible, automatically resilient to environmental change and inherently scalable but they are difficult to accurately predict and can run away with themselves if not given the right information to from which to make deductions about state. The biological way. In a web-scale environment where availability, scalability, and the capability to ship frequently are such vital attributes of any product, it seems to me that we benefit more from thinking in a loosely coupled, compartmentalized, organic way than an interlocked, highly dependent, production line way.

The Business Value of the Back End

In my experience, most companies are good at recognizing the value of customer facing, revenue generating features and not so good at recognizing the value of back end, platform features.  Which is a real shame, because without the platform there can be no customer facing features, and beyond that, if you invest in platform you can squeeze even more value from it through quicker time to market, more availability, and growth at better marginal cost.

I can see why we're in this situation - it's easy to judge the business value of customer facing features; you can count revenue, wallet share, registrations/conversion, ARPU and stickiness quite simply.  Capability, which is the primary deliverable of platform, is tougher to describe in measurable ways.

Perhaps at this point we could just agree to disagree, but it's not that simple.  As engineers, we know that there is a certain amount of effort we have to spend on platform just to keep the business running.  We also know there is a certain amount of effort we should spend on platform, which would result in tools that lead to better quality.  And, to be fair, we also know there is time we could spend on platform that would take us into diminishing returns.  Ensuring the business commits to the have to effort and helping to get better trade-offs for the should do investment is why we can't settle for mutual incomprehension - we need to get better at selling this.

It should be easy, because there is business value in the platform things that we, as engineers, want to do - after all, if there wasn't, we wouldn't be proposing them.  To me, the answer is in how we describe what we want to do.  If we can articulate the business value better then we level the playing field between features and platform.  This enables decision makers to compare apples with apples when deciding how to spend the organisations resources.

Some simplified examples:

• Refactoring some old products into loosely coupled services = a more granular failure model = more consistent service to customers (you can't take revenue through a 404).
• Improving the throughput of a database server = quicker market settlements = customer funds returned sooner (they'll trade again more times that day).
• Separating UI components and content management = more front end flexibility and editorial control = more marketing and promotional opportunities without software changes.
• Providing a simple, abstracted data storage and retrieval interface = less complexity for engineers building new products = quicker time to market for new features.
• Implementing a messaging infrastructure which presents core components as APIs = more reuse and isolation of features = more concurrent workstreams and easier outsourcing (because of the natural boundaries created).

We can never guarantee people will make the decisions we consider "right" but if we use the language of the business when we're describing the platform capabilities we want to build we'll at least have removed the primary barrier to understanding.

For any web business to work, everything has to be a balance of customer facing features and back end capability - the key is that balance being determined by a true comparison of value.  I think we'll know that we're on equal footing when capability gains are celebrated when released, and pursued when missed, with the same ferocity and dedication customer facing features are.

Tweet Tweet

Today I decided I'm going to play with twitter; so I signed up like so.  I can't really picture myself having the time to twit (or tweet?) every few hours, I signed up for slightly less conventional reasons...

It all stems from how I use this blog.  When I started blogging, I foresaw a channel for my original thoughts, a way for me to share my experience in the industry - real life problems and solutions from the world of running webscale engineering.  It was my opinion and what's worked for me; I didn't want to get into simply posting links to other people's opinion, unless I can substantially build on them, and thus add some value to the idea being discussed.  I wanted there to be some substance, some usefulness and some kind of conclusion to what you read here.

I have stayed true to this vision, but these days I am increasingly coming across content I want to share in a briefer 'check this link out' kind of format.  This of course poses the question; do I dilute the purpose of this blog by posting shorter, less meaningful messages with links or embedded external content or do I find another way (or, that oft-forgotten option we always have, do nothing)?

A tool that seemed fit for purpose to me was micro-blogging.  Small snippets of text pushed out as regularly as you see fit and a whole culture which prohibits verbosity (how will I cope).

So I'm going to play whack-a-link on twitter.  Anytime I see something I like or agree with (but don't want to more formally expand on) I'm going to demonstrate my support for it by posting the link into my feed.

For me this is one of those kind of experimental things - start using the technology and see what value emerges.

While we're talking about twitter I want to give the fail whale an honorable mention for achieving the pinnacle of error message accomplishment - being a popular sight.  No kidding.

Failure joins death and taxes in the hallowed halls of unavoidable inevitability.  I spend a lot of time working out how to detect it, avoid it, and recover from it but sooner or later it gets us all.  This is where architecture stops operations and customer service starts; personally I consider the likes of web server 404's and 500's to be the middle finger of the internet - if you can turn these into disarming, apologetic messages then you'll at least have a chance to keep your customers on your side while you work out your issues.

The fail whale is almost too good at this.  It's grown into some kind of phenomenon of it's own.  People have made fail whale models, you can buy fail whale t-shirts and mugs, and there is even a fan club.  Remember; this is a holding page they show when their site is down!

SAI 25

A while back the SAI 25 was published on Silicon Alley Insider and we're sitting at number 4.  There is a reasonably scientific approach to how these companies are measured and, based on that, I can see how we're climbing the list.  Growth, margin and market share are all strengths of ours.  Kick ass.

Do Some Good

One of our engineers posted this link on our internal forum and made a spirited appeal to our charitable sides in an attempt to stir up some volunteers.  I like to help good causes (we all have our favorites) and I would always rather donate my time - help out in a practical way - than simply giving money away.  I get more of a sense of satisfaction doing that and I don't feel like I've 'bought' my conscience off!

I’ve helped some charities in a similar capacity in the past and I have to say I found it curiously rewarding in a way I wasn’t quite expecting.  But even if you're just not that charitable then among the many, worthwhile philanthropic reasons to help out like this there are also some perfectly good selfish motivations - or maybe symbiotic might be a better word:

The thing you have to remember about charities is they are almost always terminally short of resources – most significantly cash and people – yet they still have the same IT challenges that a lot of small/medium businesses have. That means you’ve got to work with constraints you won’t be used to because you can’t just buy hardware, you can’t just use something commercial or licensed and you [often] can’t use expensive network connections or hosting. This means you have to be really creative with what you put together and you also have to exercise your end-to-end solution muscles because, chances are you won’t be able to assemble a reasonable team either – it’ll be all down to you.  This will be a totally different environment for you to learn to be effective in because most of us are in the fortunate position of having a budget that’s appropriate to the problems we're trying to solve.  Sometimes this 'plenty' can make you lazy – necessity is, after all, the mother of invention.

I guess the summary is you'll get exposure to a very different size and type of problem to the one you're used to working with every day and, because of the unique constraints, you'll really get to exercise your problem solving skills.  So give it a go, you’ll probably find it quite refreshing and might even learn something too.

Definition of Scalable

We talk a lot about scalability but what is it that we really mean when we refer to a system or service as scalable?

"A service is said to be scalable if an increase in system resources results in a proportional increase in performance."

To webscale computing increased performance typically means serving more units of work (pages, TPS) but it can also mean larger units of work (bigger datasets, many-where-clauses).

The main reason I like this definition of scalability is it separates the scaled from the scalable.  I've seen plenty of really ugly systems go big - but vertically and at ludicrous capital costs.  Yes, you managed to squeeze scale that thing but that doesn't automatically earn you the right to refer to it as scalable.

To me, scalability is an economic thing as much as it is a technical thing.  You have to build wide and grow complex IP across a commodity platform - but if you can't maintain (reduce!!!) marginal cost while you're at it then you still haven't earned the right to call your system truly scalable.