The power and the pitfalls of test and learn

My core philosophy on product is that product/market fit is a journey; it is a function of discovery and learning over time, and it is impractical to expect to be able to make a set of ideal decisions up front (i.e. in advance of any product development and real feedback from real customers).

We now live in the world where the fast eat the slow, and I am going to argue that learning is speed:

At Hotwire we’ve dramatically improved our business performance in the last year by focussing on learning as a primary goal.  It’s a first order metric, along with the ‘hard’ KPIs most businesses are familiar with (revenue, room nights, activations, etc) but - as any good teacher will tell you - measuring learning is extremely hard to do.  Fortunately for us, there’s another feature which is both easy to observe and highly correlated with learning: experimentation.

Whenever you see a high rate of improvement (in nature and in science) you will almost always see a high rate of experimentation.  There’s a great Thomas Edison quote which goes something like; “None of my inventions came by accident. I see a worthwhile need to be met and I make trial after trial until it comes.”  Most of our early learning as human beings is heavily experiment driven; it is interactions with the world that teach us how to behave effectively within it.  But my favorite story about the value of pursuing learning as a primary goal is the Kremer Prize:

In 1959 Henry Kremer, an industrialist and patron of early aviation, established a prize of £50,000 for the first human-powered aircraft to achieve a controlled flight.  Hundreds of attempts were made over 20 years with no success.  And most of these were well-informed experts; aviation companies and universities etc.  The prize went unclaimed until 1977 when Paul MacCready won it with his Gossamer Condor.  The secret to his success wasn’t being smarter than any of those who previously attempted flights or knowing something they didn’t, it was his approach to the problem being fundamentally different.

Most teams spent months designing and building their craft, then they took it out to a field or an airstrip to try it out, then they crashed, then they swept all the pieces up and returned to the hanger for another few months to rebuild.  MacCready focused not on the airframe that would work, but on a cheap construction which was easy to assemble and disassemble by hand in the field.  Using this he was able to try more designs out in one single day than every preceding team added together for the entire previous year.

His formula for success had 2 main features; variation and repetition.  You can see this in nature too; natural selection tries out variations of organisms over and over again (optimizing to the ultimate KPI - life itself!) as those organisms improve their suitability to their environments.  While not strictly mathematical, Fisher’s theorem is a useful formalization of this (and the other examples we’ve discussed here).  It goes something like this; “The rate of increase in fitness of any organism at any time is equal to its genetic variance in fitness at that time.”  Or, more simply:

The capability to try out more hypothesis at any given time is highly correlated with faster improvement.

So learning is speed, and number of experiments is a useful approximation of learning.  But before you just count all a/b tests etc, there are some subtleties to success here:

The first thing I like to watch out for is confirmation bias.  As you embrace test and learn and make experiments cheaper to run you lower the bar for organizational participation.  This is unquestionably another benefit - harnessing the innovation of a larger slice of your org - but not everyone is as academically disciplined as you’d hope, and there is an underlying human tendency to like our own ideas and see them in a less-critical light than the ideas of others.  A while ago I was fortunate enough to be able to spend some time with Alan Kay, and he told me that science is the process that stops people from falling in love with their own ideas.  This is more than just a principle; if you come up with a hypothesis which you’re trying to prove instead of disprove, you will tend to discount contradictory evidence (i.e. proof points that suggest customers do not like it) and waste a lot of time trying marginally different manifestations of the same core idea in the desperate hope that you can somehow make them love it.  You lose speed, not gain speed, this way.

A fun way to look at this is to examine the difference between scientific thinking and religious thinking.  Jack Cohen is another wonderful human being, with a vast amount to teach the world, with whom I had the pleasure of spending a little time.  He argues that; in religious thinking all that matters is how hard you believe, especially in the presence of contradictory evidence (it’s there to test your belief).  In scientific thinking, all that matters is how hard you doubt, especially in the presence of confirmatory evidence (easy answers there to trick you into overgeneralizing your observations).

People will naturally come up with ideas that they like, but good product hygiene is about coming up with ideas that your customer likes.  Rigor in testing is the ultimate arbiter which can make that distinction clear to you, if you let it.  If you’re not open to being wrong - in fact expecting it and actively seeking to make it so - then you cannot, by definition, learn.

Another big-picture mistake is confusing iteration with learning.  It’s a common problem in the whole ‘transition to agile’ world; break up a big-up-front-plan into a number of predetermined phases, label them ‘iterations’ and then cash in your huge cheque as a profound agile coach.  Starting with the same inflexible ideas and delivering them across a higher number of software releases has some benefits in terms of quality and system risk, but it does nothing to improve your product/market fit.  You had determined the level of product/market fit at the beginning of the project and you have not improved that fit over the whole, say, year you were working on it whether it’s one single release at end of the 12 months on a whole series of incremental drops every two weeks.  Learning is about regularly shipping something customers can touch, then watching their interactions with that thing.  Looking for where it enriches and where it detracts, and only then deciding on the final scope for that next iteration.  The point is that each iteration should reflect the learnings from the previous iteration and improve upon it with respect to the customer experience, and therefore cannot be rigidly determined in advance of that feedback.

To be inclusive and stimulate the organizational pathways for innovating, a pretty broad filter is useful in the beginning.  All ideas are valid, and anything can be reduced to a testable hypothesis.  But, as you execute on a test and learn backlog, selecting ideas which are both coherent with the existing product and carry a higher probability of resulting in an improved experience for the customer becomes important.  Think back to our Kremer Prize example; while MacCready focused on the ability to explore a large number of variants, he was not randomly experimenting with geometry in the hope that he got lucky and found something that would fly.  He was an engineer who ran an aerospace company and he had a detailed understanding of all the mechanics involved in lift and control etc.  In user experience terms that means figuring out what customers are sensitive to, and using those things as inspiration points for ideas.  On the internet these sensitivities are rooted in behavioral science (decision theory, nudge theory etc) and are often things like social signals (how many others have bought the same item or currently viewing this page), urgency messaging (this is a popular item or this deal expires soon), and recommendation (if you like x you might like y) etc.  Discovering what these sensitivities are for your particular product can get you more ideas aligned to the physics of your particular business.

When talking about ways to increase the likelihood of ‘winning’ tests I always like to reiterate the value of ‘losers’ too.  A losing test is essentially an idea someone had for a new product or interaction which, without the experiment disproving the hypothesis, would have resulted in a project that would have taken away valuable product development resources for no (or negative) return.  What did you save right there?  How much customer attrition did you avoid by staying away from that unpopular option?

There is much more to doing this well - a whole series of posts wouldn’t do it justice - but I like to focus on the quality of the thinking first.

So does learning == speed?  At least in the special case of how rapidly you can get to the right product/market fit and grow, it certainly does here.  Is the number of experiments a good heuristic for organizational learning?  If you chart the improvement in our real business performance and the number of concurrent tests over time, you see very similar growth curves, just as Fisher’s theorem predicted...

Steriods for Signups

Anyone who runs a business on the web should be intimately familiar with the registration funnel (sometimes called the recruitment or conversion funnel) so - except for the briefest review to frame the problem - I'm not going to rehash that here. Instead I'm going to cover some common and some not-so-common ways to boost registration numbers.

So, for the quick review, the registration funnel is basically the set of steps a potential customer traverses on their way to becoming an actual customer (converted) and typically starts with a site hit and ends with an active user account. What these steps consist of varies with the nature of each business and might include things like a shipping address or a credit check along with the basics such as username and password.

The problem every web business faces is that the registration funnel is essentially a very lossy process. For example; of every 100 site visitors, 50 will start the registration process, of that 50 25 will complete the process, and of that 25 10 might go on to place their first order. Even with the best husbandry many web businesses have single digit conversion rates. With marketing on the web becoming more sophisticated and SEO/semantics driving smarter access to content getting the numbers in the top of the funnel usually isn't the problem and - just like any other lossy process - there can be a big impact on the bottom line by improving the rate by just a couple of points.

So how can you juice this up?

The very first thing you need is data. Just like everything else you need to understand the flow and then you need to instrument the heck out of it before you should be fiddling with it. If you haven't already got this in place then stop reading now and get on with it - the rest of this is useless without insight.

Separate your registration process out in your head; this will help you understand and instrument it. Think through all the steps in your funnel make it as granular as possible - this gives you much more insight into where potential customers drop out and much more flexibility in how you modify the process in response. Crudely; if you're thinking visitor -> credentials -> personal details -> checkout then you should be thinking visitor -> credentials -> name and email address -> shipping address -> checkout. This doesn't necessarily mean more pages or more data for customers to enter (in fact the goal is the opposite) but it does mean than you can count these things individually to see exactly what bit of information your potential customers get reluctant about handing over and it allows you to stage things a little more (see below).

The general principle is to put the fewest barriers between your potential customers and your product as possible. This means looking at what information you're asking for, when you're asking for it, and how it's captured. In some businesses there can be constraints which have to be acknowledged; with a product which is regulated or controlled in some way there may be some external rules which can reduce your options - such as the requirement for identity verification or credit checking - but the keyword here is reduce. The registration pipeline in these types of business is often poorly managed for this reason but it doesn't always need to be; take the time to understand the details of your governance and you will usually find that you have more flexibility than you assumed. For example you might need to verify your customers identity somehow before allowing them to make a withdrawal but does that have to stop you from granting them access to the rest of your product entirely?

Nice segue into staging. One of my favorite techniques in businesses with more complex registration requirements is to focus on capturing the absolute bare minimum during an initial signup (usernames, email addresses, passwords, etc) and capturing the rest 'just in time' in line with access to features. For example; why not delay asking for a physical address until the first time a user reaches the checkout? Why not ask for payment information just before their first purchase? You don't need that data until those events and having to enter it all up front can be prohibitive especially when potential customers might not be 100% sure that you're the service they want. Also think carefully about when you want to trigger the 1st step of the registration process - there can be benefit to letting potential customers play with a little bit more of your product before hitting them up for some credentials (but you have to balance this with less complete early contact information and the risk of taking it too far and having fewer accounts because the majority of the value is available anonymously).

Usability is an important aspect here. Considering the number of screens in your signup process, the number of fields, and even the phraseology of the questions can improve your conversion rates.

Also consider the order that you're capturing information in; if you start with email addresses and you're automatically storing fields as they're completed - a highly recommended Ajax trick - then you have some actionable data to go with your telemetry. You might want to use it for a follow-up contact to see if you can claw a customer back from the edge of registration oblivion.

Another consideration when you're designing your forms is to keep in mind the sort of information users are likely to have close at hand when signing up for your site. The more esoteric (in day-to-day terms) the information you ask for is the higher the chances that a potential customer will need to go elsewhere to retrieve it during the signup process. Data always shows that anytime a customer leaves the page(s) for any reason the changes that they'll return and complete them drop through the floor. What might you not even need users to enter at all? You can pick up things like language and locale from the browser and tools like geo-IP; adjusting a probably-correct field from a list of common choices is far preferable to data entry.

While we're on forms; short and sweet is the way to go. Whenever you have to go 'below the fold' then you're better off splitting the form into multiple pages (otherwise it appears too daunting) and, when you're going across multiple pages, then label the progress. A simple 'page X of Y' can work wonders setting expectations.

And finally, if there is a relevant infrastructure for your line of business (meaning that you can trust it and that a portion of your users are likely to be members) you can consider using an identity service, such as OpenID, to implement a kind of 1-click signup and then just capture the additional information you need to operate the account.

Change it and see what happens - after all you're collecting all the data you need, right? If you're crafty enough then you can do some A/B testing with alternate pages or different text. As long as you're measuring the process you will really quickly find out what's better and what's worse.

Encouraging Profitable Usage

A couple of weeks ago a friend sent me this article from a mutual acquaintance at CCP. It's all about the continuous struggle that the MMO guys have keeping their in-game economies balanced amongst all the real-world trading of items and game currency.

I was particularly interested in the stats near the end - in a crackdown on gold farming they managed to drop CPU utilisation 30% through a 2% drop in players (banned accounts who were violating these terms). To paraphrase that; 2% of the player base [revenues] were creating 30% of the load [costs] and that my friends, is what I call unprofitable usage of a product.

When you read that last sentence it makes perfect sense, however so few of us on the web think this way. How often have you looked up your top workload creators (page impressions, transactions) and your top revenue creators (purchasers, subscribers) and compared the two? How sure are you that your costliest customers - that you are probably supporting a collection of CPUs to service - are not marginal or worse in terms of value to your organisation? With sites more data rich than ever (read: more back end work per view) and screen-scraping a more widespread way to acquire content it's worth working to discourage this behaviour or at least make conscious decisions to allow it.

This thinking should be taken all the way back to product design - on the web we tend to emphasise users/visitor/subscribers without really thinking through what constitutes profitable usage.

It's not just about nailing people who are out to exploit you either - as a long time data provider my biggest issues have usually been well-meaning customers with lazy integrations. If you're going to build a client on top of a web API it's quick and easy to poll the bejesus out of it round the clock and then react to data in the responses rather than, for example, get a schedule of 'interesting' data and dynamically set up threads pick up smaller changes to discreet items on separate and appropriate timers. Take trades for example; why hammer a Hang Seng Index Options feed after 4:15pm? Or how about our feed - do you need 5 prices a second for this weekend's football fixtures on Tuesday or would hourly pre-match odds and squad changes be more than sufficient that far out? A slightly trickier initial integration but a better and cheaper product (for both parties) thereafter.

And finally I've always found it more effective to encourage good behaviour than discourage bad behaviour. Sometimes this can be as simple as an SLA which guarantees a superb latency up to a certain number of requests and creeps out in bands beyond that. Other times this can be purely commercial - why not extend better pricing to customers who genuinely cost you less to service or a tiered revenue share to those who contribute more to the partnership?

Screen Scraping for Dummies

If you run a website with any sort of valuable content, then you are almost guaranteed to run into scraping sooner or later. Screen Scraping is more or less an automated program taking an impression of a web page and then parsing it to pull out some specific bits of data that the scraper is interested in - which theoretically is then stored or used in some other way.

The piece of software that does this scraping is commonly called a robot, or bot, and it is really just an automated web client that accesses and uses sites in the exact same way as it's fleshy counterparts, just with machine precision and repetitiveness. A bot may be a large and complex program running on a server with it’s own database etc, or as simple as a script running in a browser on a desktop.


This is typically regarded as undesirable behavior by many sites because, in most cases, it’s a source of load and associated with unprofitable usage. Whenever we draw a page impression, which we’ll do for every bot hit just as we do for every human visitor, we consume web server time and, worse, back end time. When we structure highly functional pages loaded dynamic content we can create a very engaging user experience, and all that functionality is built on plenty of back end work and data. When used at regular human pace that's usually OK, but under the relentless rate of hammering robots are capable of, it starts to become expensive.

If you've got a case of bots, you have to start by identifying them. Their repetitiveness and flawless precision is, in this case, their downfall and we can usually spot them easily through proper analysis of web logs - no human user is as mechanically regular, millisecond quick, and consistent in journey as your average droid.

Spotting droids isn’t too hard, but then you've got to decide what you're going to do about it. Most scrapers aren't malicious and often don't realise the headache they're giving you. In the first instance it's best to try some detective work, see if you can find out who they are, and get in touch. Domain registrars can be a great resource for this, but don't overlook the obvious - maybe they even have an account with you if data they're using requires signup to view.

Beyond that it gets tricky, and can easily turn into an IP address blocking arms race. With some caching finesse or smart layer 7 rules you can throttle bot activity to more palatable levels, or persist them to their very own node that they can thrash all day without impacting the experience for the rest of your users.

If robots are a very big problem for you, then try and take it as a compliment on the value of your data and perhaps consider publishing it via a productised API or feed - if you make it simple enough to consume, then most scrapers will willingly change over to a more reliable integration mechanism and perhaps even pay you for it.

The Parable of the Big Hands

This is an old story, retrieved from the dusty recesses of my memory, but it's as relevant today as it ever was. Names changed to protect the incompetent...

A number of years ago I was called in to help an integration company who had been struggling for a number of weeks with a manufacturing client. They had delivered a factory automation system, introducing computer-controlled fabrication and handling machines, replacing the manually operated mechanical kind. The project had been justified upon increased productivity and reduced material wastage - yet reduced productivity and increased metal wastage were being observed. Not really the results anyone was hoping for, customers blaming the vendor, vendor blaming the users etc.

My job was to "troubleshoot the new software" and "find the bugs" before the manufacturing company started firing live ammo.

Step 1 was to get a coherent problem statement. It seemed that in the vendor's lab everything went according to plan - sheet metal machines could be reconfigured in tens of seconds via a new software load - yet at the factory it was taking minutes and materials were frequently being cut to the wrong dimensions.

So what was different between the factory and the lab? I was assured that there was no differences at all - machines had been swapped, the exact same builds had been uploaded to the machines at both sites, error logs had been downloaded etc. But clearly there was a difference, otherwise it would be working (or broken) in both places.

I figured there must be something environmental - how else could the exact same machine, software, and configuration work perfectly in one location and not another. "So, tell me about this factory" I asked. "Never been there" was the answer (believe it or not). Ah-ha!

So off we go to the warehouse/factory thingy. We never even got through the tour of the floor before we made the critical observation - these metalworking dudes have really big hands! If your job consists of throwing around huge blocks of steel, cranking big heavy leavers, and turning giant bolts, then you soon develop a pair of mitts like shovels.

Why was this important? Quite simple. The keys on the small numpad-type keyboard that had been specially made for entering the program parameters on each machine were a standard keyboard size, and were therefore simply far too small for the bunch-of-bannans fingers of the guys that had to use it. These guys are hardcore welders and whatnot, they don't have our delicate little programmer hands!

So what's the lesson here? Know Your Users. It's as simple as that. You have no business even touching a keyboard (of any size) until you know who your users are, where they work, how they'll interact with the system, and what successful use is. Remember - you're not building the software for yourself or your business analysts, you're building the software for them. Get to know them.