Saturday 3 December 2011

Eachan's 5 Laws of Platform Architecture

Architecture - the shape of the system, the patterns used - is by far the most meaningful thing to get right in any system.  Architectural decisions have far more influence on the capabilities and limitations of any given system than the implementation decisions - frameworks and languages - ever can.

This effect is hugely magnified when you're building a platform, because you're creating tools and raw materials for other developers, and your decisions can enable them to easily build apps any way they can imagine or limit their options to a handful of pre-configured choices.

Side note here; the internet is becoming full of platforms.  It has never been easier to programmatically consume services on the web, even through interfaces originally only intended for humans, and combine existing data in new ways to create whole new experiences.  You might put an app out there for customers, which you consider to be a finished product, and then discover that someone else is using your app as a building block in their app.  One man's front end becomes another man's back end.  I think this is a good thing for momentum and creativity; allowing new takes on current ideas to be spun up and explored quickly and cheaply.

But if you intend to build a platform, if that's what you're explicitly setting out to do, then there are some design guidelines which will lend it greater commercial utility.  Not patterns per se, more like principles, which I'm going to call the 5 laws of platform architecture:

1st Law of Platform Architecture - the value of a platform is directly proportionate to the amount of work it causes to be unnecessary.

Most web businesses are more similar than they are different.  There is a set of web basics they all need (identity, profile, cart, payment, analytics...) and there are usually a set of line-of-business basics that most of them have in common (in my case that's things like inventory, price, geography, booking, weather...) and that's an awful lot of code that we're all writing.  If a platform solves the 'common problems' via a handful of simple web service calls, then partners using said platform can put the majority of their time and passion and investment where it will make the biggest difference to their business; building the unique features which distinguish them from the rest of the marketplace.  I like to ask "what don't my partners have to do because we already did it?" and I like to have a long list of things to put into that bucket.

2nd Law of Platform Architecture - ease of adoption maps directly to business momentum.

Most serious platforms are monetised in the same handful of ways, and in most cases the sooner you can have partners up and running the sooner you're counting revenue from them, ergo time to market for your partners is a lever in your own P&L and if you focus on ease of adoption you will bring on more partners, they will each 'go live' quicker, and it is cheaper than more traditional incentives such as offering improved commercial terms in exchange for faster rollouts etc.  The kinds of things you ought to consider here go beyond the simplicity of the services you expose (which must be easy for the most pedestrian of developers to grok - in the platform game you are penalised for externally-observable complexity) and into how much you invest in documentation, SDK and client code, and developer community support.  These things pay you back.

3rd Law of Platform Architecture - entrypoint depth multiplies the business flexibility of any given service.

Higher layer abstractions always infer business logic.  That's what a composite service really is; someone decided that for use case alpha, you always do X then Y then Z, and therefore aggregating calls out to services A and B and C solves that problem in a single call.  You have just created a simpler solution for alpha - which is an admirable achievement and it belongs in your platform exposed to everyone else who has use case alpha.  But let's say I have use case alpha prime?  A basically similar business process but with key differences that no longer require data from B?  That's why your underlying services belong in your platform too; if a client has some business logic which maps quite closely to the higher layer abstractions then that's great orchestration for that app.  If a partner want to do something a little different, then he might have to consume those composites and do a bunch of his own transformations on the data, perhaps even store the data himself, and then present it to the client apps through his own interfaces.  With the option of consuming services from lower down the stack clients can build unique apps on top of the platform with far less 'back end' work (stripping out the inferred but irrelevant business logic) on the client side.

4th Law of Platform Architecture - compute work becomes exponentially cheaper and faster as it approaches the edge.

At last something more technical!  If you're building a platform, and you're not planning on comprehensive global coverage, then I would question your commitment to your idea on a web where ubiquitousness has never been cheaper to claim.  So let's assume that you are doing this globally.  Without getting into a discourse on distributed systems, you're going to want to deliver straightforward and performant access to every consumer regardless of where they're based.  That is a distributed system problem, and so your platform should run on a system with a diameter greater than 1, and you should try to service requests as close to the edge (and as high up in your stack) as you can.  There are plenty of mature grid and edge computing infrastructures out there to provide the scaffolding for this.  What does it mean for your partners?  They're using a platform with better durability and lower latency than alternatives that they might choose, which translates into better SLAs and better service which also happens to be cheaper to deliver.

5th Law of Platform Architecture - as services become more atomic the addressable market segments become more broad.

My 3rd law deals with entrypoints, and this is nearly the same thing.  Lower layer abstractions allow platform consumption without constraining the use cases, but another critical property of those granular services is their atomicity.  Your low level granular services must perform generic, stateless (independent) operations, all be individually consumable, and each should be of business value individually.  When I say individually consumable I think I really mean individually usable.  If one has to consume a handful of other services to be able to make sense of the data returned from a single service, then it clearly isn't too useful in it's own right.  Perhaps it needs slightly more data or metadata around it or maybe it is just noise in your stack - keeping the catalogue manageable is actually a harder problem than it may seem.  When your lower layer abstractions are individually useful you find that more kinds of businesses can make use of them.  To round this out with an example; exposing our geography data distinctly (i.e. outside of a composite which applies it to a booking flow) enables us to power navigation and mapping apps, instead of just travel apps.

These are guidelines for architecting a platform to maximise its utility to developers and its commercial success as a set of business building blocks for a range markets.  But what about the shape of the system(s) behind each service?  That dimension of architecture - how you manifest each of your APIs - is just as important.

We all have our favourite ways of doing things, and pushing one pattern over another without an understanding of the business problem and environment is just bad science, so the best I can do here is suggest establishing some engineering principles to give engineers a decision making framework within which they can own the problem and still consistently meet the higher standards expected from platforms.  Scalability, Maintainability, Quality, Security, Availability, and User Experience have always worked for me.

Whatever you do keep in mind that if you operate a platform, then you're partly responsible for the availability and integrity and brand and revenue of n number of businesses (i.e. the partners building their apps on your stuff) so you have an even greater responsibility for quality than you do when your building your own customer-facing apps.  I'm a pretty casual guy about most things, but that's a responsibility I take very seriously indeed...

Saturday 5 November 2011

The difference between taking a shortcut and cutting a corner

There are two very different things to talk about here, with two very different implications for your software, but I'll admit that which way around you label them can simply be language pedantry.  So for the purposes of this discourse, let's say that:
  • Taking a shortcut is using your knowledge and experience in place of studying things from first principles every time.
  • Cutting a corner is a compromise in quality or good practices in the place of the prudent actions otherwise required.
Both are used to speed progress through a given piece of work.  Using knowledge and experience is a time to market competitive advantage gained through having talented people, compromising quality is a time to market advantage gained through process; albeit purchased at the price of later cost.

Taking Shortcuts

Let's talk about taking shortcuts first.  When you can do this confidently it is a magical place to be.  I like to hire really smart people and give them the freedom to use the experience they've gained throughout their careers to leap frog us straight over the other organisations who had to do it all from scratch, the sources of the experience I want to tap into.

Experienced people have been through years of doing everything by careful analysis and research, gradually proving out a larger and larger set of assumptions which they can build up from, and using life's great feedback loops to tune their thinking until their instinctive decisions are as a good as a noobie's 3 months of science.  That's why we hire the kind of people we hire.

Having all that collective wisdom at hand and not using it is irresponsible leadership and suffocating to the innovators you should be growing.  There are still times when the right thing to do is take things back to first principles and do that 3 months of science, but the beauty of experience is it also tells you when it's right to use your instincts and when it's right to figure it out.

Cutting Corners

And now cutting corners.  Reducing time to market but doing something less well (sacrificing quality) or by skipping validation steps such as doing less testing (taking more risk) works in the sort term, but really just exchanges a short term temporal win for a lot more trouble later.  Often more later trouble than you really think, which is why this accelerent is so frequently misused.

I say 'freqently misused' quite deliberately, because in a pragmatic real world it is not always the wrong thing to do.  Occasionally it can even be the path to very big wins, when time is absolutely of the essence and every day counts materially.  But it is always the wrong thing to do to unconsciously blunder into cutting corners, to not be actively managing your quality, and to incur this type of technical debt without having a credible plan to come back to fix it up before it festers too long.

If you are not grown up enough to manage the payback, then you are not grown up enough to incur the debt.


Hire people smarter than you are.  Empower them to use what they know and what they've done for your benefit - that's what they want anyway - and to use their time with you to further expand their horizons.  Carefully manage your quality, incur technical debt strategically, and never do it without paying it back.

You'll do a good job and everyone will have a good time, most importantly your consumers.

Sunday 18 September 2011

Nobody cares about your tickets

As engineers we like to speak in tickets and stories and bugs IDs etc - those units of currency we find easy to grok; they're usually finite, easy to define, narrowly scoped, and uniquely identifiable.

Our customers - for the most part - like to speak in outcomes and business plans and commercial objectives and projects.  The longer term, more vague, abstracted-from-the-work things that we like to digest by breaking them down into those stories etc that we're so comfortable with.

This can lead to a weird situation where you have all your iterations scoped out and your releases forecast (with painstaking detail in every story) yet the business still lacks clarity about what's being delivered by who and when.  Hmmm.

I think a bit of generalisation can help us to understand this.  Lets appreciate that engineers, for the most part, are deeply practical people who are comfortable handling actionable tasks.  Since we're also geeks - again for the most part - this descends quickly into technical details.  You'd be surprised how unintelligible this can all be when you're not a developer.

What I've learned over the years is that stakeholders are interested in the meta problem, the commercial plans and outcomes, not the tangible problem, the technical tasks and code which need to come together to solve it.  As technical leaders we need to talk about the meta problems with the business, and the technical problems with the engineers.

This comes with two obvious challenges - whenever we're talking about the same things but using different currency we rely on a degree of interpretation and some things can be lost in translation.  This most commonly surfaces as that old "unclear requirements" thing.  The other challenge is making sure that, as we solve our technical problems, we're solving our meta problems.  This isn't easy and this is why great product managers are so crazy hard to find!

This kind of interpretation of work - the aggregation of agile units into bigger picture project plans - is usually an early casualty of agile adoptions.  It is reminiscent of waterfall so we label it bad.  But it is a helpful mechanism which bridges the gap between agile execution and regular business planning.  It gives you both currencies.

Get awesome product people - they are the only other ones who live in this gap with you and have to speak both languages.

Monday 12 September 2011

The secret of managing software projects

As a technology manager there is almost unlimited discourse you can read about how to successfully manage delivery and control projects.  The real secret here is you don't manage projects, you manage the circumstances around a project...

This assumes that you have a team of talented engineers, a project manager as strong of character as he is on process, and an architecturally sound plan with some slack in your estimates.  If you don't already have this then don't worry, you're not about to fail, you've already failed.  What you're about to experience is you doing your best to mitigate the consequences of poor leadership.  Good luck with that.

But let's just say that you've got the structure right - now what?  If that's true, then the best contribution you can make as a leader is managing the environment around the project.  Provide a vision and simple insight into what moves the dial for the business and then move the obstacles, keep the distractions away, make sure the priorities don't change, and protect the productivity of the team members.  Keep the project fed and watered; get it the budget and skillsets it needs, get it the right visibility and attention from external teams who may be dependencies.  You're usually on the hook for the whole thing, so the temptation to dive into the details is often overwhelming.  Do it if you must but know that you probably aren't helping.  Your guys won't feel trusted and you won't be encouraging them to take ownership of their own space.  If you have a good plan on a sensible horizon, don't fiddle with it.  Every time you do you increase the chances of things not working out.

That doesn't mean you should be disinterested or uninvolved - quite the opposite.  Be there 100% for every engineer every day and give them everything they need to give you what you need.  Be easily accessible and take on any potential problem, consequence-free, that your team feels like raising.  They don't want to waste your time any more than you want to waste theirs, so if they come to you with something, it will usually be because there is a genuine risk that you're not going to get what you planned and there is something you can do to put things back on track.

I love technology.  We all do.  That's why we got into this business.  I love what my guys do, and I love to talk to them about it, but I am always aware of when prudent interest in the organization's deliverables crosses the line and becomes micromanagement and interference.

The most important thing you can give to any project is certainty.  Make a good plan based around people better than you are and then defend them.

Monday 22 August 2011

Great Quote and Greater Source

This article has a great take on developer-husbandry, and I particularly like the last paragraph:

"I’ve gone back and forth on whether managers should code and my opinion is: don’t stop coding. Each week that passes where you don’t share the joy, despair, and discovery of software development is a week when you slowly forget what it means to be a software developer. Over time it means you’ll have a harder time talking to engineers because you’ll forget how they think and how they become bored."

Even better is who sent it over to me - one of my commercial stakeholders.  Being in a tech savvy business makes my job a lot easier in a million little ways.

Friday 22 July 2011

Why don't we invent more?

I sat in on a conference call today (actually I'm still on it!) where we talked a lot about innovation - specifically why we haven't done more of it.  Lots of different stories but the common theme seemed to be governance.

When I say governance I mean things like roadmaps, product councils, architectural oversight, PMO, etc.  Basically the constructs we've set up to steer our technology investments and spend our resources wisely.  Properly applied, they are critical to the success of any significant engineering endeavour.  Improperly applied they are critical to the failure of any significant engineering endeavour!  So are they being properly applied to invention and experimentation?

I don't think it is a case of proper application as much as it is a case of application at all.  I think any engineer has two jobs; serve the roadmap - build the applications and systems defined and managed by our governance processes - and serve the technology - discover new ways of solving our business problems, advance your own knowledge by experimenting, come up with new ideas and see if they fly.  If you're only doing the former then you're only doing half your job.

These are two very distinct types of activity and they shouldn't be governed by the same controls.  It just makes no sense to apply ROI and risk controls and rigid scheduling to a journey of discovery with a totally unknown destination.  The difference between teams who innovate and teams who just talk about it is recognising the immense value in the journey alone...

Besides, you don't need all that bureaucracy.  All you need to be great at this is to leave a space for it to grow and - if you have the right people - it will expand to fill it.  That is how I roll and that is what I want to see.

Wednesday 13 July 2011

Big Roadmap seeks Innovative Thinkers for long term relationship

I haven't posted much recently - my new gig has been keeping me quite busy - but life is returning to a more sustainable pace now so I'll be able to start sharing more often again.  In the meantime I once again find myself drawing up battle plans in the war for talent.  So what are my 3 top weapons I can bring to the fight?

Step 1 - oops almost out of desks; let's get a really kick ass place to house all these smart engineers.  So we're moving to a sweet new spot in Angel.  We're growing fast and it's already nearly standing room only at the current office.  Covent Garden has its charms, but our new spot is being custom fitted - just for us - to be one of the best work spaces in the UK.  Check out the progress:

The list of features includes eco-geekery such as biomass boilers and heating and lighting driven by room occupancy sensors, high ceilings with 360 degree sunlight, huge storage space for bikes with plenty of showers and lockers, a variety of breakout areas for informal meetings, and a nifty cafe and Jamie's Italian restaurant on the ground floor.  All 80,000 sq ft of it (a huge uplift to accommodate our growth) is coming along nicely and is 30 seconds walk from Angel tube station.

Cool.  Now for step 2 - set up the right environment and culture.  So what is it actually like to work here?  Well, you could take our word for it, or I could give you the inside scoop...

When you get right down to it technology is core to what we do - our whole business is built on it and it drives our partner's businesses too.  Therefore we have to have a culture which supports great engineering in order to succeed.  That means bringing in smart developers, letting them own the problems, and then giving them the space to find the best ways of solving them.  It means creating time for innovation, it means allowing technologists to make technical decisions, and it means permission to work in the implementations which make sense (and maybe even a little pushing to keep you expanding your horizons) instead of being dogmatic about any particular stack.  Add to this a sustainable business model with plenty of big wins left to achieve and some clear priorities and shared steering of the business, and you have yourself the right ingredients for putting together some really successful software.

In the macro, Expedia is a very large place but we operate in a kind of group structure.  So yes, it is a very big company and that [fortunately] comes with some big company benefits, but the work experience is more similar to working for a well funded startup.  We work as small, independent, cross-functional teams, with each team having a good mixture of autonomy and support.  We believe that our guys should have end-to-end ownership of the systems they build - everyone is part of a small elite team and feels like they're personally making a difference.  If you've read a few of my posts before then you probably have a fair idea about how I like to run things.

Step 3 - challenges that really make you think.  This is an exciting time to be part of creating something pretty unique; I've said recently that we're about to change the rules in web travel and it's going to be fun.  Obviously I won't share my roadmap here, but I will talk about the sort of technology we'll be working with.  How about creating an eventually consistent distributed datastore latency tolerant?  What about peer-to-peer cluster management in the cloud?  Making an n+1 node in-memory cache topology aware?  Automated provisioning based on realtime system utilisation feedback?  Inventing new sorting algorithms and queuing strategies?  Or your own anti-entropy protocol?  Collecting obscene amounts of performance data and rebuilding your tools to watch those numbers move?...

You'll notice that I didn't say 'Java' or 'C#' or 'SQL' or 'PHP' at all.  That's because, to a certain extent, those things aren't what's important.  What's important are the patterns, the fundamentals of how we logically solve business problems using computer science, and that is what great engineers do.  Then they write some code as a result.

If that sounds awesome to you, and you can show me some aptitude in this kind of space, then we should talk (especially if you're Cassandra/MapReduce curious).  Get hold of me or my recruitment ninja, Roopesh Panchasraright now.

PS - I am also hiring in the US, so drop me a line if you're stateside and want to work with smart folks on some sharp stuff.

Thursday 28 April 2011

a geographic sense of urgency

A friend of mine recently visited the Boeing factory just outside of Seattle and came back with a story I thought was worth sharing; something they called 'a geographic sense of urgency' over there.  It's a nifty story about motivation and incentive.  I'll do my best to retell it 2nd hand.

A few years ago Boeing was struggling with efficiency on their assembly lines.  They needed a way to build airplanes faster but were approaching limits on what could be done in parallel, how long a working day could be extended, or how many more resources could be applied to each assembly (outside of the economic ceiling I guess there must be some limitation to the number of people who can fit around the airframe).

To grossly oversimplify - with apologies to aircraft mechanics everywhere - a plane started at one end of the production line and moved through a series of stages; stopping at each 'station' to fit whatever parts that station fitted until eventually arriving at the other end of the hanger complete.  Another way to describe this might be to say that a plane moved through a series of bottlenecks, with all pressure focused on the current station until its work was complete.

The answer borrowed heavily from lean but was all anchored around a single, creative idea; the plane never stops from one end of the hanger to the other.  Instead of rolling forward a few dozen meters and then stopping to have work done (loop until plane=true) the plane is constantly moving forward by a few centimeters an hour and only stops when it reaches the end of the hanger - hopefully as a complete aircraft!  Flow.  Work happens more fluidly and consistently all the way along the plane's terrestrial journey.

This simple change to constant motion had a remarkable impact on the teams.  I wish I had more data (and if I can find any I will post it) but the improvements in throughout were astounding.  The physiological effect was a sense of pace - this is what they call a geographic sense of urgency.  The results were significant increases in efficiency and total throughput without extending crews or hours or resorting to the traditional but seldom as effective as expected measures such as pay increases.

Smart idea Boeing.