Monday 31 March 2008

I Don't Brake for Meetings

That's the bumper sticker I'd have on my car if I was the sort of person who affixed adhesive labels declaring how I feel about stuff to my vehicle.  But I'm not.  But I do write about it...

Meetings can be the single biggest black hole for time (and spirit) created by man.  Left unchecked they constitute a gradually increasing tax on your time giving very little back in return.  Before you know it all you do is attend meetings and you wonder why you never achieve anything you set out to.  After every meeting ask yourself a few simple questions - what do we know now that we didn't before we went into that room?  What conclusion did we reach and if we didn't do we know exactly what comes next in the journey?  If you can't honestly justify it to yourself then chances are it wasn't the best use of your time.

I have a few little rules I try to live by that I think have freed me from the shackles of an agenda-driven descent into madness and put me back in control of the lion's share of my time:

Always have a goal for a meeting or just don't go

Before you call a meeting think carefully about exactly what result you want to achieve by getting that group of people you've targeted together, make them aware of the conclusion you want to reach (so they can all do their homework and you don't have to adjourn until later for people to consider it) and don't let the discussion stray off course.  If you call a meeting; chair it.

Don't just accept invitations to meetings without question

Ask inviters about their goal and make sure it's actually a goal.  For example "discuss roadmap" is not an outcome, that's a method.  What do you want to change on the roadmap?  What do you want me to agree with?  Tell me or you'll be lonely in that room.

Never book recurring meetings with no end date

Because if you do you will be there until then end of time.  Granted, there are things that take time to work through - if you know when the activity you're governing with that meeting is due to end then book the series to end at the same time.  If you don't know when something is likely to end then book it for 4 or 6 weeks at a time (whatever horizon makes sense).  That can be a pain if you have to keep re-booking it but at least it provides a checkpoint for you to stop and think - do I really need to keep doing this or can everyone have that hour back?

Try to wriggle out of a meeting if at all possible

The first thing I do when I receive a meeting invite (or when I feel tempted to send one) is phone the other party - or better yet, if they're in the same office wander over to their desk.  Try and reach the outcome the meeting was intended for on the spot - so often mine are as simple as "can we do this?" to which all I need say is "yes as long as X and Y are true" and the job is done.  5 minutes instead of 1 hour - plus the time lost in context switching.

Be brutal and consistent about this.  You might initially offend a few people and seem pushy but only until they realize how much more effective you are.  But even if they don't who cares?  You'll feel better when you're time is spent making a difference instead of rattling through endless agendas.

Thursday 27 March 2008

Balancing Capability vs. The Perfect Design

A while back I read this post and it got me thinking about how we design solutions - an activity that architecture is a [big] subset of.  On the one hand we're after the ideal answer to the business problem but on the other hand we've got to be able to actually deliver it, extend it in the future and support it too.

The guys on codingthearchitecture were talking more about their own personal experiences transitioning from lead developer to formally designated architect but I think the fundamental continuum implied in the discussion - that line with "what we're best at" at one extreme and "what's best for this requirement" at the other - is worthy of some management attention.

If you examine the "what we're best at" extreme you find there are some advantages to designing solutions in that space; you'll typically have higher quality software, the project will be quick and operationally you'll have a lot of nice, peaceful nights sleep running technology you're already tooled and trained to support.  The downside here is that if you always limit yourself to the hammer every problem is a nail.  You'll end up with a whole lot of products that are done very well but quite plain and indistinguishable (where's the real USP's?) and you're likely to spend far too much on some projects and not quite enough on others - because if everything follows the same basic pattern/technology set then you need to size that for your biggest, baddest problem.

At the "what's best for this requirement" end of the spectrum you'll end up tailoring the most appropriate, specific solutions to every business problem and when your technology is your product that creates the most competitive advantage.  It also allows you to really capitalize on the creativity and innovation you have locked away in those engineers and that will in turn keep the team growing, developing and improving - your customers will see benefits from this as your execution of new technology improves.  The principal downside here is risk - you'll occasionally be stepping out into complete unknowns that won't always work out.  And your project is likely to take a little longer to get done if engineers need to do some on-the-fly learning.

So which one is right?

I think the answer here is balance.  Per-project you should find an optimum point along that continuum - sometimes you'll be able to take a little more risk, push the team forward a bit and try something new and other times you'll need to fall back on today's expertise to get something to market quickly or guarantee an SLA.

As management you should enable your engineers to make these decisions and encourage them to lean towards the side that's most beneficial to your organization.  Ask yourself whether creativity or commodity is more valuable to you; whether innovation or stability will give you more of a competitive advantage.

Tuesday 25 March 2008


I like Dan's analogy on this, simply motoring along blind will almost always have a fiery end.  I want to apply it to planning on a more practical, daily basis though - I know this isn't exactly what Dan meant but I think the concept scales down with equal validity.

We know that whenever we break projects down into good quality user stories, then into individually-ownable tasks, collaboratively estimate these and then assign and track them in daily stand-ups we get consistently good results and predictable delivery.

The problem is that when the heat is on - for example one of Dan's crazy deadlines looms ominously - proper planning seems to be the first thing cast overboard to save the ship.  After all, whatever time you spent on planning is time you could be coding right?

We also know that whenever we try to tackle a huge volume of work in a very short time by just diving into it head first and all working like mad we get consistently poor results and low quality (and quite often we don't even get there at all).  We tend to leave the bigger, harder things to the end so chances are once we've failed we've left the most valuable bits behind.

Given that we know these 2 things why is it we keep doing this?

With proper planning not only do you have the highest chance of success in the first place you'll also be driving the car (rather than just switching the headlights on) so if you do run out of time you can take some consolation in the fact that whatever is left probably represents the least business value of everything you could have done for that project.

I am the veteran of many foolhardy deadlines with fixed scopes and I've observed that teams tackling this type of work in a controlled, ordered list tend to feel less stressed and make fewer mistakes.  It doesn't feel like the weight of the world is coming down on their shoulders and they're facing a huge, amorphous mountain of work that just can't be done...

Friday 21 March 2008

Do the Right Thing

I got a few emails about this post and I guess it might have been a little too vague.  The conclusion I was trying to draw is that we need more than one tool in out toolkits.

As engineers it feels intuitively right to have a single process, a single way to tackle every activity.  Even though we say we don't we secretly quite like a good ole' fashioned IF/THEN.  IF code like this THEN test like that.

Sure, we always need a framework to work in, a reference process to fall back on - just don't get dogmatic.  Recognize that not everything you do needs to be treated exactly the same way (unless you have a really boring job) and stay flexible enough to be able to do the right thing in the right circumstances and take advantage of opportunity as it presents itself.

The best sort of engineering is about creating competitive advantage through technology - it's uncovering business value as you see it so whatever flavor your favorite bureaucracy comes in always make sure you're not structured to ignore opportunity.

Tuesday 18 March 2008

A Revenue Share != an SLA

When white labeling etc with 3rd parties a revenue share as a commercial structure often seems attractive.  There are usually some obvious benefits (depending on your specific terms) such as a low entry cost to a new market/product and shared risk - perhaps even a painless exit strategy if it doesn't work out for you.

One mistake a lot of people make when negotiating these deals is being a little too lightweight on the service levels governing the part of the product supplied by the 3rd party.

On the surface it seems intuitive that with a revenue share agreement in place you can afford to be a little less draconian on any SLA - after all you have a shared interest in keeping the top line flowing right?  Possibly you can.  But first you need to take into account the differences between you and your partner.

A rev share usually is almost always top line based (as that's the most transparent thing you can both count) but what really matters to you is bottom line; the bottom line for you and your partner might be quite different.  If you are in a heavily regulated industry like online gambling then you'll have a whole lot of taxes and levies that need to come out of your share - ergo you're both operating at a different margin.  Difference number 1.

The other thing to remember is that a rev share motivates partners to do the most profitable thing - that's not always the same as the best thing for your business.  Let's say you have a bug on the 3rd party side (undocumented feature?) that you find particularly distressing.  It might be very costly for them to resolve and this, coupled with the difference in operating margin, means they just might not because it isn't the most commercially viable use of their resources.  Difference number 2.

I guess the summary here is just because you're "in it together" doesn't mean you should skimp on the SLAs - treat it like any other contractual agreement or you just might not get what you want.

Monday 17 March 2008

Closing the Gap

Last week I posted about the gap between infrastructure and software.  I figured it might be a nice idea to talk about how we could address that gap instead of just having a whinge.

Thinking about why this gap exists in the first place it seems to me that most organizations are structured to create it.  They have this "developers in one team, infrastructure guys in another" kind of philosophy - organization by trade.  Then battle commences and they throw work back and forth over an imaginary brick wall between the groups.  Here is my code, no we can't support that it doesn't have monitoring feature X, OK it's back, no it doesn't work on this server, but it worked in the development environment... ad nauseam.

The next thing most companies try is fixing the problem with a hefty application of good ole' fashioned process.  This inevitably brings with it gates (signoff points in the development cycle) that engineers need to navigate.  Bob needs to sign off the monitoring capabilities, Tim needs to sign off the failure testing, Steve needs to sign off the compatibility with the live environment etc.  Congratulations, you've now built natural choke points (and if you're not careful key man dependencies) into your product delivery process - and we all know a machine can only move as fast as it's slowest part.

So what did you learn through your use of gates?  You learnt who needed to have a say in your product development, that's what.  This means the journey wasn't a complete waste of time - unless you stop here.  Now that you know who needs to work together to build your product in the best way possible why not team them up like that?

Organize ownership by product or feature or service - whatever the basic unit of your engineering output is.  Cross discipline teams containing all the skillsets necessary to design, build, test, deploy and support your system on a per service/feature basis will yield you the most appropriate solutions in the shortest possible time.  Embed all the necessary capability in every team - you'll need it every time anyway the only difference is the size of the communication gap.

I've suggested that product = software + infrastructure + operational know-how to run it.  That being the case why shouldn't a product team be made up of developers, testers, system administrators, architects and product owners?

Friday 14 March 2008

Epistle to the Vendors

Before doing what I do now I was always in system integrators/consultancies.  Now that I'm on the other side of the boardroom table I've seen the whole picture and I have to say - we're playing right into the hands of the resellers, integrators and consultancies.  If we're going to do this anyway we may as well be a bit more open about it.  Let's go in up front and do it properly in a more formal, planned manner rather than stumbling from one bad decision to the next.

Therefore I have written this open letter to vendors everywhere outlining exactly what they need to do to take us to the cleaners:


Dear Sir/Madam

We're a web enterprise just starting out and we're teetering on the brink of profitability - we have a popular product and the business is expanding at a fearsome rate.  We're running around like madmen trying to work out how to get the right balance between scaling our system, shipping more features and protecting our time to market.  You've come in to meet us disguised as help - nice touch - and you're trying to work out how you can squeeze the largest possible cross-section of what you sell into our requirements over the longest possible timeline.  Using these 3 big phases you can really stretch it out.

Phase 1 - hardware (AKA brute force)

Popular product, lots of customers, increasing load - get some bigger computers, we'll buy those.  Next sell us some faster network, we'll need gigabit everywhere, some bigger firewalls and then how about load balancers?  Let's go brand crazy and throw together some F5, Netscaler, Cisco and Foundry.  Load increasing a bit more?  Next size up computers please.  Now we're a little scared too (evil hackers!) so let's have some Top Layer, maybe Netscreen and Juniper too.  Uh-oh load is up again so more memory, more CPU please.  Wait a minute, we're now out of bigger boxes in the x86 space?  Can't address any more memory in a single node?  I guess we'd better move into something like SPARC, jumping a whole class of computer - ka-ching!

Phase 2 - middleware (AKA short sighted)

OK we're still hitting capacity ceilings but now we worry equally about uptime.  How can you help [yourself to our cash] here?  The best tactic is going to be some really expensively licensed software that contorts scale and availability from products that just weren't built for it.  Ideally we're looking for some second-rate results by forcing artificial distribution using things like GemStone, Tangosol, Veritas, CA, TimesTen.  It'll also get us some failover (yay) but without delivering too much more actual availability (boo) since our application is blissfully unaware of the state of it's infrastructure.  Tricky but lucrative times.

Phase 3 - consultancy (AKA told-you-so)

Now's the best bit - you get to sell us your most expensive, highest margin product - consultancy [meanwhile in the real world we finally agree on something; we're in a bind we can't buy our way out of].  So what's your advice?  Decouple and distribute, scale horizontally, version data and services, isolate features and automate the hell out of the whole thing.  Thanks for that (and ka-ching again).  The most important thing to remember is not to ever let on that you knew this all along.  Oh and if you're clever you can usually also work in some sort of licenses or something in this phase as you never really want to completely wean us off you.

So there you have it, think of this as your 12-step programme compressed into 3 easy stages - just do your bit (turn up with PDF's and nicely ironed shirts) and we'll do ours (make appalling decisions in our early product development) and together we'll milk us dry.

Yours insincerely

N. E. webscale business


Hang on this all sounds pretty sweet, why did I ever leave consultancy?  But it's not too late!  You don't have to put Capgemini, Dimension Data or Logica's children through college if you don't want to!  You just have to think differently now.  You're making a product you intend to be successful right?  You intend for it to be popular, right?  Well decouple, distribute, scale horizontally, version data and services, isolate features and automate the hell out of the whole thing now, today!  You will end up doing that (or die trying) anyway, it's just a matter of how much it costs you on the way.

Tuesday 11 March 2008

The Great Divide

I'm into availability at the moment, so much so that I'm leading an organization-wide change initiative that targets cultural, technical and process habits (busting established, forming new) in order to deliver a more consistent experience.  Our goal is to always have something to offer our customers regardless of any maintenance we're performing or failures we're experiencing.

This is how I discovered The Great Divide.  The namesake of this post is that gap between the infrastructure and the software - because of which we offer our product to our users much less often than we could.

Here is how it works:

We've got a pairs of firewalls that can fail over while maintaining session state.  We've got tiers of load balancers that can reroute traffic around down network devices.  We've got clustered databases that can move active systems between nodes in a couple of minutes.

But guess what else we've got?

We've got applications that lose session information without contiguous sequence numbers.  We've got applications that cant match users to activity if their traffic suddenly comes from another IP address.  We've got applications that depend so heavily on their databases that death occurs within a few seconds of separation.


You don't get any partial credit in product uptime - your customers will not award you a bonus point if your site is down but your servers are up.  If they cant log in they cant log in, if they cant place orders they cant place orders; they're quite a binary bunch.

For us product = infrastructure + software + operational know-how to run it.  We need to stop worrying about server/network availability and start worrying about product availability - because guess what, that's what our customers are measuring us on.

Close that gap and let your customers see the benefit of those cool devices.

Monday 10 March 2008

A Good Quote

I talk a lot about failure in systems - it's inevitability and the things we can do to minimize it's impact and recover in a quick, automated fashion (note I never said "prevent" failure).

I recently came across a good quote on the topic that I think is really punchy:

"failure is not an option - it's bundled with your software"

I would credit it's author but I can't seem to work out where it came from - I've seen it tacked on the end of a few people's forum posts. I can't help but think they meant it a bit more cynically than I'm taking it but it's a good thinking point when you're delivering online services.

Sunday 9 March 2008

A Change of Screenery

Continuing the good work started here...

[for a given value of "good" I guess]