The difference between taking a shortcut and cutting a corner

There are two very different things to talk about here, with two very different implications for your software, but I'll admit that which way around you label them can simply be language pedantry.  So for the purposes of this discourse, let's say that:

• Taking a shortcut is using your knowledge and experience in place of studying things from first principles every time.
• Cutting a corner is a compromise in quality or good practices in the place of the prudent actions otherwise required.

Both are used to speed progress through a given piece of work.  Using knowledge and experience is a time to market competitive advantage gained through having talented people, compromising quality is a time to market advantage gained through process; albeit purchased at the price of later cost.

Taking Shortcuts

Let's talk about taking shortcuts first.  When you can do this confidently it is a magical place to be.  I like to hire really smart people and give them the freedom to use the experience they've gained throughout their careers to leap frog us straight over the other organisations who had to do it all from scratch, the sources of the experience I want to tap into.

Experienced people have been through years of doing everything by careful analysis and research, gradually proving out a larger and larger set of assumptions which they can build up from, and using life's great feedback loops to tune their thinking until their instinctive decisions are as a good as a noobie's 3 months of science.  That's why we hire the kind of people we hire.

Having all that collective wisdom at hand and not using it is irresponsible leadership and suffocating to the innovators you should be growing.  There are still times when the right thing to do is take things back to first principles and do that 3 months of science, but the beauty of experience is it also tells you when it's right to use your instincts and when it's right to figure it out.

Cutting Corners

And now cutting corners.  Reducing time to market but doing something less well (sacrificing quality) or by skipping validation steps such as doing less testing (taking more risk) works in the sort term, but really just exchanges a short term temporal win for a lot more trouble later.  Often more later trouble than you really think, which is why this accelerent is so frequently misused.

I say 'freqently misused' quite deliberately, because in a pragmatic real world it is not always the wrong thing to do.  Occasionally it can even be the path to very big wins, when time is absolutely of the essence and every day counts materially.  But it is always the wrong thing to do to unconsciously blunder into cutting corners, to not be actively managing your quality, and to incur this type of technical debt without having a credible plan to come back to fix it up before it festers too long.

If you are not grown up enough to manage the payback, then you are not grown up enough to incur the debt.

So...

Hire people smarter than you are.  Empower them to use what they know and what they've done for your benefit - that's what they want anyway - and to use their time with you to further expand their horizons.  Carefully manage your quality, incur technical debt strategically, and never do it without paying it back.

You'll do a good job and everyone will have a good time, most importantly your consumers.

Mechanical Sympathy

In the world of motorsports ‘mechanical sympathy’ is the measure of a driver’s impact on their vehicle. As well as fuel economy it is probably well understood by most people that your driving habits do have some impact on how long your car is likely to last, how often it needs attention, and how quickly the parts will wear out. The world of professional drivers is no different, except that excess punishment of the car can mean that vital few more seconds spent in the pit lane – which can cost a racing team a lot more than just this year’s annual service coming up sooner.

Mechanical sympathy attempts to quantify that impact and allows teams to acknowledge it and work on balancing car wear against squeezing that last little bit extra out of it.

I did a motorsports driving course a while back (number of formula 1 events participated in since training: 0 – number of speeding tickets around town since training: 1) and was mercilessly rated in this category. I guess that explains the frequency of servicing that my long-suffering saloon seems to require…

As an outsider to the world of motorsport the importance of certain metrics is immediately obvious – lap times or time spent under power etc – but why measure drivers on something mechanical? These guys don’t fix their own engines, swap out their own gearboxes, or tune their own brakes; that’s someone else in the team. But, as drivers, they do have a direct impact on how often this needs done, how severe it’s going to be, and how expensive it will be to do – and that has a direct impact on winning and losing.

So why don’t we measure the non-IT people in our businesses on their impact on technology? I’d like to introduce the concept of technical sympathy.

I think that people in our businesses should have at least a small part of their KPIs relate to technology. We’re not expecting them to be developers or sysadmins – in the same way that drivers aren’t expected to be mechanics – but we have to acknowledge that they do have an impact on this stuff (probably more than they realise).

Good stakeholdership can make the difference between a well-run project hitting time and budget and a chaotic piece of work drifting on endlessly. Considerate use of systems can avoid the need for expensive environmental controls and the wasting of shared resources such as disks and tapes. The costs and turnaround times of support can be dramatically different with more personal responsibility and better use of self-service at the desktop.

In almost every company I have worked in we’ve measured engineers on business knowledge and company performance to some degree, yet we never seem to have thought of the reverse even though business behaviours can have a profound effect on the cost and quality of technology.

The hidden wisdom of Kevin McCloud

I was lucky enough to get a copy of the Grand Designs Handbook for Christmas - it's a compendium of building projects from the channel 4 show (the book, not Christmas). The first few chapters have a lot of lessons learned and good practices picked up from 9 seasons worth of experience with, well, variously successful building projects.

Reading some of the advice, there is a lot that can be applied directly to software projects. There is a section on 'how to behave with your team' which is aimed at helping people keep their building sites productive - but the advice translates well to technical projects. Here's a couple of my favorites along with my join-the-dots:

"Don't give verbal instructions on site unless they've been agreed with your consultants and they're backed up in writing. If you just issue verbal instructions, you risk confusion and possible extra costs."

This is a top one for me - especially in the agile world where many interpret 'agile' as 'make it up as you go along' and then end up disappointed. I think maximizing interaction between delivery team and customer team is a good thing especially when both sides recognise the boundary between clarification and change in in-flight work.

"Do take time to prepare detailed written briefs setting out all your requirements for all your consultants. A good brief is an essential part of the design process."

One of the simplest formulas the govern all project delivery (hardhat or mousepad) is that the quality of what you get out with never exceed the quality of what you put in - yet this seems to escape so many of us so often. What you invest in clear instruction and regular contact with developers is a key variable in how happy you'll be with the results.

"Do allow your experts the freedom to do their jobs. Stand back from the project and enjoy your role as client; making decisions and hanging around, generally being a good egg and imbuing the world with optimism and excitement."

I don't exactly advocate stakeholders 'standing back' from a project, and I don't think that's what Kevin means either. Don't micromanage your delivery team and, as a customer, be sure to perceive the difference between decisions you should make (and don't let the team wait for those) and decisions the team should make (and let them use their skills and experience to make them).

"Don't listen to other people outside the professional team. You'll only end up getting very confused."

External feedback and extra ideas are good, but remember that an engineer knows his business best and don't fall into the trap of listening to someone else simply because you like their answers better. I consider myself relatively smart but even I'll say all sorts of wacky things when I don't have any skin in the game...

There are a couple of other sections with close analogies to technical projects - how to be a perfect customer and how to put a good brief together - which I think are 2 other critical ingredients to any successful engineering project (construction or software) but they'll keep for another post.

conferences - jolly good or just a jolly?

It has always interested me that two different people can look at the exact same conference, and one declare it to be an unmissable event relevant to their work and the other consider it a waste of time; a free day out of the office at best.

It always leads to a discussion about which events are worthwhile and which aren't, but here's the secret - whether a conference is worthwhile or not is only 10% the event itself and 90% what you do while there and afterward.

You could carefully plan the sessions you attend, take the time to meet people with similar technical problems or who have done worked with relevant technology and follow up with them later and, when you return to the office, share your new knowledge around and adopt some better practices.

Or

You could have a few days out, enjoy some vendor's hospitality (free beer always tastes better), and check out some new bars and, when you return to the office, settle comfortably back into your old habits.

That, ladies and gentlemen, is the difference between a jolly good conference and just a jolly.

So who is in a team anyway?

I caught up with a friend the other day; he’s in the system integrator business and is struggling a little with recent growth. Long story short, things are a bit a disorganized and efficiency is low. They’ve got some kind of consultant in, and we were talking over his advice.

So how does their guy propose they fight the first battle in the war for organizational effectiveness? Setting aside certain days when people in one team aren’t allowed to talk to another. No, you didn’t read that wrong, the ‘big plan’ is to actually create barriers to communication.

I just hope that wasn’t expensive.

In the reseller/systems integrator game it’s all down to the cooperation between sales and their technical teams, and so I really struggle to see how driving intentional wedges between them is going to help. So my theory? Perhaps the wrong people are teamed up together in the first place…

In any mission there are multiple people with different skillsets doing different jobs all coming together around a central project/customer/business problem. Isn’t that a team? The people who actually need to work together on a daily basis for the direct benefit of the customer? I don’t know why we keep insisting on grouping people by job or by skillset (sales team and technical team, or dev team and test team) because the cold, hard facts are that these teams can’t achieve anything end-to-end for a customer on their own. Maybe it just appeals to our human need to nicely categorize things.

When you put people together in a team, you create a tight loop of communication and cooperation between those people (it’s one of the reasons we do it in the first place) – but you do something else too – you also create a little organizational fence between them and other teams. A fence that we absolutely should work at keeping as low and friendly as possible, but a fence nonetheless. To me it just comes down to where you want to put your fences – does it make sense to create boundaries between the individuals whose effort must by synchronized daily to complete units of work?

In what I do, the unit of organizational delivery is the project (or, as a superset, the product). That’s that unique object around which we all gather and apply our individual skills and experience to create tangible value for the company. So by default, I have a nice, simple method for grouping my guys together in a way that creates the fewest barriers to the most common communication and coordination lines that they’ll be following each day. I suppose I could put all the developers in one team, all the testers in another team, all the project mangers in another team again etc – but if I did, then what’s the very first thing I’ll have to do as soon as I want to do any work? That’s right, pull them back together into virtual or temporary teams based on the projects they’re delivering against. Hmmm.

From my experience in the SI and reseller game, I’d suggest that the unit of organizational delivery is the customer. That’s the internally-ownable entity which performance and profit is measured against, margin is made against, and time is billed against. Yet almost every one of these companies is, at best, a matrix management organization. So you group people together into a sales team, a tech team, a presales team etc – and then wonder why, whenever they all have to get together around that common indivisible business problem (the customer) there are all sorts of communication issues, competing priorities, complexities planning time...

Perhaps a team in this type of organization should be something like a sales guy, some sales support, a presales guy, a couple of engineers, and a share of some administration – all grouped together around a set of similar customers that they own the end-to-end P&L for.

The inevitable argument against this kind of cellular organization is the same one I’m used to elsewhere – developers like hanging out with developers, infrastructure engineers like hanging out with infrastructure engineers and so on. My only response to this is that I don’t see how organizing your people into the teams which best reflects the true structure and purpose of the company precludes communities of interest, and again, just ask yourself where you want to put the barriers, and whether you consider the upside of keeping people grouped into teams of similar individuals worth the downside of trying like mad to make them work together efficiently and effectively whenever you have to get anything done.

Good ops guys are hard to find...

Good operations is about staying one step ahead of the state of the system; taking proactive actions based on quality telemetry.

I’m reinventing how my sites are supported – if you’re an awesome one of these or a kick-ass one of these, then we should talk.

Making Decisions

I thought I'd start with this as my first 'proper' post of the year, mainly because decision making isn't usually considered a skill in it's own right, despite being deceptively simple and so commonly done poorly.

Making a decision isn't a stand alone task, when you make one, you really do a small collection of things together. You make choice from some options, you make a plan and delegate it, and then you follow up and review. Most people think that once they've picked their option they're done, but like Drucker says, you haven't made a decision until you have an action plan - you've only made a choice.

So let's talk a little more about effective choosing, as that's the first hurdle to trip over. The behaviors you want to avoid here are essentially procrastination. How many times have you been staring an obvious course of action in the face (or been presented with a fantastic opportunity) but instead of making progress, you're taking bets on which is growing faster - your number of choices or the list of data points people want to see for each one! You have to reduce your number of options to a manageable amount (3 is a magic number if you're stuck for inspiration), and while you should never ignore great last minute ideas just for the sake of rules, most business don't fully appreciate the cost of inaction. The point marked "doing anything is better than nothing" creeps up a lot quicker than you think.

The next thing you have to worry about is the ballooning amount of information on each of your choices. Too many and you risk it being treated like a smorgasbord (can we have this from option A, and that from option B) or being paralyzed by the sheer volume of statistics and technicalities. Try removing things that are the same for all choices - for example, if A, B, and C are all similarly expensive, why compare cost? And how about things you don't feel strongly about? I don't know how many times I've seen stakeholders agonize over details like the difference in reusability between 2 choices in a decision, and then when asked where else they plan to use the technology, answer "nowhere". If it's an attribute you don't feel strongly about, then it's unlikely to influence your decision, so why clutter the process with it?

We've talked a lot about paring things down, and now here's something to add in. Don't forget that for every choice you're comparing you need to know what you don't get if you don't pick it. Most people are generally pretty good at stacking up the costs and benefits, but opportunity cost is often neglected.

Hurdle number 2 is action plans and delegation - the behavior you want to avoid here are those deja vu meetings where you all sit around and 'review' the decisions you made, wonder at the lack of progress and then agree to 'update next week' all over again. Once you've made your choice, don't let yourself off the hook until you've agreed who is actually going to do the work, and by when. Follow the usual rules on delegation - make sure whoever you're going to hold accountable clearly understands their obligations, is empowered to do the work, and has the available resources.

Don't forget to follow up. You spend your time on the things that are important to you, and you're not making unimportant decisions, are you? The things that you focus on will get done - keep track of what's happening, make major milestones public, and use your sponsorship to see it through. And once whatever it is has been delivered (or has crashed and burned), you're still not quite finished. Now comes the feedback loop - what did you learn? How can the experience improve the rest of the organization? What pitfalls can you now help others avoid?

And finally, you shouldn't be afraid to revisit your decisions and change your mind. You should be keeping an eye on your customers, your portfolio, and your team, and continually assessing what any changing circumstances may mean for you. That said, you must resist being flighty as you need to show commitment, but keep in mind that "because we said we would" isn't a business benefit to completing a project - it's a sleepwalker's justification for not being in touch with the market.

Speed is in the Eye of the Beholder

Yesterday I was coming through Heathrow again, and while I was waiting for my luggage to surprise me by actually turning up, I was ruminating on performance, user experience, and the iris scanner (old BBC link). For those who haven't had the pleasure, we're talking about a biometric border security system based on the unique patterns in an individual's eye, implemented at certain UK borders as an optional alternative to traverse passport security in a self-service way.

Flying internationally at least once a month for the last couple of years, I've been a regular user of the iris scanner since it's early trials. Like all new technology, it went through a period of unreliability and rapid change in its early days, but its been pretty good for a while now. Step into booth. Have picture of eyes taken. Enter the UK. Awesome.

The only problem is that...
It...
Is...
Just...
Too...
Slow...

Don't get me wrong - you don't have to queue up for it (yet) and you do spend less time overall at the border - it's just that, for a high tech biometric solution, it seems to take an awful long time to look me up once it snaps what it considers to be a usable image of my retinas. You know, in real time it isn't even that expensive - probably an average of about 5 or 6 seconds - but it's just long enough for you to want to keep moving, hesitate, grumble to yourself, briefly wonder if it's broken, and then the gates open up.

It occurred to me that maybe my expectations as a user were perhaps unfair, but then quick as a flash I realized - hey, I'm the user here, so don't I set the standard?

So that's when I started to think about how the performance of this system could be improved, and caught myself falling straight into rookie trap number 1:

I started making assumptions about the implementation and thinking about what solutions could be put in place. I figured there must be some processing time incurred when the retinal image is captured as the machine picks out the patterns in my eyes. That's a local CPU power constraint. Once my retinal patterns are converted into a machine-readable representation, there must be some kind of DB lookup to find a match. Once I've been uniquely identified, I imagine there will be some other data sources to be consulted - checking against the various lists we use to keep terrorists and whatnot at bay.

Well this all sounds very read intensive, so it's a good case for having lots of local replicas (the iris machines are spread out over a number of terminals). Each unique 'eye print' is rarely going to come through more often than days or weeks apart, so most forms of request caching won't help us much with that. Of course there is also write load - we've got to keep track of who crossed the border when and what the result was - but we can delay those without much penalty, as it is the reads that keep us standing in that booth. Maybe we could even periodically import our border security lists to a local service if we observe a significant network cost in checking against them for each individual scanned through the gate (assuming they are currently maintained remotely by other agencies).

Ignoring the fact that I'm making all sorts of horrible guesses about how this system currently works, these seem like reasonably sensible patterns to me, so what's the rookie trap?

Simply that I didn't start by understanding the basic problem I am trying to solve, instead rolling up my sleeves and diving straight into the technical complexities. In doing so, I might have overlooked a plain, simple solution and that's usually bad - why solve a problem in a complex, expensive way when a simple, cheap way will do just as well?

In this example, the problem is not that 5 seconds is an unacceptable time to process the image, compare the data against the registered users, and then pass all the additional border security checks - the problem is that 5 seconds feels like a long time when you're standing in a glass box waiting for it all to happen!

So what else might we have done instead of a major re-architecture of the iris back end? How about just lengthening the booth to form slightly more of a corridor, with the scanner at one end and the gates at the other? With this simple trick, it still takes 5 seconds before the gates can open, but it doesn't feel like it - you haven't been standing still waiting, there is a sensation of progress.

Same level of system performance, very different user experience.

General engineering lesson 1 - customer experience is king. Monitoring and other data driven metrics are important, but how it really looks and feels to your users matters way more than whatever you can prove with data. They'll judge you by their own experiences of your system, not by your reports.

General engineering lesson 2 - you don't have to solve every problem. Sometimes it's better/cheaper/faster to neutralize or work around the issue instead. You might not be able to build a different floor layout, but you can do things like transfer data in the background (not locking out the UI) and show progress 'holding' pages during long searches etc.

Oh and just for the record, I really have no idea how the back end of the iris system works - you are supposed to be seeing the analogy here...

Is Leadership a Noun or a Verb?

Is technical leadership something you know or something you do? Is it a finite thing you can point to, and check off a list, or is it an ongoing process, more akin to a journey? Let's look at what it means in practice...

Firstly you have to have some answers - or better yet know the right questions. You need to have a vision, and preferably some ideas on how to break that down into Big Goals. You need to define what success looks like for your team, and know what sorts of things contribute to that success and what sorts of things distract you from it. Maybe this is leadership the noun?

All good work so far, but worth nothing to your organization if no one knows about it - who is it that's supposed to be working towards these goals? Who is it that you depend upon to realize your vision?

So the next thing you need is for everyone to proverbially Get On Board. You have to communicate that vision as simply as possible and help every individual understand exactly how they can contribute to it. The whole team has to get emotional about it, believe in it, and live it through how they do their jobs each day. Perhaps this is leadership the verb?

So perhaps the answer is both? I certainly think so. Share an empty vision or set the wrong goals and you won't get where you need to be. Keep quiet about your plans and how they can be achieved and you rob the organization of huge value.

Oh and don't forget to keep your eye on the horizon - things change and from time to time that means adapting your strategy or losing ground!

IASA Connections - Day 3

Day 1, day 2, and now day 3:

Scott Ambler – Practice Leader in Agile Development, IBM Software Group
• Data modelling from an agile perspective.
• Don’t get too hung up on purist agile, good techniques are good techniques regardless of where they come from.
• Software development is more of a craft or an art than engineering – hence we tend to apply some practices that don’t make things the easiest for us.
• Agile is becoming mainstream – you cannot dodge it – and data is not going to change from being an asset, so how can these things coexist?
• Data and quality are 2 words that are hardly ever seen together, yet this is so important.
• In most “Big Requirements Up Front” projects, 45% of features never used, 19% of features rarely used, so this is effectively a waste of over half the development budget.
• 66% of teams work around internal data groups, most common reasons are difficulty in engagement, responses too slow, and development teams don’t understand the value. So data architects have work to do.
• Modelling is like anything else, it just needs to be good enough for the task at hand (repeatable results, not repeatable processes).
• If you can’t get active stakeholder participation, then cancel the project because all that will happen is disaster
• Clear box testing is just as important as black box testing, because you can test triggers, views, constraints, and referential integrity rather than just observable functionality.
• Do continuous integration of databases – there is nothing special about data so why shouldn’t it be treated the same?
• Make it easy for people to do the right thing – loads of verbose documentation and white papers are less likely to be effective than a small number of run-able assets and some 1:1 support to integrate them.

What I think
• Obviously Scott was wearing his data hat this time, but clearly the whole session was predicated upon the fact that you believe relational databases are themselves the right solution…
• Really like the “repeatable results not repeatable processes” phrase, it is such a powerful idea – I am always battling the ‘1 process to rule them all’ crowd.
• Probably best whiteboard walkthrough of the “karate school” example I’ve ever seen.
• My approach to modelling has always been to define things in order of ‘most difficult to change later’ to ‘least difficult to change later’ so when you run out of steam the unknowns you’re left with are the cheapest ones.
• Abstractions – a lot of the examples in the talk were based around the assumption that applications directly accessed data, we need to think more about how we can build systems that don’t need to know details about schemas etc (access via object etc).
• Totally agree with the active stakeholder participation thing, if it’s important enough for them to expect you to deliver it then its important enough for them to invest themselves in it.

Dr Neil Roodyn - independent consultant, trainer and author
• A session titled “software architecture for real time systems” was about patterns for performance and performance tuning.
• Hard vs. soft real-time distinctions important.
• Time is so often not considered in system design – we think about features and quality and failure but so little about latency.
• Automated failure recovery is so much more important in real time computing because you cannot stop to allow human intervention upon failure.
• There are some strong similarities between real time computing thinking and distributed systems thinking:
1. Consistency
2. Initialisation
3. Processor communication
4. Load distribution
5. Resource allocation
• Asynchronous is kind of “cheating” as it leads to the illusion actions have completed before responses are returned.
• The 3 most important considerations in real time computing are time, time, and time (haha very good Neil).
• Common software tricks and patterns for real time systems (obviously assuming real time performance is your overriding requirement):
1. Use lookup tables for decision making
2. Used fixed size arrays
3. Avoid dynamic memory allocation
4. Reduce number of tasks in the system
5. Avoid multithreaded design
6. Only optimise popular scenarios
7. Search can be more efficient than hash
8. Use state machines
9. Timestamps instead of timers
10. Avoid hooks for future enhancements
11. Avoid bit packed variable length messages
12. Reduce message handshakes
13. Avoid mixed platform support
14. Minimise configurable parameters
• Overall know your system and approach performance tuning scientifically, observe where time is lost and spend energy there, don’t just guess.

What I think
• When we think about SLAs for latency we have to make sure we consider time from the users perspective – if you have a very fast back end but it takes ages for results to render for users, then is it really high performance?
• Even if you have a few processes in your system that need to be real-time, chances are the majority of your system does not, so don’t be afraid to mix memes because if you make the whole thing real time end-to-end you might be unnecessarily sacrificing some scalability or maintainability.
• Totally agree with points on developers needing to know more about the tin their systems run on and how this will lead to better software overall.
• I cant’ help but think we’re getting lazy from our (relative) lack of constraints – back when you had 32K to play with you really thought hard about how you used that memory, when you had to load everything from tape you really planned that storage hit…

Neal Ford – Software Architect and Meme Wrangler, Thoughtworks
• Fairly by-the-book session on “SOA won’t save the world” but via the humorous and engaging analogy of Chuck Norris facts.
• “Peripherally technical manager” types are the key causes for SOA overpromising.
• Good definition of a service as a unit of course grained, self-contained business functionality.
• The tension will always be on between centralised planning and tactical action, so you need to learn how to plan ahead without the planning becoming a constraint on business agility.
• Beware of “cleaning up” the messy communications paths in the enterprise by hiding them inside a box (ESB) – you still have same underlying problems, but you suddenly have less visibility of them.
• Beware of the difference between ‘standards based’ and ‘standardised’ i.e. most vendor ESB solutions share some common basic standard foundations but the functionality has been extended in a proprietary way – so it can still mean lock in.
• Keep track of the number of exceptions you’re granting against the governance you have in place – too many and you might have excessive governance.
• Using ubiquitous language is a must; perhaps even have a formal dictionary for each project.
• The business must be in integration projects not just initiate them.
• We all like a bit of ESB-bashing but they can still be useful for connecting up things like mainframes and 3rd party systems to your nifty REST/WS fabric.
• Exchanging metadata is a great way to negative communication parameters (reliability, security, synchronicity etc) between services and consumers.
• SOA is undebugable – but it is testable - so a good testing discipline is essential.

What I think
• The insurance industry is the place I’ve worked with the most legacy out there (you cant retire certain mainframe systems until all the policyholders die!) and the ESB attachment is just the right milk for this cookie.
• We as engineers contributed to the overselling as much as the Neal’s peripherally technical manager did – I think we got carried away with the interest that a bit of technology was suddenly getting and we were happy to be on the bandwagon as we usually struggle so hard to get technical things on the agenda – how could we not ride this wave?
• There are more benefits to SOA than reuse yet that’s all we ever seem to talk about. How about what it can do for scalability? Failure isolation? Concurrent change? Hot maintenance?
• Yes. BPEL. Terrifying.

The Big Summary
Overall I think this was a very good flagship event, my thanks to the organisers. The turnout was a good size – small enough to feel personal and allow plenty of networking opportunities, yet big enough to ensure a variety of engaging discussions.

The IASA mission is a worthwhile one, and one that I think we don’t do enough about in any technology discipline. Whether we’re talking about architects, system administrators or developers, how much time do we spend investing in our community? In ourselves as a profession? When was the last time you looked for someone to mentor? Went out of your way to share your ideas beyond your organisational boundaries? Visited a university to see what the talent you’ll have to work with in 5 years is actually learning? Investing in ourselves as an industry is undervalued, and I’m happy to be part of a group trying to address this.

If there is 1 thing I would change for next year, it would be the format of the talks. There are already enough conferences you can go to and watch someone meander through a slide deck (this was great meandering though!). If we change the focus so that speakers only use 50% of the allotted time and treat their role as setting the scene, then we could use the other 50% to get a real group discussion going on the topic at hand. I would certainly find that more valuable, and I would suggest that promoting high intensity peer discussions on tough technology and strategy issues would probably better serve the mission of establishing architecture as a profession.

I have been assured that you will eventually be able to find the slides from the formal sessions and keynotes here.

So that was IASA Connections. Tomorrow I’m off to UC Berkeley for the day, so probably best ease back on the beer this evening (it makes it harder to maintain the illusion of cleverness I’ll need). Sayonara.

IASA Connections - Day 2

Ceteris paribus, what comes after day 1 of a conference? Well, day 2 of course…

David Platt - teacher and author (and mad professor?), Rolling Thunder Computing
• Day 2’s keynote on why software sucks, which was adapted from his book by the same name.
• Lots of statistics on why users hate our software, mostly boiling down to the theory that software is built by geeks for geeks but actually used by regular folks.
• 2 million web users in 1994 has become 1200 million in 2006; and almost all are now laymen (in 1994 almost all were geeks themselves).
• Platt’s rule, “know thy user for he is not thee”
• Users don’t want to use your application, they want to have used it – so understanding that you provide a means to an end rather than an end in itself is a helpful thought pattern.
• Customers don’t buy software, they buy the outcome it’s intended to create, e.g. you don’t buy a drill from B&Q, you are really buying a hole.
• Transferring extra clicks into cash (salary costs of time spent clicking) is a useful way to demonstrate this to management.
• Suggested solutions:
1. Add a virgin to your design team, i.e. get someone never used your system before to try it and observe their challenges.
2. Break convention when needed – for example, why have a save button if you could just have the system save things as you go?
3. Don’t allow edge cases to reach mainstream – if something is really unlikely to be used by more than 1% of visitors, why include it in the main real estate?
4. Instrument carefully – watch user experience, track clicks, see what users actually click and how frequently, how do they arrive at certain pages/features?
5. Always question each design decision, “is it taking us closer to ‘just working’ or further away?”

What I think
• Some of the talk didn’t apply to Betfair per se, as online gaming/gambling etc joins the ranks of things like porn (can I say that?) in software that’s used for the experience, rather than to go through a process to arrive at a destination.
• Most of the examples are based on software that shipped with pointless features or usability issues. What wasn’t addressed was how architects and engineers relate to this – I agree that you can make terrible usability errors, but this is usually a decision making process by product managers and other stakeholders rather than the result of developers work. More material on how engineers can push back on these decisions would help us as a profession.
• Instrument carefully is good advice, and in my opinion is critical to effective A/B testing. Knowing what your customers like best coupled with the ability to try new layouts or versions in a lightweight way is a pretty important to keeping the best software out there.

Ted Neward - independent software development architect and mentor, Neward & Associates
• What are architects really about and what does architecture mean?
• We tend to get carried away with “what if’s” like what if I want to change logging system later so build in 96 logging APIs and pretty much use 1 or 2.
• Do we have architectural styles in the same way as building architecture has styles?
• Defining pragmatism – avoiding idealism in favor of something appropriate.
• There is no such thing as NFR, they are all just requirements.
• What do architects do:
1. Understand (what’s the problem, what’s the constraints)
2. Reassess (changing goals, changing constraints)
3. Explore (more potential solutions, new opportunities)
• Beware of the coolness trap – cool technology doesn’t mean appropriate technology.
• The 3 kinds of architects:
1. Infrastructure – datacenters, networks, storage
2. Enterprise – CTO high level leadership
3. Solutions– the “per project” answers to business questions
• Patterns are not architecture; they are simply a vocabulary we can use to have high-speed discussions about technical ideas.
• When designing systems, make big decisions first, then next most detailed, then next most detailed…
• Idea of architecture catalogue (the things you do as an architect)
1. Communication – transports, exchanges, and formats.
2. Presentation – UI, style, and delivery.
3. State management – durable vs. transient, context vs. process.
4. Processing – implementation, transactions, and shared data.
5. Resource management – location, registry, discovery.
6. Tools – languages, data formats.
• If you don’t explicitly define architecture it will be implicitly defined for you!
• Architecture is a bigger picture view than a developer would typically take.

What I think
• Not sure I agree with the bigger picture assertion, as I think this promotes the ivory tower perception of architects – and besides, successful software demands that everyone involved in building it understands the bigger picture.
• You need to consider future options, but take into account likelihood and difficulty to change and watch for entering diminishing returns!
• One of the reasons traditional architects don’t suffer the same difficulties we do is that the constraints and practicalities in real world construction are so much more visible. We need to put more effort into making these understood.
• I like the NFR idea because so few people talk about scalability and availability etc as business benefits, but they most certainly are!
• Agree with the patterns point, too many people get hung up on them but they do not describe a working solution.
• While I always like to talk about defining architecture as a trade, I am not sure about the catalogue idea – some of those decisions (like presentation, data formats, and languages) and surely best made by those closest to the problem; i.e. the delivery engineers.

Dr Neil Roodyn - independent consultant, trainer and author
• Gave a talk called “it’s all wrong, change it all” a rather provocative title that attracted me to the session…
• We know things are going to change, so how can we put ourselves in a position where this is as easy as possible?
• We’re in a change heavy industry, however, humans are naturally change averse so this is something we have to work hard at.
• Abstractions actually make it harder to change things, because they bind code to a particular abstraction – and people tend to feel emotional attachment to their implementations!
• Big visions are so important - always remember that’s what you’re delivering against, not the details.
• We get carried away with reusability – make sure the expected demand for it to be reusable justifies the cost of building it in a reusable way.
• How you design your system can make it more change-friendly or less change-friendly, this is how architecture can help.
• People are the biggest factor – there is a fine line between continuous improvement and gold plating.
• Don’t underestimate malleability of code.

What I think
• People typically associate change with risk or loss. Part of the trick to getting buy in to ideas and making people happy to adopt changes is to address good-old-fashioned “what’s in it for me?”
• Abstraction layers scares me because we’re not getting smarter at the same rate that we’re abstracting away from the primitives. I think we have a problem as an industry, which is convincing people to learn to be engineers (computer scientists) over quick and dirty coders (learn Java in 28 days and get a job).

Paul Preiss – President, IASA
• Dealing with how we establish architecture as a profession in it’s own right.
• To be a profession something must have:
1. An identifiable body of knowledge
2. Able to enter this trade without having to have been something else first
• Helping to realize architecture as a profession is a key purpose of the IASA group.
• America Institute of Architects formed in 1857 because (sound familiar?):
1. It is difficult to solve real architecture problems due to lack of targeted resources
2. It is difficult to find experienced architects
3. It is difficult to be of maximum value to our employers due to lack of stable job definition
4. It is difficult to tell a qualifies architect from an unqualified one
• Architects tend to be in one of 4 major groups:
1. Corporate
2. Vendors
3. Thought leaders
4. Service integrations
• Professional perspectives and skills taxonomy are resources we’re trying to build up – providing that ‘identifiable body of knowledge’
• The skills one must master to truly deserve the title of architect are:
1. The IT environment
2. Business technology strategy
3. Design
4. Human dynamics
5. Quality attributes
6. Software architecture
7. Infrastructure architecture

What I think
• Working towards formalizing architecture as a practice, a role, and a valuable and necessary part of a team is a massive undertaking – we have work to do here as an industry.
• Perhaps the key to this is working backwards from result? Maybe we need to ask ourselves “how do we test for architecture?” If we can work out what we’d be able to observe in an organization with this nebulous architecture thing in it vs. one that does not, then we’ll have an answer and some benefits to point at!
• In the short term we can’t test for architecture so how do we find them? Well you can test for thinking patterns, values, and philosophies so look for the ones that contribute the most to your organizations definition of success.
• Perhaps we need a charter? Some publically visible set of principles that anyone engaging a software architect should expect to be met?
• Good to see the ‘ities’ discussed as central to what we do.
• I still say it’s science Mr. Preiss – but pragmatic science! We solve real-world business problems by using our skill and experience to apply the right technology.

And with that I must once again apply my problem solving skills to the problem of unconsumed beer. See you tomorrow, same bat-time, same bat-channel...

IASA Connections - Day 1

This week I skipped across the Atlantic to attend the International Association of Software Architects & Architect Connections conference in San Francisco. Between my failed attempts at getting a chapter off the ground in Romania and being a long-standing member of the London chapter, IASA has been a hugely valuable organization to be associated with. There are regular activities at most major IT hubs in the world, with a strong focus on sharing ideas and meeting other people with similar technology challenges.

For the next 3 days I have a fairly hefty schedule of people to meet and sessions to attend - so I'll try and throw up a brief summary post at the end of each day covering what's discussed. So let's dive straight into day 1:

David Chappell - serial technology writer and speaker, Chappell & Associates
• Gave the first days keynote on cloud computing, covered the basics (this is still a very misunderstood topic) and direction over the next few years.
• Cloud computing falls into 3 main categories; cloud delivered products like windows live, attached services such as spam filtering and email antivirus, and cloud platforms like EC2 and S3.
• There will always be a mix of 'onsite' software and cloud based services in every organization in the future - so clouds will not rules the world!
• Advantages of cloud computing include lower costs and less financial risk (as capacity grows with demand, rather than having to build for peak from day 1).
• Disadvantages of cloud computing include trust issues, data protection, and regulatory requirements.
• Important to distinguish between consumer (free and ad supported and usually less reliable) and business (usually paid directly and covered by an SLA) cloud solutions.
• Underlying message is really about change and how hard it is - and how this relates to making the paradigm shift we need to get the enterprise to accept cloud as a valid technology.

What I think
• The cynical side of me can’t help but think we’re throwing things we've been doing for ages (like hosted email virus and spam checking) into the cloud bucket now that we came up with a new word for it.
• I consider some of the constraints of building on a cloud platform to actually be benefits – forcing developers to abstract away from hardware means you’re building systems that are inherently more scalable, available, and hot maintainable (whether deployed in a cloud or onsite).
• Lack of enterprise strength SLAs are keeping a lot of organizations away from cloud computing. Perhaps we as engineers have an obligation to help the business start to look less at SLA and more at actual performance? Most of the time a cloud infrastructure will trash a fixed internal hardware deployment in overall availability yet he SLAs don’t yet demonstrate this.
• Cost is very much still a factor when moving to a cloud deployment, but you move to a marginal cost consideration – you don’t want to run an expensive piece of code for each hit, since that unnecessarily increases the cost of the hosting. In a traditional model it doesn’t matter so much if you have expensive code running most of the time if you’re using capacity normally idle except during peak – now that costs you every time you execute it.

Dr Neil Roodyn - independent consultant, trainer and author (and big fan of XP)
• The “capital A” Architect vs. the idea of “little a” architecture, talking about what the role really should be and how it works in agile.
• In software we sometimes borrow a little to heavily from industrial engineering.
• Architecture is a practice created to try and put predictability back into a process that is inherently unpredictable (software delivery).
• Traditionally we try to separate design from construction – i.e. have really smart architects design the system and then any idiots can code it – and how this is flawed thinking.
• Change is inevitable in software projects, most of our traditional processes require rigid and stable requirements in order to be successful but this is not real life.
• Do the best developers only end up architects because we romanticize the role too much? Or it is the only way they see they can get career progression? Wouldn’t we rather have the best coders coding?
• Maybe we need to start looking at developers more as craftsmen and less like engineers?
• Be careful how you incentivize people, because that’s what you’ll get!

What I think
• Something omitted here is that the quality of an engineering team’s output can never exceed the quality of it’s input, so there is a strong dependency on working well together with customers to truly understand requirements and make the right tradeoffs.
• Incentive is a very valid point, if you measure architects on creating verbose design documents or UML diagrams then that’s what you’ll get – try measuring them on quality, working software.
• The people that know how best to solve problems are the people closest to their effects, so architecture has to be about harnessing the individual developers skill, experience, and exposure to the domain – not trying to tell them what to do.
• I quite like the label “engineer” because it stands for solving real world problems with technology – but I do agree the process does need to be considered to be more creative than it currently is. I recently read something that likened software development to film making, which I think is a powerful analogy.
• Why does IT have such a bad reputation? I don’t think it is as simple as failed projects, I think it is how they’re ‘washed up’ afterwards too. Working closer with customers along the way makes sure they’re aware of the impact of their decisions (changing/adding requirements) so you can easily end up with projects over time/budget, which are not considered failures by the business – because they expected it and were involved along the way.

Randy Shoup - Distinguished Architect, eBay
• The architectural forces governing eBay are scalability, availability, latency, manageability, and cost.
• Their architectural strategies are partition everything, asynchronous everywhere, automate everything, and remember that everything fails!
• Data partitioning is dividing DBs by functional areas (user, items, feedback) then sharding so that each logical DB is horizontally partitioned (by a key such as itemID) which provides scalability for write heavy data sets.
• CAP - their system prefers availability and network partition tolerance and thus trades away consistency. Variable window for different features.
• Functional segmentation at application server layer follows the same pattern as the databases (selling, search, checkout).
• Most operations are stateless which means the most basic, off the shelf load balancers can be used to horizontally scale application server tier.
• When travelling though the system, what state is generated is accumulated in cookies, URLs, and a scratch (periodically flushed) DB.
• Patterns used:
1. Event dispatch (new item listed, item sold)
2. Periodic batch (auction close process, import 3rd party data)
3. Adaptive configuration (dynamically adjust consumer thread pools and polling frequency based on SLAs)
4. Machine learning (dynamically adapt experience by collecting data for the day on for example recommendations and then try different ones next day)
5. Failure detection (collect data on activity send on multicast message bus and listeners watch for certain error messages)
6. Rollback (entire site redeployed every 2 weeks and no changes cannot be undone, features rolled out in dependency tree order)
7. Graceful degradation (switch off non essentials, defer processing to more guaranteed async messages)
• Message reliability dealt with via framework.
• Code deployment decoupled from feature deployment, so new versions can be rolled out with dormant functionality (they call this wired off) that can then be turned on gradually.
• NOT agile and don’t claim to be!
• Completely redeploy entire site every 2 weeks – although this is more of a constant, rolling upgrade than the big bang it sounds like.

What I think
• A very agreeable talk, particularly around deployment, versioning, and A/B testing.
• The architectural forces are very similar to our 6 principles (availability, scalability, maintainability, quality, security, and user experience) and really are just different words to describe the same good practices needed to meet the challenges we have in common.
• State is the enemy of scalability, and it is refreshing to see someone tackling this well - instead of saying "how can we build a distributed state tracking system this big?" they're saying "how can we build our system so that the functionality does not depend upon a central state system?"

Rob Daigneau - Chief Architect, SynXis
• Anti-patterns - human behavior in software projects and how to deal with it.
• Overdependence on methodology as solution to all problems and smooth project execution – not enough focus on personalities and simple human nature.
• Individual anti-patterns:
1. Cyber addict – using only IM or email etc for communication is very efficient, but not necessarily effective as you can lose the meaning of your message.
2. Fortress – not sharing knowledge and being secretive leads to good old fashioned unsustainable codebases and key man dependencies.
3. Ugly baby – too little care taken with how and when criticism is issues can lead to valid advice and points being ignored to the detriment of the project.
4. Perfectionist – believing that masterpieces are the only way every time can lead to unnecessary gold plating, artificially inflating the cost and time to market beyond what’s beneficial to the organization.
5. Conquistador – being overzealous about "the one way to do it" can mean failure to make good tradeoffs.
6. Workaholic – not having the right work/life balance will only lead to burn out and mistakes made.
• Team anti-patterns:
1. Hazing – throwing people into the deep end with very little coaching or attention to integrating with the existing team often means they take a long time to become productive.
2. Fight club – the organization can become a casualty in pointless battles between architects and developers, developers and testers etc unless teams learn to work together.
3. Wishful thinking – working out some estimates that show a certain milestone cant be reached in a certain timeline means just that, no matter how much you’d like it not to.
4. Firing squad – if people are scared to disagree or the culture doesn’t encourage the introduction of new ideas; the organization can be robbed of great value that stays locked away in engineers’ heads.
5. Too many cooks – if there are no clear boundaries then consensus becomes a huge issue and teams become paralyzed, unable to make progress because of endless circular discussions.
• Leadership anti-patterns:
1. Deer in headlights – if a leader wont (cant?) make a decision then the company stands still and competitors gain ground.
2. Cliff jumpers – taking risks for the sake of risks or making decisions on dangerously incomplete data can waste time, money and patience.
3. Spin zone - talking talk not walking walk gets transparent pretty quick.
4. Drill sergeant – exercising authority in an aggressive way makes very short-term gains and never gets a team behind a goal.
5. False summits - artificial deadlines or objectives can be dangerous, if the activity doesn’t turn out to be committed to, a loss of morale and trust in management can result.

What I think
• Reasonably fundamental stuff for experienced management but good to see it getting airtime in this crowd, architects typically come up via technical tracks and probably don’t spend enough time on their influencing and consensus building skills given how important a part of the role this it.
• This would have been a good opportunity to talk more about incentive and how it drives behavior. A lot of anti-patterns can be traced back to KPIs that encourage/reward the wrong things.
• Preventing burnout and keeping sustainable development going are some key reasons I like SCRUM – working in sprints gives you a cyclic mix of high intensity work interspersed by more passive, thoughtful phases.
• I liked the Gerald Weinberg quite "no matter how it looks at first it is always a people problem".

Ken Spencer – Practice Manager, Solid Quality Mentors
How agile applies to traditional, formal technical projects using a current project for the US government as a case study.
Manufacturing background so most examples look back to this experience.
Looking at why most projects fail – this is because people fundamentally keep doing what they do.
Given that we almost automatically repeat what we know, the good news is that this means being successful once significantly increases the likelihood of being successful again.
Spend commensurate amount of time planning - you need plans but you need to know when to just start work and learn from doing.
Good enough – what are the testing criteria and when are you into diminishing returns?
The ‘love curve’ which is a cycle you go through when taking over a failed project; initially distrusted though bad experience, then looked upon as savior, then reality sets in, then they see sustainable success.
Getting back to basics – when things go off the rails look first at the simplest units of work the team ought to be doing and this is usually where you will find your answers.

What I think
• Largely based on the agile manifesto, so we have a pretty strong philosophical connection here.
• Slightly damming of waterfall, at the end of the day it isn’t all bad and isn’t evil, I think it’s quite suitable for very repeatable, commodity projects as long-horizon predictability is possible and the phase space of all possible features is finite.
• Not sure I like the label ‘formalized’ when used to describe waterfall vs. agile, as this kind of indicates less control, visibility, and organization around agile when in fact I think these are some of its strengths.

Whew! Time for beer now, see you tomorrow…

Don't be a developer, be an Engineer

I do part time lecturing engagements in local universities whenever I can make a break for it from my day job - dodging and weaving through a kind of webscale Logan's Run while being hunted down by product owners and various other stakeholders, each individually as determined and unyielding as a T-800 on steroids, yet all the more terrifying for their emergent pack behavior. It's nice to have friends.

Over the last year I've made a quite a few such regularly scheduled escapes, as a result of which, I've noticed a disturbing trend. It can be summed up in one simple sentence; people are being taught to be developers, not engineers. Let me tell you a little more about that, and why I think it's a bad thing.

Developers vs Engineers...

Firstly, we need to appreciate the difference between a developer and an engineer. Again simple to summarize; a developer writes code, an engineer solves problems. I think it helps to define a developer in relation to an engineer, as I think "developer" is a subset of "engineer", so:

Engineers work as part of the business, helping to define the problem, helping to flesh out the specification, and bringing the practicalities and constraints to the table. They are creative; coming up with a number of technical solutions which solve the business problem, yet commercially savvy enough to evaluate the candidate solutions against the organizations goals to determine the best one. Engineers also appreciate that when it comes time to descend upon the keyboard in anger, writing the actual code is one little part of the whole job; they need to keep themselves and the rest of the team on track, report on progress, manage stakeholders such as operations, external vendors, and IT support staff, and plan for the release, version control, maintenance and quality of the system they're building.... On top of all this, good engineers are also constantly looking at their tools and their environments; working out how to make their own lives easier by improving the build systems, repositories, and automation that supports their work.

Developers take a tightly specified piece of functionality and express said functionality in whatever programming language their proficiency is in. You'll recall that we mentioned that as part of being an engineer.

What's wrong with this...

Is this necessarily a bad thing? It doesn't have to be, but here is why I think it is; we're not increasing the intelligence of our environments at the same speed as we're reducing our appreciation of the complexity they mask. Let me expand a little more on what I mean by this; by the time you drag together a new class in your favorite java IDE a massive amount of stuff just got done for you - even assuming you haven't included any non-essential resources from the toolkit. Machines will interpret your instructions so that they're executable by a given CPU. They'll manage the memory allocation and reclamation. They'll keep track of state and they'll write data for you. they'll take care of IPC and RPC and even hook you up to a database if you want. They'll put things onto the network stack and pull them off, and when worst comes to worst they'll give you a whole bunch of debugging information to ponder. Easy. But this kind of ready-to-roll stack doesn't just occur naturally in the wild; people made these machine-helpers. Those people were engineers, most certainly not developers. So, until machines can make us continually more advanced machine-helpers, we'll need a healthy population of those people to keep us going. Sooner or later everything we do is gates, voltage, magnetism, and wavelengths of light.

If a long term gloomy vision of the future doesn't do it for you, here is something quite selfish to get upset about. Being a .net or java developer is pretty much guaranteed to get you a job somewhere pretty quickly after graduating - but what next? While you're immediately productive from the first day, a lack of appreciation of the science behind what you do (how computers store and retrieve data, how memory is referenced and logical operations performed) will hold you back. You'll always be up against a steeper learning curve than your compsci-based compatriots, forced into learning about new things as a delta from what you know (an implementation like .net), rather than as their own unique systems - and that's before we even mention the wider appreciation of technology as a whole required to make a decent architect (or even make good implementation decisions). That's a serious competitive disadvantage in the workplace.

How did we get here...

We'll I'm not entirely sure we're "here" yet, but I tell you what, we're not far off. This is a trend thats getting stronger, and I think there are 2 key causes:

Firstly, people value immediate employability over long term careers. We're in competitive times, and now more than ever people view their classmates as competition for the next big step - turning that degree into gainful employment. Courseware is changing to accommodate this. We also live in times where people change jobs every few years - and total career changes are becoming very common and happening later and later in people's careers. This means there is a lot less incentive for students to consider the longer term, bigger picture when planing their studies, after all, they might only be in software for a couple of years and then become inspired by writing, or groundskeeping, or zoology. So are students and professors right or wrong here? Perhaps individually they're doing the right thing, but that doesn't mean that the combined effects of their decisions isn't going to hurt our trade.

Secondly, there is an imbalance of power between academic and commercial concerns. We are seeing unprecedented levels of cooperation between the academic world and the commercial world - which I totally, wholeheartedly, endorse by the way - there are joint research projects, internship programmes, technical community workshops and speakers, and a whole bevy of activities which help keep technology relevant and give willing students the ability to see how the science (key word there) they study can be applied to solve real life business problems. Sounds awesome, so why is it going wrong? To put it bluntly, I think academia are bending over for companies. Organizations are obviously only interested in participating in (or sponsoring) the aforementioned events if there is something in it for them - I know, I do it (responsibly!) myself. The most immediately obvious benefits are recruitment (hey, we have to work to get employees these days) and the ability to influence study programmes to favor the needs of the company in question. Here is where I think the academic side of these partnerships has a duty to push back - a balance needs to be maintained between what companies need today and making engineering, as a profession, sustainable. If we load all our papers and study plans to suit next quarter's recruitment targets where will we be in 10 years? I can see that this is the path we're starting down.

Can we save ourselves...

Sure we can - all is not lost. We might not be able to do much about the social pressures that form inputs into people's study plans, but we can at least help them appreciate the value that sound logical thinking, reasoning, and research/presentation skills brings to any career choice.

The single biggest thing we can do today is get the message to tomorrow's engineers - focus on becoming an engineer; being a developer will come naturally after that. You might have to study a bit harder, and look more strategically for your first few roles, but believe me - you are in the strongest possible position after you and your non-engineer peers have had 5 years to average out in the real world.

Faculties need to take action too - don't forget what you're for. It's always nice to have companies interested in working with you but you have a duty to maintain that balance. It isn't in the best interests of any organization to do long term damage to the core capabilities of the engineering industry, so you have to make them understand that's what they're heading for. If they are even slightly responsibly technical citizens (and I can't think of any I know not to be) they'll appreciate this, know that it is in their own long term best interests, and support it. Don't sell papers.

Rise up, good engineers, for we are becoming few!

Business Value of Back End Redux

A few posts back I wrote about the business value of good architecture, but I didn't mention at the time what I consider to be the single most important benefit...

The greatest advantage your business will ever have from a rock solid platform is time spent closer to customers.

If you have your availability, security, scalability, maintenance, technical usability, and quality nailed; then you can afford to spend more time right up against your customers - building the features they want, understanding how they use your technology and what they'd like to see in the system.  This is the stuff that matters to them, this is why they'll give you their time, attention, and hard earned currency.

When you are able to apply your best minds and your greatest investments into things that work directly for customers, thats you sweating your platform. The creative, innovative, unique things you can get out that your customers directly interact with will bring you material, easy to measure reward. And guess what, you're only in the fortunate position to be comfortable focusing here because you got that platform business sorted out early...

You won't usually gain business by being more secure, but you will definitely lose it by violating your users trust.  You won't usually get more customers by being more available, but you will definitely turn them away by being down. You wont usually gain business by having a scalable file system, but you will definitely lose it by failing to save and retrieve critical information in busy times.

Customers don't value a good platform explicitly, they just expect it to fundamentally be there. Want proof? Just watch them leave in droves if you don't have it sorted.

Heroism Hides the Truth

A while back, my trusty compatriot Dan Creswell posted this which I consider to be pretty good advice, but an even more powerful message when you think about it in terms of bad habits to avoid forming.

You have to avoid these bad habits because they become vicious cycles that are difficult to break. Consider Dan's point on heroics:

"Some managers encourage heroic behavior when they focus too strongly on can-do attitudes. By elevating can-do attitudes above accurate and sometimes gloomy status reporting, such project managers undercut their ability to take corrective action. They don’t even know they need to take corrective action until the damage is done. As Tom DeMarco says, can-do attitudes escalate minor setbacks into true disasters."

My biggest issue with this is that you've hidden the true cost of a piece of work from the business. How can you possibly make valid, prudent decisions that are in the best interests of the business if you don't have a true picture of what things cost? Cost is one of the most basic, and critical, inputs to how a business is run. Even though these heroics are often embarked upon with the best of intentions, hiding the actual cost of a feature actually does the business a huge disservice.

Once you're in this pattern, how to you reset expectations? There is no point at all in talking about the triangle and other real life constraints if you artificially modify the values though this kind of behavior. You also lose credibility with the rest of the organization - because working this way is unsustainable you'll eventually find yourself trying to get things back on the rails, at which point you'll be met with doubt; and fair enough too - you've been doing it so far, right?

It is hard to break out of these cycles and get your team back to a better work/life balance and so, much like smoking, my advice is the best way to quit is not to start in the first place!

There's something about Darwin

Biological computing isn't very well defined in the collective mind, and it means different things to different people. To me, it is more powerful as an analogy, a way of thinking about a systems overall behavior, than as any specific piece of technology.

I'm going to argue that there are 2 fundamental tenets upon which systems can be designed and built; mechanical and biological. Both start from a different mindset and both exhibit different basic capabilities that constrain what sort of things can be achieved within a given effort. I'm also going to postulate (triple word score!) that a system exhibiting biological characteristics is much more suitable to web-scale computing than it's mechanical counterpart.

Before we get into the meat of it, let's do some simple word association:

Forgetting completely about software for a minute (we'll deal with the analogies later), what comes to mind when you think about a mechanical system? Probably things like gears, cogs meshed together, reliability, tight integration, close supervision, a single contiguous chain, unthinking automation, dependency, predictability, and control.

Again leaving software out of the picture, what comes to mind when you think about a biological system? Probably things like change, iteration, flexibility, loose relationships between units, unpredictability, variance, the hive over the individual, response to environment, awareness, and cooperation.

The mechanical analogy is how we've all traditionally built computer systems. It kind of makes sense, we've been building mechanisms for a very long time and the logical, mathematical basis of computer science lends itself to this kind of thinking - representing in software a chain of cogs, each coupled to the next, powered by and dependent upon it's neighbor.  Look inside an old watch. Now take something small and seemingly insignificant out. Still working? Thought not.

A mechanical type of system has a number of 'moving parts', all of which must be functioning for any of them to function. The system is very easy to observe and very easy to predict - you'll pretty much be able to guarantee what state it will be in under any given circumstances. Information will flow through this 'production line' from one step to the next, operated on with monotonous exactitude. You'll have to keep a close eye on it though, because no steps in the chain will ever be aware of any others, and therefore can't make a decision about whether or not it's safe to pass their output on to the next step (will it be lost?) or whether they can trust the input of the previous step (is it corrupt?). Luckily it's pretty easy to observe, but unluckily it will more than likely need manual intervention to 'realign the teeth' when a gear goes bad. Scaling is tough too, you can always use bigger cogs but everything will only ever go as fast as the slowest wheel.

Recently we've recognized some patterns of behavior that occur in the natural world that would be of benefit in running a large scale computer system, and at the cost of increased complexity, we can replicate these in software just as we replicated a chain of hardwired moving parts.

The biological analogy is a way to build systems that are aware of their environments and their fellows, possess a degree of self-governance, and can respond 'intelligently' to changing circumstances. Look at a line of ants. Block their path, take some away, move their food source. Still working? You bet.

There are a couple of equally valid analogies to draw with the natural world; firstly observe how a single organism works on the cellular level. What matters is the whole animal functions competitively, individual cells are irrelevant, but they all work together to keep the 'system' alive. They have a way to replicate themselves, kill off corruption, heal 'faults' and modify their role as requirements change. They don't need externally monitored or controlled by a central point - they all have a little bit of this distributed amongst their number. Now imagine that animal is your system and the cells are servers, nodes, units of functionality - whatever it is that makes up your system. The other way to think about biological computing is to study social insects, cooperative hunters, and hives/swarms. You end up in pretty much the same place, a group of individuals which make up a whole with the whole being more valuable than the individuals, I think which one you prefer depends on where you went to school (or how much animal planet you've watched). It can be easier to think about bees making conscious (instinctive?) choices, communicating and reaching a consensus, than trying to divine meaning in the more mysterious chemical reactions that govern the cellular world.

A biological type of system is made up of a collection of disposable nodes, the sum of which is the whole system. Nothing is more important than the survival of the system, and these 'cells' must change purpose and even 'die' to keep the system going. There is no central government or master nodes, all the 'cells' are peers and the administrative tasks in the system are distributed amongst them. The system is capable of making simple decisions for itself, responding to environmental stimulus like load, failure conditions and available upgrades. Many of these decisions are reached by consensus, using protocols which create a 'hive mind' pulling together the opinion of every relevant individual and answering questions such as; is a feature down? Where am I? Am I the right version? What are my neighbors doing? Who will service the next request? Is this safe to eat?

Biological systems are more difficult to observe and their behavior becomes harder to predict the further into the future you look. Their decisions are only as good as the data they have and the rules you give them to assess it with - so you can end up worse off than if humans manually configured everything. A cellular system is composed of a number of small, totally independent pieces of functionality, and the upsides to this are scalability and partial failure. Scale comes through being able to arbitrarily add more 'cells' to your 'organism' as it needs to get bigger and bottlenecks are easier to solve as every individual component is able to operate as fast as it's legs will carry it (these designs are usually asynchronous). Partial failure means that even if all the nodes that make up a certain feature are down, your system as a whole will still work sans that one bit of functionality. Self healing is when a system is aware of what it should look like (as a gecko I should have a tail) and is able to recognize when it does not (ouch I don't anymore) and take some corrective action (better grow it back). This itself is a double edged sword, imagine you're intentionally taking something down because of a security flaw; you have to give yourself some way to prevent the page springing back up somewhere else. The 'split brain' problem becomes even more significant; usually in a split brain you face potential inconsistency with data being written in more than one place. With a system designed to repair itself, you might just end up with 2 complete copies working independently - that may not be all bad (depending on how your system works) but the ability to kill of duplication once connectivity is returned is something that needs addressed.

So now the previously promised postulation (I am good at scrabble). We know we can build systems whose behavior is easy to predict, they are easy to observe but they are rigid and require a lot of manual oversight and intervention. The mechanical way. We know we can build systems that are flexible, automatically resilient to environmental change and inherently scalable but they are difficult to accurately predict and can run away with themselves if not given the right information to from which to make deductions about state. The biological way. In a web-scale environment where availability, scalability, and the capability to ship frequently are such vital attributes of any product, it seems to me that we benefit more from thinking in a loosely coupled, compartmentalized, organic way than an interlocked, highly dependent, production line way.

The Law of Conservation of Complexity

The law of conservation of complexity states that there is a minimum amount of trouble you need to go to in order to achieve a certain outcome. You can always make it harder, but you can never simplify beyond a certain minimum level without compromising the result. Imagine a system with 3 components all of which are at their least complex, by further simplifying one component of this system you do not reduce the minimum overall amount of complexity required, because another of the components must become more complex in compensation.

A couple of real life examples:

Something customer facing. Let's say we're selling car parts online and we need to check availability from manufacturers before we accept orders, and we'd quite like to do this in real time so we don't set our customers expectations wrongly. We could do this in the back end by making complimentary calls from our application to the manufacturers inventory system as we receive requests from our customers. This might be a smart solution because we'd be able to do some short-term caching of results for popularly looked-up items. We could do this in the front end by getting our users browsers to fetch the stock levels directly from the manufacturers website or (if we're lucky) API. This might be a smart solution because it would reduce the load our application takes.

So we can simplify the front end at the expense of the back end, or we can simplify the back end at the expense of the front end. Either way we can't escape the need to make that request somehow without changing the functionality of the system [being aware of stock levels].

Something platform. Let's say we've got a distributed application that we need to provide strongly consistent data storage for. Since consistency is one of our requirements, and assuming we don't want to centralize the system, we need a way to make sure that when we write to a piece of data in one place, a contradictory update is not being made in another. We could do this on the storage side, by using a distributed locking algorithm or electing certain partitions to be the designated 'writer' for certain data items. This might be a smart solution because it's more portable. We could do this in the application by making the storage client responsible for locking every copy of a datum, or sending a message to all it's peers advising them of updates it wants to make. This might be a smart solution because it simplifies our data administration.

So we can simplify the application at the expense of the storage system, or we can simplify the storage system at the expense of the application. Either way we can't escape the need to govern writes somehow without changing the functionality of the system [keeping consistent data].

Just as energy is never lost, the minimum amount of complexity an entire system must have to achieve its goals can never be reduced, it can only be moved around.

Leading by Example

Another basic forgotten by so many.  Early in a switch to management, it slowly dawns on you that have a very different job to that of your team.  That's exactly as it should be, but just because you don't write [much] code anymore don't get sucked into thinking you can't still set the example!  You no longer perform the same daily tasks, but there are some basic professional conducts your team will always observe in you.

As a leader, your actions tell everyone what’s important to you and what you expect from everyone else.  It doesn’t matter if what you say is in direct contradiction with what you do – people will follow the walk you walk, not the talk you talk (risking an old cliche).

Think about the way you conduct yourself.  The times you arrive and leave, the dress code you follow, the type and tone of emails you send and your use of company resources.  If you’d be happy with your team behaving in exactly the same way then great, you nailed it.  If not, then think carefully about what you’re telling them...

Specialisation vs. Pooling

A while back I posted about ways to structure teams that will get the most appropriate input into your product development inside the smallest possible communication loop.  My argument was essentially for organizing by product or service - deconstructing your domain to the smallest units of functionality that can be owned end-to-end by a multidisciplinary team of 5-10 engineers.  I haven't yet come across a webscale system that this can't be done with.

Of course there are benefits to organizing by skill (java team, DBA team, QA team etc) such as ensuring consistency in how tasks are performed and creating a sense of community among engineers in similar roles.  Grouping people by the role they perform tends to make you really good at executing those particular skillsets.  Grouping by the product or service you take to market tends to make you really good at the subset of those skillsets that make the most difference to your product - what particular part of system administration is most critical to keeping a trading exchange online?  What particular part of testing finds the most critical issues in a deposit and withdrawal service?

Like most things one isn't the 'right' answer; it's really two different sets of benefits and you need to ask yourself which are more valuable to your company.

If you use off-the-shelf commodity systems to support traditional business processes then you're probably better off organizing by trade - chances are you'll get the most competitive advantage by keeping JD Edwards online, making sure your file servers don't run out of disk space and your CRM system is always up.  If you're are a technology-led company then you really should consider organizing by product or service.  You'll still always have the business of running the business (you'll need some corporate systems) but chances are you will create the most competitive advantage by being the best at those particular engineering tasks that are specific to your product.

Hot Maintaining Communication Systems

A while ago Joe Armstrong posted a few simple ideas for hot maintenance on his blog (and I heart hot maintenance).  It starts from a position of a fair few assumptions, such as your clustering service being OK with nodes arbitrarily joining and leaving, and any central data or state storage being OK with heterogeneous nodes connecting to it.  But hey - good design principles anyway, so for arguments sake let's just call those assumptions validated...

Someone posted a very relevant question on the comments - what about upgrades that require changes to the protocol between nodes?  This is a vital issue to address, because if you can't come up with a good solution for hot maintaining your communications glue, you have a limited amount of overall hot maintenance you can practically achieve.

The answers to this question are as varied as there are design patterns, but using the same set of assumptions we read into Joe's original post, here are 3 techniques that should be fairly portable:

Dual-headed.  If you're changing transports you can introduce nodes that communicate using both protocols.  You duplicate traffic (if you do it fairly unscientifically) but it's viable for a rolling upgrade.

Versioned interfaces.  If you're changing the message format, version the messages.  This will let you gradually move nodes onto the new build and give you some added benefits like A/B testing, faster future upgrades and rollbacks.

Translation gateway.  So far we've assumed a group of nodes in a single location.  For a distributed system with collections of nodes in distinct places, a gateway that speaks protocol 'old' on one side and protocol 'new' on the other might work best, letting you upgrade cluster-by-cluster without taking down the whole system.

To get the right answer for any given system the first thing to do is study the communication pattern - how much is management vs. service, what is node to node vs. node to DB, and how much crosses geographical locations?  This gives you your architectural options, beyond which you just need to keep your state tracking heterogeneous, your data partitionable, and (to make your own life easier) your inconsistency window as long as the business rules allow.