Monday 28 April 2008

Technical Debt

As engineers we tend to look at technical debt as a uniformly bad thing.  True, in an ideal world we wouldn't have any at all - but we live in the real world and sometimes it's the right thing to do.  Incurring some debt can get something to market quicker when time is really of the essence.  That's not inherently bad, the mistakes we make are choosing the wrong tradeoffs (things that are too difficult to refactor later) and, more importantly, not paying it back.

We need to change how our entire organisations look at technical debt and we're going to have to lead the way on it.

Paring it right down to basics, let's have a look at the definition of the word debt.  That hallowed source of organic knowledge, Wikipedia, says "debt is that which is owed; usually referencing assets owed, but the term can cover other obligations.  In the case of assets, debt is a means of using future purchasing power in the present before a summation has been earned."

So, technical debt is a facility that allows us to bring forward the realisation of a piece of technology before it would otherwise have been ready.  A temporary injection of something intended to be paid back at a later time.

How about engineering's role in creating this facility?  Well since we've made this loan I'd say that makes us the creditor.  Let's see what Wikipedia says about that "a debt is created when a creditor [that's us] agrees to lend a sum of assets [technology in advance] to a debtor [the business].  In modern society, debt is usually granted with expected repayment [refactoring]; in many cases, plus interest."

Plus interest - now there's an idea, but let's win this one step at a time...

Think about your position in technical debt; you are actually in control!  You are the creditor here and as such you are free to impose terms, terms like making sure it gets paid back!  You are lending technology to the business ahead of where it would naturally be created and you need to demand payback on a sensible timescale.

If you stand up for the right solutions, make the right tradeoffs and then ensure it's repaid then technical debt doesn't have to be a bad thing at all.  For this to really work you have to have senior management commitment; but whether you've got it or not don't let that stop you trying - today.

Thursday 24 April 2008

Romanian Trends

A few days ago I posted a summary about how our offshoring adventures have turned out so far and with said posting was a promise to put something up about the recent trends I've seen emerging in Romania.  Let's run through them in no particular order:


The most immediately visible trend on a monthly basis (and the one people usually ask about first) is how costs are changing.  In the year since we've been there we've seen costs creep up by approximately 14% overall.  The biggest increases have been in rents/leases, utilities, fuel and cost of credit (making existing borrowing more expensive).  Salaries have risen by about 7% and there is probably more adjustment to come as the cost of living has risen well in excess of this.  When you combine these factors with a volatile currency (RON) and a big supply of imported goods and services priced in Euros you've got a upward spiral going on.  Some of this might not line up with what you may read on the internet or in pre-investment reports.  Why might this be?  See the next point...

Public Information

In the UK it is easy to access fairly accurate information about market prices, salaries, accounting and employment laws - you get used to being able to make pretty informed decisions based on reliable data.  Not so in Romania - historical data and public records are notoriously difficult to get hold of, usually paper-based and tend to say what they recorder thought was best.  What you read in authoritative reports is usually a kind of average of what's been discovered - so it isn't that it's unreliable, it's just that it's as reliable as you're going to get.  Another thing to bear in mind is the discrepancy between cities is a lot more significant than in the west so national averages don't really carry much weight in real life.  This is supposed to be about trends and there isn't really a trend here - I guess I included it because there should be a trend toward making more accurate, relevant market information freely available.  People need confidence before investing and facts are a good source of that.


By now everyone must know what I think about Romanian engineers so let's skip that and talk about the market.  It's been an employers market here for a very long time but the balance of power is gradually shifting to employees.  Between the increased opportunities for working abroad EU membership has brought and the rapidly growing number of companies opening offices in Romania skilled Romanians have a lot more choice in work now than ever before.  This doesn't bother us much as we've got a very unique offering but it does mean having to work harder this year to attract the right talent and I only see this becoming more competitive.


Universities and technical colleges are very good at adapting their curriculum to match what the market demands from graduates, but maybe too good...  They need to strike a balance between the practical (preparing people for employment) and the academic (teaching those fundamental scientific principles and research/problem solving skills).  At the moment more and more education is based on equipping students with immediately usable skills but it seems to be at the expense of the basics like how logical operations work, communications protocols and how machines interpret instructions.  If this keeps going in the direction it's headed it will lead to graduates who can immediately put together a basic system using (for example) rails but will struggle to pick up the next thing, and the next and the next.


On the whole Romanians consider working for western companies desirable and are attracted to our values and management philosophy.  From what I have seen traditional Romanian management styles seem to revolve around clock watching and micromanagement - it's a very 'production line' philosophy.  I haven't come across a good management school/academy here and perhaps that's why?  The flipside of this is there is no shortage of people with good leadership potential and we're experiencing a surge of interest in moving into management roles - it seems like once the right culture has been established they are keen to take it on.  We're busy planning how we'll support transitioning people into these roles and if there are any significant lessons in it I'll be sure to post about it.

Bottom Line

In summary we’ve seen prices rise quite considerably in just over a year here and the currency has been quite volatile but where it could really all go wrong for Romania is that while prices have risen there has been no corresponding increase in quality of infrastructure or improvement in central services (i.e. how the government administers/supports businesses etc).  If these things are not carefully planned I think Romania could find itself in a situation where it prices itself out of the reach of that first wave of cost sensitive companies prepared to accept shortcomings in certain areas in order to access such great talent at low cost and at the same time not be providing the stable platform that the next wave of less cost sensitive global expansionists expect before they’ll come into a new economy.  Competition for skilled employees is certainly increasing but my feeling is we've got a few years worth of recruitment tactics yet before having to reassess growth strategies.

Monday 21 April 2008

Offshoring - What's the Skinny 1 Year On?

A little over a year ago I left the tree lined and centrally-heated comfort of Surrey and ventured boldly forth to set up an offshore development center.  I toured the usual suspects (ah expenses, the last true bastion of the free lunch) and eventually settled on Romania via a journey documented by such worthy scribes as, ComputerWeekly and even myself.  While setup was all very exciting, one thing that doesn't seem to happen very often is a bit of a retrospective - once the dust has settled we should ask ourselves; what has it really been like?  Did it live up to expectations?  Did it work out as planned?  What unexpected challenges did we encounter?

Since such follow up stories never seem to emerge naturally I thought I'd take a few minutes to do a "1 year later debrief" in the time honored Clint Eastwood style.  I hope it's useful data for anyone considering setting up an offshore operation in Eastern Europe - I know I would've liked the benefit of a bit more experience before we got started!  So, Romania, what would Mr Eastwood say?

The Good

The first thing to say is the quality of the engineers over there is outstanding, they have truly exceeded our expectations.  As I've said before we have very complex business problems to solve and they can't be addressed with run-of-the-mill-3-tier-java-tomcat-and-a-DB thinking; we need innovation and a real desire to tackle the fundamentals of a problem - not just contort it to fit ruby on rails or [insert your favorite stack here].  The guys are passionate about technology and keen to get to grips with new patterns and techniques which can be a breath of fresh air when you're used to a more pessimistic, risk averse crowd.

Cost is of course the other major advantage of Eastern Europe.  I'd add that Romania isn't the cheapest place to set up, in practice things ended up costing around a third of our UK benchmark; but what it does have in spades is cost effectiveness when you weigh up what you're getting for that third.  If, like us, you've got reasonably unique requirements then you'll need to retain the domain knowledge as you build it up (avoid the low-cost high-churn models) and you'll want more engaged, creative engineers - you can do worse that here.

In general Romania has a friendly atmosphere, a unique (but hard to find) culture and almost everybody you need to deal with speaks excellent English.  I spend the majority of my time with the team there and really enjoy the buzz in the office - it's an environment conducive to solving tough problems with new technology.

The Bad

The main things I've found frustrating about Romania have largely been administrative.  From the very outset there is little central support for new businesses or companies looking to open offices.  A little light reading would suggest I'm wrong here (there are a lot of schemes publicised, particularly since EU ascension in January 07) but the reality on the ground is a little different - and what is available is inconsistent and [mostly] retrospective.  Another aspect of administration is paperwork and I can tell you, Romania scares a lot of trees!  Many everyday processes (payroll, taxes, registration and even utility bills) are manual and require the ferrying of individual pieces of paper between different buildings all over town to be stamped and signed by different human beings.  Then you get to ferry said paperwork back so the first set of human beings can stamp it again to verify the next set had stamped it after them.  Keeps the admin staff fit at least.

Another small gripe I have is the difficulty in finding reliable suppliers.  This goes for everything from computers to office furniture to shipping.  I haven't come across any of the base dishonesty a lot of those global risk management reports would have you believe is de rigueur, simply a lot of companies so desperate for your business that they make promises they can't keep.  Yes, we guarantee all your desks will be here by Tuesday.  Yes, we can assure you we have those servers in stock.  No, you clearly couldn't and I would much rather have known about it up front.

Finally an issue you wouldn't expect in such an administration-happy country; getting a consistent answer.  This is a such a big issue it even constituted a formal finding in the preparedness for EU membership report, recorded as unifying the interpretation and application of the law and progress here is slow.  For example, when trying to ascertain your obligations with respect to taxes, reporting and employment law there are as many different answers as people you are prepared to ask - and very little in the way of public records and other reference material available online.

The Ugly

There are a number of cultural differences between the UK and Romania and if you're not careful they can bite you.  An example of something we learned the hard way is the approach to public holidays (referred to as free days in Romania).  In the UK we pretty much view said holidays as bonus time off regardless of any historical significance so when a project is running behind and we need to pull some extra hours that's usually OK (as long as we get the time off later!).  Not so in Romania - generalising to the same degree we just did about London; people here tend to be quite religious and do have an attachment to taking the break on the formal occasion rather than some later, arbitrary time that better suits the company schedule.

Another thing that can be tough to get to grips with is the idea that everything is a continuous negotiation.  Initially it's a refreshing challenge - there's quite a lightweight legal framework for the more common operations (like leasing and real estate transactions) and most organizations have very few standard terms (compared to the volumes of contractual nitpicking you're used to wading through) so it means you've got a lot of scope to come to a highly tailored agreement.  Agreement being the key word here; quite often suppliers turn up every couple of months to negotiate new terms to a 2 year deal.  On the upside they do tend to have a sense of fairness (it is not always about gouging you some more) but sometimes you'd rather just have something sorted for 24 months, be able to depend on it and just get on with other things.

Perhaps it's an artifact born of sudden freedom after so many years of zero tolerance and no flexibility but, come on, a deal is a deal.

The Summary [caution: not strictly a Clint metric]

I still stand behind the decision to start our offshore development center in Romania and we'll be here for the long run.  Overall the benefits stacked up much as we predicted and we're getting great results and favorable economics.  It's cold in winter, hot in summer and they put on some excellent women and wine.  It isn't without it's challenges but the tradeoffs are good.  That said, if everything in Romania worked as smoothly as it does in London then it would be London and, well, we could have stayed at home.

I was going to include in this post a rundown of the trends I'm seeing on the ground here and my predictions for the future but it's gone on way too long already (well done if you made it this far).  So for the sake of your eyes and my fingertips I'll post something separate in a couple of days.

Thursday 17 April 2008

Corn Rules

I just watched Michael Pollan's TED talk on the relationship between humans and the plant world.  I've always been fascinated by the idea that perhaps we don't have the whole biological world totally under our control as we like to believe.  Perhaps everything is getting exactly what it wants out of our domineering programmes of domestication and agriculture.  Better yet, perhaps some of these 'subjugated' organisms are really driving us to build, farm and plant...  Worth a 17 minute work break.

Wednesday 16 April 2008


Here's a couple of classic ideas that are still good distributed system thinking exercises today - the ACID/BASE concepts are essentially a pair of acronyms describing two opposing extremes of system attributes.  ACID is a fairly established computer science principle from way back in the DB-centric days (it's what school taught us was important to systems) and BASE is what more and more of today's systems are being forced towards through web scalability problems.  Let's take a look at both:


ACID stands for Atomicity, Consistency, Isolation and Durability and usually describes the behavior of a tightly bound, data centric system.

DB people will have no trouble with atomicity, this is the property of a database operation which offers the "all or nothing" guarantee - for example we won't credit your account if we can't debit your credit card.  Works on the principle that you'd rather not do one without the other (like our banking example).

Consistency is a massive topic on its own but if forced into a one-liner I'd say it's a measurement of identicalness for any given data set no matter where or how many times it appears in a system.  Assuming ACID consistency means strong consistency (and from context that's a safe bet) an example here is an East-West replica set where if you change your password via a client of one DB our system dictates that change must appear in our other DB as instantly as the speed of light allows - because no matter when or where that unit of data is observed from every copy must always be the same.

The next property, isolation, can almost be described as locking (but can't the whole thing?).  It is the process by which data being updated is hidden away from other processes/nodes until it whatever value-changing operation is completed and the changes are committed.  Isolation ensures integrity (for example I can't spend the same money twice from my back account no matter how quickly I put my card in another ATM) but is the enemy of concurrence.  A lot of people think durability is about availability and it kind of is, just not the sort of availability I settle for.

Durability is that property of a data store that promises to keep your information safe - in other words once the system has satisfied atomicity you will always be able to recall that data accurately.  This is usually about writing to persistent storage and journalling changes in a transaction log.  If, for example, the disks that the database files reside on are destroyed then I should be able to replay said logs against my most recent copy (do backups or die) and end up right back where I was.


BASE stands for Basically Available, Soft state and Eventual consistency and describes the attributes of a loosely coupled system valuing availability and tolerance over strict consistency and isolated operations.

Basically available is recognizing that the availability of a system as a whole should never be the same as the availability of any individual node/feature/service within that system - your system should survive the death of it's component parts.  It's saying we never try to guarantee that a given data store will always be here (as ACID assumes) we just build a lot of tolerance into the system so that there is usually enough of it alive at any given point in time to present something usable.  More than that, basic availability is about saying we'd rather deal with conflict resolution and versioning issues than pay a scalability and availability price with isolation.

State [in this context] is essentially whether we're required to track, remember and somehow use preceding events in a flow of activity (stateful) or whether we can provide the right responses to requests without knowing what the immediately preceding activities and their results were (stateless).  Stateful operations tend to be unique to individuals (or groups); for example account management - you want to make sure a user has successfully logged in before you let them edit their address or payment details.  Stateless operations tend to be wider in scope; for example pushing out price feeds or updating a product catalogue - you usually want everyone to see the same thing no matter where they are.  Traditional state management is about recording a series of values in a centralized storage point so that other processes or nodes can retrieve it whenever they need to make decisions based on what you did before.  Soft state is about pushing back the use of state until the last possible moment it's needed and using techniques like leasing/refreshing and partitioning to reduce dependencies created by a single source of truth.

Now we're left with eventual consistency which says that when a distributed/replicated item of data is created or changed in one location it will be changed everywhere else after a period of time rather than immediately in ACID's strong consistency model.  Continuing with our product catalogue example you'd want to have that set in a few locations so that it was close to where your customers were (performance) and you can be sure could always show a list/search result (availability).  When you add a new iron to your homeware section does it really matter if it doesn't show up in every location for 10 minutes?  Of course the benefits are based on the premise that we'd rather have a quick answer that isn't 100% up to date than a slow one that is (or no answer at all).


Webscale systems are moving along towards BASE as we realize how valuable a commodity availability is and how difficult it is to scale systems to meet the demand the Internet creates.  We're suddenly faced with challenges that are impossible (or grossly unprofitable) to meet with traditional thinking and are forced to start making tradeoffs that just don't match ACID - relax consistency to buy availability?  Relax isolation to buy scale?

One last thing that's worth noting is different parts of the same system can occupy a different point along the ACID/BASE continuum.  Thinking about that iron we added to homeware we might not be bothered about it instantly showing in every replica of our catalogue (eventual) but what if we realise we made a mistake on the price and we're selling them for much less than they cost us?  We might want that sort of update to be immediately pushed out super quick (strong).

Saturday 12 April 2008

Pecha Kucha

Not the pokemon, the presentation style.  These days I am particularly interested in unique presentation styles that promote attention and retention as I tend to do a lot of public speaking.  And when have to steer a reasonably big organization you can't expect to be able to spend long periods of time with every single individual so being able to effectively get your message across in a more public 'batch' forum is a key leadership skill.

What's attractive to me about Pecha Kucha is the 20/20/640 concept; you use 20 slides each shown for 20 seconds (for a total of 6:40 talk time) and that's it.  I think it is particularly good for engineers because we just love verbosity.  There are so many great things you could say about your architecture, about your new product or how you solved problem X while still keeping that instant fail-over why wouldn't you want to say them all?  Because people will die of old age.  Putting such strict constraints on the normally freestyle art of presenting forces you to think carefully about the information that makes the biggest difference, culling the marginal and the good-but-irrelevant.  It's a lot harder than it sounds, as George Bemard Shaw once said "I'm sorry to have written such a long letter, but I didn't have time to write a shorter one."

There is always gold on Presentation Zen but my current favorite is punchy pace, lots of images and heavy use of repetition; like Dick Hardt's Identity 2.0 keynote.  It makes for a big deck but if you have a nice flow and keep it moving it really holds people's attention and leaves an impression.

Thursday 10 April 2008

Agile Architecture

There are a lot of opinions on how to handle architecture in an agile SDLC - so I thought why not throw mine out there too?

The first thing I think is important is to maintain a good reference architecture that readily explains the whole system including shared infrastructure and how to access it.  Having strictly defined interfaces/APIs and communications infrastructure rules is fundamental to being able to build anything with a distributed system as a platform.

Your biggest need for architecture is going to be early in a new project to build a new feature or product - so for the first few sprints reserve extra long times for sprint planning and use that time for consultative design sessions.  Lots of people use a "design sprint" or two but I've never been a big fan of that - I think a sprint is the best use of time once you've planned out exactly what tasks you're going to do and have an estimate (based on facts) of how long they're going to take.  The dedicated, concentrated burst of activity that traditionally makes up a sprint isn't the environment most conducive to the contemplative, consultative, learning process that is design.  Besides, having a designated "design sprint" tends to make you want to answer all the questions and design everything up front - and we all know that's kidding ourselves, in fact it's probably why most of us adopted agile in the first place!

Next you should make your first sprint (and perhaps even the second depending on the size of the project) long enough to build the entire width of the whole system to the barest, most skeletal framework possible (a month for example) and the depth one customer feature to [one normal sprint's worth of] completeness.  Then you can go back to your shorter, more focused, high energy sprints (2 weeks for example) and flesh out the rest of the system feature-by-feature.

This works on the theory that "deliver working, demonstrable software every sprint" overrules "many short, predictable cycles of delivery" and in any case I believe the output reflects the right mix of performing looser, more creative solution architecture activities and tightly timeboxed, ferocious task burning activities.

Tuesday 8 April 2008

There is a lesson in here if you look closely...

We had a tough weekend but if we take the right things away from it then it will be one of the cheapest lessons we could pay for...

A while back we launched a new product; starting price.  The concept is pretty simple and analogous to it's parent product (the sports exchange) where for every event backers are matched against layers.  In the case of starting price bets, we are matching starting price backers (who have specified a stake) and exchange backers with starting price layers (who have specified a liability) and exchange layers.  So just like traditional SP but with our unique peer-to-peer slant on it.  There is one more big benefit I like, SP lets you set a persistent bet at the beginning of an market which stays open once the market goes in-play (these used to be voided) which is big for us because that can be tens of thousands of potential matches depending on the event.

So what went wrong?  Depends who you ask.  Let's say I ask my monitoring systems - nothing.  How about my sysadmins - nothing.  Intrusion detection - nothing.  If I ask my developers, testers, mathematician - still nothing.  Anecdotally (the currency of monitoring) I can say nothing went wrong at all yet I am still not happy.  By "I" I really mean the business and the business wasn't happy because our SP was being calculated at around 50% of the industry benchmark - on the one hand if that's the true market value then that's the true market value but on the other hand it just isn't how we'd planned that the product would perform.

So here is what I think went wrong - we built a sunny day system.  Product managers will always write specs based on what they want to happen, the journey they have planned for their users.  They'll hardly ever write about what they don't want to happen or think about an alternative journey a user might take through the system.  It is just as important to think about what else people might do, through legitimate or malicious use, with your functionality - if the only input to your design is the desired behavior then you're doomed (unless you are unfeasibly lucky).  On top of all this you should also add common failure scenarios and some sensible behavior under exceptions and variable performance (latency, disk space etc).

What was our newbie mistake?  Most people that are new to a trading system back only and it takes them a long time to build up the courage to try some lay betting.  Good old fashioned assumption; "people will use SP to back and lay" rather than "most people will use SP to back only" and I can't help thinking that had we considered this simple piece of human behavior (that we considered undesirable) we could have built in some rule - min and max thresholds or distance from trend etc.

That's software in the real world.

Monday 7 April 2008

Get Attached to Results

It seems a little obvious to say but results matter - the reason we adopt certain processes or take certain actions is to create a desired effect, not just to exercise our procedure muscles.

All well and good but so often I run into people who are so wound up in the process, so attached to checking off the steps like a to-do list that they forget to look for the change they're supposed to be making by checking that list off.  This goes for everything from project wash-ups (looking to improve the SDLC) to performance reviews (looking improve the quality of individual's contribution and give them a workable career plan) to basic, everyday meetings (looking for a decision or consensus in a group).

Keep your eyes on the prize.  If you get to the end of a process and nothing is different you are not done.  Go back and try something else.  Get attached to creating the difference, not stepping through a script.

Wednesday 2 April 2008

Is Agile Gaining Wait?

No that's not a typo, just my rapier-sharp wit (ahem).  Reflecting on us as an industry I am worried that we're doing to agile what we did to SOA - productise it.  SOA has become such a blurry, distorted concept that there are now legions of consultants making - what do the kids call it these days - fat cash just explaining it to companies.

How did we get there?  Well we started off with classical distributed computing then came up with a thought process (SOA) we could lay over the top to help design distributed architectures that better reflected the business processes they supported.  That made the concept a little more accessible and suddenly a lot of organisations wanted it on the action.  Cue vendors because hey, if someone wants something we can always find a way to license it.  I used to call SOA distributed systems with some conventions (rules and patterns) but now I call it distributed systems with some conventions (sales conferences).  There's that wit again.

But we're on a tangent - back to the point:

Are we doing the same thing to agile?  Are we diluting the power of its concepts by trying to shrink wrap it into something we can market, ship and pay maintenance on?  If you look at the original agile manifesto (arguably where it all began) you'll read some pretty agreeable statements of principle, like "individuals and interactions over processes and tools" and "responding to change over following a plan" etc.  I'm all for that but what scares me is every day I see more tools and processes 'for agile' and more consultancies teaching and preaching it.  I'm getting asked questions like "what's our process for agile?" and "this book told me to always do X" more frequently.

Don't get me wrong I believe tools are important and as for all the preaching - well if something is valuable and can help us as a profession then we should spread the word.  Being an evil capitalist at heart I also don't see anything wrong with getting paid for that, I'm just saying we need to keep our eyes open for the boundary between this and when we start to reduce it's effectiveness.

I guess what I'm saying is I just don't want to overcomplicate and burden with process something whose greatest strengths are its lightweight simplicity and use of old fashioned common sense.