5 tips for more strategic budgeting

With the end of the financial year looming as the next major fiscal date in many UK company's diaries, IT people everywhere will soon be embarking on the wishlist-to-business plan journey that is the budgeting process.

There is already a lot of good advice out there on how to estimate numbers and prioritise what makes the cut, and that can be a brutal process, so here's a few things I keep in mind to make sure I'm not being too short sighted:

1 - Know the cost of a customer. The rest of this list isn't in any particular order, but this is definitely number 1. Most good CIOs and CTOs I meet know what this disk array costs vs. that disk array and what these licenses cost vs. those licenses, but not that many know what a customer costs to acquire and convert or to reactivate if they drift away. The relevance? Knowing that means you can work out exactly how many frustrated customers you can lose because of that bug before its worth fixing, or that slow server before its worth upgrading. In this light, things that might otherwise not have made the cut become the no-brainers that they should be.

2 - Don't be influenced by trendy buzzwords. There is always a meme that vendors will be able to exploit because of our inability to apply common sense and logic to a pitch on whatever the currently fashionable thing is. SOA, virtualisation, and cloud computing come to mind whenever I recall ill thought through spending sprees. That's not the same as saying that these - or any of the other candidates in the same category - aren't good technologies with valid applications, just that any investment in them should be handled with the same pessimism and judgement of real business value as money spent anywhere else. Just adding 'cloud' onto the name of a product doesn't change it's fundamental ROI, so don't let these slip by you unchecked.

3 - Know the cost of scale. When sizing up a new project, give growth plans some weighting rather than just initial costs. Depending on the degree of speculation involved in your plan, taking cheaper entry options can sometimes bite you when you reach scalability limits.

4 - Wider consultation. A year is a long time and a budgeting exercise inevitably involves dusting off your crystal ball and trying to foresee everything that everyone is going to want in the coming 12 months. Good luck with that and, although it seems a little obvious, spending a bit more time with each area of the business trying to coax their plans and ideas out of them will at least expose the bigger ones up front.

5 - Increase priority on core things. Another one that sounds obvious but - through its limited practice - clearly isn't. In tougher economic times it is particularly important to give the most crucial things the most attention, and it can be mistake to favor too many might-lead-somewhere-might-not initiatives over those core basics upon which the business depends today. I think it is critical to set aside time and money for innovation and to take a punt on those less-certain ventures that our instincts tell us have that commercial je ne sais quoi but it needs to be proportionate to the main line. This is important when deciding what to drop in order to bring your wishlist and the available cash resources closer together - big but critical projects can be easy targets because they pull back more cost in a single swipe than a number of smaller but less meaningful items will.

The hidden wisdom of Kevin McCloud

I was lucky enough to get a copy of the Grand Designs Handbook for Christmas - it's a compendium of building projects from the channel 4 show (the book, not Christmas). The first few chapters have a lot of lessons learned and good practices picked up from 9 seasons worth of experience with, well, variously successful building projects.

Reading some of the advice, there is a lot that can be applied directly to software projects. There is a section on 'how to behave with your team' which is aimed at helping people keep their building sites productive - but the advice translates well to technical projects. Here's a couple of my favorites along with my join-the-dots:

"Don't give verbal instructions on site unless they've been agreed with your consultants and they're backed up in writing. If you just issue verbal instructions, you risk confusion and possible extra costs."

This is a top one for me - especially in the agile world where many interpret 'agile' as 'make it up as you go along' and then end up disappointed. I think maximizing interaction between delivery team and customer team is a good thing especially when both sides recognise the boundary between clarification and change in in-flight work.

"Do take time to prepare detailed written briefs setting out all your requirements for all your consultants. A good brief is an essential part of the design process."

One of the simplest formulas the govern all project delivery (hardhat or mousepad) is that the quality of what you get out with never exceed the quality of what you put in - yet this seems to escape so many of us so often. What you invest in clear instruction and regular contact with developers is a key variable in how happy you'll be with the results.

"Do allow your experts the freedom to do their jobs. Stand back from the project and enjoy your role as client; making decisions and hanging around, generally being a good egg and imbuing the world with optimism and excitement."

I don't exactly advocate stakeholders 'standing back' from a project, and I don't think that's what Kevin means either. Don't micromanage your delivery team and, as a customer, be sure to perceive the difference between decisions you should make (and don't let the team wait for those) and decisions the team should make (and let them use their skills and experience to make them).

"Don't listen to other people outside the professional team. You'll only end up getting very confused."

External feedback and extra ideas are good, but remember that an engineer knows his business best and don't fall into the trap of listening to someone else simply because you like their answers better. I consider myself relatively smart but even I'll say all sorts of wacky things when I don't have any skin in the game...

There are a couple of other sections with close analogies to technical projects - how to be a perfect customer and how to put a good brief together - which I think are 2 other critical ingredients to any successful engineering project (construction or software) but they'll keep for another post.

Standups vs Team Meetings

When teams first start to pick up SCRUM, there is a tendency to let stand ups replace regular team meetings. The risks here are that your stand up will elongate and get off topic because it is the only chance you get to talk about issues as a group, yet your team cohesion will still suffer because no matter how long you can string out a stand up in the morning it won't be long enough to table all the things you need to get through as a team.

A stand up is a project-oriented meeting, all about running the day to day work of the team, has a tightly fixed agenda, is time bound, and specifically focused on what's required for today.

A team meeting is a team-oriented meeting, all about maintaining the team, continuous improvement, future requirements, and focussed on bigger picture topics.

They are definitely two very different meetings with two very different purposes, and both of them are necessary for a team to run smooth projects and keep productivity and job satisfaction high.

Introducing Sporting Index Super Simple Integration

This year we decided to give our B2B customers an early Christmas present - SSI.

Sporting Index (well, really Sporting Solutions, our B2B brand) provides pre-match and in-play pricing and content to fixed odds bookmakers for hundreds of markets across 16 sports with more events from more jurisdictions being added almost every month.

Traditionally we’ve done this by pushing XML documents around over reliable transports and leased lines. This works very well and there is nothing inherently wrong with this solution, but we weren’t happy that customers had to install expensive fixed lines and associated hardware, and our back end is capable of producing multiple-thousands of price changes a minute – well beyond the rate at which some of our customers can consume them.

On top of this, our customers wanted the low latency and reliable updates that come with a push solutions, but also the easy integration and flexible client-side controls that comes with a pull solution.

Well, you asked for it and you’re the customers, so let’s unwrap this bad boy.

Super Simple Integration is essentially a RESTful web services API available over the good-old-fashioned Internet. The feed is as easy to consume as you’d expect from any web API, and we’ve spent a lot of time on the data formats and doing some clever pub-sub tricks in the background – thus all the benefits of the heavyweight XML services but through an integration which is quick and easy to get to grips with.

There’s a lot to this, so let’s just cover two of the most common use cases:

Get Content

Our content products can be used to populate websites with upcoming fixtures and players, and to power pages or widgets showing points, corners, goals etc as they happen in running matches.

Content products are mostly stateless and work on a fairly simple request-response model, which can mean no integration work at all. Well that’s half true – you still need to consume the data from the API – but you don’t necessarily need to build a repository on the customer side or have to match up schemas.

Let’s say that wanted to build a little scoreboard for my site to show my visitors the progress of a live England v New Zealand rugby game;

You get the content you want, as often as you want, delivered back in name-value pairs. There are also a couple of methods to call to list out what sports and events are available, so you can pick out what to pull into your scoreboard app.

Pretty basic stuff so far, where it really gets sweet is how we implement real time price streaming.

Get Prices

Our price products can be used to price up sports betting markets, either as a risk advisory tool for human operators or a totally automated pricing service, and cover a wide range of sports in-play with high fidelity price output.

In price products - particularly once fast paced events turn in-play - state and reliable delivery become important, but we’ve enabled our customers to nicely sidestep the costs and technical complexity this usually involves. When your API client registers interest in pricing up an event we build you your very own data object, which we call a channel, and then it works like pub-sub system. Our back end pricing engine is event driven, and as events occur which trigger price changes, those changes are accumulated in the channel you’ve subscribed to and delivered out to your system in real time over secure HTTP;

This puts quite granular control of the product into the client-side’s hands, allowing you to programmatically build object(s) with any combination of markets and data that you’re interested in and then subscribe to your very own private feed over familiar transport.

By accumulating interesting data into user-defined objects this way we also buy back the robustness that dedicated WAN links gave us – if there is a network disconnection the client simply re-establishes an HTTP connection to the API and picks up where it left off – no lost updates or re-synching required.

That’s SSI in a nutshell. It’s the easiest integration in the industry, and from our perspective it scales out nicely - being just a collection of smart web services - and also enables us to accelerate the building of data-driven products for our own business too.

Development Environment SLAs

When we use the word 'production' to label our customer-facing systems we create certain expectations around how we support and manage them - careful change control, detailed monitoring, rapid response to incidents, and administration that works around user requirements.

But when we label our software delivery lifecycle environments 'development' and 'test' systems, we likewise heap upon them certain expectations. Unfortunately for the development process, those expectations are usually secondary priorities and open season for any sort of experimentation or ad-hoc change.

Perhaps this is why so many projects come across environmental issues from time to time? Development environments are in production too, they just produce something a little different to what your customer facing systems produce - more customer facing systems.

Sure, when faced with a direct choice between doing something for a lifecycle environment and something for live products the decision is easy, however handling development and test systems with a little more production-like behaviors doesn't need to impact your live system activities and will make all your projects a whole lot smoother.

Welcome to Movember

Movember is an annual event to raise awareness (and money) for men's health issues, and this year I'm joining in with a pretty sweet handlebar. Keep an eye on my Mo Space to check out the mo and back a good cause.

While we're talking about charity, remember this post from Christmas last year? We're now at £222.14 - and I guess we've found our good cause now, so this'll be our first donation.

* updated * money from the google-ads-for-charity stuff above has now been donated - check my Mo Space.

* updated again * grand total is in - we raised £1098.14 from family, colleagues, and friends. A massive thank you to all who supported men's health by donating or participating.

I think you might have it backwards

The most common criticism I hear about agile is also the most common mistake teams make in their early adoption of iterative practices - not fixing any certainty.

One of the benefits of managing projects using, for example, SCRUM is that you're not burdened with having to define all the requirements up front and plan out all the tasks and times for all the resources for 6, 12, 18 months. Trying to precisely define everything a project needs up front and trying to plan for every dependency, holiday, resource issue, structural or strategic change over a long period of time proves again and again to be an impractical task.

Enter agile.

Suddenly we don't have to worry about the detail of every change and every movement over an entire year, and we're introducing a whole bunch of flexibility. Where it goes wrong is not fixing any certainty in the delivery process at all.

Agile is about certainty and detailed planning, just detailed planning over a period which is possible to predict. Who knows what'll happen over a year, but you at least have a fighting chance of planning out the next 2-4 weeks will some degree of accuracy. Beyond that you can keep plans and requirements fluid and high level, netting you the right balance of fixing doable deliverables and keeping flexibility.

Software delivery needs certainty to happen - you have to be able to rely on some requirements and some resources over some time span in order to make progress - and agile isn't about not having any, it's about having it on terms you can foresee.

Screen Scraping for Dummies

If you run a website with any sort of valuable content, then you are almost guaranteed to run into scraping sooner or later. Screen Scraping is more or less an automated program taking an impression of a web page and then parsing it to pull out some specific bits of data that the scraper is interested in - which theoretically is then stored or used in some other way.

The piece of software that does this scraping is commonly called a robot, or bot, and it is really just an automated web client that accesses and uses sites in the exact same way as it's fleshy counterparts, just with machine precision and repetitiveness. A bot may be a large and complex program running on a server with it’s own database etc, or as simple as a script running in a browser on a desktop.


This is typically regarded as undesirable behavior by many sites because, in most cases, it’s a source of load and associated with unprofitable usage. Whenever we draw a page impression, which we’ll do for every bot hit just as we do for every human visitor, we consume web server time and, worse, back end time. When we structure highly functional pages loaded dynamic content we can create a very engaging user experience, and all that functionality is built on plenty of back end work and data. When used at regular human pace that's usually OK, but under the relentless rate of hammering robots are capable of, it starts to become expensive.

If you've got a case of bots, you have to start by identifying them. Their repetitiveness and flawless precision is, in this case, their downfall and we can usually spot them easily through proper analysis of web logs - no human user is as mechanically regular, millisecond quick, and consistent in journey as your average droid.

Spotting droids isn’t too hard, but then you've got to decide what you're going to do about it. Most scrapers aren't malicious and often don't realise the headache they're giving you. In the first instance it's best to try some detective work, see if you can find out who they are, and get in touch. Domain registrars can be a great resource for this, but don't overlook the obvious - maybe they even have an account with you if data they're using requires signup to view.

Beyond that it gets tricky, and can easily turn into an IP address blocking arms race. With some caching finesse or smart layer 7 rules you can throttle bot activity to more palatable levels, or persist them to their very own node that they can thrash all day without impacting the experience for the rest of your users.

If robots are a very big problem for you, then try and take it as a compliment on the value of your data and perhaps consider publishing it via a productised API or feed - if you make it simple enough to consume, then most scrapers will willingly change over to a more reliable integration mechanism and perhaps even pay you for it.

conferences - jolly good or just a jolly?

It has always interested me that two different people can look at the exact same conference, and one declare it to be an unmissable event relevant to their work and the other consider it a waste of time; a free day out of the office at best.

It always leads to a discussion about which events are worthwhile and which aren't, but here's the secret - whether a conference is worthwhile or not is only 10% the event itself and 90% what you do while there and afterward.

You could carefully plan the sessions you attend, take the time to meet people with similar technical problems or who have done worked with relevant technology and follow up with them later and, when you return to the office, share your new knowledge around and adopt some better practices.

Or

You could have a few days out, enjoy some vendor's hospitality (free beer always tastes better), and check out some new bars and, when you return to the office, settle comfortably back into your old habits.

That, ladies and gentlemen, is the difference between a jolly good conference and just a jolly.

Delegation from the other side

I really like Manager Tools - it's probably the best single source of personal development material that you're ever going to find bundled in easy to consume episodically content. Recently I thought I'd go back and listen to the basics series again, because you can never spend too much time on the fundamentals of good management and, like Mark and Mike say, if you don't do anything else except those basic practices, you'll probably do OK.

It was a really good refresher, and worthwhile doing from time to time. Then I got to the juggling koan cast, which was about how to handle that inevitable situation when your boss gives you another big task to work on when you're already maxed out with too much to do. It's worth listening to - so I won't spoil it entirely - except to say that the answer focuses on developing your delegation muscles, passing on some of your things to make space for the new action and helping your team to grow as an nice side effect.

The majority of the cast focused on how to handle events after you accept this new delegated responsibility, and I liked the point about deliverables gaining size as they flow down the delegation tree (i.e. your small tasks are bigger challenges for more junior members of your team), however I think there was a valuable lesson here about the delegation from the recipient's perspective.

The Manager Tools perspective on this was to accept the new delegation, and to view it as the expression of confidence and trust and the development opportunity that it really is. Many new managers would worry about whether they have the time to take it on or not (no thanks boss) or attempt to enter into a negotiation about what else to drop.* The reason I like this angle is that I don't think there is enough coverage out there on followership.

Leadership is glamorous and fashionable to write about and speak on, so we do a lot of it, but good followership is important to develop too. There is an inherent responsibility on all the individuals in an organisation to be rational participants, and I think a group who know and practice the right behaviors on both sides of that equation performs better. Anyway, isn't part of what we're doing when we're developing our people just cultivating better followership?

Something I'd suggest you ponder when you listen to the cast is clarifying the difference between being handed an action and given responsibility for an outcome. There was heavy emphasis on not entering into negotiations with your boss about whether or not you are going to take this thing on, however I think it's both valid and valuable to question the task and look for the underlying business value.

This isn't arguing about whether or not you or your department have the time, or forcing your boss to manage your priorities for you, it's simply clarifying what you're really being expected to achieve rather than what you're being asked to do. In my experience, particularly because I run technical departments, there is often a difference between what people want to achieve and what they initially ask for, as they think they have resolved the outcome they have in mind into a few actions and sometimes those actions don't add up.

If you can have that sort of open discussion, then you might agree to accept delegation of a slightly different task than was first put your way, but one that much more directly leads to the goal your boss really had in mind.

* Key caveat here about dropping stuff. Things will eventually be dropped, and when you're handling the most important things you can and have delegated the rest then so will your directs, and therefore [due to the hierarchical effect created when everyone delegates their least critical things] what gets dropped should eventually be the least important tasks overall.