The things I should have learned from years of putting together people and technology to get successful product online. We’ll probably talk about strategy, distributed systems, agile development, webscale computing and of course how to manage those most complicated of all machines – the human being – in our quest to expose the most business value in the least expensive way.
Tuesday, 22 December 2009
Standups vs Team Meetings
A stand up is a project-oriented meeting, all about running the day to day work of the team, has a tightly fixed agenda, is time bound, and specifically focused on what's required for today.
A team meeting is a team-oriented meeting, all about maintaining the team, continuous improvement, future requirements, and focussed on bigger picture topics.
They are definitely two very different meetings with two very different purposes, and both of them are necessary for a team to run smooth projects and keep productivity and job satisfaction high.
Wednesday, 16 December 2009
Introducing Sporting Index Super Simple Integration
This year we decided to give our B2B customers an early Christmas present - SSI.
Sporting Index (well, really Sporting Solutions, our B2B brand) provides pre-match and in-play pricing and content to fixed odds bookmakers for hundreds of markets across 16 sports with more events from more jurisdictions being added almost every month.
Traditionally we’ve done this by pushing XML documents around over reliable transports and leased lines. This works very well and there is nothing inherently wrong with this solution, but we weren’t happy that customers had to install expensive fixed lines and associated hardware, and our back end is capable of producing multiple-thousands of price changes a minute – well beyond the rate at which some of our customers can consume them.
On top of this, our customers wanted the low latency and reliable updates that come with a push solutions, but also the easy integration and flexible client-side controls that comes with a pull solution.
Well, you asked for it and you’re the customers, so let’s unwrap this bad boy.
Super Simple Integration is essentially a RESTful web services API available over the good-old-fashioned Internet. The feed is as easy to consume as you’d expect from any web API, and we’ve spent a lot of time on the data formats and doing some clever pub-sub tricks in the background – thus all the benefits of the heavyweight XML services but through an integration which is quick and easy to get to grips with.
There’s a lot to this, so let’s just cover two of the most common use cases:
Our content products can be used to populate websites with upcoming fixtures and players, and to power pages or widgets showing points, corners, goals etc as they happen in running matches.
Content products are mostly stateless and work on a fairly simple request-response model, which can mean no integration work at all. Well that’s half true – you still need to consume the data from the API – but you don’t necessarily need to build a repository on the customer side or have to match up schemas.
Let’s say that wanted to build a little scoreboard for my site to show my visitors the progress of a live England v New Zealand rugby game;
You get the content you want, as often as you want, delivered back in name-value pairs. There are also a couple of methods to call to list out what sports and events are available, so you can pick out what to pull into your scoreboard app.
Pretty basic stuff so far, where it really gets sweet is how we implement real time price streaming.
Our price products can be used to price up sports betting markets, either as a risk advisory tool for human operators or a totally automated pricing service, and cover a wide range of sports in-play with high fidelity price output.
In price products - particularly once fast paced events turn in-play - state and reliable delivery become important, but we’ve enabled our customers to nicely sidestep the costs and technical complexity this usually involves. When your API client registers interest in pricing up an event we build you your very own data object, which we call a channel, and then it works like pub-sub system. Our back end pricing engine is event driven, and as events occur which trigger price changes, those changes are accumulated in the channel you’ve subscribed to and delivered out to your system in real time over secure HTTP;
This puts quite granular control of the product into the client-side’s hands, allowing you to programmatically build object(s) with any combination of markets and data that you’re interested in and then subscribe to your very own private feed over familiar transport.
By accumulating interesting data into user-defined objects this way we also buy back the robustness that dedicated WAN links gave us – if there is a network disconnection the client simply re-establishes an HTTP connection to the API and picks up where it left off – no lost updates or re-synching required.
That’s SSI in a nutshell. It’s the easiest integration in the industry, and from our perspective it scales out nicely - being just a collection of smart web services - and also enables us to accelerate the building of data-driven products for our own business too.
Monday, 14 December 2009
Development Environment SLAs
But when we label our software delivery lifecycle environments 'development' and 'test' systems, we likewise heap upon them certain expectations. Unfortunately for the development process, those expectations are usually secondary priorities and open season for any sort of experimentation or ad-hoc change.
Perhaps this is why so many projects come across environmental issues from time to time? Development environments are in production too, they just produce something a little different to what your customer facing systems produce - more customer facing systems.
Sure, when faced with a direct choice between doing something for a lifecycle environment and something for live products the decision is easy, however handling development and test systems with a little more production-like behaviors doesn't need to impact your live system activities and will make all your projects a whole lot smoother.
Sunday, 1 November 2009
Welcome to Movember
While we're talking about charity, remember this post from Christmas last year? We're now at £222.14 - and I guess we've found our good cause now, so this'll be our first donation.
* updated * money from the google-ads-for-charity stuff above has now been donated - check my Mo Space.
* updated again * grand total is in - we raised £1098.14 from family, colleagues, and friends. A massive thank you to all who supported men's health by donating or participating.
Saturday, 3 October 2009
I think you might have it backwards
One of the benefits of managing projects using, for example, SCRUM is that you're not burdened with having to define all the requirements up front and plan out all the tasks and times for all the resources for 6, 12, 18 months. Trying to precisely define everything a project needs up front and trying to plan for every dependency, holiday, resource issue, structural or strategic change over a long period of time proves again and again to be an impractical task.
Suddenly we don't have to worry about the detail of every change and every movement over an entire year, and we're introducing a whole bunch of flexibility. Where it goes wrong is not fixing any certainty in the delivery process at all.
Agile is about certainty and detailed planning, just detailed planning over a period which is possible to predict. Who knows what'll happen over a year, but you at least have a fighting chance of planning out the next 2-4 weeks will some degree of accuracy. Beyond that you can keep plans and requirements fluid and high level, netting you the right balance of fixing doable deliverables and keeping flexibility.
Software delivery needs certainty to happen - you have to be able to rely on some requirements and some resources over some time span in order to make progress - and agile isn't about not having any, it's about having it on terms you can foresee.
Sunday, 27 September 2009
Screen Scraping for Dummies
The piece of software that does this scraping is commonly called a robot, or bot, and it is really just an automated web client that accesses and uses sites in the exact same way as it's fleshy counterparts, just with machine precision and repetitiveness. A bot may be a large and complex program running on a server with it’s own database etc, or as simple as a script running in a browser on a desktop.
This is typically regarded as undesirable behavior by many sites because, in most cases, it’s a source of load and associated with unprofitable usage. Whenever we draw a page impression, which we’ll do for every bot hit just as we do for every human visitor, we consume web server time and, worse, back end time. When we structure highly functional pages loaded dynamic content we can create a very engaging user experience, and all that functionality is built on plenty of back end work and data. When used at regular human pace that's usually OK, but under the relentless rate of hammering robots are capable of, it starts to become expensive.
If you've got a case of bots, you have to start by identifying them. Their repetitiveness and flawless precision is, in this case, their downfall and we can usually spot them easily through proper analysis of web logs - no human user is as mechanically regular, millisecond quick, and consistent in journey as your average droid.
Spotting droids isn’t too hard, but then you've got to decide what you're going to do about it. Most scrapers aren't malicious and often don't realise the headache they're giving you. In the first instance it's best to try some detective work, see if you can find out who they are, and get in touch. Domain registrars can be a great resource for this, but don't overlook the obvious - maybe they even have an account with you if data they're using requires signup to view.
Beyond that it gets tricky, and can easily turn into an IP address blocking arms race. With some caching finesse or smart layer 7 rules you can throttle bot activity to more palatable levels, or persist them to their very own node that they can thrash all day without impacting the experience for the rest of your users.
If robots are a very big problem for you, then try and take it as a compliment on the value of your data and perhaps consider publishing it via a productised API or feed - if you make it simple enough to consume, then most scrapers will willingly change over to a more reliable integration mechanism and perhaps even pay you for it.
Sunday, 20 September 2009
conferences - jolly good or just a jolly?
It always leads to a discussion about which events are worthwhile and which aren't, but here's the secret - whether a conference is worthwhile or not is only 10% the event itself and 90% what you do while there and afterward.
You could carefully plan the sessions you attend, take the time to meet people with similar technical problems or who have done worked with relevant technology and follow up with them later and, when you return to the office, share your new knowledge around and adopt some better practices.
You could have a few days out, enjoy some vendor's hospitality (free beer always tastes better), and check out some new bars and, when you return to the office, settle comfortably back into your old habits.
That, ladies and gentlemen, is the difference between a jolly good conference and just a jolly.
Friday, 4 September 2009
Delegation from the other side
It was a really good refresher, and worthwhile doing from time to time. Then I got to the juggling koan cast, which was about how to handle that inevitable situation when your boss gives you another big task to work on when you're already maxed out with too much to do. It's worth listening to - so I won't spoil it entirely - except to say that the answer focuses on developing your delegation muscles, passing on some of your things to make space for the new action and helping your team to grow as an nice side effect.
The majority of the cast focused on how to handle events after you accept this new delegated responsibility, and I liked the point about deliverables gaining size as they flow down the delegation tree (i.e. your small tasks are bigger challenges for more junior members of your team), however I think there was a valuable lesson here about the delegation from the recipient's perspective.
The Manager Tools perspective on this was to accept the new delegation, and to view it as the expression of confidence and trust and the development opportunity that it really is. Many new managers would worry about whether they have the time to take it on or not (no thanks boss) or attempt to enter into a negotiation about what else to drop.* The reason I like this angle is that I don't think there is enough coverage out there on followership.
Leadership is glamorous and fashionable to write about and speak on, so we do a lot of it, but good followership is important to develop too. There is an inherent responsibility on all the individuals in an organisation to be rational participants, and I think a group who know and practice the right behaviors on both sides of that equation performs better. Anyway, isn't part of what we're doing when we're developing our people just cultivating better followership?
Something I'd suggest you ponder when you listen to the cast is clarifying the difference between being handed an action and given responsibility for an outcome. There was heavy emphasis on not entering into negotiations with your boss about whether or not you are going to take this thing on, however I think it's both valid and valuable to question the task and look for the underlying business value.
This isn't arguing about whether or not you or your department have the time, or forcing your boss to manage your priorities for you, it's simply clarifying what you're really being expected to achieve rather than what you're being asked to do. In my experience, particularly because I run technical departments, there is often a difference between what people want to achieve and what they initially ask for, as they think they have resolved the outcome they have in mind into a few actions and sometimes those actions don't add up.
If you can have that sort of open discussion, then you might agree to accept delegation of a slightly different task than was first put your way, but one that much more directly leads to the goal your boss really had in mind.
Sunday, 30 August 2009
A better way to manage equity?
The problem with this approach is that it doesn't easily cater for all the things that really happen during the course of a startup. For example, you can't always predict who's going to stick with it or how many people are ultimately going to be involved. As the idea evolves (because you almost never end up with the same plan you started out on) the bias of work can shift to a point where the equity split no longer reflects the effort split, and of course the value of effort to a business changes over time.
I don't think there is such a thing as a flawless mechanism for this, however I think I might be onto something a little more elegant. Here's how it works...
Break all the work - this same approach works for commercial and marketing tasks just as well as it does for the technical stuff - required to bring your idea to market into tasks or jobs which can be done individually by the members of your team. Decide on a number of points for each task, based on a sensible blend of value to the business and effort required to deliver. Points are the key to the whole thing; do the task -> earn the points -> exchange the points for shares.
Because you haven't fixed the equity allocations up front, it doesn't matter if you gain or lose co-founders and you minimise those 'who isn't pulling their weight' discussions, as you only get the shares you directly earn.
But why points when you could just do the same thing with the shares themselves?
Points give you a couple of extra controls you might not have if you immediately issued the shares; for example you can set an exchange rate between points and shares, allowing you to increase the relative value of tasks not being picked up, or offer individuals a better exchange rate once they have accumulated a large number of points (thus encouraging fewer greater contributors which makes for a more coherent project and a simpler shareholders register).
Setting specific milestones (for example beta launch or first partner signed up) at which points can be exchanged for equity ensures that everyone still focuses on bringing together an overall business delivery as well as their own contributions.
Forming your company with a healthy allocation of shares and some fairly accurate record keeping on tasks and points (I can recommend zen if you're an agile/lean thinker) is all that's required administratively.
There are a number of more subtle points to this, such as making sure you've got strong enough acceptance criteria on each of these tasks, but that's the meat of it. I'm running a project using this method now and - while we're not done yet - it's going pretty well and has already afforded us some flexibility we wouldn't otherwise have had.
Wednesday, 19 August 2009
Doing it on the side
Trying out "their own thing" will teach your guys about responsibilities well beyond those that you can reasonable expect to expose them to in their role, and give them some insight into the difficulties of running a business, taking care of finance, marketing to customers, and making product decisions. There is so much to the operational side of an organisation (even a tiny fledgling one) that they would otherwise not have many chances to experience first hand. This can help your team members develop their creative, entrepreneurial side and give them a much better appreciation of the challenges their colleagues elsewhere in the business face. This is so beneficial that I'm even found guilty of supporting such ventures with advice and coaching from time to time.
The caveats? There are only 2 key things that must be true to qualify what I’ve said above:
1 - You still have a job. Outside interests should not impinge on the quality, quantity, or timeliness of work, and if they do then one of these things must go. My policy is to provide a flexible working environment, but I'm not here to subsidise your startup either.
2 - You shouldn't compete with your employer. For most people in our industry, our roles come with a duty to innovate and develop new product to further the companies interests. Whenever something extracurricular can be even slightly interpreted as related, it needs to be raised with management and explicitly categorised into 'company IP' or 'fair game' for everyones benefit.
Regardless of which side of the employer/employee relationship you sit on it is always worth finding out what your company policies are on this kind of thing before getting underway. Being uninformed might mean you start out with the best intentions and end up having to make difficult choices later.
Tuesday, 16 June 2009
The nuances of Planning Poker
What I want to address is the spirit in which group estimation happens.
Every potential task owner (read: qualified opinion) estimates each relevant task as it comes up, but the reason for this is commonly misinterpreted. The purpose is not to force the eventual task owner to reduce his or her estimate, or to adopt an estimate that others in the group ‘prefer’ and nor is it a criteria for selecting an owner for a task, based on the estimate the product owner or scrum master likes the most.
The purpose is to draw out knowledge from the outliers, so that the whole team can benefit from their experience. When someone in the group pulls a card miles away from the rest (assuming a consistent understanding of the task) that’s something to explore. You might have a much higher estimate – perhaps that person has been burnt before by that same inconspicuous looking requirement masking significant complexity. You might have a much lower estimate – perhaps that person knows a good work saving shortcut or pattern uncovered during a previous project.
If you take the time to ask those questions and consider that information, the group learns from one another and everyone gets better at estimating.
And whose estimate is used in the sprint? The person who is going to be doing the work...
Sunday, 14 June 2009
Right First Time
That’s clearly unreasonable.
So why does it seem reasonable that the first iteration of a new system you build should be the one that your business depends upon?
That’s why we prototype. That’s why we iterate. That’s why we beta. Don’t rob yourself, and your organisation, of the wisdom you’re owed from those lessons.
Monday, 8 June 2009
So who is in a team anyway?
So how does their guy propose they fight the first battle in the war for organizational effectiveness? Setting aside certain days when people in one team aren’t allowed to talk to another. No, you didn’t read that wrong, the ‘big plan’ is to actually create barriers to communication.
I just hope that wasn’t expensive.
In the reseller/systems integrator game it’s all down to the cooperation between sales and their technical teams, and so I really struggle to see how driving intentional wedges between them is going to help. So my theory? Perhaps the wrong people are teamed up together in the first place…
In any mission there are multiple people with different skillsets doing different jobs all coming together around a central project/customer/business problem. Isn’t that a team? The people who actually need to work together on a daily basis for the direct benefit of the customer? I don’t know why we keep insisting on grouping people by job or by skillset (sales team and technical team, or dev team and test team) because the cold, hard facts are that these teams can’t achieve anything end-to-end for a customer on their own. Maybe it just appeals to our human need to nicely categorize things.
When you put people together in a team, you create a tight loop of communication and cooperation between those people (it’s one of the reasons we do it in the first place) – but you do something else too – you also create a little organizational fence between them and other teams. A fence that we absolutely should work at keeping as low and friendly as possible, but a fence nonetheless. To me it just comes down to where you want to put your fences – does it make sense to create boundaries between the individuals whose effort must by synchronized daily to complete units of work?
In what I do, the unit of organizational delivery is the project (or, as a superset, the product). That’s that unique object around which we all gather and apply our individual skills and experience to create tangible value for the company. So by default, I have a nice, simple method for grouping my guys together in a way that creates the fewest barriers to the most common communication and coordination lines that they’ll be following each day. I suppose I could put all the developers in one team, all the testers in another team, all the project mangers in another team again etc – but if I did, then what’s the very first thing I’ll have to do as soon as I want to do any work? That’s right, pull them back together into virtual or temporary teams based on the projects they’re delivering against. Hmmm.
From my experience in the SI and reseller game, I’d suggest that the unit of organizational delivery is the customer. That’s the internally-ownable entity which performance and profit is measured against, margin is made against, and time is billed against. Yet almost every one of these companies is, at best, a matrix management organization. So you group people together into a sales team, a tech team, a presales team etc – and then wonder why, whenever they all have to get together around that common indivisible business problem (the customer) there are all sorts of communication issues, competing priorities, complexities planning time...
Perhaps a team in this type of organization should be something like a sales guy, some sales support, a presales guy, a couple of engineers, and a share of some administration – all grouped together around a set of similar customers that they own the end-to-end P&L for.
The inevitable argument against this kind of cellular organization is the same one I’m used to elsewhere – developers like hanging out with developers, infrastructure engineers like hanging out with infrastructure engineers and so on. My only response to this is that I don’t see how organizing your people into the teams which best reflects the true structure and purpose of the company precludes communities of interest, and again, just ask yourself where you want to put the barriers, and whether you consider the upside of keeping people grouped into teams of similar individuals worth the downside of trying like mad to make them work together efficiently and effectively whenever you have to get anything done.
Monday, 1 June 2009
Good ops guys are hard to find...
I’m reinventing how my sites are supported – if you’re an awesome one of these or a kick-ass one of these, then we should talk.
Friday, 29 May 2009
Hi, I’m Eachan and I’m a fail-a-holic
All this talk of innovation, and how you measure its ROI, was starting to tickle my aforementioned spider sense, but since a couple of the panel were from the public sector, I couldn’t help but want to explore the effect that the general public’s reaction to failure has on the culture within those organisations. This is why I’m either never invited back to these things or I’m on the panel next time.
Firstly, let’s just agree that the cost of innovation is not in salaries paid to folk you can point to and say, “he’s doing some innovating” or in capital expense on new widgets and bleeding edge technology. The price of innovation is failure. In fact, I would even suggest that the price of learning anything material is failure.
So then I look at the slating that any public service gets when things go wrong. We bay for blood in the most public of forums, newspapers, radio, and even (opportunistically) on the opposition benches. A major mistake was made, so someone has to go, right? There are countless examples of this; ranging from embarrassing (eg HMRC data loss), to genuinely horrific (eg Baby P tragedy), to that thin line between the rules and the spirit of the rules (eg MP’s expenses scandal).
It’s difficult to come to an agreement on what should have happened in any one incident, as we will all feel differently about each incident and will be affected differently by it, but that’s OK, because it is the pattern that matters. The pattern is what sets the culture.
If, every time things go wrong, we excise every last individual involved, how then do we ever hope for the lesson to be learnt? Who has felt the pain, and is carrying that forward as a driver for change? Where is, to quote a buddy of mine, the feedback loop?
Is it right for us to do this, to create such epic risk aversion, and then still expect these organisations to deliver even the most basic of improvements let alone giant innovative leaps forward? Are we just robbing them of the best ingredients needed to create the huge changes we’re demanding?
When I think more about it, I wonder if I really am a fail-a-holic at all. Perhaps I only embrace failure so much because I’m actually a success-a-holic…
Tuesday, 10 March 2009
Don't out-specify your knowledge
Standardisation is important, particularly in the context that Armstrong frames it in - if the interaction between a client and a server closely adheres to a predetermined protocol, then the systems overall behaviour becomes more predictable and easier to adapt and fault find.
But then this depends on the maturity of your understanding of your problem and your solution. As Creswell says at the beginning of his post, defining too much up front can be a trap. Coming up with a set of standards that are too rigid too early might start you down a road of building that system instead of building the right system.
Standards, patterns, and contracts are all important - but timing is key.
Sunday, 8 March 2009
Choosing Recruitment Agents
The typical approach to this pretty much involves coming up with some lower rates than you currently pay (cost control is usually the motivation behind the PSL exercise) and seeing who wants to sign up. Not a bad start, especially if you come up with some tiers which (for example) encourage recruiters to take a lower fee for a period of exclusivity, but you should probably add some metrics to ensure quality.
Firstly, think about why you use an agency. Hiring strangers is the worst recruitment strategy ever, so when you use an agency, you are basically using their network as a replacement for yours.
So how about finding out some other stuff too; do they meet candidates in person before they put them forward? How long do they typically have a relationship with a candidate for before they start to place them? How often do they place the same candidate in consecutive roles? Anything you can come up with to give you confidence that their network is going to be a passable proxy for yours is going to be a worthwhile comparison point.
Wednesday, 4 March 2009
No problem too small, no reward too big
Rushing to the nearest browser in case a patent application was necessary, we became a slightly despondent upon seeing a bunch of articles like this one on microgrids and what the EPRI are getting up to. That said, it's great to see this kind of stuff getting mindshare.
So what, if anything, is still missing? To see this come true, it seems like we'd have to work our how the power companies would still make money in our community-spirited utopian electricity supplying future. So perhaps we can still contribute by bringing some economics into the solution.
Someone has to provide that infrastructure - the grid we'd all have to be connected to in order to exchange electrical capacity. That sounds like something someone should be paid to provide. So perhaps there is a subscription fee? Or maybe we can do one better - how about a variable charge based on each household's net balance at the end of every month/quarter/year? The closer you are to netting out (producing the same number of kilowatt hours you consume) the closer to zero the bill gets.
The beauty of that model is it incentivizes consumers to be as self-sufficient as possible and disincents (is that even a word?) waste. The expected impact of such a price control is for participating individuals to want to align their production with their consumption - probably by boosting the capabilities of the former while working out efficiencies in the latter.
Reducing overconsumption is probably a no-brainer, but you could look at reducing overproduction both ways. On the one hand, overproducers supply overconsumers and we'd need that, but on the other hand we wouldn't want participating individuals to profit from overproduction. Ultimately all overproduction is wasteful (by definition if nothing else) so there should still be a price for it, a lower price than overconsumption, but not as low as netting out.
Well that was all kind of off-topic, but I hope you enjoyed it nevertheless. There are some intriguing engineering challenges in the basic problems inherent in sustaining our society, and if we gave this kind of stuff a little more attention, we'd all be living in a much better world before you know it.
Friday, 27 February 2009
Gartner Economic Downturn Briefing - Part 2
Back on track - the economic downturn. It's kind of uncomfortable to think about the upside of something that is affecting so many so negatively, however we're well positioned to take advantage of the opportunities that are coming to the surface.
With a battening-down-the-hatches mentality, people are preparing for the worst. Sites like Fool and Lovemoney pick up as we try to get as many money-saving tips as we can. The first preventative action we're likely to take is reviewing utilities and other basic household expenses, and sites like Gocompare and Moneysupermarket see a lot more visitors.
On top of this, there are a whole slew of creative new ideas starting to come through, all predicated on our current need to squeeze a little more value out of things we otherwise tolerated waste in.
At the worst end of the scale, people are being made redundant. Again this is a terrible tragedy affecting people I know, and it feels unfair to be thinking about the opportunity at a time like this. Nonetheless, the internet is now the de facto tool of choice for finding your next role, and this means more actives on Jobserve and Monster. Whether you've been affected yet or not, it is now more important than ever to get an up-to-date profile out there and build up strength in your network (before you need it). Off we go to the professional social networking sites like Plaxo and Linkedin.
Lining my own pockets for a moment, online gaming and gambling is fairly recession-resilient, but even simple ad-supported business models can do OK. You're still turning page views and actives into revenue, and if you're providing the service or content that's currently in demand, then that's not insurmountable in any times.
Thursday, 26 February 2009
Gartner Economic Downturn Briefing - Part 1
The sessions were focussed on how CIOs and CTOs are responding to the current economic circumstances, and the techniques IT organizations can use to keep cost under control and contribute to the businesses downturn survivability. So here goes - in no particular order...
• Focus on projects that are "shovel ready". Bang for buck is more critical than ever, and projects ready to be actually done are worth a lot more than potentially better activities that are still in the planning phase.
• Expect to deal with massively escalated regulation. In the post-credit crunch world, there will be a lot more regulation aimed at preventing repeat performances. This is likely to mean increased reporting and compliance overhead, as well as more constraints on how technology can be deployed and data used.
• In the worst of cases, there may well be a sharp rise in the number of legal actions. This is going to put increased demand on e-discovery and BI as organizations scrabble for the information they need for defence.
• A general loss of trust is going to lead to more diligence efforts. In a climate where any number of suppliers could go bust at any time, you'll need to spend extra time reviewing the financial integrity of key partners you depend on, and perhaps even making backup plans.
• The downsizing mentality could rob you of the top talent that you'll need on hand to steal a march on your competitors once times start looking up. Don't sack them, squirrel them away any way you can!
• Expensive oil means old assumptions on locations, topology, and transport need to be challenged. Get better at communications, and be prepared to accommodate supply chain changes aimed at closing the distance between steps.
• Check out your PR infrastructure. During these times the amount of media attention on companies is at a peak. Where is your corporate site hosted? Who edits the content? It is often an overlooked side-system, and right now you don't need the attention you'll get if it goes down.
• Take a look at your HR systems too. When companies get told to lay off 12,000 people over 12 months, their HR processes face a task they were never intended to handle. Pay special attention to data accuracy - mistakes in calculations on this scale will just kill you.
• Watch out for desperation moves. These are the times when you'll be the most likely to get stitched up by some crazy marketing plan, an over the top offer that gets madly oversubscribed, and so you'll want to keep your interdepartmental communication flowing and your eye on capacity.
There is some good sense in this, and some stuff I want to look at from a slightly different angle. My side of it - and what I think it means for us web guys - in the next post.
Saturday, 21 February 2009
Scalability is not just a technical problem
There is so much content out there about how to scale out web sites, platforms, and databases – but it all focuses on the production system architecture. Do a Google search for scalability – go on, I’ll wait for you... See what I mean?
Now I know that’s the fun bit to talk about, but being practical for a minute, if you’re starting to scale out systems using techniques like Digg, Flickr, Xbox Live, or the usual suspects like Amazon, Google, and eBay, then chances are that you’ve got a whole lot more scalability challenges than just the product.
If you’re dealing with at least hundreds of thousands of daily actives, then we can probably deduce a bunch of other stuff about your circumstances. We can guess that you’re after reasonably frequent feature drops, have a significant amount of horizontal distribution, a healthy sized engineering organisation, and a strong bias towards availability. And if even half of that stuff is true, then what are the other scalability challenges will you be up against?
How about multiple concurrent projects? If your estate is divided into more than one product then you will more than likely be working on more than one new feature in parallel. This gives rise to all sorts of version control and regression test problems, which demand process and infrastructure quite different to a single effort.
What about disparate teams? You might have people in a number of locations branching, or depending on, the same codebase. That’s a communication barrier which can be tough to solve. Large enough organisations also tend to sprout specialist disciplines, such as user experience and IA – this changes the nature of how teams engage and how work is specified, estimated, and delivered.
And how do you manage environments, tools, and documentation? A complex production architecture begets a complex development infrastructure, as there is a lot more interoperability to test for. Don't forget that with more teams working concurrently, managing contention for these expensive environments also becomes a tricky balancing act. As your products increase in popularity (a good proxy for profitability on the web) NFRs like performance and capacity will become more important and will require specialised tools to measure.
I’d like to see us sharing a little more about our experiences with this side of highly scalable systems – it might not be as sexy as memcached, CAP, and Gossip, but the reality is it is just as important a part of the solution nonetheless.
Tuesday, 17 February 2009
The Good and the Bad of Google Product Management
In a way it's a little sad to think that these harsh economic times are starting to have an effect on even that most hallowed of engineer-driven organization. Still, there are those of us probably a little less melancholy - I can just imagine how much fun serial cloud hater, John C Dvorak, is having - "told you not to keep stuff in the cloud, one day they just switch it off and then what do you do?"
The decision to shelve a product is a difficult one, and one that inevitably leads to some degree of unpopularity both internally and externally. What I wanted to do here was take a look at what this article tells us about product management practices in general.
• Willingness to take risk. Especially in these times, it can be tempting to stick to safe territory. That can be good advice, but what are you missing out on? Another way to look at it is when everybody else is playing it safe, it’s easier to stand out with something unique.
• Recognise the capability gain in products even if the product itself doesn’t work out in its current incarnation. Taking what they gained from Dodgeball and turning it into the Maps add-on Latitude is a great example of the sort of utility you can get from something that doesn’t work out in its initial form.
• Customer feedback. Accessing the community through techniques like product development blogs and using that knowledge to shape the features and user experience is a powerful way to make sure your system ends up meeting expectations as closely as possible.
• Attitude to failure. Viewing things that don’t work out as contributing to overall success rather than detracting from it helps you really internalise what you learn and encourages the innovation that fear of failure will crush.
• Internal testing. Eating your own dog food is always helpful, after all, if you find your product difficult to use then what are you asking of your customers?
• Opportunity cost. I liked the part where they consider the opportunity cost of a project as well as its real cost; i.e. what is the value of the things that the engineers are not working on because they’re working on this?
• Incentives. Paraphrasing, the article talks about difficulty attracting engineers to work on certain products. There are some reasonably well publicised shared reward schemes in place, and in this free market-like environment management needs to consider how it might need to provide external subsidies to make sure that less glamorous, but perhaps strategic, projects get attention too.
• Perpetual betas. I really couldn’t decide whether this one belonged in the good or the bad section, so I let the need for balanced paragraphs drive the decision. On one hand, I think there is no better way to refine user experience and focus the tail end of your development efforts on polishing the things users like the most, but on the other hand there are circumstances under which you need to show the commitment of a production-ready seal of approval.
• Communication. While not exclusively a product management issue, communication across geographical boundaries was a major issue here. This shows how important collocation is in a product development process.
• 20% time. I think we’ve all heard mixed reports about how this really works, but regardless of that, if you have a product that you’re committed to and have set targets for then you simply must formally assign it resources. Relying on product management to recruit people from their 20% time is setting it up for failure.
The article also mentions internal performance targets for new products, and in all businesses objectives and measures are the true meat of how product decisions are driven, so it's a shame we didn't learn more about that. Also some kudos due for how they ended Jaiku, the Twitter clone, it's good to see them pass that on to the community rather than just confine it to the shelf.
Wednesday, 4 February 2009
Obviously I’d never advise flying in the face of company policy, but I really like giving references. In fact, I consider being able to give a good reference to be a lagging indicator of great success with people.
You see, if I’m able to give a good reference for someone, it means two things:
1. We’ve had a successful, effective working relationship in which we’ve helped each other deliver tangible value to the business.
2. I’ve been able to help that person develop professionally, and I’m now helping them to continue to grow their career.
Is there any greater satisfaction to be had in management? It’s why I do the job.
Thursday, 29 January 2009
Why are they never happy?
Anyway, I'm listening to this buddy complain about how busy he is, how hard he's working, how many hours his team are putting in, and how much stuff they're getting out - yet still his customers aren't happy. They're always finding something to complain about, pointing out the things that are missing rather than thanking him for the things that are there.
When I hear this, what I think most people are really saying is "my team and I are working really hard and really long hours doing the things we want to do for this customer". Whenever I've seen this issue on the floor, it has almost always been because the guys were dedicated and hardworking, but being dedicated to and working hard on the wrong stuff.
Think about it like this; if I really want A, and you give me B, C, D, E, F, and G, then I might be impressed by how busy you've been and how much work you've done, but I still won't thank you for it. Yes, B through G were relevant to my department and they were things I wanted done - no argument there - but if A was the thing that meant the difference between getting that big contract or not, or making this quarters numbers or not, then I would have happily traded it all to have just A.
I'm not saying that there aren't ungrateful people in the world, but I am saying that until you've adopted a partnership-based, customer-centric view of how you deliver technology, you haven't earned the right to complain about them.
Look frequently and carefully at your priorities, and compare them against what your business is being driven to achieve - you never know - you might even end up doing less work and having happier customers.
Sunday, 25 January 2009
You ALWAYS Pay for Quality
But before you condemn my point entirely, consider this:
If you're buying in software, consultancy, per-project services etc, then the primary levers you can adjust are commercial terms. You might feel exceptionally good about reducing the cost of a project by £5K or shaving a couple of points off margin, but do you really think those suppliers are going to produce exactly the same output for you and just make £5K less on the deal? Good luck with that.
If you're building your own products, and assuming you've got a fixed size team, then the primary levers you can adjust are tradeoffs and shortcuts - i.e. where you spend your time. How you pay for quality is then divided up between how much you're prepared to pay for now up front, and how much you're going to pay later - typically through higher operational or technical support effort (less automation), increased cost of the next features (technical debt), and opportunity cost (revenue lost to product unavailability).
So I'd suggest that you always pay for quality, whether you intend to or not, and I'd add that it's usually less painful to pay for it overtly through a proper project plan, than inadvertently through reactive support, interest payments from future projects, and your customers patience.
I'm not saying that you should never try to sharpen a deal to save some budget or take a shortcut to get a product out quicker, simply that you should keep in mind the longer term costs as you congratulate yourself on the shorter term gains.
Thursday, 22 January 2009
The thing to keep in mind about your proposal is your goal; you're not there to do a really great presentation, you're there to convert someone into a mad raving fan of your idea, and how you do that is going to depend on who you're dealing with.
Step 1 for me is always start with the audience, as everything else flows from this. It drives your content (which benefits you accentuate, which risks you focus on) your delivery method (maybe powerpoint slides in the boardroom is best, maybe a casual discussion over a coffee is sufficient), and even the duration you're going to talk for.
Never simply have "your presentation on project X". Instead, have "your presentation on project X for the architecture steering group", and "your presentation on project X for the accountants", and "your presentation on project X for the board" etc. They're all going to want to see a different mixture of things before they get behind it. With some of them, you might want budget, so you'll want to spend a slide on the concept, then spend the balance of the time breaking down cost and benefit over the next few years. From others you might need their change control approval, so the commercials will be lightweight with much more detail about how your system will be deployed and how it will integrate with the rest of the enterprise. You might even be displacing the roles of operational people through a tool you're driving forward, so these guys will want to know how new processes will change their tasks and you'll need to spend a fair bit of time addressing concerns about the team, their jobs, and training.
That's all very good, but something people often ask me is how to handle mixed audiences - when you have to present your proposal to a collection of people in different roles. My answer is - don't. It might seem like a smart idea, get a large number of areas out of the way in one go, but it simply is not effective. You cannot adequately address each persons individual focal points, and you risk convincing nobody.
Granted, sometimes you don't have a choice, and if that's the situation you're in (and you can't get out of it) then keep it brief, simple, factual, and to the point. But do have all the detailed information right there with you - the same stuff you'd have if you were doing the customized versions for each individual - and that way you'll be knowledgeable and organized should anyone wish to drill down into any of the areas they represent. If you've been asked to do something for a multidisciplinary audience like this, then chances are it's at some kind of regular session the participants have anyway, which means they'll have other business to discuss, which means you can't expect a long slot, which means your brevity will be appropriate.
On the mechanics of presenting - and by mechanics I mean how to deliver slides - I will let David Rose tell the story for me. It's worth watching the whole clip, but at the very least watch the last few minutes in which he covers slide design, stance and tone, and using presenter mode in powerpoint etc. Second only to actually doing it yourself, watching great presenters in action is the best way to get better at it, so I'd recommend watching a few more TED talks and definitely subscribing to Presentation Zen.
When drawing up a slide deck to accompany a presentation (note that a deck of slides is not your presentation) I think it's important to resist incorporating all the whizzy features powerpoint has to offer. Animations, sounds effects, charts, and clever transitions are all entertaining, but they distract from your message. Also, avoid complex, cluttered slides loaded with detail and don't, ever, just write a speech onto slides and read it out along with your audience. Slides aren't cue cards, they're important material which backs up what you say, reinforces your message, and helps add impact to the story you're telling face to face.
A couple of final points - firstly, when you're "considering your audience" and thinking about who you'll need to get onside with your idea, don't forgot about thought leaders. It's usually easy to spot the people who will be directly affected by your plan, and decision makers tend to be obvious too, but so many neglect this final category. By thought leaders, I mean people who might not have any official hierarchical power or authority to help you, but nonetheless they set opinion and exert an influence. Getting these people onside with your plans will make adoption everywhere else much smoother.
Last of all, don't overdo it. You are kidding yourself if you don't believe you're selling when you're presenting a plan, but you must also be realistic and present a balanced view of whatever it is you're asking us for. As executives, we know that there is no such thing as an idea with no downsides, and pitching that way will just make us suspicious.
Tuesday, 20 January 2009
Don't Tell the plan, Sell the plan
Just remember the title of this post - don't tell the plan, sell the plan. As engineers, we come up with some pretty clever stuff. We're up against hard problems and we deliver smart solutions. Where we get unstuck is when we go about sharing these with the decision makers that we need to get behind them.
As engineers, we tend to think that "selling" an idea somehow devalues it - sacrifices it's integrity - perhaps we think it is sneaky, cunning, or politically motivated to properly pitch something rather than just retell the facts behind it. But believe me, for better or for worse, the things that actually end up getting done in organizations aren't the best ideas, they are the ideas that were sold the best.
Don't get me wrong - you still need that great idea, a solid business case, and a well thought through execution plan - it's just that those things alone rarely turn your proposal into a funded priority.
On a personal note, back when I was an engineer I always told myself that I'd be different, however as my career moved towards decision maker, I found that I ended up pretty much doing the exact same thing. It's not through any lack of technical knowledge or understanding of how a proposed system will work, it's simply that I do a different job now - and that job is to weigh up all the possible things we could do as a team, work out which ones will make us the most successful, and then not lose sleep over the rest.
I bet you anything your boss does exactly the same thing - he's not an idiot that doesn't seem to trust you - he just wants to quickly understand the core concept, then make a judgement based on the real cost you want today vs the potential future return against a background of all the other things he might choose to do instead. And you know what? We don't need to know exactly how the older cached inserts will be invalidated in the remote nodes once a newer update is detected locally (or whatever) in order to make that decision - but if we ask you then you'd better know!
Even if you look at it cynically - you have the best idea anyway right, so why not have the best presented idea too? If it really is the best proposal, then don't you owe it your organization to give it the best chance of becoming reality? The only danger you'll face by being better at presenting your plans is that they're more likely to get done...
So I hope I've convinced you that the presentation of your ideas is as critical to their success as the ideas themselves, so the next most obvious question is how do you do it effectively? I was planning to write something on doing presentations anyway, so come back in a few days and I'll tell you what I think.
Friday, 16 January 2009
No Pain No Gain?
A couple of data architects were talking about a consolidation project they were kicking off. "Can you believe what they're doing in the Japanese office? Their schema is so basic" one complained to the other. "Yeah, they clearly don't know what they're doing" his colleague replied, "they should leave it up to us." A couple more minutes of the same and I couldn't really let it go.
Gathering up my most diplomatic tone, I asked the resident geniuses: "OK lads, if you're saying that they have a basic, simple solution which meets the business requirements, and you have a complex, difficult solution which meets the business requirements - then isn't it you who have something to learn from them?"
You see this is why I don't have any friends.
But seriously, what's the attraction to building big, creaky, behemoth solutions where something easy and elegant would meet everybody's expectations? Why is it that we seem to derive satisfaction from making our lives hard? Unless you're very lucky (lazy?), you'll have to troubleshoot, maintain, and support whatever you put in place.
I've noticed before that good engineers can build vast systems with webs of dependencies and many moving parts, but it's the great engineers who realize that this is a bad idea.
Outside of the gym, there are very few places in the world where you'll be better rewarded for increasing the effort necessary to achieve a given outcome.
Tuesday, 13 January 2009
The Parable of the Big Hands
A number of years ago I was called in to help an integration company who had been struggling for a number of weeks with a manufacturing client. They had delivered a factory automation system, introducing computer-controlled fabrication and handling machines, replacing the manually operated mechanical kind. The project had been justified upon increased productivity and reduced material wastage - yet reduced productivity and increased metal wastage were being observed. Not really the results anyone was hoping for, customers blaming the vendor, vendor blaming the users etc.
My job was to "troubleshoot the new software" and "find the bugs" before the manufacturing company started firing live ammo.
Step 1 was to get a coherent problem statement. It seemed that in the vendor's lab everything went according to plan - sheet metal machines could be reconfigured in tens of seconds via a new software load - yet at the factory it was taking minutes and materials were frequently being cut to the wrong dimensions.
So what was different between the factory and the lab? I was assured that there was no differences at all - machines had been swapped, the exact same builds had been uploaded to the machines at both sites, error logs had been downloaded etc. But clearly there was a difference, otherwise it would be working (or broken) in both places.
I figured there must be something environmental - how else could the exact same machine, software, and configuration work perfectly in one location and not another. "So, tell me about this factory" I asked. "Never been there" was the answer (believe it or not). Ah-ha!
So off we go to the warehouse/factory thingy. We never even got through the tour of the floor before we made the critical observation - these metalworking dudes have really big hands! If your job consists of throwing around huge blocks of steel, cranking big heavy leavers, and turning giant bolts, then you soon develop a pair of mitts like shovels.
Why was this important? Quite simple. The keys on the small numpad-type keyboard that had been specially made for entering the program parameters on each machine were a standard keyboard size, and were therefore simply far too small for the bunch-of-bannans fingers of the guys that had to use it. These guys are hardcore welders and whatnot, they don't have our delicate little programmer hands!
So what's the lesson here? Know Your Users. It's as simple as that. You have no business even touching a keyboard (of any size) until you know who your users are, where they work, how they'll interact with the system, and what successful use is. Remember - you're not building the software for yourself or your business analysts, you're building the software for them. Get to know them.
Sunday, 11 January 2009
Making a decision isn't a stand alone task, when you make one, you really do a small collection of things together. You make choice from some options, you make a plan and delegate it, and then you follow up and review. Most people think that once they've picked their option they're done, but like Drucker says, you haven't made a decision until you have an action plan - you've only made a choice.
So let's talk a little more about effective choosing, as that's the first hurdle to trip over. The behaviors you want to avoid here are essentially procrastination. How many times have you been staring an obvious course of action in the face (or been presented with a fantastic opportunity) but instead of making progress, you're taking bets on which is growing faster - your number of choices or the list of data points people want to see for each one! You have to reduce your number of options to a manageable amount (3 is a magic number if you're stuck for inspiration), and while you should never ignore great last minute ideas just for the sake of rules, most business don't fully appreciate the cost of inaction. The point marked "doing anything is better than nothing" creeps up a lot quicker than you think.
The next thing you have to worry about is the ballooning amount of information on each of your choices. Too many and you risk it being treated like a smorgasbord (can we have this from option A, and that from option B) or being paralyzed by the sheer volume of statistics and technicalities. Try removing things that are the same for all choices - for example, if A, B, and C are all similarly expensive, why compare cost? And how about things you don't feel strongly about? I don't know how many times I've seen stakeholders agonize over details like the difference in reusability between 2 choices in a decision, and then when asked where else they plan to use the technology, answer "nowhere". If it's an attribute you don't feel strongly about, then it's unlikely to influence your decision, so why clutter the process with it?
We've talked a lot about paring things down, and now here's something to add in. Don't forget that for every choice you're comparing you need to know what you don't get if you don't pick it. Most people are generally pretty good at stacking up the costs and benefits, but opportunity cost is often neglected.
Hurdle number 2 is action plans and delegation - the behavior you want to avoid here are those deja vu meetings where you all sit around and 'review' the decisions you made, wonder at the lack of progress and then agree to 'update next week' all over again. Once you've made your choice, don't let yourself off the hook until you've agreed who is actually going to do the work, and by when. Follow the usual rules on delegation - make sure whoever you're going to hold accountable clearly understands their obligations, is empowered to do the work, and has the available resources.
Don't forget to follow up. You spend your time on the things that are important to you, and you're not making unimportant decisions, are you? The things that you focus on will get done - keep track of what's happening, make major milestones public, and use your sponsorship to see it through. And once whatever it is has been delivered (or has crashed and burned), you're still not quite finished. Now comes the feedback loop - what did you learn? How can the experience improve the rest of the organization? What pitfalls can you now help others avoid?
And finally, you shouldn't be afraid to revisit your decisions and change your mind. You should be keeping an eye on your customers, your portfolio, and your team, and continually assessing what any changing circumstances may mean for you. That said, you must resist being flighty as you need to show commitment, but keep in mind that "because we said we would" isn't a business benefit to completing a project - it's a sleepwalker's justification for not being in touch with the market.
Saturday, 10 January 2009
Welcome to 2009
At this time of year it's customary to make a few new year's resolutions - fresh commitments you somehow feel renewed fervor towards keeping just because we incremented the year counter by 1 - and they almost always involve gym memberships.
If I take a brief look back at last year's resolutions I think I can safely say I hit my top priorities, but just as predicted, the exercise part seems to have slipped through the cracks. No matter, that's what these annual counter-flips are for, getting invigorated about your health for 30 days or so.
I'm always interested in hearing from people out there in the technical community for any reason, and from time to time I am lucky enough to receive some feedback on my posts (thanks by the way!). Over the last 12 months the articles that have generated the most interest have been on management, dealing with people, and being effective in organizations. It seems like you want to hear more about getting things done around technology than about the technology itself. Well there's some resolution input right there...
This year I'm going to try changing the balance of the content I write, aiming for a higher ratio on project management and professional disciplines without completely dropping the things I like to bring up on architecture and computer science. After all, a hundred little things happen every day, and it's just a matter of being a bit more selective about what's captured and shared.
So yeah, welcome to 2009, you can make it whatever you what you want it to be.