The most common criticism I hear about agile is also the most common mistake teams make in their early adoption of iterative practices - not fixing any certainty.
One of the benefits of managing projects using, for example, SCRUM is that you're not burdened with having to define all the requirements up front and plan out all the tasks and times for all the resources for 6, 12, 18 months. Trying to precisely define everything a project needs up front and trying to plan for every dependency, holiday, resource issue, structural or strategic change over a long period of time proves again and again to be an impractical task.
Enter agile.
Suddenly we don't have to worry about the detail of every change and every movement over an entire year, and we're introducing a whole bunch of flexibility. Where it goes wrong is not fixing any certainty in the delivery process at all.
Agile is about certainty and detailed planning, just detailed planning over a period which is possible to predict. Who knows what'll happen over a year, but you at least have a fighting chance of planning out the next 2-4 weeks will some degree of accuracy. Beyond that you can keep plans and requirements fluid and high level, netting you the right balance of fixing doable deliverables and keeping flexibility.
Software delivery needs certainty to happen - you have to be able to rely on some requirements and some resources over some time span in order to make progress - and agile isn't about not having any, it's about having it on terms you can foresee.
Saturday, 3 October 2009
Sunday, 27 September 2009
Screen Scraping for Dummies
If you run a website with any sort of valuable content, then you are almost guaranteed to run into scraping sooner or later. Screen Scraping is more or less an automated program taking an impression of a web page and then parsing it to pull out some specific bits of data that the scraper is interested in - which theoretically is then stored or used in some other way.
The piece of software that does this scraping is commonly called a robot, or bot, and it is really just an automated web client that accesses and uses sites in the exact same way as it's fleshy counterparts, just with machine precision and repetitiveness. A bot may be a large and complex program running on a server with it’s own database etc, or as simple as a script running in a browser on a desktop.
This is typically regarded as undesirable behavior by many sites because, in most cases, it’s a source of load and associated with unprofitable usage. Whenever we draw a page impression, which we’ll do for every bot hit just as we do for every human visitor, we consume web server time and, worse, back end time. When we structure highly functional pages loaded dynamic content we can create a very engaging user experience, and all that functionality is built on plenty of back end work and data. When used at regular human pace that's usually OK, but under the relentless rate of hammering robots are capable of, it starts to become expensive.
If you've got a case of bots, you have to start by identifying them. Their repetitiveness and flawless precision is, in this case, their downfall and we can usually spot them easily through proper analysis of web logs - no human user is as mechanically regular, millisecond quick, and consistent in journey as your average droid.
Spotting droids isn’t too hard, but then you've got to decide what you're going to do about it. Most scrapers aren't malicious and often don't realise the headache they're giving you. In the first instance it's best to try some detective work, see if you can find out who they are, and get in touch. Domain registrars can be a great resource for this, but don't overlook the obvious - maybe they even have an account with you if data they're using requires signup to view.
Beyond that it gets tricky, and can easily turn into an IP address blocking arms race. With some caching finesse or smart layer 7 rules you can throttle bot activity to more palatable levels, or persist them to their very own node that they can thrash all day without impacting the experience for the rest of your users.
If robots are a very big problem for you, then try and take it as a compliment on the value of your data and perhaps consider publishing it via a productised API or feed - if you make it simple enough to consume, then most scrapers will willingly change over to a more reliable integration mechanism and perhaps even pay you for it.
The piece of software that does this scraping is commonly called a robot, or bot, and it is really just an automated web client that accesses and uses sites in the exact same way as it's fleshy counterparts, just with machine precision and repetitiveness. A bot may be a large and complex program running on a server with it’s own database etc, or as simple as a script running in a browser on a desktop.
This is typically regarded as undesirable behavior by many sites because, in most cases, it’s a source of load and associated with unprofitable usage. Whenever we draw a page impression, which we’ll do for every bot hit just as we do for every human visitor, we consume web server time and, worse, back end time. When we structure highly functional pages loaded dynamic content we can create a very engaging user experience, and all that functionality is built on plenty of back end work and data. When used at regular human pace that's usually OK, but under the relentless rate of hammering robots are capable of, it starts to become expensive.
If you've got a case of bots, you have to start by identifying them. Their repetitiveness and flawless precision is, in this case, their downfall and we can usually spot them easily through proper analysis of web logs - no human user is as mechanically regular, millisecond quick, and consistent in journey as your average droid.
Spotting droids isn’t too hard, but then you've got to decide what you're going to do about it. Most scrapers aren't malicious and often don't realise the headache they're giving you. In the first instance it's best to try some detective work, see if you can find out who they are, and get in touch. Domain registrars can be a great resource for this, but don't overlook the obvious - maybe they even have an account with you if data they're using requires signup to view.
Beyond that it gets tricky, and can easily turn into an IP address blocking arms race. With some caching finesse or smart layer 7 rules you can throttle bot activity to more palatable levels, or persist them to their very own node that they can thrash all day without impacting the experience for the rest of your users.
If robots are a very big problem for you, then try and take it as a compliment on the value of your data and perhaps consider publishing it via a productised API or feed - if you make it simple enough to consume, then most scrapers will willingly change over to a more reliable integration mechanism and perhaps even pay you for it.
Sunday, 20 September 2009
conferences - jolly good or just a jolly?
It has always interested me that two different people can look at the exact same conference, and one declare it to be an unmissable event relevant to their work and the other consider it a waste of time; a free day out of the office at best.
It always leads to a discussion about which events are worthwhile and which aren't, but here's the secret - whether a conference is worthwhile or not is only 10% the event itself and 90% what you do while there and afterward.
You could carefully plan the sessions you attend, take the time to meet people with similar technical problems or who have done worked with relevant technology and follow up with them later and, when you return to the office, share your new knowledge around and adopt some better practices.
Or
You could have a few days out, enjoy some vendor's hospitality (free beer always tastes better), and check out some new bars and, when you return to the office, settle comfortably back into your old habits.
That, ladies and gentlemen, is the difference between a jolly good conference and just a jolly.
It always leads to a discussion about which events are worthwhile and which aren't, but here's the secret - whether a conference is worthwhile or not is only 10% the event itself and 90% what you do while there and afterward.
You could carefully plan the sessions you attend, take the time to meet people with similar technical problems or who have done worked with relevant technology and follow up with them later and, when you return to the office, share your new knowledge around and adopt some better practices.
Or
You could have a few days out, enjoy some vendor's hospitality (free beer always tastes better), and check out some new bars and, when you return to the office, settle comfortably back into your old habits.
That, ladies and gentlemen, is the difference between a jolly good conference and just a jolly.
Friday, 4 September 2009
Delegation from the other side
I really like Manager Tools - it's probably the best single source of personal development material that you're ever going to find bundled in easy to consume episodically content. Recently I thought I'd go back and listen to the basics series again, because you can never spend too much time on the fundamentals of good management and, like Mark and Mike say, if you don't do anything else except those basic practices, you'll probably do OK.
It was a really good refresher, and worthwhile doing from time to time. Then I got to the juggling koan cast, which was about how to handle that inevitable situation when your boss gives you another big task to work on when you're already maxed out with too much to do. It's worth listening to - so I won't spoil it entirely - except to say that the answer focuses on developing your delegation muscles, passing on some of your things to make space for the new action and helping your team to grow as an nice side effect.
The majority of the cast focused on how to handle events after you accept this new delegated responsibility, and I liked the point about deliverables gaining size as they flow down the delegation tree (i.e. your small tasks are bigger challenges for more junior members of your team), however I think there was a valuable lesson here about the delegation from the recipient's perspective.
The Manager Tools perspective on this was to accept the new delegation, and to view it as the expression of confidence and trust and the development opportunity that it really is. Many new managers would worry about whether they have the time to take it on or not (no thanks boss) or attempt to enter into a negotiation about what else to drop.* The reason I like this angle is that I don't think there is enough coverage out there on followership.
Leadership is glamorous and fashionable to write about and speak on, so we do a lot of it, but good followership is important to develop too. There is an inherent responsibility on all the individuals in an organisation to be rational participants, and I think a group who know and practice the right behaviors on both sides of that equation performs better. Anyway, isn't part of what we're doing when we're developing our people just cultivating better followership?
Something I'd suggest you ponder when you listen to the cast is clarifying the difference between being handed an action and given responsibility for an outcome. There was heavy emphasis on not entering into negotiations with your boss about whether or not you are going to take this thing on, however I think it's both valid and valuable to question the task and look for the underlying business value.
This isn't arguing about whether or not you or your department have the time, or forcing your boss to manage your priorities for you, it's simply clarifying what you're really being expected to achieve rather than what you're being asked to do. In my experience, particularly because I run technical departments, there is often a difference between what people want to achieve and what they initially ask for, as they think they have resolved the outcome they have in mind into a few actions and sometimes those actions don't add up.
If you can have that sort of open discussion, then you might agree to accept delegation of a slightly different task than was first put your way, but one that much more directly leads to the goal your boss really had in mind.
Sunday, 30 August 2009
A better way to manage equity?
When we get together to start a new venture, the structure of the equity is always an early topic. The usual format is something along the lines of 'you do X for 20%, he does Y for 10%, and I do Z for 30%' and then we carve up the company up front.
The problem with this approach is that it doesn't easily cater for all the things that really happen during the course of a startup. For example, you can't always predict who's going to stick with it or how many people are ultimately going to be involved. As the idea evolves (because you almost never end up with the same plan you started out on) the bias of work can shift to a point where the equity split no longer reflects the effort split, and of course the value of effort to a business changes over time.
I don't think there is such a thing as a flawless mechanism for this, however I think I might be onto something a little more elegant. Here's how it works...
Break all the work - this same approach works for commercial and marketing tasks just as well as it does for the technical stuff - required to bring your idea to market into tasks or jobs which can be done individually by the members of your team. Decide on a number of points for each task, based on a sensible blend of value to the business and effort required to deliver. Points are the key to the whole thing; do the task -> earn the points -> exchange the points for shares.
Because you haven't fixed the equity allocations up front, it doesn't matter if you gain or lose co-founders and you minimise those 'who isn't pulling their weight' discussions, as you only get the shares you directly earn.
But why points when you could just do the same thing with the shares themselves?
Points give you a couple of extra controls you might not have if you immediately issued the shares; for example you can set an exchange rate between points and shares, allowing you to increase the relative value of tasks not being picked up, or offer individuals a better exchange rate once they have accumulated a large number of points (thus encouraging fewer greater contributors which makes for a more coherent project and a simpler shareholders register).
Setting specific milestones (for example beta launch or first partner signed up) at which points can be exchanged for equity ensures that everyone still focuses on bringing together an overall business delivery as well as their own contributions.
Forming your company with a healthy allocation of shares and some fairly accurate record keeping on tasks and points (I can recommend zen if you're an agile/lean thinker) is all that's required administratively.
There are a number of more subtle points to this, such as making sure you've got strong enough acceptance criteria on each of these tasks, but that's the meat of it. I'm running a project using this method now and - while we're not done yet - it's going pretty well and has already afforded us some flexibility we wouldn't otherwise have had.
The problem with this approach is that it doesn't easily cater for all the things that really happen during the course of a startup. For example, you can't always predict who's going to stick with it or how many people are ultimately going to be involved. As the idea evolves (because you almost never end up with the same plan you started out on) the bias of work can shift to a point where the equity split no longer reflects the effort split, and of course the value of effort to a business changes over time.
I don't think there is such a thing as a flawless mechanism for this, however I think I might be onto something a little more elegant. Here's how it works...
Break all the work - this same approach works for commercial and marketing tasks just as well as it does for the technical stuff - required to bring your idea to market into tasks or jobs which can be done individually by the members of your team. Decide on a number of points for each task, based on a sensible blend of value to the business and effort required to deliver. Points are the key to the whole thing; do the task -> earn the points -> exchange the points for shares.
Because you haven't fixed the equity allocations up front, it doesn't matter if you gain or lose co-founders and you minimise those 'who isn't pulling their weight' discussions, as you only get the shares you directly earn.
But why points when you could just do the same thing with the shares themselves?
Points give you a couple of extra controls you might not have if you immediately issued the shares; for example you can set an exchange rate between points and shares, allowing you to increase the relative value of tasks not being picked up, or offer individuals a better exchange rate once they have accumulated a large number of points (thus encouraging fewer greater contributors which makes for a more coherent project and a simpler shareholders register).
Setting specific milestones (for example beta launch or first partner signed up) at which points can be exchanged for equity ensures that everyone still focuses on bringing together an overall business delivery as well as their own contributions.
Forming your company with a healthy allocation of shares and some fairly accurate record keeping on tasks and points (I can recommend zen if you're an agile/lean thinker) is all that's required administratively.
There are a number of more subtle points to this, such as making sure you've got strong enough acceptance criteria on each of these tasks, but that's the meat of it. I'm running a project using this method now and - while we're not done yet - it's going pretty well and has already afforded us some flexibility we wouldn't otherwise have had.
Wednesday, 19 August 2009
Doing it on the side
Something new managers often struggle with is what their response should be when members of their team are involved (or are considering being involved) in external, perhaps even related, business activities. I encourage it wholeheartedly, with a small number of key caveats, on personal development grounds.
Trying out "their own thing" will teach your guys about responsibilities well beyond those that you can reasonable expect to expose them to in their role, and give them some insight into the difficulties of running a business, taking care of finance, marketing to customers, and making product decisions. There is so much to the operational side of an organisation (even a tiny fledgling one) that they would otherwise not have many chances to experience first hand. This can help your team members develop their creative, entrepreneurial side and give them a much better appreciation of the challenges their colleagues elsewhere in the business face. This is so beneficial that I'm even found guilty of supporting such ventures with advice and coaching from time to time.
The caveats? There are only 2 key things that must be true to qualify what I’ve said above:
1 - You still have a job. Outside interests should not impinge on the quality, quantity, or timeliness of work, and if they do then one of these things must go. My policy is to provide a flexible working environment, but I'm not here to subsidise your startup either.
2 - You shouldn't compete with your employer. For most people in our industry, our roles come with a duty to innovate and develop new product to further the companies interests. Whenever something extracurricular can be even slightly interpreted as related, it needs to be raised with management and explicitly categorised into 'company IP' or 'fair game' for everyones benefit.
Regardless of which side of the employer/employee relationship you sit on it is always worth finding out what your company policies are on this kind of thing before getting underway. Being uninformed might mean you start out with the best intentions and end up having to make difficult choices later.
Trying out "their own thing" will teach your guys about responsibilities well beyond those that you can reasonable expect to expose them to in their role, and give them some insight into the difficulties of running a business, taking care of finance, marketing to customers, and making product decisions. There is so much to the operational side of an organisation (even a tiny fledgling one) that they would otherwise not have many chances to experience first hand. This can help your team members develop their creative, entrepreneurial side and give them a much better appreciation of the challenges their colleagues elsewhere in the business face. This is so beneficial that I'm even found guilty of supporting such ventures with advice and coaching from time to time.
The caveats? There are only 2 key things that must be true to qualify what I’ve said above:
1 - You still have a job. Outside interests should not impinge on the quality, quantity, or timeliness of work, and if they do then one of these things must go. My policy is to provide a flexible working environment, but I'm not here to subsidise your startup either.
2 - You shouldn't compete with your employer. For most people in our industry, our roles come with a duty to innovate and develop new product to further the companies interests. Whenever something extracurricular can be even slightly interpreted as related, it needs to be raised with management and explicitly categorised into 'company IP' or 'fair game' for everyones benefit.
Regardless of which side of the employer/employee relationship you sit on it is always worth finding out what your company policies are on this kind of thing before getting underway. Being uninformed might mean you start out with the best intentions and end up having to make difficult choices later.
Tuesday, 16 June 2009
The nuances of Planning Poker
I’m going to skip right over explaining what planning poker is and how it works - especially since there are so many simple and effective explanations out there - and assume that you’ve at least given it a bash.
What I want to address is the spirit in which group estimation happens.
Every potential task owner (read: qualified opinion) estimates each relevant task as it comes up, but the reason for this is commonly misinterpreted. The purpose is not to force the eventual task owner to reduce his or her estimate, or to adopt an estimate that others in the group ‘prefer’ and nor is it a criteria for selecting an owner for a task, based on the estimate the product owner or scrum master likes the most.
The purpose is to draw out knowledge from the outliers, so that the whole team can benefit from their experience. When someone in the group pulls a card miles away from the rest (assuming a consistent understanding of the task) that’s something to explore. You might have a much higher estimate – perhaps that person has been burnt before by that same inconspicuous looking requirement masking significant complexity. You might have a much lower estimate – perhaps that person knows a good work saving shortcut or pattern uncovered during a previous project.
If you take the time to ask those questions and consider that information, the group learns from one another and everyone gets better at estimating.
And whose estimate is used in the sprint? The person who is going to be doing the work...
What I want to address is the spirit in which group estimation happens.
Every potential task owner (read: qualified opinion) estimates each relevant task as it comes up, but the reason for this is commonly misinterpreted. The purpose is not to force the eventual task owner to reduce his or her estimate, or to adopt an estimate that others in the group ‘prefer’ and nor is it a criteria for selecting an owner for a task, based on the estimate the product owner or scrum master likes the most.
The purpose is to draw out knowledge from the outliers, so that the whole team can benefit from their experience. When someone in the group pulls a card miles away from the rest (assuming a consistent understanding of the task) that’s something to explore. You might have a much higher estimate – perhaps that person has been burnt before by that same inconspicuous looking requirement masking significant complexity. You might have a much lower estimate – perhaps that person knows a good work saving shortcut or pattern uncovered during a previous project.
If you take the time to ask those questions and consider that information, the group learns from one another and everyone gets better at estimating.
And whose estimate is used in the sprint? The person who is going to be doing the work...
Sunday, 14 June 2009
Right First Time
Imagine if the first wall you ever built had to form an integral part of your dream home, or if the first time you ever drew a picture it had to hang in the Louvre? Or if the first cake you ever baked had to be for your daughters wedding?
That’s clearly unreasonable.
So why does it seem reasonable that the first iteration of a new system you build should be the one that your business depends upon?
That’s why we prototype. That’s why we iterate. That’s why we beta. Don’t rob yourself, and your organisation, of the wisdom you’re owed from those lessons.
That’s clearly unreasonable.
So why does it seem reasonable that the first iteration of a new system you build should be the one that your business depends upon?
That’s why we prototype. That’s why we iterate. That’s why we beta. Don’t rob yourself, and your organisation, of the wisdom you’re owed from those lessons.
Monday, 8 June 2009
So who is in a team anyway?
I caught up with a friend the other day; he’s in the system integrator business and is struggling a little with recent growth. Long story short, things are a bit a disorganized and efficiency is low. They’ve got some kind of consultant in, and we were talking over his advice.
So how does their guy propose they fight the first battle in the war for organizational effectiveness? Setting aside certain days when people in one team aren’t allowed to talk to another. No, you didn’t read that wrong, the ‘big plan’ is to actually create barriers to communication.
I just hope that wasn’t expensive.
In the reseller/systems integrator game it’s all down to the cooperation between sales and their technical teams, and so I really struggle to see how driving intentional wedges between them is going to help. So my theory? Perhaps the wrong people are teamed up together in the first place…
In any mission there are multiple people with different skillsets doing different jobs all coming together around a central project/customer/business problem. Isn’t that a team? The people who actually need to work together on a daily basis for the direct benefit of the customer? I don’t know why we keep insisting on grouping people by job or by skillset (sales team and technical team, or dev team and test team) because the cold, hard facts are that these teams can’t achieve anything end-to-end for a customer on their own. Maybe it just appeals to our human need to nicely categorize things.
When you put people together in a team, you create a tight loop of communication and cooperation between those people (it’s one of the reasons we do it in the first place) – but you do something else too – you also create a little organizational fence between them and other teams. A fence that we absolutely should work at keeping as low and friendly as possible, but a fence nonetheless. To me it just comes down to where you want to put your fences – does it make sense to create boundaries between the individuals whose effort must by synchronized daily to complete units of work?
In what I do, the unit of organizational delivery is the project (or, as a superset, the product). That’s that unique object around which we all gather and apply our individual skills and experience to create tangible value for the company. So by default, I have a nice, simple method for grouping my guys together in a way that creates the fewest barriers to the most common communication and coordination lines that they’ll be following each day. I suppose I could put all the developers in one team, all the testers in another team, all the project mangers in another team again etc – but if I did, then what’s the very first thing I’ll have to do as soon as I want to do any work? That’s right, pull them back together into virtual or temporary teams based on the projects they’re delivering against. Hmmm.
From my experience in the SI and reseller game, I’d suggest that the unit of organizational delivery is the customer. That’s the internally-ownable entity which performance and profit is measured against, margin is made against, and time is billed against. Yet almost every one of these companies is, at best, a matrix management organization. So you group people together into a sales team, a tech team, a presales team etc – and then wonder why, whenever they all have to get together around that common indivisible business problem (the customer) there are all sorts of communication issues, competing priorities, complexities planning time...
Perhaps a team in this type of organization should be something like a sales guy, some sales support, a presales guy, a couple of engineers, and a share of some administration – all grouped together around a set of similar customers that they own the end-to-end P&L for.
The inevitable argument against this kind of cellular organization is the same one I’m used to elsewhere – developers like hanging out with developers, infrastructure engineers like hanging out with infrastructure engineers and so on. My only response to this is that I don’t see how organizing your people into the teams which best reflects the true structure and purpose of the company precludes communities of interest, and again, just ask yourself where you want to put the barriers, and whether you consider the upside of keeping people grouped into teams of similar individuals worth the downside of trying like mad to make them work together efficiently and effectively whenever you have to get anything done.
So how does their guy propose they fight the first battle in the war for organizational effectiveness? Setting aside certain days when people in one team aren’t allowed to talk to another. No, you didn’t read that wrong, the ‘big plan’ is to actually create barriers to communication.
I just hope that wasn’t expensive.
In the reseller/systems integrator game it’s all down to the cooperation between sales and their technical teams, and so I really struggle to see how driving intentional wedges between them is going to help. So my theory? Perhaps the wrong people are teamed up together in the first place…
In any mission there are multiple people with different skillsets doing different jobs all coming together around a central project/customer/business problem. Isn’t that a team? The people who actually need to work together on a daily basis for the direct benefit of the customer? I don’t know why we keep insisting on grouping people by job or by skillset (sales team and technical team, or dev team and test team) because the cold, hard facts are that these teams can’t achieve anything end-to-end for a customer on their own. Maybe it just appeals to our human need to nicely categorize things.
When you put people together in a team, you create a tight loop of communication and cooperation between those people (it’s one of the reasons we do it in the first place) – but you do something else too – you also create a little organizational fence between them and other teams. A fence that we absolutely should work at keeping as low and friendly as possible, but a fence nonetheless. To me it just comes down to where you want to put your fences – does it make sense to create boundaries between the individuals whose effort must by synchronized daily to complete units of work?
In what I do, the unit of organizational delivery is the project (or, as a superset, the product). That’s that unique object around which we all gather and apply our individual skills and experience to create tangible value for the company. So by default, I have a nice, simple method for grouping my guys together in a way that creates the fewest barriers to the most common communication and coordination lines that they’ll be following each day. I suppose I could put all the developers in one team, all the testers in another team, all the project mangers in another team again etc – but if I did, then what’s the very first thing I’ll have to do as soon as I want to do any work? That’s right, pull them back together into virtual or temporary teams based on the projects they’re delivering against. Hmmm.
From my experience in the SI and reseller game, I’d suggest that the unit of organizational delivery is the customer. That’s the internally-ownable entity which performance and profit is measured against, margin is made against, and time is billed against. Yet almost every one of these companies is, at best, a matrix management organization. So you group people together into a sales team, a tech team, a presales team etc – and then wonder why, whenever they all have to get together around that common indivisible business problem (the customer) there are all sorts of communication issues, competing priorities, complexities planning time...
Perhaps a team in this type of organization should be something like a sales guy, some sales support, a presales guy, a couple of engineers, and a share of some administration – all grouped together around a set of similar customers that they own the end-to-end P&L for.
The inevitable argument against this kind of cellular organization is the same one I’m used to elsewhere – developers like hanging out with developers, infrastructure engineers like hanging out with infrastructure engineers and so on. My only response to this is that I don’t see how organizing your people into the teams which best reflects the true structure and purpose of the company precludes communities of interest, and again, just ask yourself where you want to put the barriers, and whether you consider the upside of keeping people grouped into teams of similar individuals worth the downside of trying like mad to make them work together efficiently and effectively whenever you have to get anything done.
Monday, 1 June 2009
Good ops guys are hard to find...
Good operations is about staying one step ahead of the state of the system; taking proactive actions based on quality telemetry.
I’m reinventing how my sites are supported – if you’re an awesome one of these or a kick-ass one of these, then we should talk.
I’m reinventing how my sites are supported – if you’re an awesome one of these or a kick-ass one of these, then we should talk.
Subscribe to:
Posts (Atom)