“We don’t have enough failed sprints,” my boss said during his staff meeting. The comment was said completely out of the blue as the meeting was winding down and I have to admit it threw me for a loop. Surely not failing a sprint was a good thing right?
Our group was pretty new to scrum at the time, and I didn’t really know what he expected me to do. The comment wasn’t related to any of his managers, just a general observation. It was his way of saying that he didn’t think our teams were pushing the envelope hard enough.
Some years, and bosses, later I was sitting in on a planning meeting. The team would calculate available hours by adding up the man days for the sprint and multiplying by four (the number of available hours per day). They’d then compare that to the sum of the task estimates to get an idea as to whether a backlog item was achievable in a sprint.
The thing was I thought the 50% availability was a bit conservative. And, I could tell that the task estimates were padded too. So to my mind the team was running at a capacity of about 25%. OK, that’s probably unfair to the team, but you get my point. I bit my tongue at the time because an in progress planning meeting is not the place to tweak process.
But it reminded me of my old boss’s remark, and to his credit I couldn’t recall the last time that any of my team’s sprints had failed. It was pretty clear that the team was scared of failure, and that was translating into very conservative estimates, and excessive time padding. They’d lost that gung-ho frontier attitude.
The fact is, that to get the most out of a team they need to be encouraged to push themselves and be aggressive about what they can pull off the backlog. But that will mean failure at some point. And if that failure has consequences they may not push as hard next time.
And remember, even thought you may not practice executive censure for failure, if they are a group of high achieving individuals -as these guys were- then just the fact that they’ve failed, and that their peers know it, is probably as bad as anything you can dish out.
So what do you do?
The official scrum process is that the team should be conservative in what they commit to, but if they have time left over then they should work with the product owner to possibly pick up some additional backlog items. To me this seems like a good litmus test of an engaged enthusiastic team. Make sure your scrum master is checking for this.
Looking back I struggle to identify the source of their fear of failure. It certainly wasn’t punishment on my part. We’d been through numerous management changes and had a new fire breathing VP, so that could have been it. In hind sight I should have encouraged a more aggressive approach to counter that cultural pressure.
And that’s the thing with self managing teams. While you inspect and measure their output and they optimise themselves appropriately, they are also subject to other pressures from the organisation. If you’re not diligent about inspecting all the results of the team they could easily be optimising themselves in a direction you don’t expect.
And the fact that they fail, or don’t, is another output that should be inspected as well. And never failing could actually be the less desirable result.
My last post touched on developer productivity, and while we’re in that space we really should address the elephant in the room. It’s easy to take a micro economic view of productivity and pick away at individual issues, like I did in the previous post. The hope is that as you eliminate each problem the sum of your efforts will build up and eventually you’ll double or triple the productivity of your team.
After all that hard work, you might be tempted to measure productivity, and this is where you might miss the elephant. The elephant I’m talking about is that it’s normal to see huge variation in individual developer productivity.
Let me say that again. It’s OK to have one developer three, five or even ten time as productive as another.
Paul Graham introduces this idea in his Great Hackers essay which is where I first came across it. Sure before then I realised that different devs have different levels of productivity, but Paul introduced the idea that orders of magnitude of difference is not only possible, but it’s usual.
The next question that springs to mind is of course is that OK? Paul talks about individual productivity, but what’s the impact on the team of this imbalance. Is it healthy for twenty percent of the team to be producing eighty percent of the output.
The first situation you might compare this to is your archetypal gang of council workers. Six guys leaning on shovels, and one guy actually digging the ditch. But to continue the analogy, the situation you really have is more like a mechanised freeway construction crew. Different workers use different tools and contribute different things, and no one metric can properly measure the scope of an individual’s contribution.
Imagine if the productivity of each member of the aforementioned construction crew was measured using e.g. volume of dirt moved. The front end loader and tipper operators would leap to the top. The grader and roller operators who more than anyone else will define the quality of the road move mere fractions of dirt.
As much as it might be tempting to try to build teams of only high performing individuals. I suspect that such a team might well implode. I’m not suggesting that you should have laggards. But having solid performers and smart but inexperienced juniors is healthy. The best work I have seen from the senior devs was when there was a young up and coming junior snapping at their heels.
And don’t discount your good solid middle of the range developers. When you have an arduous and potentially dull merge to do that must be right they are your go-to guys. The senors will probably rankle at doing such a task and ironically do a worse job. These developers are the support network that allow the high performers the freedom to pick and choose the most challenging and engaging tasks.
So choose your metrics carefully. Counting lines of code might be OK for comparing teams over time, or maybe trying to identify external issues that might be affecting a team’s productivity. But it’s not much good for individuals.
And if you mold your team around a single metric, over time you’re likely to loose the diversity that makes your team an interesting place to work. Which will drive your high performers away faster than you can say “performance review”.
In fact, engineers love engineering. They love creating, and anything that gets in the road of that is beneath contempt. Meetings are reviled, but so are performance reviews, surveys from HR, pointless e-mails from euro sales droids sent to the ALL-Company mailing lists… you get the picture.
So what about the daily stand up? Is there room for improvement there? I say yes.
There are some good aspects of the stand up, apart from the fact that because the engineer is standing up it feels like they’re about to leave at any moment. Personally I think that whoever came up with the concept of the stand up, did that on purpose. It doesn’t feel like a meeting, it’s short, and the engineers know exactly what’s expected of them. Also they can probably prepare for it in the walk from their desk to the room.
But the meeting interrupts their morning, and they have to give status.
Now we can’t do much about the status because that’s really what the meeting is for. Not so much for management because frankly I can get all the status I want from the system. But by giving their status verbally they’re making themselves accountable to their peers for what they’ve been doing. And while they may think that they can bamboozle their PHB, they know their colleagues will see straight through any dissembling and they usually won’t want to look stupid or lazy in front of their colleagues.
But the price of this daily accountability is the interruption of the meeting. And, engineers hate being interrupted.
Coding requires that a huge amount of complex detail is kept in short term memory. A single interruption can flush that detail, and once it’s lost it can take fifteen minutes to get back to the same point. Developers are at their happiest when they’re in that fugue state where algorithms are leaping fully formed from their fingers. But it takes time to get into that zone, and only seconds to be pulled from it.
That is why companies have meeting free days, or e-mail free days. Ideally creatives, engineers, writers and developers should schedule their time in four hour blocks if they’re going to be productive. For a manager, whose day is interrupt driven, this can be difficult to understand.
So that’s easy to fix right? let’s just schedule the stand up first thing. Except maybe you have flexible working hours. OK, you think, just about everyone is in by ten, let’s have it then. But then you realise that if you have it at ten you’re pulling the guys who got in at eight right in the middle of their morning.
For the early starters, this is the most productive part of their day. This is the only time they can code in peace. What is worse, if they know that a meeting is coming up at ten they may even choose not to start a big task because it will be interrupted in a few hours time. Maybe they’ll use that time to read e-mail instead. In a worst case scenario your choice of meeting time may well have halved the productivity of all your early starters.
My solution was to have the stand up just before lunch. In fact I had this policy for all team meetings right from when I first became a manager. If at all possible any team meeting should be between 11:30 and 12. The exception might be something like a big planning meeting that you know will take half a day.
The reason is simple. If the standup is at 11:45, with lunch looming the engineers are already in the mind set to take a break anyway, so it’s a lot less disruptive. They might make a phone call, or check a couple of e-mails afterwards, but they’re not going to start anything big. They’ve just had a good three or four hours of (hopefully) uninterrupted productive work and once they’re done with lunch they can get back to another substantial task without worrying about a meeting killing their afternoon.
The fact that you also unconsciously associate the stand up with food can’t be a bad thing either.
In the end the more uninterrupted development time you can give your engineers, the happier they will be and the more productive they will be as a result. You may not be able to control that spam e-mail from the euro sales team, but you can control when your stand ups are. The timing of a meeting may seem like a little thing, but to quote John Wooden;
“It’s the little details that are vital. Little things make big things happen”
Back when I was a young whipper snapper I’d often work on some project in my father’s shed. By project I mean some kind of adaption to my car like grafting a couple of Webbers to the engine. Dad had better tools than I did, so I’d often save a particularly nasty job for the university holidays.
Of course at some point something would go wrong, and a stream of profanities would issue out from under the hood of my car. Dad would wander over and look at the rounded nut, or stripped thread or whatever problem I’d created and say, in his laconic bush drawl, “I dunno, maybe you could try XXX”. Invariably XXX would be something odd-ball like patching a cracked oil sump with epoxy, or using a pipe wrench to undo a nut. Something I’d never have thought of, but it would usually work.
Dad’s solutions were resourceful and creative, he’d developed those skills through years of farming, and growing up during the war when everything was scarce. Sometimes he just didn’t have the right tool and had to make do.
And, when you’re part of a large development organisation you need to develop that kind of mentality as well. There will be things beyond your control, and mandates you will have no choice but to comply with. On some issues it may be worth campaigning for organisational change, but for some it won’t and you have to pick your battles.
For example my last employer had an internal all-of-company bug tracking system. Now while this was not the most cutting edge or fully featured system, it worked. And, when there’s a lot of inter product integration and dependency, it makes a lot of sense to have all the development teams on the same system.
So what happens in that situation when you’re evaluating an Agile tool that has bug tracking built in but you’ve already got one. There’s three options;
- Buy the tool and don’t use the feature. This causes an awkward workflow and potential confusion. Not to mention loss of value.
- Use the feature and write integration to the existing system. This could be a lot of work.
- Limit yourself to tools that don’t include bug tracking. This reduces your choice.
This is just one example. And if you’re trying to implement agile within an organisation that already has processes, policies and procedures governing development you will be coming up against this kind of thing all the time.
One of the smartest things that an old boss of mine did was assign one of our junior Indian developers the job of tool automation. At the time his organisation probably had around a hundred engineers and there were twelve to fourteen sprints running across a number of products.
All that engineer did was write tools and scripts that integrated various systems like bug tracking, sprint management, wiki’s etc. His work allowed the tool we used for managing our sprints to smoothly mesh with the overall development process that was never designed for scrum. Sometimes he used API’s the tools exposed, and sometimes he had to query directly against a tool’s database.
Like the pipe wrench on the nut, many of the integrations were a hack but they got the job done.
So before you decide on using that all singing all dancing cloud based tool consider the kinds of integrations you might have to do. Then ask yourself how, or even if, they’d be possible using just the API’s they provide.
To return to the car analogy, if the hood of your car is welded shut there’s no way you’ll be able to bolt those Webbers on. And my car so needed those Webbers.
For the last few weeks I’ve been involved with helping a start-up get moving. You may have noticed that I’ve been a little fixated on tools for the last few posts and that’s why. I’m currently choosing tools and procedures, which is a nice change from having them foisted on you, and I want to be sure to avoid the mistakes I’ve seen in the past.
The reason tool choice is important is because your tools define metrics you can collect. For this post I will focus only on the kind of metrics you might use when managing a Agile/Scrum team or teams. But the concepts are more generally applicable.
The internet is abound with stories of bad metric choice, setting pay rises based on the number of lines developers checked in is just one example. I’ll leave that kind of metric for a later post, if I ever address them at all.
But that example of a bad metric does serve to introduce some general principles for metrics.
- When a metric is gamed, it should still produces an outcome you desire.
- Metrics should be generated automatically using published methods.
- Metrics should be generated and published frequently, if not on demand.
- Metrics should be visible to everyone on the team or in the group.
I’ll discuss each of these concepts shortly, but in general if a metric does not satisfy those four conditions, it either needs to be replaced, or abandoned.
The first principle is probably the hardest to stick to. It’s hard because unless you’re a fool you’re hiring clever developers, ideally cleverer than you. They like solving complex puzzles and playing games. Any metric you create will be gamed. The lazy manager puts rules and policies in place to avoid the obvious gaming. This not only creates an us-and-them dynamic, but it also just makes the metric more complex, more challenging and thus more interesting to game.
A better approach is to anticipate the gaming, embrace it even. But this is not always possible. e.g. Years ago, during a bug-fix crunch period, before we’d adopted Agile, we had a product with a lot of bugs to fix to hit a release readiness milestone. We experimented with some gamification (that’s a topic for another post) but the practical upshot was that we offered prizes for the more days in a row that devs marked bugs as fixed. They only had to fix one bug in the day for it to count.
Now the senior devs could probably fix more than one bug a day, and we were concerned that they’d fix bugs and then sit on them, doling them out each day to get the maximum number in a row. But the thing was, over six weeks, we’d still get 30 bugs out of each of them, so why care.
The second principle is a little easier to stick to. It’s one of the reasons I’ve been harping on about ensuring that the tools you use can be queried and automated. If a metric is generated by someone counting the sticky notes on a wall, human error will always be a factor. Specifically, having a person generate a metric allows the veracity of the metric to be doubted. You can argue with a person and question their ability but you can’t argue with the results of a SQL query.
The third principle is tied closely to the second. Only automated metrics can be produced on demand. Generally if a lot of human processing is required to produce a metric it might be produced weekly or even worse monthly. Imagine how useless a sprint burn-down chart would be if it was only produced at the end of the sprint. That metric is only useful because it’s reviewed every day by the team. The moment someone is having trouble it’s obvious and the team pitches in to help.
The fourth principle is probably the most contentious. Metrics influence behavior most strongly when the team can access their metrics on demand and they know that you, and their peers can also access the metric. Think of all the sales teams that have leaderboards, or call centers that show how many calls operators answer. It’s the visibility of the metric that makes this work.
But this is a two edged sword. If the metrics focus down to individuals then you most definitely get developers locally optimising their own numbers even if that’s at the expense of their peers. Software development is a team activity, so in general you only want metrics that are measured for a whole team. Not wanting to let your peers down is probably one of the strongest motivators you have. You can leverage this by measuring the whole team, not individuals.
What are some examples of good and bad metrics that are specific to sprint team management, I hear you ask.
- Good: burndowns. As long as they’re generated by a system and not a scrum master with a spreadsheet. Putting it up on a dashboard or portal is good. But having every team’s burndown compiled into a single graph heavy email and sent out is the best use I’ve seen for a burn down.
- Good: Bugcounts. Using the sprint starting point as a baseline is good if you need to focus on quality and your teams have a tendency to focus on the features and leave the bugs for later. This kind of thinking is common for teams who’ve been doing waterfall for years.
- Good: Automated Test Counts and Coverage. Daily build results are good for this. As teams add features they should be adding tests and not introducing regressions. Even a simple percentage of code covered as generated by automated tools would work. Of course, beware of them gaming the system by having empty tests.
- Bad: Story Point Consumption. This one is easy to get from most tools, and as a manager you might want to know for planning what your teams are capable of consuming but do you really want them cherry picking the easy backlog items to boost their averages? It also encourages shipping features over quality.
To sum up, metrics are about communication. They not only tell you what your teams are doing, they tell your teams what you care about. Careful metric choice can also encourage teamwork. When a team’s metrics are published they can’t help but compare themselves to other teams. So metrics need to be chosen with thought because like it or not they influence behavior.
At one point as our group dived into our particular miss-implementation of scrum we had, as you would expect, one wall of a meeting room covered in yellow sticky notes. And we’re not talking here about a poky little phone room. This was the room we used for “all hands” meetings and for team pizza lunches. You could fit a good twenty pizzas on that meeting table and still have room for drinks and garlic bread.
That wall and those post-it’s managed the work of twenty five developers. And as I was shortly to discover, it was the bane of our scrum master’s life.
One day I wandered in to see what my developers were up to. I only managed one of the four teams at that point in time and the wall was great to see what my guys were doing. M (I’ve abbreviated her name to protect her right to anonymity) was sitting at the table, and even my stunted geek emotional radar could tell that she was seconds away from bursting into tears.
She was in the process of reconciling the wall with the bug tracking system and the escalation system. This took an inordinate part of her day and to add insult to injury she’d just discovered fluttering around under the table like autumn leaves, a whole pile of orphan post-it’s.
So I grabbed my laptop, sat down and helped her sort through the mess. Then for the next few weeks we would spend a good hour each day keeping on top of everything. It was the end of the release and our director and his peers needed the bug and escalation systems to be correct.
Now you would have thought that we should have been able to rely on the developers and the QA’s to keep all the systems updated. But when, as a manger, you’ve got processes that require your developers to update three different systems when they fix a bug, you have to take some of the blame for the inevitable mistakes and oversights. Hence I was helping keep everything straight.
Not long after that we abandoned the wall and it’s plethora of fluttering yellow notes. And eventually we managed to merge the escalation and bug tracking systems as well. As a learning experience it taught me not only to choose my tools more carefully, but also not to blindly trust the crowd. At the time the web was abound with people waxing lyrically at how post-it’s had transformed their development processes and we’d bought into that.
To be fair, if you have no formal development process, no bug tracking system and no feature management system then post-it’s would be an improvement. But when you do have existing systems, introducing a new manual system that has no potential for integration with your existing systems doesn’t really make sense.
In an enterprise it’s important to be able to back up your assertions with hard numbers. VP’s like spreadsheets. During planning you need to be able to say things like;
“Based on the last three months of development my sprint team consumes an average of 23 story points per sprint, with a minimum of 18 and a maximum of 27. Feature ABC is 100 story points in total so it will take at least 4 sprints.”
That can be used in a planning exercise. Saying instead;
“I think ABC will take about two months.”
Is just inviting your VP to order you to do it in one.
When your data is in a structured system that keeps history around the epic user stories, the stories they were broken down into, the tasks they became, and the estimates at each step of the way, you have a corpus of data that cannot be argued with.
So what happens instead in the above scenario is that the conversation you have with the VP is directed to the feature set and what can be trimmed from that if a one month deliverable is a hard requirement for feature ABC. That is a more constructive outcome for everyone.
From previous posts you, dear reader, will have gleaned that I’m not an advocate of post-it based task boards. As a visual aid, and to motivate teams and provide accountability and visibility within the team they are great. But they are a one trick pony. You can’t usefully distill data from them for other purposes. You can’t integrate them with other systems, and you can’t see them unless you’re in the room.
Not to mention that post-it’s fall off the wall and get lost under the table.
One of my colleagues Anthony Langsworth has recently documented his experiences and opinions on remote working. He deals with it specifically from the perspective of an architect, an individual contributor, and I would agree with all the points he brings up.
Before I get to the purpose of this post I want to add that his point about video conferencing really resonated with me. A good setup, like the halo suite we had, can certainly substitute for many face to face meetings and could easily be used for a lot of scrum meetings. Unfortunately that kind of suite becomes a victim of it’s own success. Once our sales organisation became aware of how good the halo suite was, it was booked solid in that crucial two hour overlap that we had with the US office.
As Anthony says, try it yourself. I would add that if you can get a setup dedicated for your engineering department’s use, then by all means build processes around it, but don’t rely on it if you have to share.
But my purpose in this post is not to rail on about video conferencing. However it does introduce the idea of how, and in fact why you should incorporate remote workers into a scrum/agile team.
So why even bother?
There are several reasons why you should endeavor to include the facility for remote work in a scrum/agile team, or for that matter in any software development team.
The first reason to consider remote workers is that in enterprise scale organisations distributed teams are a fact of life. To get to the size they are most enterprise scale organisations grew with some level of merger/acquisition. That inevitably results in teams that are spread out over the country and indeed over the globe. In large organizations, outsourcing is a fact of life. So as an engineering director you will either inherit, or be required to develop, capability in India, China, Russia, or wherever else skilled labour can be had at a reasonable fraction of a US salary.
Even if you have all your engineers co-located, it’s inevitable that you yourself will probably report to someone on a different continent, or at least in a different timezone. You will also have shared engineering services like localisation, and architecture that you need to include in your processes. And let’s not forget technical technical support who will probably have some presence in at least three timezones.
So, if your development processes only support co-located contributors because they consist of sticky notes stuck to a meeting room wall, you are reducing your options for staffing, and excluding key contributors that make your product better.
The second reason to consider remote workers is that it gives you an edge in a competitive labour market. Unless you’re a merchant bank with deep pockets you can’t just buy the best developers. And if you want to keep your best people from being poached by organisations with more budget that you have, you have to change the game.
Usually in engineering organisations this is done by encouraging casual dress, having break rooms with X-Boxes etc. But working from home for one day a week is a privilege that is a lot more sticky. Not only can you equate it more easily to money, but once your staff have structured their family lives around it, they will think twice before taking a higher paid position at a much more conservative organisation like a bank. And sometimes that all you need to keep them.
I’m not suggesting that you should neglect your developer’s remuneration, but the harsh reality is that the only time in my career as a manager that I didn’t have to worry about competing with financial institutions for staff was during the first two years of the GFC.
The third reason to consider remote workers is because of a side effect that supporting them generates. Engineering is not only about engineers. You have to provide status up the management chain, and you need input from architecture, sales, support and product management. As I mentioned earlier in large enterprises you are unlikely to have all these on site. The more of your processes you can expose externally, the better your chances of getting the key people involved early. I’m not saying that everyone should see everything, what you expose is up to you. But if your processes are sticky notes on walls and you decide to email a daily scrum burn-down. Generating that e-mail will be a horrible, unreliable, manual process that will either send your scrum masters mad, postal, or more likely somewhere else.
What is the key enabler for remote workers?
Right, so you’ve seen the light and don’t want to rule out remote workers. You might have gleaned by now that I’m not a fan of post it notes on walls. As a way of introducing scrum, for small projects, and possibly in planning meetings, they are a fluid, tactile tool that encourages discussion. But for big ongoing projects you should use some kind of planning system like Version One, or X-Planner.
Your needs will vary, but as a bare minimum you should ensure that the system has the following features;
- Some kind of backlog for user stories and a way to break epic’s down into items that are consumable by your team/teams.
- Sprint task list management.
- A back end database, or accessible API’s
- (Optional) Automatic generation of basic status information like burn down and sprint summaries are useful both for yourself and your stakeholders.
The reality is that almost any system will hit this minimum list and in fact you will get a veritable plethora of additional stats and graphs and sprint planning, bug tracking etc. Before you start implementation ensure that it has sufficient API’s and hooks to integrate it with systems you have already in place that you don’t want, or can’t replace (e.g. bug tracking, or release planning).
The beauty of using an online system over sticky notes is that when a dev has competed their current task early (probably because they’re at home in a distraction free environment) they can log on and pick up another. Also, they can dial into the daily stand ups, and still get a sense of who is working on what. I have seen a junior dev pick up a task in a standup, only to have one of my principals chime in from the phone with an idea, or a request to review the code because they’re familiar with that particular area. They can do this when they have the task details in front of them.
Like bug tracking systems, and source code control systems, I consider an online backlog/sprint management system essential to the working of a well run software development team. And as long as it’s web accessible then your engineers and managers, who will be using it every day, can do so whether they’re at the office, at home, or on the road.
You’ll unshackle your developers from their desk’s, and you’ll get a number of other benefits as well, which I’ll go into in a later post.
I’ve said before that large enterprises are each unique and have situations that your typical scrum cookbook cannot anticipate. Because of their size enterprises tend to centralise specialist functions. Product Management is one such function.
Before I delve into the issues, I had better define what my company calls a product manager. You will hear names like; Product Manager, Program Manager, Project Manager, and Development Manager used differently even within software companies.
Product Managers are responsible for determining the product direction. They talk to customers to determine needs. They put together road maps and do competitive analysis. They work with sales to find out issues blocking adoption. They work with Engineering to produce product requirements, and determine priorities. I suppose you could say that they are the public face of the product.
That sounds like the definition of a product owner in scrum right?
Yes, and No. Yes they own the direction of the product, but they’re not part of the engineering organisation. At least in Symantec they’re not. They have their own VP who is a more or less a peer with the engineering VP’s who are ultimately responsible for the products they produce.
They’re also not a large group. There would be about one Product Manager for about every fifty developers. Because they need to spend a lot of time with customers, it makes sense to place them all together in whichever region has the most customers. And here is where I get to my point, in large outsourced organisations, that will probably be a different region to the development team.
For waterfall this doesn’t really pose a problem. Product Managers do a lot of travel, and getting them out to the dev team two or three times a year for a planning session works fine. Time those trips right and Development Managers and Program Managers can then run the plan through to release.
But Scrum requires a more week to week involvement from the Product Owner. Planning sessions at the start of each sprint, and handovers at a minimum. Regular backlog grooming needs to be factored in as well. Scrum pretty much assumes that you’ve got the product owner on hand.
Let me tell you right now that we never really solved this issue to my complete satisfaction. But there are some things you can do to mitigate the effects;
- Do not use paper based sprint tracking tools like post it notes or index cards. You need to move to an all electronic form of tracking. Use Wiki’s like Confluence (http://en.wikipedia.org/wiki/Confluence_%28software%29) and requirements tools like Version One or Feature Plan.
- If you’ve got high end video conferencing available, by all means try it.
- Once the backlog is all electronic, backlog grooming can be scheduled as a weekly con call between Development Directors, Architects and Product Managers.
- Do everything possible to get the Product Managers to attend sprint handover meetings via con-call. Handovers are more structured meetings with a fixed agenda and are easier to follow over the phone. They should be less than an hour, and make sure the QA lead prepares a demo and uses screen sharing technology like Webex.
Planning meetings are too long and unstructured for a Product Manager to add value on the phone. It’s possible to get Development Managers and Architects to act as a proxy for the Product Owners for the planning and operational aspects of a sprint as long as they’ve prepared in concert with the Product Managers.
Before I close I would like to make one more observation. Even though the best mapping to the scrum Product Owner seemed to be Product Managers, I’m still not convinced it was the right one in our case. In their role supporting sales, our Product Managers have never been as available to engineering as we felt they needed to be. Especially as in our case they were not co-located.
I find myself thinking back to the early Altiris days before we’d grown to the point where we needed a Product Management organisation. Our CTO, (later VP) performed that role, and did some coding occasionally as well. He was the very epitomisation of a scrum product owner. Unfortunately CTOs and VPs don’t scale. As their organisation grows they need to create a structure to delegate functions to, and unless that structure is created specifically with scrum or agile in mind we will always have this problem.
So be prepared to map the Product Owner role to several different people. And if you’re a Development Director one of those people will probably be you. Because as far as your VP is concerned you are the Product Owner.
OK so you’ve seen the Google Ken Schwaber videos on http://video.google.com/videoplay?docid=-7230144396191025011 and you’ve sent your program managers and dev managers off to scrum training. And your whole team is full of enthusiasm for the new processes.
And they make a lot of sense. Taking a chunk of functionality and getting it to the point where it could actually go out the door is a good idea. Adding lots of half baked features just to show progress and then bug fixing in a death march for months or years is no-ones idea of fun.
So you vow that from now on features will not be released until they’re done, done and done. Technical debt will be a thing of the past.
But, there’s always a ‘but’. What about all that code you already have in your enterprise application. You’ve got –lets say– ten million lines of code developed over ten years. The quality of that code will likely be highly variable, depending on which developers coded it, who QA’d it, and what management pressure the team were under at the time.
You will in all likelihood have a bug list as long as your arm. There will also be customer and management escalations to deal with, if you haven’t got a good solid sustained engineering team covering your back (as I was). In fact if you’re strapped for resources your ability to get new functionality to market is probably groaning under the weight of just supporting what you have already.
The thing is, dealing with bugs is not something I’ve seen addressed in Agile/Scrum. Possibly because training and books seem to focus on small new development projects that don’t have this problem.
The first thing you need to do is get agreement from the product owners how much new development, vs technical dept, vs bug fixing you need to do. These will not be easy conversations. But if you focus on the customer benefits you should be able to come to an agreement on what percentage of your teams should be fixing bugs.
Second you should add bug fix items to the backlog, giving each one enough story points to be consumed by one team for one sprint. It’s important that these items are visible to everyone.
Thirdly you should schedule each of your sprint teams for a bug fix sprint on as fair a rotational basis as your critical path analysis will let you. Spread them out over your whole release cycle and if you have enough teams you should aim to always have one team on this duty at all times.
What you get from this approach:
- You’ve always got a go to team for escalations that you can hit up without frustrating them, or having them lose momentum on a feature.
- Having teams do nothing but bug fixing for two or three weeks give you good data on their fix rates.
- Once you have data you can predict ahead of time how many bugs your team will fix for a given level of resourcing. This will be invaluable when it comes to producing service packs. You will be able to use hard numbers when having conversations planning releases and how many bugs the business gets for what level of resourcing.
- Your teams are going to be a lot happier to do a bug fix sprint if it’s only every three or four sprints.
Things to watch out for:
- Don’t let your teams fall back into old habits. They should be fixing bugs that come up in sprints within that sprint, they should not be knowingly contributing to a net increase in bugs as features are implemented. Make one of your “Done, Done and Done” criteria a metric based off your bug tracking system and use that same system to track all sprint bugs.
- The bug fix sprints in the backlog will be the first in the firing line when a new feature has to be added to a release. You will just have to fight for these if you think it’s warranted. One thing that may help is putting a focus on the bug fix sprints e.g. performance or reliability. Choose areas that are known customer pain points for your sprints and you’ll get better buy in.
- Resist the urge to pick up a couple of extra “bits and pieces” in a bug fix sprint. It will dirty your metrics, and they team will work better if they’re all working on bugs together.
- Don’t over plan the bug fix sprints, and don’t make the team’s commit to a specific number of bugs. Make the team’s total (not individuals) part of the handover and openly discuss the count and how they compare to your other teams. That will generally be enough to spur them on to doing their best especially when they know that they (the team) is being compared against their peer teams.
Bug fixing is not one of the most pleasant aspects of software development. But when I implemented the above approach it became less of a thorn in my side. Sure, the teams didn’t love it when they were doing it, but once they’d got a couple of completely interruption free feature sprints they understood the value of it. And they loved the ten minutes planning meeting for those sprints.
It was good to have a team I could hit hit up for any escalations without the guilt of pulling them off a task. Plus, giving each dev in your org the opportunity to help their director out personally at some point, and getting the opportunity to personally and genuinely thank them for it has a value to morale that is incalculable.
This post is about organising large teams, specifically large teams working on a single product. One of the defining characteristics of enterprises is their size and the development teams, and the products they work on, within such an organisation often share that. Our VP used to brag that he had five hundred and fifty people in his engineering organisation, five hundred and fifty one including him. And apart from a few PA’s for the directors, and a small IT team most of that count were developers, QAs or their managers.
In many cases the teams will be organised on functional lines and broken down until you get to a manageable size. In that case each team can use pretty traditional scrum methods with little trouble. They’ll each have a backlog, four or five developers, two or three QAs and they’ll run their sprints as a single team.
But what happens when the product is large and there are 25 developers and 15 QAs working on it? And here’s the crucial bit, any developer can (within reason) work on any part of the product. Teams aren’t divided functionally, nor are they broken up into front end, services, and DB teams, or any other arbitrary division of responsibility.
This pretty much describes my team, and before you denigrate that arrangement there are good reasons for it;
- You can bring more developers to bear on problem areas and new features and bring them to market sooner. i.e. you’re more agile.
- Product knowledge is spread throughout the whole group as opposed to being siloed in functional teams.
- Developers are more mobile within a team allowing them to find their own niche. And happy developers are productive developers.
- The team is more efficient. You don’t have one team cruising with an empty backlog doing priority 3 or 4 items while another is under the gun.
- A single backlog.
Now there are downsides as well, I won’t go into that now. however, with regards to scrum I was fairly sure that a single sprint team with forty members wasn’t going to work.
The solution was to break the team up into six smaller sprint teams with 4 or so developers and two or three QAs each. How we chose the makeup of those teams I’ll leave to a later post. In fact there are many aspects of this arrangement that warrant later posts. The immediate issue we had to solve however was scheduling.
Put simply, with a single product and a single backlog and a single product owner, no two sprint teams could have a planning meeting or a handover simultaneously.
Remember that we have six sprint teams in this scenario, and that a planning meeting is usually a half day event. Doing the math on a standard two week sprint that meant that the only time the product owner would have outside of the sprint process was Monday morning and Friday afternoon.
This is just one example of where slavish application to doctrine fails in the enterprise. Two week sprints are too short. So we made the sprints three weeks long. That meant that by staggering the start of the six teams, we could arrange two planning meetings and two handovers each week.
The reality is that most scrum advice says to start with 14 days, or 30 days and adjust the sprints to whatever works for your team and product. But the reality of our situation was that three week sprints were the only option we had if we wanted to keep a sustainable level of load on management.
So, where other teams might have stretched their sprint cycles out to deal with large or complex backlog items inherent to their product, we had to stick with three weeks. The reality was that often we would admit that some items were just not possible to implement in three weeks. Or, the effort to break them down into chunks that could be consumed by the teams in three weeks was not warranted or simply artificial.
So sometimes we broke the scrum rules about having artifacts of one sprint carry forward into another and having no concrete deliverables from the first sprint. I’m sure everyone in the enterprise has done it at some point. And as long as you can arrange things so that you don’t branch for release between the first and second sprint, where’s the harm.
Of course, when your fresh-out-of-certification-training scrum-master sees this happening you’ll catch an earful, but only the first time. Once you’ve broken another half dozen or so rules this one won’t even be a bump in the road.
Now, would I use that arrangement again? Only if the product can’t be broken along functional lines. It’s not easy. But if you have a big homogenous product and are short on staff it’s worth considering.
If you take this approach you need to consider the following;
- Scheduling is really important. You need to have your project managers create a schedule for your sprints and teams need to stick to it.
- Stagger the start of the sprints, and account for a dead couple of weeks at the beginning, and that you will have sprints still running when you hit milestones.
- If at all possible avoid Mondays for planning or handover meetings. Only because a public holiday will cause a reschedule, and this will invariable clash with one of the other teams and there will be flow on effects.
- Keep the backlog well groomed and the sizing estimates there normalised and reviewed. You will have enpough trouble tracking velocity metrics for all the teams without the additional complication of adjusting for backlog variance. (This is good advice for any backlog really).
- You will be have several backlog items consumed per week, so you need to keep ahead. Remember there are two planning meetings per week.
- Keep track of runs of related backlog items. If you have the runway on the project plan it’s better to have the same team pickup related items. It’s good to give teams choice and let them self organise the work, but that still needs oversight.