Why does new product/feature development take so long?

This is a post for non-technical readers (particularly non-technical product and high level feature owners) to explain why technology is "so slow" to deliver your new products and features.

Comparative baseline.  You need to ask yourself, "Why do I think something is taking too long?" What's my baseline, what am I comparing to when I think "slow"?  More often than not, I find that people are just displacing their general frustration in not having something they want (much like a child does), and take it out on the deliverer (the parent).

That, or you're just the type of person that moans about most things generally, so please stop.

I find that some stakeholders either delight in or don't realize they're only selectively comparing one organization to another.  Why can Company X deliver a new feature in a week when it takes our tech organization 3 months?  They rarely seek to understand why, they just want to selectively pick comparison points to criticize the delivery team.

Of course what they may not seek to understand is that Company X:

• May have more and better technologists (perhaps at a higher cost base)

• Invested in a technology delivery system that is much more efficient to extend, scale, and maintain (i.e., relatively less technical debt accumulated)

• Enables a significantly different approach to IT due to a different revenue and cost structure (e.g., can afford better kit, replaced more often, serviced by better more expensive support channels)

True switching costs.  Perhaps you're the person that was frustrated with the rate of in-house delivery, and sought out a big supplier to deliver what you're looking for.  Hey, cheaper right?  And you got to teach those in-house slackers a lesson.  You get bigger economies of scale, and a lot more people to deploy to build your solutions.  But then it turns out your knowledge domain is new to the supplier.  Congratulations!  You just paid for the privilege of bringing a bunch of new and external people up to speed that didn't know your domain from a hole in the ground.  And then the supplier takes that knowledge out to other customers, finally creating that economy of scale you were sold on in the first place (Supplier: "Hey, thanks very much for giving me a new area to expand my business and dilute your priorities and control!").

A really good technologist or really thorough domain knowledge, especially both together, is rarely a commodity.  A "Java programmer" is a commodity.  A "really good Java programmer who understands our customer's requirements and has several year's experience developing our products within our business culture" is not.

You're a "get what you want", "bulldog", "win-lose" negotiator.  You might be the type of stakeholder that always demands things be done more quickly than what's been presented to you.  You might think that technical trade-off discussions are just technobabble to justify a "heavily padded", "low risk", "plenty of surfing time" schedule.  Perhaps you think your a tough negotiator and your stripping out the "fluff" the technologists inserted into the plan to save the shareholder's money.  Either way, that means you think you understand better how long things should take as compared to one or more people that just spent days or weeks thinking about the problem.

Unfortunately, you really need to understand and accept four laws of software development physics:

• Good technologists rarely over-estimate.  Most either want to please you by giving you an aggressive schedule, they think their team is better than they actually are (they take their own personal estimations and extrapolate it for their entire team, one aspect of the Dunning-Kruger Effect), and/or don't think about whole solution delivery timings.

• A qualified and professional team of people who spend time thinking about a solution will most likely know more about how long it will take to construct the solution than you do

• Technical debt can be ratcheted up and quality down to deliver speed

• Assuming the technology team is reasonably competent, dedicated, and professional, the only way to reduce the schedule is to alter some other project dimension

Here's the thing, technology teams can sometimes seemingly magically strip time out of a schedule.  However, if you take the time to understand the trade-offs, it's not magic.  Technologists can generally remove quality, stability, best practices and "this is how it should be done" dimensions from a project and deliver more quickly.  However, by doing so they're increasing technical debt and/or business risk.

As the stakeholder, you'd be wise to track technical debt just like you track project budgets.  Unless of course you plan to hand the technical debt over to the next management team and move on to another project.  Say, aren't you clever getting that big bonus for an on-time delivery.  Too bad the new administration didn't know about all that debt they're acquiring...

Servicing the debt.  If you don't manage the product and technical debt, if you don't seek to understand the tradeoffs that are being made when you pressurize delivery schedules, things will start to move even slower than they did before.  It will appear to you that your team is getting "worse" over time, their deliveries slower.

This won't initially make sense to you if you don't understand the trade-offs that have been made.  The team's technical, domain, and cultural knowledge should be getting better and better so why is everything taking longer now to deliver?  When the technologists try to explain why, it's very complicated.  In fact, it sounds like the same old technobabble they were trying to use to con you into a heavily padded delivery schedule.  Probably best to continue to ignore it like you've always done.  Your approach has served you and your delivery schedule well so far.  The team is probably just getting burned out and it might be time to switch suppliers!  And a new job has opened up elsewhere you'd be perfect for, let someone else struggle with this declining, unmotivated delivery team...

Excuses, Excuses

Let's face it, some individuals and teams are really really bad at what they're supposed to be doing.  Maybe, just maybe, you have been an attentive, detail oriented, engaged stakeholder, you do understand the tradeoffs that have been made, and the team you work with simply isn't delivering.  One or several of the following may have happened.

First, maybe the team really is bad (as compared to other similar teams).  How can that be?

• Unaligned expectations - you really can't teach a pig to sing; you've hired someone that while good in their own right, can't possibly be good at what you've hired them to do
• Burnout - people really have burned out (guess what, managing burnout is just another technical debt to manage)
• Bad hiring - some people really are just incompetent and/or lazy; these types also tend to be good at lying as well
• People change - they've just tuned out, perhaps due to personal issues; maybe their worked well in your business' previous culture and context a few years ago but don't in the current one
• Bad alignment - maybe someone else in the business is poaching their time
• Different priorities - they're delivering just enough not to get fired while they work on their own business or day trade
• Poor management and leadership - people don't know what to do and/or aren't enabled to do it; their manager simply isn't managing, the organization isn't enabling them
• Sewing seeds of discontent - a few people are really just negative, nasty, and unpleasant; they spread FUD to create discontent in the team and then take pleasure in the results

Second, I have to admit it, a lot of good technologists put architecture, future-proofing, tools, process efficiency, frameworks, scalability, extensibility, and maintainability against actually delivering a single feature.  They want to build a double-super-awesome application to dominate all other applications, and it takes a lot of architectural work to do that.  The key here is a technology leader that pushes for incremental improvement and delivery along all dimensions (product and feature delivery first and foremost) and makes technical debt levels transparent to stakeholders.

Third, you really have been screwed by a delivery team or supplier.  They're farming their alleged 100% allocated team out to three different customers like you.  Your SLAs have no teeth.  You've been sold using Ruby on Rails for development and even now a team of 30 people in several other countries are reading Ruby for Dummies and wondering why they've been hired to evaluate precious gems and lay railroad tracks but not write software.  50% of the time and budget are gone and you're too heavily invested to change.

Perhaps you really are in one or more of these situations.  If you are, you're probably justified in making drastic changes.  However, you have to ask yourself how things got that way, and what role you played in it.  Because if you're the one that created the bad situation in the first place, are you really qualified to fix or replace it?  Do yourself and your shareholders a favor and get some help.

What's the right way?

So what is the formula to speed up delivery?  Assuming that you prioritize speed over cost (see The Trinity Extended for more on this):

• Acquire a good technology leader.   Find someone with credentials and references you trust, and then let them get on with it.  If you made a bad choice, then really, that's your or maybe your Boss' fault, isn't it?

• Acquire good technologists.  Unfortunately, they're rarely cheap, because they know what they're worth.  Also unfortunately there are a bunch out there that will take advantage of you so you can also easily end up not getting what you paid for.

• Domain knowledge, and even better, interest.  Acquire technologists that understand what you want built.  They should "get it" when you give a high level overview on what you want.  Ideally, you'll find people that have an actual personal interest in what you want done.  They want to build software that they would use themselves.

• Delivered something similar.  Acquire technologists that have a proven track record delivering something similar to what you want to do.  It will certainly help with scheduling and anticipating risks.

• Cultural alignment.  Acquire people that live and breathe within your target market.  It's much more likely they'll have an implicit understanding of your customers and what's required right at the start.  Also hire people that have worked in companies like yours:
• Start-ups vs big companies
• New development versus support and extend existing, inherited, purchased products
• New development versus integration and middleware
• Internal versus external customer facing
• b2b versus b2c
• Dedicated resourcing (generally apolitical) versus programme "beg-borrow-steal" persuasion/shared model of resourcing (generally political)
• Static (not much change, driving down costs) versus dynamic (fast moving market, environment; lots of change)
• Reactive (operationally oriented; opportunity led) versus Proactive (project oriented; strategy led)

• Knowingly manage technical debt.  Sure, you can accelerate now and pay later.  Commercial realities may dictate this behavior.  Just make sure you knowingly stay aware of how much technical debt you're creating as you go.  Set budget, strategy, constraints, horizon expectations clearly and have your tech team explain where debt is being accumulated and likely impacts that will result.

There are many other and similar views on how to deliver fast and well.  The whole family of Agile, XP, Scrum, DSDM, FDD, Kanban and other similar methodologies all take views on this.  They are all interesting and worthwhile to understand and use as appropriate.  But to me most of them don't really emphasize enough the human component, the true difference a really excellent technology individual and/or team can make.  Instead if closely followed they tend to treat people equally, reward mediocrity, and put process ahead of people.  All technologists are definitely not created equal, and yes, they're not machines they're people.

If you are unable to effectively make a judgment about the situation (good for you, at least you recognize this in yourself), bring in an external IT consultant you trust or with a good reputation to perform an IT audit.  Have the consultant take a view on what is and isn't working.  And whatever you do, don't bring in a consultant who is a stealth sales implant for a large professional services arm at the same company.  Make it very clear that you would use someone else to do any follow-up remediation.

Conclusion

It is possible that you have a poor, slow speed delivery team.  Perhaps you do need to significantly alter how you deliver with new people, new suppliers.

However, before you do anything radical, do your business and shareholders a favor.  Sit back and think for a minute where the "slow" designation comes from and how objective it really is.  If the only common denominator between the in-house and out-sourced speed assessment is you, perhaps it's your judgment of "slow" may be flawed and/or that you really aren't managing your technical debt.  And if you're really not sure, get a project/IT audit done by a trusted resource to give you an outside view.

Highly Agile Product and Project Management for Small Teams

No rocket science here, just a quick entry on a very light way to manage a small and self-contained project team frequently receiving new and modified feature requests while building a new and unproven product.  Goals are minimal process, managing feature creep, and getting the new product into the hands of real customers.

Assume the team is composed of:

• There are several stakeholders acting as product manager, making feature requests and requiring a high level view of the project.  They are the work creators.
• One project manager coordinating everything.  The PM vets, routes, value-adds, chases up, and drives completion of work.
• Handful of technologists and suppliers the project manager is coordinating.  They get work done.

There are two main control documents: product backlog, project plan.

Product Backlog

The first is a product backlog, used to capture all the new ideas and feature requests.  The backlog should be kept in a wiki, google doc, or similar shared location.  For a small team and project the project manager is the chaperone of the backlog, but anyone can put new entries into it.

The backlog is the only way that new feature requests can enter the project.

Backlog entries have these attributes:

• Who requested
• When requested
• Enough detail so everyone roughly knows what is being requested (but no more, not at this point)

As you start adding in backlog requests, put them into a few simple groups:

• Required to "make the sale", acquire investment, unblock the team, or otherwise business critical in nature ("It's game over if we don't do this right now!")
• Required as part of the release currently being worked on ("We can't deliver the new release without this.")
• High want for the next release coming up after the current release ("I really want this in the current release, but I'll settle for the next release coming up."  "All our competition has this feature.")
• Futures, nice-to-have, everything else ("Cool idea, but we don't absolutely need it right now")

If people other than the PM add new entries, they should throw them into the bottom of the list but can note where they would like the request.  Let the PM move the request into the main backlog as part of their vetting and value add functions.

As backlog items are completed, edit them out of the backlog and into a completed list.  Note when they were completed and by whom.

Tactical Plan

The tactical plan is a simple document written by the PM for the stakeholders that covers:

• Two lists of tasks:
• Last week - what got done, who delivered the task, when done
• What tasks are on-going from last week, who is involved with each, a rough estimate to complete each, reasons for date changes (if any)
• List of (anticipated) free resources and when available
• Suggestion of next tasks to put on the tactical plan as taken from the backlog and from other sources
• List of dates and description for delivery aspirations, commitments and estimates.  Highlight changes from the last meeting
• List of roadblocks, issues, and risks
• List of budget/spend requests

Weekly Review

About once per week, the project manager updates a tactical plan for review with the stakeholders.

Before the meeting the PM has done their homework, getting fresh information from the rest of the team and suppliers.  The PM has taken a quick scan through the backlog, looking for fresh, urgent additions and is prepared to make recommendations of which should become new tasks.

The project manager sends out the updated backlog and tactical plan in advance of the meeting.

The stakeholders use the weekly meeting to make changes to the tactical plan.  For efficiency, unless it is an absolute crisis or a critical mistake is being made, the stakeholders agree to not change the tactical plan more frequently than weekly.

The PM and stakeholders also try to avoid creating a long list of partially done tasks, instead allowing tasks to run to completion and not interrupting them with a new task assignment unless the work is really of zero value.

The PM and stakeholders review the recommended promotions from the backlog to the tactical plan and agree on which items to add to the tactical plan in the next 1-2 weeks based on expected free resources.  As new backlog items are selected, makes sure expectations are aligned about what it means to be "done" for each item.

The PM and stakeholders review the roadblocks, issues, and risks to problems solve and make changes to the project and team.

Spend requests are reviewed and approved/denied.

The PM revises the tactical plan and backlog after the meeting is complete and circulates it.

Intentional Exclusions

Other items could be put into the project plan such as QA test coverage and bug lists, code stats, and operational concerns.  However, with a really small team and unproven product, these may not be of much value yet.

Summary

Keep things as simple and light as possible when managing a small team working on an unproven product.

Use the backlog to give the stakeholders a voice and to not clutter and thrash the tactical project plan.

Use a one week review cycle to balance flexibility with efficiency.

Project Management - the trinity extended

Project management fundamentals teach us that that project management was a trade-off between the trinity of:

• Schedule
• Scope (requirements)
• Resource (people, competency of people, and money)

The three dimensions are in equilibrium and each dimension is proportional (inversely or directly) to the other two.

Formulas like the Ideal Gas Law (PV=nRT) work well to simply communicate the basic relationship between pressure (P), volume (V), and temperature (T).  Very abstractly, project management fundamental values might look like the mathematical formula:

Schedule * Resource = Scope

For example, if Schedule is decreased, then Resource must increase (inversely proportional) and/or Scope must decrease (directly proportional).

Each of three values are only so elastic. To be a practical formula, none of the values can be extremely low (0) or infinite.

Customers/stakeholders may only control one or two of the above three. Not all three. If stakeholders try to control all three and the estimates for all three are reasonably correct, the project will fail.

For example, "scope creep" results when the scope is increased by stakeholders that don't want schedule or resources to increase from your original estimates. The formula indicates that this simply isn't possible and scope creep alone (without compensating by increasing Resource and/or Schedule) will cause project failure.  Note that increasing Resource can also be delivered as improved Resources (e.g., same headcount, higher quality but more expensive staff).

There are three other important dimensions. They are derivative from the above three, but important enough to warrant their own explicit treatment. They are:

• Quality
• Risk
• Technical "Balance"

Expanding the formula, it now looks like this:

Schedule * Resource * Risk * Technical Balance = Scope * Quality

(Note: I use "Technical Balance" rather than "Technical Debt" as I believe in at least some areas you can "future proof" or "over architect" to create a positive/credit balance that may or may not create yield at a later time.  This is akin to the agile tenant of "only build what is required" versus "I *know* we're going to need this in a year, and it's a lot more efficient/cheaper to build it now."  It is also similar to cash sitting in a bank account that you're not investing in a way to maximize yield.  Regardless, a tip-of-the-hat to Ward Cunningham and others on the concept of technical debt, it's brilliant.)

Examples of putting these derivative values into play:

• Decrease quality to reduce schedule (e.g., launch sooner and let the customer test for us; we only fix what the customers complain about)
• Create technical debt to reduce resources (e.g., don't bother with code/build tools, don't bother with reducing code duplication or code quality refactorings)
• Reducing the scope allows us to reduce the risk (e.g., of delivery) - the less to deliver, the lower the risk

The above 6 project management dimensions are about managing participant and customer expectations - making sure the trade-offs you are making in the project are transparent to the customer/stakeholder so they can make better decisions to guide the project.

You could use these 6 items in project status reports or at least during major review points with customers.

Update May 2010

There are emerging approaches in software development coming from the manufacturing world (Toyota) which claim to be tearing down the "time, quality, money - pick two" formula and re-writing the laws of engineering and hence software development.  These practices have been adopted into so-called "lean" practices.

Observations:

• The old schedule-scope-resource formula holds, but only if you don't include a time dimension.  Schedule-scope-resource continues to apply in a short period of time, but not in a long period (e.g., due to trading out scope/quality for few resources and tight schedule).  If you are delivering a single shot software, the formula works.  If you are going to live with that software for a long time, evolving and extending it, then you have to include.... time... in the formula.  The time horizon is a business decision.  You educate your business counterparts on technical debt levels, and let them decide.  
• Good non-tech business leaders responsible for a business that uses tech to power the business must develop an understanding of how to make these tradeoffs, no different than being able to assess balance and P&L sheets.  If they can't do this, they won't be effective leaders of a tech heavy business.  I have yet to work with a business leader that has a time horizon greater than a year, and it's probably closer to a quarter (especially with publicly traded companies: shareholder horizon = board horizon = CEO horizon, and so on).  At least in Internet Gambling, the business doesn't seem to prioritize "built to last" over "get me X right now!!".
• By including "technical balance" in the above formula, which captures changing technical debt due to scope/quality versus resource and schedule, there is an implicit temporal aspect to the formula.
• Some technologists prefer to root cause all defects, maximize learning as the business progresses, and keep a very firm, low-debt foundation under them.  However, the reality is that many early stage businesses have to make trade-offs every day between enabling future scalability versus doing whatever it takes that week to get a sale made or acquire a few more customers.  The alternative is the business runs out of money and its game over for everyone.
• Unless business/board leaders are extremely confident of the business proposition, they are not going to invest heavily in future-proofing the business.  This comes down to the business leaders making the right guesses and being in the right place at the right time (note here the debate between user centered design versus good/lucky business leadership - either is valid in the right circumstances, but bring both together and wow).  Great business leaders being in the right place at the right time is a fairly unique combination.

The formula could have a stronger time basis added to it.  You can't (e.g.) inject money into the formula and instantly increase quality and scope.  It generally takes time for a change in one variable to ripple out to adjusting the other variables.  The same applies by gradually downsizing the team - things don't necessarily start failing immediately.  This is an inertia, latency, or lead time effect.  However, you can minimize the time required to (e.g.) inject new talent onto a team by prioritizing as a requirement the ongoing creation and maintenance of tools, processes and techniques that facilitate insertion/deletion of talent onto a team.

The time basis could also be included from the perspective of the time-value of the scope being delivered.  From a competitive viewpoint, it's likely that having the output right now is of highest value, with scope value decreasing as schedule increases.  This is no different than building a new building - once the decision to build is made, every day the building isn't earning money is lost revenue.  The faster you try to build it, the higher the costs - there is a cost/revenue sweet spot here, but difficult to identify in advance.

Another huge variation is that not all resources are created equal.  Some resources can even create a net productivity loss.  Similarly, as resources mature in a team and learn the codebase/technologies, their efficiency increases.  Therefore resource effectiveness becomes it's own formula that replaces the simpler "resources" in the above main formula.

The formula also doesn't fully capture project "success".  Success comes from successful project execution (formula captures this well enough), but not whether the resultant delivery is successful (e.g, with a customer, in a market, creating positive revenue).  A successful delivery is half or less of the battle.  Creating a product valued by customers resulting in healthy and sustainable net profits is really the battle to be won.

How to keep IT aligned with the business

1. Stay in close mental alignment with your CEO and other key strategy and opinion leaders in the business (they are "the business"!). Written strategies and project plans are too cumbersome at this level (they are useful for some across and most down management into the organization). Talk frequently with them - where is their head at today and is IT supporting them as best as possible at that moment?

2. Keep those same opinion leaders engaged in IT priorities and deliverables. The IT leader should be able to prioritize with 95+% accuracy because of (1), but you still need their help and support. Keep discussions in their terms, not technical ones. Talk primarily about costs, benefits, and really material risks. For bigger projects, keep reminding them of commitments made between IT and the business in terms of upcoming benefits.

3. Strike a balance of short and long term investment with your opinion leaders. Some IT work only indirectly support the business over the longer term (typically "wiring and plumbing" infrastructure type projects). Visibly balance these with projects that help you "make the sale" or get short term revenue in the door.

4. Align speed of IT delivery with speed of the rest of your business and the market you play in. Speed costs - align views of time-to-market with budget, quality, risks, and aspirations. Kill projects that have missed the market - don't fiddle while Rome burns. If you can't deliver to market requirements or beat the competition, you and your business are in trouble.