p2p File Sharing - a Dropbox Killer?

I've been a big fan of Dropbox since it came out, even with the security ups and downs along the way.    They offer plenty of storage for free and it Just Works.  However, two recent announcements have got me thinking about how a competitor might go after Dropbox and other similar (mostly inferior!) products like Box.Net, Microsoft Skydrive and Google Drive.

Background - Some Recent Announcements

The first announcement was from a company called LogMeIn, and their new network storage and sync product Cubby.  Cubby's subtle twist versus Dropbox is that they offer p2p syncing called "DirectSync" (no "cloud" centralised data store required).  LogMeIn has been around for a long time and their product/service Hamachi has been a semi-tech way to set up a VPN between systems - basically a "Goto My PC" for a slightly more technical audience.  Here is the important bit: they have a long history as a p2p communications enabler (UDP hole punching, STUN, TURN, and related tricks) to make the p2p Hamachi service (and now Cubby) work in a behind NAT and firewalls.

The second announcement was by BitTorrent and their new Sync product.  BitTorrent is of course the eponymous creator of the bit torrent protocol, very successful at creating a broadly adopted way of sharing files.  It is also a p2p, no cloud required, network sync product.  BitTorrent, the underlying protocol, also has a very well established p2p communications enabler.

(Postnote: See also subsequent blog article A Closer Look at BitTorrent's SyncApp for related information.)

Now, who is the biggest, well-known p2p service out there, one with a very effective p2p communications enabler scheme, plenty of money and technical capability behind them?  Skype of course, with Microsoft's deep pockets and Skydrive capability.

Dropbox: You've been warned.

Network storage and sync: Cloud vs p2p

Of course, we don't really need a cloud service to sync directories and files between multiple devices.  rsync and boatloads of scripting around it has been been around for many years.  What Dropbox and others did was combine rsync, a cloud-based backup and control location, and It Just Works software with sufficient and simple usability to make it all work.

Skype has shown us that a p2p approach, with highly functional p2p communications enablement  to hook everything together, works just fine.  Articles on Skype discuss this very point of how they keep their infrastructure costs down by only setting up communications between two people (two devices), not actually handling the communications data itself unless they have to (setting aside group coms, super nodes and must-do coms relays to simplify a bit).  Skype wisely focused on value-added services like Skype Out to generate revenue.

BitTorrent has shown us that a flexible and lightweight p2p connection broker service can copy (sync!) massive amounts of data quickly.  And because p2p "cloud-based" components (mostly) only manage connection and data location coordination and don't manage the actual content like Dropbox does, they are light, inexpensive, and resilient (especially against content-rights litigators!).

What's the Minimum Viable Product feature set to push p2p sync over the line?

In addition to the fundamental idea of p2p sync, I think there are a few additional key features to create this new, hypothetical, revenue-generative p2p-based Dropbox killer:

1. Just Works

• Dropbox absolutely gets this one right
• With less reliance on the cloud, this should be even easier in a p2p world
• Stable (no Microsoft BSODs), fast
• Doesn't crush system performance by pigging all available disk I/O for indexing, CPU for sync calculations, Internet or local network I/O for sync transfers (if possible, sync via directly on a local LAN with no public Internet bandwidth required to shovel data between devices)
• At least the efficiency of rsync in the sync approach (e.g., block level not file level syncing)
• Works in corporate environments with strong firewalling (e.g., relay if necessary, uses typically unblocked outbound ports 80/443)

2. Pervasive - available on all high-use platforms and applications

• Desktop, Laptop - OS X, Windows, Linux
• Mobile - OS X, Android
• Application access: simply local filesystem use where possible (desktop/laptop) or an API for sandboxed environments like mobile

3. Secure

• All data sent over the network is encrypted, both to peers and to the cloud based service coordinator (e.g., public/private key)
• Service coordination:
• Authenticates you as a valid user, your account information, including billing information for billable value-add services
• Authorises your nodes you want to keep in sync
• Authenticates and authorises other's nodes you are sharing data with
• Knows nothing about your data (no relay function unless required - and even then only passes through encrypted information)
• All data is encrypted at rest in each participating node (see BoxCryptor - a big hole and opportunity in Dropbox's offering)

4. Usability

• BitTorrent really struggles here (indexing, trackers) but a viable connection coordinator (e..g, Skype, Cubby) would solve this problem.
• Tight filesystem integration (like Dropbox)
• Async recognition of folder/file changes
• Iconic representation of sync status in Finder/Explorer windows
• Clear, simple communication of sync status between devices within a client and potentially iconically:
• Which device(s) has the master/newest version of a file (think BitTorrent's Seed/Leech)?
• How synchronised is is a file, directory, or everything? (as a % complete)
• How "safe" is a file?
• # of master/complete copies (# of devices on which file fully exists)
• Are devices geographically distributed or co-located? (client asks OS for location info if available)
• Basic management of sync conflicts
• Visual notification of conflicts (Notification, Finder/Explorer iconics)
• Filesystem filename changes (as per Dropbox)
• Client logfile of existing, unreconciled conflicts with basic guidance on how to clear them
• Can flag and manage favourites and high-priority sync choices
• Camera/photo find and sync - I use this for all photo sources now

5. Sharing folders/files with others via syncing

• BitTorrent excels at this, but not in a secure way
• Use the cloud service coordinator to authenticate invited users; local node authorises access to authenticated users

What does this new p2p network storage and sync world look like?

Fast forward to a world where a company like BitTorrent Sync or LogMeIn Cubby has successfully deployed a working, free + billable services p2p sync product that has the required MVP features to compete with and beat Dropbox.

The user downloads/buys the software for each device they want to sync/copy their data between - just like Dropbox.  The client includes a simple dashboard indicating sync status and data "safety".  The client can administer all participating nodes and shared folders.  Client is a gateway to buying additional services.  User can see status of all devices, summary of content all on devices, when device was last sync'ed/seen.

Offer a user a "free" version of something that feels close to what Dropbox is today, but enables the synchronisation of an unlimited amount of data across any number of personal devices.  In fact, in general the more devices you store the data on, the faster sync happens and the "safer" the data is.

Why can't Dropbox and similar existing services do this today?

They could certainly do some of it, particularly around simplified usability.

However, Dropbox and most others have a legacy business model and investment in cloud storage, not p2p.  This will hold them back from p2p.

I'm not sure Skype can pull this off either.  Now that Skype is owned by Microsoft, one can guess the internal politics between Microsoft's Skydrive and the new Skype team will slow any progress to develop this to a crawl.  Besides, Microsoft rarely demonstrates Internet-first thinking.

What that means of course is that a new player like Cubby or BitTorrent Sync may be able to slip in.

How to make money in p2p

BitTorrent never made any money on p2p.  Just a big "thanks" from the tech-savvy Interent community for the protocol that has enabled fast and resilient file sharing for years.  However, I have a penchant for actually making money as well as technical elegance, so here are some ways that might happen with a p2p sync product.

Software sales.  Clients must be purchased for each platform you want to synchronise between.  I think users would generally accept a single shot, nominal software fee for client software purchases.  Certainly after beta and pre a mass-market ramp.  Each new device type is a new software purchase.  BoxCryptor and 1Password successfully use this model.

Services.  Much to the frustration of the media rights holders, p2p (a la BitTorrent) doesn't enable a path for them to make money.  They key of course is offering truly valuable services on top of the p2p service.

Cloud based service coordination.  I think a small fee per year could be charged after the first year for data relay, authentication and similar coordination/security services.  Certainly enough to cover related operational costs.  Looks like Cubby is doing this today.

The most obvious way to generate revenue is to create value-add services on top of cloud storage and data transfer to/from cloud storage.

A backup service is the most obvious value-add offering.

Basic backup service.  User's want to be confident that their precious documents and digital photographs are backed up to a safe and secure location.  Particularly if you build in having a separate geographic location being an important criteria to receive a "Your'e safe!" rating in the status dashboard for data redundancy.  From a security perspective, cloud based backups must be a "locked box" with only the encrypted format of the file backed up to the cloud and with only the user having the key to decrypt the files (client side de/encrypt) - not like Dropbox that store all your data in their cloud servers in an unencrypted format.

Versioning service.  Deliver as an extension to backup, perhaps like Apple's Time Capsule as a growing number of people understand the Time Machine concept.  Pay as you use for frequency and size of backups/versions.

High value local application data backup services.  Some of the below are structured data files meaning backup may not just be a simple copy.  Seek and suggest high value targets for sync and backup by offering a billable backup extension:
- Photos, iPhoto DB

- Contacts
- Calendar

- Email (local)
- Video
- 1Password DB
- BoxCryptor DB
- iTunes purchased music
- Purchased software
- ...

High value social/internet backup services.  Offer each as a billable backup extension.  A little like Facebook's timeline, but platform neutral.  Use APIs to pull out social media contributions and references:
- Email (hotmail, google, ...)
- Facebook
- Twitter
- LinkedIn
- Blogger
- ...

Web-based publication service.  Provide cloud storage and controls to Internet publish and share your content.

Specialised hardware.  Offer dedicated NAS devices that directly support the p2p sync protocol or license the protocol to NAS makers (BitTorrent Sync hits this point).  Offer from 1 to 5 disk chassis.  The sync client will automatically discover any new device connected to the local network and offer to configure it for you.  Usability will be critical.

Technical Challenges

I didn't say there aren't technical challenges here.  A few of them might be:

1. How does each node determine which file is the master among all nodes?  When to sync the newest versus flag a sync conflict and rename files.  Will have to develop a strategy around inaccurate system clocks.

2. Mobile versus desktop/laptop.  How do you manage limited space and CPU access on mobile devices versus always-connected and plenty-of-horsepower desktop/laptops?  Caching and sync prioritisation is tricky in a mixed node environment.

• User can explicitly mark high/low priority data (BitTorrent client "Transmission" has this feature) - always a high priority or just a high priority until everything in sync then back to normal priority
• User can explicitly mark favourites to imply an on-going high priority
• Automatically identify "hot" files through regular/frequent/recent use.  Files you are actively working on are implicitly prioritised for sync across all nodes.
• Amazon Kindle's approach to "cloud" vs "device" location of books is a good usability model to consider and will educate mainstream users on this model
• Always prioritise what user is asking for right now in the front of the sync work stream, ahead of what is pending (for other reasons) to be sync'ed.

Bits and pieces

There are a few other factors to consider when looking for Dropbox weaknesses.

Dropbox incurs a competitive disadvantage bourne of their very successful "share with a friend" referral model to build up membership - a whole bunch of freeloaders aren't paying for the service but still creating operational costs.  Of course, freeloaders don't cost Dropbox - the costs are borne by customers who pay for Dropbox services by paying somewhat more than the true cost of their service.  Assuming you don't sync media and just stick with documents, 5GB of free storage is room plus the refer-a-friend space bonuses can store for *a lot* of documents.

Unfortunately, Cubby appears to have copied the Dropbox business model instead of offering (e.g.) a limited duration trial and aggressive shutdown of freeloaders.  I would recommend that Cubby emphasise unlimited space free syncing for one year then charge for value-add services like data relaying and cloud backup.

Dropbox also has a higher cost by hosting their storage in AWS' S3.  They must be at a point where a dedicated in-house equivalent service with an S3 simulation wrapper around it would be cheaper.

Dropbox is still semi-technical in nature - further usability improvements are possible.

BitTorrent including their name in marketing their Sync product will be a mistake if mass-market usage is their goal.  BitTorrent file sharing has in part been inhibited from wide-spread public acceptance because of its association with illegal activity (media rights violation).  However, so long as I'm sharing my own (rightfully obtained) files… at least between my own devices… for my own use... there should be no violation.  There will have to a be a temptation with the BitTorrent approach of course to co-mingle media sharing with sync which will also inhibit mass-market acceptance.

The p2p approach to storage will take a long time to be adopted in corporate environments.  Just look at the struggle of Skype and Dropbox in the Enterprise continuing even today.

Conclusion

There is an emerging trend now to think "mobile first" in development.  Companies that are oriented toward browser based "traditional" Internet service consumption are at risk because a mobile first equivalent can come along and end-run them.  Similarly, p2p for network file storage and sync could easily become a disintermediating force for file sync and share services that currently think cloud first.  Is p2p sync a viable product in the "post cloud" world?  At least in this use case?

So who might pull this off?
- Although Skype *should* be the best horse to bet on, the Microsoft purchase, "Internet Last" thinking and internal politics may kill all hope
- I don't think BitTorrent Sync will be relevant to go mass market - too much baggage.

So that leaves LogMeIn Cubby can pull this off and steal market share from Dropbox and other "last year's tech" cloud storage and sync providers.  Of course, there are all the other MVP features above they need to get right as well, which is chancy.  Or perhaps some other startup or early/quiet competitor whose ramping up their operation right now…

(Postnote: A blog article specific to Bit Torrent's SyncApp was written a few days after this one.)

Flapping Tell-tales: Over-Management of Products and Priorities

If you've ever sailed with a bit of curiosity, you've learned about tell-tales.  Essentially they're the little flapping ribbons on a sail that help you know whether you're sail is working efficiently or not.  If they're flapping about all different directions, you're sail isn't doing much for you.

Now imagine one sailing boat with one steering wheel, and 15 people tugging the wheel different directions trying to get the tell-tales to settle down and make efficient use of sail and wind to to move you toward your destination.  Some of these people jostling about the helm are bigger than others, can shove the others aside and can really yank the wheel one direction.  Some just stand there in the way thinking about other wheels they need to stand around later in the day.  Others ask nicely for a go at the wheel and nudge it a bit one way or the other (n.b.: not to worry, per Darwin we'll evolve out the namby-pamby collaborative team players soon enough).

The same thing can happen in an internet gambling business with lots of opinionated non-technical product and other manager types trying to control the priorities of a small team of technology developers.  There is a lot of "flapping" and the sailing, if you're moving at all, is not in the right direction.

The Tell-tales

Here are common tell-tales of organizational brokenness around product and prioritization:

• Product managers are complaining that their "must have" "make or break" feature won't be delivered for months.  Sales execs are blaming missing targets on not having a good product to sell and instead are selling products that don't exist.  Instead of looking for work someplace where they can make sales and create new products, they plod along selling vapor and complaining about slow IT delivery.
• It's also taking a lot of IT management to keep track of and prioritize a rapidly increasing backlog.  There are 30 people across the business spinning out one liner requirements into a backlog maintained by 4 managers being worked on by 3 developers.  
• What's considered urgent one month in the backlog drops to no interest the next month.  Features, once delivered, hit the market "too late".  Product managers and sales execs are viewed by the technologists as being fickle, lacking follow-through and constantly changing their minds on what they want.  IT is blamed for not keeping up with changing market requirements.
• You have no bench capacity to chase emergent opportunities.  These opportunities pass you by.
• People comment that the "politics" in the company is increasing.  What uncomfortable behaviors are labeled as "political"?
• Conflicting priorities: there are more people to "drive" the organization than there are people to do the work.  The people defining the priorities have an inability to agree a common set of priorities and stick to those priorities once they walk out of the "alignment" meeting. The software development manager receives multiple conflicting requests and priorities from two or more people and is then in the position of deciding between them.
• Not sure who to follow: Lack of clear who-owns-what decision making structure.  Seemingly arbitrary and often passive-aggressive punishment for following the "wrong" person.  The "right" person is flavor of the month, then out the next month.
• If people complain about a lack of strategy and clear priorities, they're criticized for not having the "mental agility" to work in "ambiguous situations" and requiring everything to be "black and white" and spelled out for them.
• Managers are using persuasion, asking for "favors" and a "bit of extra work on the side".  What's worse is they're then enabled and applauded for working this way.
• People are careful to maintain plausible deniability and other CYA behaviors reducing overall business efficiency.
• No one is actively nurturing an environment of trust and indeed such activities are viewed as a waste of time.
• You're on a conference call with 15 people and realize:
• Of the 15 people, one is there to ask question, listen carefully, and assess what is going to make high quality decisions.  This is the big boss in the meeting.  Unfortunately, this one person tends to do most of the talking and when they're not talking, they're "listening" while reading and answering their boss' emails on their Blackberry.  This one person has the special ability to pay only marginal attention but then to hear certain keywords and then talk a lot about them.
• One of the 15 people actually does the work being discussed.
• One manager owns and directly manages the one worker.
• Five people are trying to influence the one person who owns resources to do work for them, each one at the loss of the other four.  These are product, project, and programme managers.  One is the flavor-of-the-month alpha dog while four are rummaging for scraps and taking notes for ammunition against the alpha dog in the future.  They alternate between threatening the one worker's manager and sucking up to the worker.
• Two of the people distrust someone working for them and want to attend the meeting as their employee doesn't communicate a useful meeting summary with them or doesn't appear to pay much attention or take notes.  The two people don't want the person working for them to embarrass the department or create more work with screw-ups.  They don't contribute much unless they think their employee is about to screw up or conducting damage control afterwards.
• Two people whose motivational imperatives haven't been crushed out of existence yet occasionally pipe in with genuinely helpful "cross-pollination", "how to do that" advice.
• Five people on the call think the worker should be grateful to have been brought to a "high level" meeting to receive so much wisdom.  The worker isn't one of the five.
• There are at least four people who are in other groups, only vaguely associated with this project, have been invited to make sure they're "aligned", and really don't care much about what is being said.  They're probably paying no attention and answering email, but no one will include them in the conversation anyway because no one is particularly sure why they're there.  These people will often write "lack of communications" as a systemic corporate problem during annual HR surveys.
• There are at least five "could you repeat that" requests during the call as people are caught out not paying attention.

What went wrong?

Fundamentally, the company has over-staffed Captains and purchased too few boats and crews.  Communications are inefficient and durable alignment is non-existant.  It's typically not the fault of the one worker involved (unless they aren't effective at the job they signed up for) - it's a leadership, organizational, and strategy failure.

Ok, so what went wrong?  These are the reasons I've come across:

• If you're a senior person in the organization, you understand senior people.  You have less of an understanding of more junior staff proportional to how far you're removed from them.  You hire a more senior employee because you relate to them better.  This is ok (in fact, preferred) if the person you hire is going to build a team under them or at least have control over enabling budget/resources.  If you find yourself stepping across and obviating an employee, you've created a problem.
• Don't hire people you aren't planning to empower
• (Humanely!) remove people that are no longer empowered
• General wisdom is that a senior person is more skilled, knows more, can get more done.  They can cover the more junior responsibilities if they need to.  
• This should apply to senior employees, but often doesn't.  They either can't or don't want to operate at a junior level and are more interested in bizdev, strategy and commercials.  They didn't enter the business with strong domain knowledge nor seem interested in developing it (why were they hired again?).
• Although the "hire senior emlpoyee" wisdom may apply in some business functions, it's often wildly wrong in technology.  The scope of knowledge in technology is enormous, changes rapidly, and can take a while to pick up.
• Regardless, as a result the senior employee owner can't work at depth, so hires more junior product owners and/or other managers step in to fill the gap.
• The product partitioning and ownership in the organizational strategy was wrong and the original strategy's advocates resist changing the strategy as they don't want to be seen having made a bad decision.
• Not many execs with a non-tech upbringing understand IT.  They don't understand how products are created and how long it takes to build IT and product organizations.  They think increasing sales, project, product, and programme managers are needed to "keep the IT team focused" when deliveries are late and of low quality.  They stay with what they know rather than drilling into detail they can't or don't want to understand.
• The product(s) involved are sold in different "flavors" to different categories of customers.  Each category will have it's own requirements and priorities.  There is a lack of overall priority and alignment between them.  The flavoring creates new late stage technical requirements.

Considerations

Some assumptions about the right way of to get your ship in order:

• Don't have the software development manager making strategic business decisions by arbitrating priorities.  Great that the manager is a little bit commercially aware and is sensitive to business and customer needs.  However, the more time they're building this knowledge, they're not building software.  Enable them to focus on software construction.
• Don't expect the product managers, often peers, to effectively communicate with each other and prioritize between each other.  They're typically measured on delivery of their product line, not playing nice with other product owners.  A consistent and simple approach to business cases and approval process will help.
• Don't have product managers that have to beg, borrow, steal resources.  They shouldn't have to rely on persuasion to acquire core enabling resources as it leads to politics, distrust, confusion and all the related inefficiencies highlighted above.
• Do assign one product manager to be the final decision maker accountable for overall product priorities.  
• Do have the software development team align their backlog priorities to the senior product manager's decisions.
• Do hire product managers that understand at least at a high level the underlying technology and can sensibly evaluate technical debt tradeoffs.  Hire product managers that are interested in how to most effectively use the technology and can appropriately prioritize non-functional technical requirements.

The Solution

Scale the number of people who want to own and drive product to people who can build those products. This balance is essential.  You don't want 15 people behind one helm on one tiny boat.

If you have a big software development team with several potentially unaligned product managers creating work and issuing priorities, use a resource allocation model so that each product manager has to justify and fund a set of resources for their product responsibilities within the software development team.  Preferably these are intact teams, like a scrum team.  Resource allocations should be based on business value and/or risk management.

Make sure the software development team and manager have their own internal set of resources that they can internally prioritize to take on overall architecture, integration, code reviews, refactoring, mentoring, tools, how to scale the code base and personnel, and other (mostly) non functional requirements.

What about maintaining bench capacity?  I have yet to work someplace where there is any bench capacity for very long - everyone is busy.  The danger with bench is the organization may degenerate into a beg-borrow-steal model to grab these resources.  Ideally try to keep some bench in the internal set of software development resources and balance technical debt versus emergent benefit.  It really is powerful in an organization to have a few highly competent innovation-minded resources available to jump on the new market opportunity that just popped up.

Periodically review the resource allocations.  Rebalance them  between project deliveries to match changing business requirements.  Also, commercial realities can change suddenly so you may need to re-allocate resources at short notice.  So long as quick changes happen as an exception and not a rule and the reasons for it are clear and consistent, no one is going to object.  Indeed it can be exciting and foster improved teamwork.

Identify and eliminate any point in the organization where product and team priorities, ownership and decision making is ambiguous or shared.

Consider carefully whether a project or programme manager should get between a product owner and the lead/manager of the software development team delivering to that product owner.  Why does this extra layer exist?  Who really understands what is in the backlog?  Sometimes this additional management layer is justified if the project/product is big, complicated and/or there are heavy customer-driven process/audit/document requirements.

Here are two basic rules of thumb on whether someone is needed in the management layer:

• If a team member forwards email to a peer more frequently than answering it or dealing with it, remove them from the team.  If they forward it to someone working for them without adding value, are they delegating appropriately?
• If a recurring meeting member has contributed nothing of value for more than a few meetings in a row, remove them from the meeting.  They can read the meeting minutes or have a hallway conversation to catch up.

Conclusion

Fundamentally, an organization will be more efficient if there is a good match between Captains, boats and crews.  It's not easy to get the balance right, and the need for all three will change frequently.

If you do have to error on one side or the other, error on the side of having more boats and crews.  Better to have a little bit of spare execution capacity that can be thrown at an emergent opportunity than to degenerate into politically charged, rudderless anarchy.

Why does new product/feature development take so long?

This is a post for non-technical readers (particularly non-technical product and high level feature owners) to explain why technology is "so slow" to deliver your new products and features.

Comparative baseline.  You need to ask yourself, "Why do I think something is taking too long?" What's my baseline, what am I comparing to when I think "slow"?  More often than not, I find that people are just displacing their general frustration in not having something they want (much like a child does), and take it out on the deliverer (the parent).

That, or you're just the type of person that moans about most things generally, so please stop.

I find that some stakeholders either delight in or don't realize they're only selectively comparing one organization to another.  Why can Company X deliver a new feature in a week when it takes our tech organization 3 months?  They rarely seek to understand why, they just want to selectively pick comparison points to criticize the delivery team.

Of course what they may not seek to understand is that Company X:

• May have more and better technologists (perhaps at a higher cost base)

• Invested in a technology delivery system that is much more efficient to extend, scale, and maintain (i.e., relatively less technical debt accumulated)

• Enables a significantly different approach to IT due to a different revenue and cost structure (e.g., can afford better kit, replaced more often, serviced by better more expensive support channels)

True switching costs.  Perhaps you're the person that was frustrated with the rate of in-house delivery, and sought out a big supplier to deliver what you're looking for.  Hey, cheaper right?  And you got to teach those in-house slackers a lesson.  You get bigger economies of scale, and a lot more people to deploy to build your solutions.  But then it turns out your knowledge domain is new to the supplier.  Congratulations!  You just paid for the privilege of bringing a bunch of new and external people up to speed that didn't know your domain from a hole in the ground.  And then the supplier takes that knowledge out to other customers, finally creating that economy of scale you were sold on in the first place (Supplier: "Hey, thanks very much for giving me a new area to expand my business and dilute your priorities and control!").

A really good technologist or really thorough domain knowledge, especially both together, is rarely a commodity.  A "Java programmer" is a commodity.  A "really good Java programmer who understands our customer's requirements and has several year's experience developing our products within our business culture" is not.

You're a "get what you want", "bulldog", "win-lose" negotiator.  You might be the type of stakeholder that always demands things be done more quickly than what's been presented to you.  You might think that technical trade-off discussions are just technobabble to justify a "heavily padded", "low risk", "plenty of surfing time" schedule.  Perhaps you think your a tough negotiator and your stripping out the "fluff" the technologists inserted into the plan to save the shareholder's money.  Either way, that means you think you understand better how long things should take as compared to one or more people that just spent days or weeks thinking about the problem.

Unfortunately, you really need to understand and accept four laws of software development physics:

• Good technologists rarely over-estimate.  Most either want to please you by giving you an aggressive schedule, they think their team is better than they actually are (they take their own personal estimations and extrapolate it for their entire team, one aspect of the Dunning-Kruger Effect), and/or don't think about whole solution delivery timings.

• A qualified and professional team of people who spend time thinking about a solution will most likely know more about how long it will take to construct the solution than you do

• Technical debt can be ratcheted up and quality down to deliver speed

• Assuming the technology team is reasonably competent, dedicated, and professional, the only way to reduce the schedule is to alter some other project dimension

Here's the thing, technology teams can sometimes seemingly magically strip time out of a schedule.  However, if you take the time to understand the trade-offs, it's not magic.  Technologists can generally remove quality, stability, best practices and "this is how it should be done" dimensions from a project and deliver more quickly.  However, by doing so they're increasing technical debt and/or business risk.

As the stakeholder, you'd be wise to track technical debt just like you track project budgets.  Unless of course you plan to hand the technical debt over to the next management team and move on to another project.  Say, aren't you clever getting that big bonus for an on-time delivery.  Too bad the new administration didn't know about all that debt they're acquiring...

Servicing the debt.  If you don't manage the product and technical debt, if you don't seek to understand the tradeoffs that are being made when you pressurize delivery schedules, things will start to move even slower than they did before.  It will appear to you that your team is getting "worse" over time, their deliveries slower.

This won't initially make sense to you if you don't understand the trade-offs that have been made.  The team's technical, domain, and cultural knowledge should be getting better and better so why is everything taking longer now to deliver?  When the technologists try to explain why, it's very complicated.  In fact, it sounds like the same old technobabble they were trying to use to con you into a heavily padded delivery schedule.  Probably best to continue to ignore it like you've always done.  Your approach has served you and your delivery schedule well so far.  The team is probably just getting burned out and it might be time to switch suppliers!  And a new job has opened up elsewhere you'd be perfect for, let someone else struggle with this declining, unmotivated delivery team...

Excuses, Excuses

Let's face it, some individuals and teams are really really bad at what they're supposed to be doing.  Maybe, just maybe, you have been an attentive, detail oriented, engaged stakeholder, you do understand the tradeoffs that have been made, and the team you work with simply isn't delivering.  One or several of the following may have happened.

First, maybe the team really is bad (as compared to other similar teams).  How can that be?

• Unaligned expectations - you really can't teach a pig to sing; you've hired someone that while good in their own right, can't possibly be good at what you've hired them to do
• Burnout - people really have burned out (guess what, managing burnout is just another technical debt to manage)
• Bad hiring - some people really are just incompetent and/or lazy; these types also tend to be good at lying as well
• People change - they've just tuned out, perhaps due to personal issues; maybe their worked well in your business' previous culture and context a few years ago but don't in the current one
• Bad alignment - maybe someone else in the business is poaching their time
• Different priorities - they're delivering just enough not to get fired while they work on their own business or day trade
• Poor management and leadership - people don't know what to do and/or aren't enabled to do it; their manager simply isn't managing, the organization isn't enabling them
• Sewing seeds of discontent - a few people are really just negative, nasty, and unpleasant; they spread FUD to create discontent in the team and then take pleasure in the results

Second, I have to admit it, a lot of good technologists put architecture, future-proofing, tools, process efficiency, frameworks, scalability, extensibility, and maintainability against actually delivering a single feature.  They want to build a double-super-awesome application to dominate all other applications, and it takes a lot of architectural work to do that.  The key here is a technology leader that pushes for incremental improvement and delivery along all dimensions (product and feature delivery first and foremost) and makes technical debt levels transparent to stakeholders.

Third, you really have been screwed by a delivery team or supplier.  They're farming their alleged 100% allocated team out to three different customers like you.  Your SLAs have no teeth.  You've been sold using Ruby on Rails for development and even now a team of 30 people in several other countries are reading Ruby for Dummies and wondering why they've been hired to evaluate precious gems and lay railroad tracks but not write software.  50% of the time and budget are gone and you're too heavily invested to change.

Perhaps you really are in one or more of these situations.  If you are, you're probably justified in making drastic changes.  However, you have to ask yourself how things got that way, and what role you played in it.  Because if you're the one that created the bad situation in the first place, are you really qualified to fix or replace it?  Do yourself and your shareholders a favor and get some help.

What's the right way?

So what is the formula to speed up delivery?  Assuming that you prioritize speed over cost (see The Trinity Extended for more on this):

• Acquire a good technology leader.   Find someone with credentials and references you trust, and then let them get on with it.  If you made a bad choice, then really, that's your or maybe your Boss' fault, isn't it?

• Acquire good technologists.  Unfortunately, they're rarely cheap, because they know what they're worth.  Also unfortunately there are a bunch out there that will take advantage of you so you can also easily end up not getting what you paid for.

• Domain knowledge, and even better, interest.  Acquire technologists that understand what you want built.  They should "get it" when you give a high level overview on what you want.  Ideally, you'll find people that have an actual personal interest in what you want done.  They want to build software that they would use themselves.

• Delivered something similar.  Acquire technologists that have a proven track record delivering something similar to what you want to do.  It will certainly help with scheduling and anticipating risks.

• Cultural alignment.  Acquire people that live and breathe within your target market.  It's much more likely they'll have an implicit understanding of your customers and what's required right at the start.  Also hire people that have worked in companies like yours:
• Start-ups vs big companies
• New development versus support and extend existing, inherited, purchased products
• New development versus integration and middleware
• Internal versus external customer facing
• b2b versus b2c
• Dedicated resourcing (generally apolitical) versus programme "beg-borrow-steal" persuasion/shared model of resourcing (generally political)
• Static (not much change, driving down costs) versus dynamic (fast moving market, environment; lots of change)
• Reactive (operationally oriented; opportunity led) versus Proactive (project oriented; strategy led)

• Knowingly manage technical debt.  Sure, you can accelerate now and pay later.  Commercial realities may dictate this behavior.  Just make sure you knowingly stay aware of how much technical debt you're creating as you go.  Set budget, strategy, constraints, horizon expectations clearly and have your tech team explain where debt is being accumulated and likely impacts that will result.

There are many other and similar views on how to deliver fast and well.  The whole family of Agile, XP, Scrum, DSDM, FDD, Kanban and other similar methodologies all take views on this.  They are all interesting and worthwhile to understand and use as appropriate.  But to me most of them don't really emphasize enough the human component, the true difference a really excellent technology individual and/or team can make.  Instead if closely followed they tend to treat people equally, reward mediocrity, and put process ahead of people.  All technologists are definitely not created equal, and yes, they're not machines they're people.

If you are unable to effectively make a judgment about the situation (good for you, at least you recognize this in yourself), bring in an external IT consultant you trust or with a good reputation to perform an IT audit.  Have the consultant take a view on what is and isn't working.  And whatever you do, don't bring in a consultant who is a stealth sales implant for a large professional services arm at the same company.  Make it very clear that you would use someone else to do any follow-up remediation.

Conclusion

It is possible that you have a poor, slow speed delivery team.  Perhaps you do need to significantly alter how you deliver with new people, new suppliers.

However, before you do anything radical, do your business and shareholders a favor.  Sit back and think for a minute where the "slow" designation comes from and how objective it really is.  If the only common denominator between the in-house and out-sourced speed assessment is you, perhaps it's your judgment of "slow" may be flawed and/or that you really aren't managing your technical debt.  And if you're really not sure, get a project/IT audit done by a trusted resource to give you an outside view.