Using MobileMe's iDisk as an interim backup while traveling

Introduction

I use an Apple laptop hard disk as my primary (master) data storage device.  To provide interim backups while traveling, I use Apple's MobileMe iDisk for network backups to supplement primary backups only available to me when I'm at home.

Having dabbled with iDisk for a few years, I have two key constraints for using iDisk:

• I don't always have a lot of bandwidth available (e.g., a mobile phone GPRS connection) and I don't want a frequent automatic sync to hog a limited connection.
• I don't trust MobileMe with primary ownership of data or files.  Several years ago I switched to using the iDisk Documents folder (with local cache) for primary storage but then had several files magically disappear.

I've now evolved to using iDisk as a secondary backup medium.  I manually run these steps when I have plenty of bandwidth available.  There are two steps to this:

• rsync files/folders from specific primary locations to a named directory under iDisk
• Sync the iDisk

How to do it

The rsync command I use looks like this:

for fn in Desktop dev Documents Sites; do

   du -sk "/Users/my_username/$fn" | tee -a ~/logs/laptop_name-idisk.rsync.log

   rsync -avE --stats --delete "/Users/my_username/$fn" "/Volumes/my_mobileme_name/laptop_name/Users/my_username" | tee -a ~/logs/laptop_name-idisk.rsync.log

done

The rsync flags in use:

-a         archive (-rlptgoD no -H)

           -r    recursive

           -l    copy symlinks as symlinks

           -p    preserve permissions

           -t    preserve times

           -g    preserve group

           -o    preserve owner

           -D    same as "--devices --specials" (preserve device and special files)

-v         verbose

-E         preserve extended attributes

--stats    detailed info on sync

--delete   remove destination files not in source

Explanation:

• I'm targeting specific locations that I want to backup that aren't overly big but tend to change frequently (in this case several folders from my home directory: Desktop, dev, Documents, Sites)
• A basic log is maintained, including the size of what is being backed up (the "du" command)
• I use rsync rather than copy because rsync is quite efficient - it generally only copies the differences, not the whole fileset.
• The naming approach on the iDisk allows me to keep a backup by laptop name allowing me to keep discrete backup collections over time.  My old laptop and backups sit beside my current laptop backups.
• The naming approach also means I don't use any of the default directories supplied by iDisk as I'm not confident that Apple won't monkey with them.
• ~/Library/Mail is a high change area but not backed up here (see below for why)

The rsync updates the local iDisk cache.  Once the rsync is complete (after the first rsync I find it takes less than 10 seconds for subsequent rsyncs), manually kick off an iDisk network sync (e.g., via a Finder window, clicking on the icon next to iDisk).

An additional benefit to having a network backup of my important files and folders is that I can view and/or edit these files from the web, iphone, or PC.  I find that being able to access email/IMAP from alternative locations is the most useful feature, but I have had minor benefit from accessing files as well when my laptop was unavailable or inconvenient to access (e.g., quick check of a contract term in the back of a taxi on an iphone).

Other Backups

I have two other forms of backups:

• Irregular use of Time Machine to a Time Capsule, typically once a week if my travel schedule permits.
• MobileMe's IMAP for all email filing (and IMAP generally for all email).

Basically, if I'm traveling, I rely on rsync/iDisk and IMAP for backups.  I also have the ability to recover a whole machine from a fairly recent Time Machine backup.

Success Story

In June 2009 I lost my laptop HDD on a return flight home after 2 weeks of travel.  I had a Time Machine backup from right before I'd left on travel, and occasional iDisk rsyncs while traveling.

Once I got home I found an older HDD of sufficient size and restored from the Time Machine image from the Time Capsule.  This gave me a system that was just over 2 weeks "behind".  Once IMAP synchronized my mailboxes, that only left a few documents missing that I'd created while traveling.  Luckily I'd run an rsync and iDisk right before my return flight, so once I'd restored those, I'd recovered everything I'd worked on over the two weeks of travel, only missing only some IMAP filing I'd done on the plane.

Weakness

The primary flaw in my approach is that you have to have the discipline to remember to manually kick off the rsync and iDisk sync after you've made changes you don't want to lose.  I certainly don't always remember to run it, nor do I always have a good Internet connection available to enable it.  However, I find that remembering sometimes is always better than not having any recent backup at all.

Alternative Approaches

An obvious alternative is to use the MobileMeBackup program that is preloaded onto your iDisk under the Software/Backup directory.  Using this tool, you should be able to perform a similar type of backup to what I've done here.  I've not tried it as it was considered buggy back when I first started using iDisk for network backups.  I'll likely eventually try this and may shift to it if it works.

A viable alternative approach is to carry around a portable external hard drive, and make Time Machine backups to it more frequently than you would otherwise do over the network via iDisk.  You could basically keep a complete system image relatively up-to-date if you do this.  More hassle, but lower risk and easier recovery if your primary HDD fails.  However, if you get your laptop bag and external HDD stolen, you'll be worse off.

While on holiday recently, I was clearing images off of camera SD card memory as it filled up.  I put these images both on the laptop HDD and an external HDD.  This protects me from laptop HDD failure, but wouldn't help if both the laptop and external HDD was stolen.

iDisk Comparison to DropBox

DropBox is a popular alternative to iDisk.  I find DropBox to be better at quickly and selectively sharing files, it has better cross-platform support (particularly with a basic Android client), and it's sync algorithm seems to work better than the iDisk equivalent.  You could certainly do everything described here with DropBox.

The downside with DropBox is having to pay $120 per year for 50GB of storage versus $60-100 per year ($60 on promotion, e.g., with a new Apple laptop; otherwise $100) for 20GB of storage with MobileMe.  I find 20GB to be plenty for IMAP, iDisk and photos providing I filter out big auto-generated emailed business reports (store on laptop disk not in IMAP), and only upload small edited sets of photos.  I'll probably exhaust the 20GB in 2-3 more years at my current pace, but I'd expect Apple to increase the minimum by the time I would otherwise be running out of space.

MobileMe is of course more than just iDisk, so if you use more of it's features, it increases in value relative to DropBox.

Both iDisk and DropBox are usable choices, the differences are not sufficiently material to strongly argue for one or the other.  I have seen iDisk improve over the last few years and I'd expect Apple to eventually catch up with DropBox.

Conclusion

While I'm not confident in using MobileMe's iDisk as a primary storage location, I have found it useful as a network backup.  Combined with normal backups using Time Machine and Time Capsule, it provides a high-confidence recovery from damaged or lost primary use laptops.

Using wget to ask jspwiki to re-index its search DB

For whatever reason, our installation of jspwiki (v2.8.2) decides to ignore or lose pages out of its index (hey, what do you want for free?!).  With our jspwiki hitting 2000 pages, search is the main tool to find pages. Unfortunately, I've taken to keeping my own links page to important pages just so I don't lose them as the search indexing seems to break regularly.  While a re-index solves the problem, but it requires going into the site, authenticating, and clicking a button - way too much work.

Here is a quicky to use wget to log in to jspwiki and force a re-indexing of pages:

# POST to log in and get login and session cookies
wget --verbose --save-cookies=cookie --keep-session-cookies --post-data="j_username=myuid&j_password=mypw&redirect=Main&submitlogin=Login" "http://wiki.mydomain.com/JSPWiki/Login.jsp" --output-document "MainPostLogin.html"

# POST to kick off reindexing using cookies
wget --verbose --load-cookies=cookie --post-data="tab-admin=core&tab-core=Search+manager&bean=com.ecyrd.jspwiki.ui.admin.beans.SearchManagerBean&searchmanagerbean-reload=Force+index+reload" --output-document "PostFromForceIndexReload.html"  "http://wiki.mydomain.com/JSPWiki/admin/Admin.jsp"


Tweak myuid, mypw, and wiki.mydomain.com in the above to have them be what you need.  Drop the output once you're comfortable it's working (I was saving it in the above to make sure I could see artifacts of being authenticated in the output).

Put the above into a cron'ed script and run it hourly.

Note that all versions of wget are not created equal as 1.10 didn't seem to work but 1.10.2 and 1.12 worked fine for the above.

QCon London 2010 - Miscellaneous Topics

There were no shortage of interesting topics at QCon London 2010.  Although I'm writing in some depth about a few of them due to personal interest and/or applicability to Internet gambling, there are many others I'll highlight here briefly.

Shared nothing architecture
- Each node of a system is stand-alone and shares nothing with other nodes
- Great horizontal scalability
- Shared databases, data stores, caches are constraining
- Great for stateless, single-shot request-response, and content oriented services; less so for multi-state transactional systems

Industry Consolidation driving big boys architectures
- Internet traits such as the network effect and rapid feedback loops accelerate consolidation on a single market-dominant (defacto monopoly) services (e.g., ebay, betfair)
- Big consolidated services require a big compute capacity.  Market convergence on a single supplier isn't possible if that supplier can't scale to meet demand.
- Big compute capacity requires a lot more thought on the "-ilities" (non-functional attributes) of service delivery.  Functionality becomes commodity.
- Web technologies are embracing "traditional" approaches to increase compute capacity: asynchronous message oriented design, greater attention to maximizing hardware
- Consolidation also means longer life of legacy software
- The CAP theorem (see below) is coming into play for big systems that want to be highly available and need to massively scale

Programmer Quality of Life wins over Abstraction and Separation
- XML is painful
- Co-locate configuration with code (annotations)
- Convention over configuration (even Java coming on board with apps like Roo on Spring)
- Repetitive coding requirements should be built in (no boilerplate or scaffolding) - aspects relatively for free
- (Where does that leave dependency specification?  Hello maven pom.xml my nemesis!)

HTML 5, CSS 3, and Javascript versus rich client interface technologies
- Native executables (Wintel binaries) used to be way ahead of the browser on usability and richness but HTML/CSS/JS continues to move the browser experience closer to native executable experience
- Major new browser advances are right around the corner
- Flash, Air, Silverlight - great interfaces but browser continues to advance

- Mobile causing a renaissance of RIA and native executables - but browser continues to advance
- Innovation areas will tend to use an RIA and then the browser will catch up
- High touch experience (e.g., game graphics with high performance requirements) will require native executable performance for some time to come
- For most enterprise and business requirements, the browser experience is already sufficient today

Power efficiency, carbon credits and trading
- Assuming carbon trading advances, we might see a day where well written (more efficient, less energy consumptive) applications are important again
- Energy efficient HW (e.g., Sparc v Intel) may be more valued
- Some odd things may happen such as shifting compute capacity (carbon emission) to third world "carbon dumping grounds" due to economic incentives

Right tool for the right job versus efficiency from limited technology choices
- Although Java is dominant in the enterprise, Ruby is making inroads.  Recognition of productivity boost of a pleasant coding environment that encourages DRY and good programming techniques.
- Functional languages that facilite multi-core (parallel) computing are increasing in popularity as currently popular languages in the enterprise do not (Java!)

- Advent of language neutral information passing protocols to better enable innovation within components (but not forcing between components)
- As of today, homogeneous technology choices for the enterprise are still winning

Software Developer can "do it all"
- Moving test into development through TDD (and from unit to functional and some end-to-end)
- Cloud services abstracting operational systems (the specific HW and OS don't matter)
- Moving live deployment into development (Continuous Integration leading to Continuous Deployment)
- Better to use a shared nothing architecture under developer's control than reliance on specialty approaches like a cache in BigIP F5s or a shared in-memory cache

And a grab bag of others:

• OSGi and Java.  JARs lack versioning and dependency declarations and therefore lack safe coupling.  OSGi defines bundles to make integration/upgrade safer.  Feels complicated versus using a Convention over Configuration approach.  Could we use co-located annotations in the code instead to describe dependencies?  What about dependencies outside a specific application/JVM?
• SOA (Service Oriented Architecture) is dead, long live SOA!  (No one seems to like SOA but a lot of practices from SOA are in prevalent and growing use)
• TDD (Test Driven Development) is pretty much assumed now even for the smallest teams and projects.  CI in varying states but clearly the next development practice that will be an assumption shortly
• Log everything (Google, Facebook) - both customer actions and internal systems and be able to compare anything to anything
• CPU clocks hitting speed limits.  Until some new as yet unidentified technology breakthrough, CPU clock speeds have hit about as fast as they're going to be.  From now forward it will be about parallel processing on a growing number of cores.
• DDD (Domain Driven Design) - Design software with the interests of specific stakeholder's interests at heart, using the stakeholder's terms ("Ubiquitous Language") Let stakeholder interest area ("Bounded Context") warp a "perfect" implementation to one that is tailored to the stakeholder's needs.  In a complex system, identify the Domains of interest, and design around each of them in parallel with figuring out how to glue together these Domains.
• CAP theorem - pick 2: Consistency, Availability, and Partition tolerance (CAP).  Business will generally pick Availability and Partition tolerance, so that leaves Consistency as the odd man out and implies that more attention is then needed on identifying and recovering from inconsistent states.  Eventual consistency for some functions is sufficient.
• New persistance models - Social networks with their many-to-many relationships in the data are driving the use of new persistance models to supplement their relational databases
• Dreyfus model of skill acquisition - a good way to take a view on how people pick up skills and as a way to assess how skilled/mature your staff actually is

QCon London 2010 - Cloud Computing

Cloud computing and virtualization was a popular topic at QCon London 2010.

Background/primer/proposition:

• Cloud marketing suggests that hardware and/or systems administration is now a commodity that you shouldn't have to think about too much and can safely outsource. 
• Just like TDD (Test Driven Development) decreases the need for QA, CI (Continuous Integration) with direct deployments into an operational environment will decrease the need for systems administration.
• Outsourced pay-as-you-use cloud propositions will likely cause costs to switch from capex to opex to budget for computing capacity (was traditionally HW and SW in capex)
• Grossly simplifying, there are four interesting cloud propositions available:
• In-house hardware virtualization - cloud under your control, in your data centre (e.g., VMware, Xen, Solaris Zones)
• Outsourced hardware virtualization (IaaS - Infrastructure as a Service) - cloud as an "infinite capacity" of generic computing and you define the systems from the OS up (e.g., Amazon's AWS EC3)
• Outsource compute capacity (PaaS - Platform as a Service) - cloud as a place to deploy software components into a fairly tightly defined (constrained) operating environment (e.g., Google's App Engine)
• Pure services (SaaS - Software as a Service) - cloud as a source of "commoditized" services to be used when you construct an application (e.g., Google's web analytics, Facebook OAuth API for user credential management, AWS's S3 for storage)
• Cloud means that you can cost effectively create and delete computing resources as needed for parts of your IT environment that don't require regular use.  For example testing and in particular load testing.
• Non-tech business types get excited by cloud because:
• If your an entrepreneur type, you get bonus points for running your infrastructure from the cloud when looking for funding (more-so in the last two years, this is declining some now)
• Finance and P&L owners get exited any time they can commoditize something to drive down costs.  Tech has mixed feels about this as "drive down costs" tends to imply redundancies.
• Easier to justify upfront costs for a new business case if you only pay for what you use (a failure is easy to delete, no sunk capex expenditures)
• Both tech and non-tech types get excited about not having to generate a lot of paperwork then wait for authorizations and shipping times to get new kit.  Assuming company bureaucracy doesn't shackle down cloud controls too vigorously, a new virtual platform can made available very quickly and at low costs.
• If you can maximize utilization of HW you buy, then it's no different than buying cloud resources (likely cheaper)

General Observations on Cloud and Virtualization

Virtualization enables us to achieve that solutions architecture ideal of "one box one purpose", it just that it's become "one virtual box one purpose".

Virtualization enables us to take applications that don't have a good threading model to take advantage of boxes with many cores and use up all the cores (application per VM; VMs added until all cores are utilized)

Cloud does imply a lack of control over your core infrastructure.  Do you need this control?

The cloud is still just a bunch of hardware systems in a data centre.  There is no magic.  Their DC and systems admins will have their share of problems as well.  If the cloud sysadmins can provide more uptime than your own techops can provide at a similar cost point, the argument for cloud increases.

Similarly, there is debate over how good the SLAs are for cloud.  But really, how enforceable are the SLAs you have anyway?

Your choice of virtualization or cloud will enforce a way of creating applications and handling services.  You may not like it.  Conversely, it may force you to be disciplined in a new way otherwise missing when you create applications.

You will make an investment to learn the systems and make your applications work in the cloud environment.  This will cost and create some lock-in.  This is more true for PaaS than IaaS.

The cloud is being used to "long tail" a number of services.  Service "particles" are appearing you can use to provide an aspect of functionality in your overall solution.  The more of these partners you use that are in the same cloud with you, the greater the efficiencies and hence lower costs.  Combined with first mover advantage and vendor lock-ins, this is a network effect that should drive toward having just a few cloud suppliers in a few years.

Relating Cloud to Internet Gambling Business

The use of an in-house cloud like VMWare makes good sense.  We're regularly adding in new products that need to undergo development and test yet we don't need permanent capacity to service these requirements.  While a VMWare setup can't fully proxy a production environment (unless you use VMWare in production as well), it is very suitable for most types of functional verification other than load and low level device compatibility.

Being able to hand the keys over to a set of virtualized servers enables more entrepreneurial behavior.  For example, if you have a larger business that has a heavy layer of process, you can still work effectively with start-up partners.  Give them the keys to their own set of systems and they can do whatever they want with them without impacting your core systems.  At which time they're proven successful, their revenue stream can justify improved risk management.

Handling flash crowds with cloud probably isn't possible for our industry today.  In-house clouds don't really handle flash crowds (Why not just have the capacity there anyway? What do you want to cripple to support that big marketing campaign?).  Outsourced cloud generally isn't possible as the bigger cloud providers may not allow internet gambling to be run within their clouds (AWS restriction anyway; and yes, this will likely ease up at some point, just look at Akamai's behavior on Internet Gambling).  Also a CDN (Content Distribution network; an SaaS of a sorts) will take care of a lot of the flash crowd load we experience.

Using an outsourced cloud PaaS for data analytics doesn't seem likely.  Data analytics crunching benefits from close proximity to the data set being crunched.  Bandwidth to upload big data sets into the cloud from higher connectivity costs locations (lots of internet gambling in offshore locations with expensive ISP costs) doesn't make sense.

SaaS however is quite interesting.  Services like Google Analytics that enable almost real-time data analysis are clearly the way to go for an Internet gambling site.  Highly bespoke business analytics will likely stay inside the business or use a SaaS for commodity analytics.  

Depending on who you ask, the following may be real risks or just FUD:

• Taxation - as services are sourced from someplace other than the tax advantaged place you have your business in, you are at risk of emerging taxation implications
• Centralized point for governments to enforce legal compliance.  By hosting in the cloud (which is actually going to be one or more physical data centres), you've given the governments that have oversight of those data centres a good choke point to use against you.  They could use taxation, inappropriate content, or services not in compliance with regulation.

Conclusion

Virtualization makes complete sense for Internet gambling companies, all the way from development through to production.  That's not news, most in our sector have been using virtualization for a few years now.

On Cloud/IaaS provisions, AWS (a clear IaaS market leader) have flatly disallowed any internet gambling related operations inside their service.  While it is likely you could get away with internal use (dev, test) of cloud in these services, do you want to create a dependency and then have it suddenly shut off on you?  AWS of course isn't the only show in town for IaaS  There are other providers -  you would have to evaluate them versus related risk factors and re-development costs to integrate their use into your environment.

There is no clear use yet of Cloud/PaaS for standard Internet gambling products.

There are plenty of emergent opportunities to use Cloud/SaaS for Internet gambling.

(Index of emergent technologies applied to Internet Gambling)

QCon London 2010 - Themes and Trends

Last week I had the good fortune to attend QCon London which bills itself as an "enterprise software development conference".

I thought the conference struck a good balance between maybe 40% academic/futures/ideas from the ivory towers versus 60% practical, grunty software development from the trenches.  That of course varied by what sessions you attended as there were various tracks and tutorials available.

QCon was fairly software development centric.  Although there were tracks on technical operations and QA, both felt more like "what software development thinks how techops and QA should work" versus hardcore QA and techops experts running the tracks and presenting.

Although billed as "enterprise" software development, QCon was new media centric.  Less about enterprise and more about entrepreneurship using (recently) new tools and techniques to deliver and manage software.  I found this quite suitable for igaming that is still more entrepreneur land than it is enterprise.

The following are themes and trends that were in the air at Qcon that captured my interest.  Some will be old and familiar (yet receiving continued attention), others are relatively emergent in the last year or so.  Each item may eventually lead to a blog entry with detailed commentary on the subject and as relevant a view on how they apply to internet gambling systems.

• Cloud Computing
• NoSQL versus Relational Databases
• RESTful architecture
• Functional programming languages
• Post Scrum
• Mobile Computing
• Event based architectures, asynchronous messaging
• DevOps, particularly Continuous Deployment
• Miscellaneous topics

Internet Gambling Jobs in Gibraltar

Gibraltar has plenty of on-line gambling companies and there is almost always some form of related recruitment going on.

Whether you want to make a fresh start in Southern Spain or just got made redundant, the following are some good starting points if you're interested in working in Gib.  While I've got an IT bias, none of the companies below specialize only in IT.

(Please note I'm not affiliated with any of the companies listed below although I've talked with all of them over the years.)

Gibraltar Local Recruiters

Quad has been in Gibraltar for a long time now (at least in dog years), lists Gibraltar online gambling jobs, and has plenty of information about Gib and Spain on their site.

Ambient has been around for a fair amount of time as well.  They are not Gibraltar based, but close enough (up the Costa).

SRG has just opened up their office next door in Europort in Gib.  Their website covers some basics about Gib like living in Gib/Spain and local Income Tax.

Other Recruiters Operating into Gibraltar

There are plenty of other recruiting companies outside of Gib (typically UK) that operate into Gib.  They come and go, and the recruiters themselves change over time.

There are also a variety of headhunters that typically work other sectors that come and go.

There are two companies that have been around for a long time that have done plenty of work for Gib based companies:  BettingJobs.com and Pentasia.  Both of these companies place world-wide but you can find Gibraltar jobs on their sites as well.

Other Job Sources for Gibraltar

It's traditional (but not cheap!) to post jobs in The Gibraltar Chronicle newspaper on Fridays.  Yes, this is an actual paper newspaper, just like the ones Grandpa used to read.  They don't cross-post jobs to their website.

I keep an eye on jobserve using category IT and keyword "Gibraltar" to create an RSS feed to see what my IT colleagues around Gib are up to.

The GRA (Gibraltar Regulatory Authority) thoughtfully provides a summary page of all "remote gambling" operators with Gib licenses.

Wildcards

I've not personally worked with the following, but they at least had a few listings or Gib or along the Costa.  YMMV.

gibraltarportal.com lists a few local jobs.

The surinenglish.com delegates their recruitment to myservicesdirectory.com, it's very Spain oriented, not too much interesting on the Gib and IT side.

I'm not familiar with Andalucia Technology Recruitment.  I've not seen anything for Gib on their site, but they do have a few IT roles along the Costa.

Bits and Pieces

There is an Excel sheet you can download from the Gib government site to calculate your potential income tax.  With the same salary, you'll typically be better off in Gib than most other European countries.

Other starting points

EGR has published a short list of nominees for their 2010 igaming awards, it's also a good source of companies to look at, although certainly not limited to just Gibraltar.

Why does new product/feature development take so long?

This is a post for non-technical readers (particularly non-technical product and high level feature owners) to explain why technology is "so slow" to deliver your new products and features.

Comparative baseline.  You need to ask yourself, "Why do I think something is taking too long?" What's my baseline, what am I comparing to when I think "slow"?  More often than not, I find that people are just displacing their general frustration in not having something they want (much like a child does), and take it out on the deliverer (the parent).

That, or you're just the type of person that moans about most things generally, so please stop.

I find that some stakeholders either delight in or don't realize they're only selectively comparing one organization to another.  Why can Company X deliver a new feature in a week when it takes our tech organization 3 months?  They rarely seek to understand why, they just want to selectively pick comparison points to criticize the delivery team.

Of course what they may not seek to understand is that Company X:

• May have more and better technologists (perhaps at a higher cost base)

• Invested in a technology delivery system that is much more efficient to extend, scale, and maintain (i.e., relatively less technical debt accumulated)

• Enables a significantly different approach to IT due to a different revenue and cost structure (e.g., can afford better kit, replaced more often, serviced by better more expensive support channels)

True switching costs.  Perhaps you're the person that was frustrated with the rate of in-house delivery, and sought out a big supplier to deliver what you're looking for.  Hey, cheaper right?  And you got to teach those in-house slackers a lesson.  You get bigger economies of scale, and a lot more people to deploy to build your solutions.  But then it turns out your knowledge domain is new to the supplier.  Congratulations!  You just paid for the privilege of bringing a bunch of new and external people up to speed that didn't know your domain from a hole in the ground.  And then the supplier takes that knowledge out to other customers, finally creating that economy of scale you were sold on in the first place (Supplier: "Hey, thanks very much for giving me a new area to expand my business and dilute your priorities and control!").

A really good technologist or really thorough domain knowledge, especially both together, is rarely a commodity.  A "Java programmer" is a commodity.  A "really good Java programmer who understands our customer's requirements and has several year's experience developing our products within our business culture" is not.

You're a "get what you want", "bulldog", "win-lose" negotiator.  You might be the type of stakeholder that always demands things be done more quickly than what's been presented to you.  You might think that technical trade-off discussions are just technobabble to justify a "heavily padded", "low risk", "plenty of surfing time" schedule.  Perhaps you think your a tough negotiator and your stripping out the "fluff" the technologists inserted into the plan to save the shareholder's money.  Either way, that means you think you understand better how long things should take as compared to one or more people that just spent days or weeks thinking about the problem.

Unfortunately, you really need to understand and accept four laws of software development physics:

• Good technologists rarely over-estimate.  Most either want to please you by giving you an aggressive schedule, they think their team is better than they actually are (they take their own personal estimations and extrapolate it for their entire team, one aspect of the Dunning-Kruger Effect), and/or don't think about whole solution delivery timings.

• A qualified and professional team of people who spend time thinking about a solution will most likely know more about how long it will take to construct the solution than you do

• Technical debt can be ratcheted up and quality down to deliver speed

• Assuming the technology team is reasonably competent, dedicated, and professional, the only way to reduce the schedule is to alter some other project dimension

Here's the thing, technology teams can sometimes seemingly magically strip time out of a schedule.  However, if you take the time to understand the trade-offs, it's not magic.  Technologists can generally remove quality, stability, best practices and "this is how it should be done" dimensions from a project and deliver more quickly.  However, by doing so they're increasing technical debt and/or business risk.

As the stakeholder, you'd be wise to track technical debt just like you track project budgets.  Unless of course you plan to hand the technical debt over to the next management team and move on to another project.  Say, aren't you clever getting that big bonus for an on-time delivery.  Too bad the new administration didn't know about all that debt they're acquiring...

Servicing the debt.  If you don't manage the product and technical debt, if you don't seek to understand the tradeoffs that are being made when you pressurize delivery schedules, things will start to move even slower than they did before.  It will appear to you that your team is getting "worse" over time, their deliveries slower.

This won't initially make sense to you if you don't understand the trade-offs that have been made.  The team's technical, domain, and cultural knowledge should be getting better and better so why is everything taking longer now to deliver?  When the technologists try to explain why, it's very complicated.  In fact, it sounds like the same old technobabble they were trying to use to con you into a heavily padded delivery schedule.  Probably best to continue to ignore it like you've always done.  Your approach has served you and your delivery schedule well so far.  The team is probably just getting burned out and it might be time to switch suppliers!  And a new job has opened up elsewhere you'd be perfect for, let someone else struggle with this declining, unmotivated delivery team...

Excuses, Excuses

Let's face it, some individuals and teams are really really bad at what they're supposed to be doing.  Maybe, just maybe, you have been an attentive, detail oriented, engaged stakeholder, you do understand the tradeoffs that have been made, and the team you work with simply isn't delivering.  One or several of the following may have happened.

First, maybe the team really is bad (as compared to other similar teams).  How can that be?

• Unaligned expectations - you really can't teach a pig to sing; you've hired someone that while good in their own right, can't possibly be good at what you've hired them to do
• Burnout - people really have burned out (guess what, managing burnout is just another technical debt to manage)
• Bad hiring - some people really are just incompetent and/or lazy; these types also tend to be good at lying as well
• People change - they've just tuned out, perhaps due to personal issues; maybe their worked well in your business' previous culture and context a few years ago but don't in the current one
• Bad alignment - maybe someone else in the business is poaching their time
• Different priorities - they're delivering just enough not to get fired while they work on their own business or day trade
• Poor management and leadership - people don't know what to do and/or aren't enabled to do it; their manager simply isn't managing, the organization isn't enabling them
• Sewing seeds of discontent - a few people are really just negative, nasty, and unpleasant; they spread FUD to create discontent in the team and then take pleasure in the results

Second, I have to admit it, a lot of good technologists put architecture, future-proofing, tools, process efficiency, frameworks, scalability, extensibility, and maintainability against actually delivering a single feature.  They want to build a double-super-awesome application to dominate all other applications, and it takes a lot of architectural work to do that.  The key here is a technology leader that pushes for incremental improvement and delivery along all dimensions (product and feature delivery first and foremost) and makes technical debt levels transparent to stakeholders.

Third, you really have been screwed by a delivery team or supplier.  They're farming their alleged 100% allocated team out to three different customers like you.  Your SLAs have no teeth.  You've been sold using Ruby on Rails for development and even now a team of 30 people in several other countries are reading Ruby for Dummies and wondering why they've been hired to evaluate precious gems and lay railroad tracks but not write software.  50% of the time and budget are gone and you're too heavily invested to change.

Perhaps you really are in one or more of these situations.  If you are, you're probably justified in making drastic changes.  However, you have to ask yourself how things got that way, and what role you played in it.  Because if you're the one that created the bad situation in the first place, are you really qualified to fix or replace it?  Do yourself and your shareholders a favor and get some help.

What's the right way?

So what is the formula to speed up delivery?  Assuming that you prioritize speed over cost (see The Trinity Extended for more on this):

• Acquire a good technology leader.   Find someone with credentials and references you trust, and then let them get on with it.  If you made a bad choice, then really, that's your or maybe your Boss' fault, isn't it?

• Acquire good technologists.  Unfortunately, they're rarely cheap, because they know what they're worth.  Also unfortunately there are a bunch out there that will take advantage of you so you can also easily end up not getting what you paid for.

• Domain knowledge, and even better, interest.  Acquire technologists that understand what you want built.  They should "get it" when you give a high level overview on what you want.  Ideally, you'll find people that have an actual personal interest in what you want done.  They want to build software that they would use themselves.

• Delivered something similar.  Acquire technologists that have a proven track record delivering something similar to what you want to do.  It will certainly help with scheduling and anticipating risks.

• Cultural alignment.  Acquire people that live and breathe within your target market.  It's much more likely they'll have an implicit understanding of your customers and what's required right at the start.  Also hire people that have worked in companies like yours:
• Start-ups vs big companies
• New development versus support and extend existing, inherited, purchased products
• New development versus integration and middleware
• Internal versus external customer facing
• b2b versus b2c
• Dedicated resourcing (generally apolitical) versus programme "beg-borrow-steal" persuasion/shared model of resourcing (generally political)
• Static (not much change, driving down costs) versus dynamic (fast moving market, environment; lots of change)
• Reactive (operationally oriented; opportunity led) versus Proactive (project oriented; strategy led)

• Knowingly manage technical debt.  Sure, you can accelerate now and pay later.  Commercial realities may dictate this behavior.  Just make sure you knowingly stay aware of how much technical debt you're creating as you go.  Set budget, strategy, constraints, horizon expectations clearly and have your tech team explain where debt is being accumulated and likely impacts that will result.

There are many other and similar views on how to deliver fast and well.  The whole family of Agile, XP, Scrum, DSDM, FDD, Kanban and other similar methodologies all take views on this.  They are all interesting and worthwhile to understand and use as appropriate.  But to me most of them don't really emphasize enough the human component, the true difference a really excellent technology individual and/or team can make.  Instead if closely followed they tend to treat people equally, reward mediocrity, and put process ahead of people.  All technologists are definitely not created equal, and yes, they're not machines they're people.

If you are unable to effectively make a judgment about the situation (good for you, at least you recognize this in yourself), bring in an external IT consultant you trust or with a good reputation to perform an IT audit.  Have the consultant take a view on what is and isn't working.  And whatever you do, don't bring in a consultant who is a stealth sales implant for a large professional services arm at the same company.  Make it very clear that you would use someone else to do any follow-up remediation.

Conclusion

It is possible that you have a poor, slow speed delivery team.  Perhaps you do need to significantly alter how you deliver with new people, new suppliers.

However, before you do anything radical, do your business and shareholders a favor.  Sit back and think for a minute where the "slow" designation comes from and how objective it really is.  If the only common denominator between the in-house and out-sourced speed assessment is you, perhaps it's your judgment of "slow" may be flawed and/or that you really aren't managing your technical debt.  And if you're really not sure, get a project/IT audit done by a trusted resource to give you an outside view.

Using wget and google to download and translate websites

There is a website for the neighborhood I live in that is all in Spanish (cuartonparque.com).  So that's useful if your Spanish is good, which mine isn't.  Google's translate function is great, but I wanted an archive of the site both in Spanish and English in case the site disappeared or was substantially altered.

wget is a great command line *nix utility to recursively download a website providing the links are statically constructed.  I use wget on OS X (install xcode and macports to enable installation of wget if you don't have it).

For cuartonparque.com, the wget command is straight-forward and well documented.  The site uses simple static links and only has a few levels of linking. To download the site, I used:

wget -rpkv -e "robots=off" 'http://cuartonparque.com' 2>&1 | tee cuartonparque.com.wget.log

This command creates a cuartonparque.com directory with a browsable website.

To download a translate.google.com version of the site was trickier.  Although various googled pages helped a bit, I couldn't find find an example that actually worked.  After some hacking about, I uncovered the required tricks to make this work:

• Google appears to only process requests from browsers it's familiar with (use -U Mozilla)
• Google uses frames and changes it's domain name a bit as it translates (find out the final URL of interest by digging around in the page source)
• Safari really likes a .html extension on files it opens (use --html-extension)

My pain is your gain.  Here is the wget command that downloads the translated version of the website:

wget -rpkv -e "robots=off" -U Mozilla --html-extension 'http://translate.googleusercontent.com/translate_c?hl=en&sl=es&tl=en&u=http://cuartonparque.com/&rurl=translate.google.com&twu=1&usg=ALkJrhjabXZlzJpBCZeWpsmLaKss09lCuQ' 2>&1 | tee -a cuartonparque.com.En.wget.log

wget creates a translate.googleusercontent.com directory with a browsable website, localized from Spanish to English with a horrific URL for the index.html page:

file:///Users/xyz/Downloads/Web%20Sites/cuartonparque.com.En.Google.Trans/translate.googleusercontent.com/translate_c%3Fhl=en&sl=es&tl=en&u=http:%252F%252Fcuartonparque.com%252F&rurl=translate.google.com&twu=1&usg=ALkJrhjabXZlzJpBCZeWpsmLaKss09lCuQ.html

A quick browse around on the downloaded version suggests everything came through, nicely translated to English with wholly-formed pages.  Enjoy!

DBA Evolution and Specialization

Introduction.  DBAs are an expensive resource.  Part of the art of IT management is to manage IT costs against revenue.  Here are the growth phases of businesses I've seen and how the DBA function evolves and specializes against changing revenues, risk tolerance, and work load.

Phase 1 - Startup, limited cash, keep the burn rate down.  When a business first starts out and afford only a few technical resources, chances are you'll have an all-rounder technologist also acting as a DBA.  As a little more money comes available or a risk factor hits (typically a crippling systems, security or change management failure), an all-rounder systems admin may be hired that picks up some of the DBA responsibilities, typically at the edges of the database (e.g., backups, storage).

Phase 2A - Revenues increasing, risk tolerance slightly decreasing.  Revenues and risk management warrant hiring a full-time dedicated DBA.  The DBA starts reviewing developer driven database changes and eventually the software developers get shut out of production databases to better control change.  The DBA takes over backups from the all-rounder systems admin.  Confidence in DB recovery goes up.  The DBA takes over ownership of the fundamental data model and supplements the report generation otherwise being performed by the sw devs.  The DBA can't ever go on vacation as they are on the support escalation call-out and are business critical.  Your one DBA is an all-rounder and supports all databases throughout the business - some well, some not so well.

Phase 2B - Marginally net profitable, and risk tolerance decreasing to protect profits.  The DBA is becoming overloaded and would really like a stress-free vacation.  You've probably increased your number of schemas and instances of your main database and a few other database packages have appeared.  DBAs are expensive, so rather than hiring DBAs to cover a 24x7 rota, you hire maybe one more DBA to deal with overall growth of requests.  You have to outsource database services to provide escalated issue support and to support non-primary database packages.  Your 1-2 DBAs are all-rounders with respect to the database packages you support.

Phase 3 - Business successful, profits increasing and scalability is challenging.  Now you have to take the plunge and split up your all-rounder DBAs responsibilities and ownership.  Generally, there are three areas of DBA specialization:

1. Specific product(s) orientation, business-facing.  These DBAs understand the data and business model by product (product composed of one or more inter-related applications, schemas).  Products may be in-house developed or from third parties but either way require IT depth of expertise to meet business objectives.  These DBAs understand how data ties together and business logic as it related to data manipulation.  They communicate well and are more business then technically oriented.  They can produce reports against the data model and these reports are consistent with each without super-detailed report requirements definition due to the DBA's understanding of the business and its product data model and business logic.
2. Specific product(s) orientation, developer facing.  These DBAs are more technical.  They work closely with the software developers of in-house developed products.  They are application developers in that they write code within the database platform itself, such as stored procedures.  They take developer database code, DML, and DDL and review, clean-up, and optimize it before it enters production.  They manage change of the application within the database and write the scripts to migrate/upgrade the application's DB based data structures.  They troubleshoot application problems that are within the database and an application layer.  Depending on the systems architecture, they may also have a passing familiarity with application persistence engines like Java's Hibernate to be able to troubleshoot issues between Oracle and the application persistence layer.
3. Specific DB platform(s), operationally oriented, systems facing.  These DBAs are even more technical.  They work closely with the systems administration team at the "edges" of the database platform.  These DBAs aren't so much concerned with the application or data structures within the database as they think of the database as a complex "black box".  They are responsible for backups, performance tuning of the DB platform as a whole, OS touch points (e.g., shared memory), storage (e.g., filesystem type, performance, allocation and placement), clustering for fault tolerance and/or scaling, change control around the database (e.g., config files), root cause DB faults (e.g., Oracle's ORA errors), upgrade and patching of the DB software, and maintaining the DB operational runbook.  They are skilled at troubleshooting the database as a whole, in and around it's operating context and in the space between clients and DB itself.

To further scale the DB function, each of the three above can be split out by product/application and by DB platform.

Hopefully one of those all-rounder DBAs you have from Phase 2 has team management aspirations and can step into an overall DBA leadership and management role as you'll need roadmap and backlog management for all the DB platforms and products/apps you have in Phase 3.

Phase 4 - No idea!  The above 3 phases should support an IT team size in my industry into upper 100s.  I look forward to the opportunity to see and maybe help shape something bigger in the future!

Conclusion and Recommendation.  Use the above phases to figure out where your organization is at and manage your DBA capacity against revenue, budget, and risk tolerance accordingly.  DBAs are expensive, but also one of the most business critical functions IT provides.

Supplier Reviews

The following is a sample agenda for a supplier review.

Objectives for Review

• Agree a plan with supplier to solve key issues
• All ownership, processes and contact points well understood and up-to-date
• Understand where the supplier is going, their strategy and roadmap

Areas to Consider

• What areas does the business want to see improved; what are the pain points with this supplier?
• Number of new features delivered (e.g., new products and/or services made available, functional improvements, elements of previous supplier review agenda delivered).  Consider both velocity and acceleration of deliveries.
• General process improvements (e.g., escalation of issues back into supplier)
• Review major product/service failures and downtime events – root cause?
• Review product/service roadmap, both current and from last time.  Assess delivery ability against stated roadmap as a way to evaluate currently presented roadmap
• Evaluate other customers of the supplier - their situations, requests; how are they driving the suppliers strategy and priority choices?
• Roadmap of technical/internals
• Upgrade plans
• Review who owns what and escalation details; general review of support and ability to escalate
• Review recent and related project retrospectives - any issues to progress with supplier?
• Can the supply/delivery chain be optimized? Come into session with a summary of the delivery chain, owners and timing - what can be tuned?
• Supplier costs summary.  Any invoice issues.
• Alternatives to supplier

Output of Review

If this is a major review, consider setting up a wiki page that pulls together all the information related to the review, e.g., specification, bug lists, retrospectives, API specs,

Documentation updates, e.g., a supplier support change names and phone numbers

Action items between now and the next review.

Spectrums of website control - from tech to non-tech

Sometimes it's not clear to a non-technical consumer of a web site what is involved to change content, style, layout, and functionality that make up the website.  In particular you're interested in enabling websites to be built, updated and extended by non-technical staff.  This entry is about explaining the spectrum of possibilities  in change in each of these areas for non-technical users, for example a non-technical website product manager.

Static Content

The static content of the website is all the text and graphics you see on a page.  It is not about functionality (e.g., customer registration), navigating between pages, or other browser-user interactions.  "Static" is a bit of a misnomer in that the text can be changed using the two methods below.  However, it is not about dynamic content that changes on a page without you requesting it to change.

At a basic level, content can be inserted directly into a web page that sits in a file on a filesystem.  To make these types of content changes, one must know at least some basic HTML and maybe CSS.  These types of content changes are typically done by a website "designer", more aptly called an HTML/CSS author.  Many, many websites start out with this approach - it's easy if you know how to do it.  There are no special costs for purchasing, integrating, and/or learning specialized content administration tools.  This could be considered a "technical" activity as it's generally not simple enough for someone without some basic HTML/CSS and related tools training to do.

At a more advanced level, there is some types of forms based interface that allows a non-technical user to make content and image changes on the website.  There are many levels of variation and complexity at this level.  Workflow, publication control, localization flow, version control and security become differentiating features.

Website Design - Style

The design style of a webpage is typically brand and "look and feel" attributes of the page.  At a simple level, this is colors and type/size/modifications of fonts.  The goal is to have a common style and brand feel across all pages of a website.

At a basic level, design may simply not exist, using browser page rendering defaults to style pages.  This is quite common early on in website/page development as you want to focus on other factors and not design.

As the design approach matures, style may initially be embedded directly in a webpage and closely associated with (located with) the content the design is being applied to.

The next level of maturity is to separate style from what is being styled.  One style view, as defined in a globally used CSS definition, is used for all pages in the website.

For all of the above, it is a technical activity with an HTML/CSS author working with files at a filesystem level.

As with static content, the jump is then made to having a forms based interface to change style dynamically.  The syntax of CSS is somewhat hidden away and simplified by the forms and fields a non-technical user interacts with.

Website Design - Layout

Separate from website style is website layout.  Layout is about how the various blocks of information on a webpage are placed on a page.  The goal here is to balance a uniform customer experience across all pages of the website versus contextually unique information and functionality that appears on specific pages.  For example, all pages of a website may have the same top bar, left navigation, and bottom set of links, but then have significant variation of information and functionality in the other parts of the page.

As above, layout can be done by hand by a HTML/CSS author for each page in a website.

As the manual process matures typically driven by a growing number of web pages, some type of organizational framework is implemented so that a uniform layout is used for all web pages.  Developers who implement the framework start working with the HTML/CSS authors to support them when implementing the framework.

The solutions complexity then jumps up significantly to enable a backoffice to manage layout of web pages.  There are basically two varieties of these currently available:

• Technical staff create a page template where the blocks of content and functionality on the page have a fixed layout.  Non-technical users can create new pages using the template as a base, but can't otherwise change the layout.  This is like working with MS-Powerpoint master templates.  
• A little more advanced is creating a fixed layout of "containers" of functionality on a web page where subordinate blocks of functionality are contained in a larger "container" block.  Popular blogging software such as Wordpress take this approach.  You can't change the overall layout, but you can pick the type and order of functional blocks within the container.  The blocks require technical work to create, typically developer and designer.  Nicer interfaces allow a drag-and-drop placement of the blocks.
• The most advanced layout approach is to allow full construction of a page or parts of a page by drag-and-drop from a toolbox of functional blocks.  The blocks may be placed, sized, and configured via a forms based interface by a non-technical user.  The page may have a completely free-form layout, or inherit certain fixed attributes (e.g., to enforce a common header and navigation at the top of all pages of a website).  A non-technical user may define a "template" layout then used by actual website pages, again like a Powerpoint slide master.  This approach may enable non-techs to create and manage pages, but uses very complex technology underneath to support this capability.  Google Sites is an example of this layout approach.

You can see from the above that layout starts technically easy to implement but doesn't enable non-tech users to manage layout.  At the most advanced, there is a lot of complex technology required, but non-tech users can control the layout of web pages.

Note that sites built using the more advanced tools tend to be blocky, linear, and very functional looking.  This is because a good designer isn't working on the layout, only a limited layout software and non-techincal (non-designer) is building the pages.

Website Functionality

We've covered the non-interactive attributes of constructing websites - content, design, and layout.  We've noticed that layout can place functional blocks, and we'll now take a closer look at them.

Functionality must be built by technologists.  As we've seen above, it's possible to enable a non-tech user to drag and drop functionality into a web page layout, but non-tech users can't create new functionality.  You want to register, login, conduct a transaction, or display dynamically updated content on a website?  Technologists will build that.  Yes, there are interesting projects like Yahoo Pipes, but they're more of a toy, very cool, but not fit for small business or little less enterprise level websites.

Given that functionality must be built by technologists, what are the various ways to build that functionality?

At a basic level, a lot of website functionality can be built directly in a web page.  This is a simple and quick way to introduce many classes of new functionality, but is quite limiting when extending and maintaining the functionality.

A much better practice is to reference functional requirements in a page, but implement the functionality completely separately.

In order to support a "page made up component blocks" for example as required by advanced page layout tools for non-technical users, functionality must be constructed in a very specific way that will play nice with the layout and design frameworks in use.

The Spectrum from and end user perspective

With all the hype around what was user generated content and what is social media (user generated content within defined networks of consumers), what are some practical implications of enabling non-technical user outside the business to manage their own pages and content within the business' website?

The first thing to think about is delegation of control.  For example, how much control do you want to pass on from the technology team to internal non-technical users?  One typical consequence when this happens is that site quality drops as the QA process is decreased but the increased flexibility and scale is deemed to be worth it.

The next step of delegation of control is to trusted external partners and suppliers outside the business.  "Trusted" is no doubt a stretch when describing some of them.

The final step of delegation is to pass control to end users.

For all of these steps of delegation, you have to decide what you can "safely" delegate, balancing scale and customer control versus internal governance and quality controls.  And of course your ability to delegate safely is based on how you've implemented the functional blocks you have to work with.

Let's take a look at a specific end user example.  In the beginning of a website/service, everything is controlled by the business - the user journey, what they see on each page, the functions they have available to them and how they are visually organized.  They interact with the site only to submit information and see results.

The next step is user generated content such as a user homepage with simple "about me", a photo, and blog/status functionality in a fixed format.  Then some user interaction functionality is added like forums and chat.

Then one or both of the following happen.

Path #1: As the site evolves and more functionality and content becomes available the customer really wants to control what they see and do.  The site's one-size-fits-all approach is breaking down.  The only thing left to do is to enable the user to create and manage their own experience by selecting functionality and content important to them, organized in a way that makes most sense to them.

Path #2: The website product manager, realizing the site can't be all things to all people, can't be another Facebook, separates out and emphasizes the core business proposition functionality from all the non-core me-too "extraneous" functionality they've worked so hard to create.  They have a Starbucks moment and realize that they need to be where their customer is and not make the customer come to them.  Enter web services into Facebook and the like and exit stand-alone business websites.

Summary

Websites are made up of content, style, layout, and functionality.

The technology exists today to empower non-technical users to create and manage websites.  There are limitations to this, and some of the underpinning technology is complex, immature, and proprietary.

Like with most technology areas, the more you want to simplify technology so that technology specialists are not required to operate the technology after implementation, the complexity and cost for the project increases.

Highly Agile Product and Project Management for Small Teams

No rocket science here, just a quick entry on a very light way to manage a small and self-contained project team frequently receiving new and modified feature requests while building a new and unproven product.  Goals are minimal process, managing feature creep, and getting the new product into the hands of real customers.

Assume the team is composed of:

• There are several stakeholders acting as product manager, making feature requests and requiring a high level view of the project.  They are the work creators.
• One project manager coordinating everything.  The PM vets, routes, value-adds, chases up, and drives completion of work.
• Handful of technologists and suppliers the project manager is coordinating.  They get work done.

There are two main control documents: product backlog, project plan.

Product Backlog

The first is a product backlog, used to capture all the new ideas and feature requests.  The backlog should be kept in a wiki, google doc, or similar shared location.  For a small team and project the project manager is the chaperone of the backlog, but anyone can put new entries into it.

The backlog is the only way that new feature requests can enter the project.

Backlog entries have these attributes:

• Who requested
• When requested
• Enough detail so everyone roughly knows what is being requested (but no more, not at this point)

As you start adding in backlog requests, put them into a few simple groups:

• Required to "make the sale", acquire investment, unblock the team, or otherwise business critical in nature ("It's game over if we don't do this right now!")
• Required as part of the release currently being worked on ("We can't deliver the new release without this.")
• High want for the next release coming up after the current release ("I really want this in the current release, but I'll settle for the next release coming up."  "All our competition has this feature.")
• Futures, nice-to-have, everything else ("Cool idea, but we don't absolutely need it right now")

If people other than the PM add new entries, they should throw them into the bottom of the list but can note where they would like the request.  Let the PM move the request into the main backlog as part of their vetting and value add functions.

As backlog items are completed, edit them out of the backlog and into a completed list.  Note when they were completed and by whom.

Tactical Plan

The tactical plan is a simple document written by the PM for the stakeholders that covers:

• Two lists of tasks:
• Last week - what got done, who delivered the task, when done
• What tasks are on-going from last week, who is involved with each, a rough estimate to complete each, reasons for date changes (if any)
• List of (anticipated) free resources and when available
• Suggestion of next tasks to put on the tactical plan as taken from the backlog and from other sources
• List of dates and description for delivery aspirations, commitments and estimates.  Highlight changes from the last meeting
• List of roadblocks, issues, and risks
• List of budget/spend requests

Weekly Review

About once per week, the project manager updates a tactical plan for review with the stakeholders.

Before the meeting the PM has done their homework, getting fresh information from the rest of the team and suppliers.  The PM has taken a quick scan through the backlog, looking for fresh, urgent additions and is prepared to make recommendations of which should become new tasks.

The project manager sends out the updated backlog and tactical plan in advance of the meeting.

The stakeholders use the weekly meeting to make changes to the tactical plan.  For efficiency, unless it is an absolute crisis or a critical mistake is being made, the stakeholders agree to not change the tactical plan more frequently than weekly.

The PM and stakeholders also try to avoid creating a long list of partially done tasks, instead allowing tasks to run to completion and not interrupting them with a new task assignment unless the work is really of zero value.

The PM and stakeholders review the recommended promotions from the backlog to the tactical plan and agree on which items to add to the tactical plan in the next 1-2 weeks based on expected free resources.  As new backlog items are selected, makes sure expectations are aligned about what it means to be "done" for each item.

The PM and stakeholders review the roadblocks, issues, and risks to problems solve and make changes to the project and team.

Spend requests are reviewed and approved/denied.

The PM revises the tactical plan and backlog after the meeting is complete and circulates it.

Intentional Exclusions

Other items could be put into the project plan such as QA test coverage and bug lists, code stats, and operational concerns.  However, with a really small team and unproven product, these may not be of much value yet.

Summary

Keep things as simple and light as possible when managing a small team working on an unproven product.

Use the backlog to give the stakeholders a voice and to not clutter and thrash the tactical project plan.

Use a one week review cycle to balance flexibility with efficiency.

Sith Mind Tricks, Episode One - The Management Menace

Prologue

Sure, all good managers and leaders develop a collection of Jedi mind tricks to create business value and further business objectives.  Unfortunately, bad managers are doing something similar, using their Sith mind tricks to further themselves and their agenda, at a sharp cost to the business and those around them.

What are these anti-patterns of reasonable behavior, good management and so-called leadership?  What are the signs and how can you defeat them?

To start with, you need to understand the Sith.  What motivates them?  Like most of us, they want to further their career to earn more money.  But to do this, the Sith put themselves first, furthering their own progress at the cost of their co-worker's progress.  They don't think about win-win, only win and who cares.  They are short-term thinking capitalists.  They make sure they look good even if that means painting others as looking bad.

Attributes of the Sith

Let's consider general attributes of tricks used by the Sith and how to recognize and defend against them.

Attribute #1 - Siths think in the short term.  Sith mind tricks can be used to great effect in the short run, but they generally result in a win-lose situation overall.

So that gives us the first general general way to defeat them - make sure you're taking the complete picture into account, not just the picture the Sith is trying to frame the conversation in.  Like in martial arts - if you fight the way your opponent fights, you will lose.  Draw them into your way of fighting, your framing of the situation, to win.

Attribute #2 - Siths shrivel in the light of thoughtful logic and complete information.  The powers of robust logic and complete information are anathema to the Sith.  They thrive on speaking rapidly, pulling in whatever facts support their position and silently ignoring or discounting the rest.  They lightly thread one-sided "facts" together with just enough veneer of logic to create the illusion of a solid basis for the uninformed around them.

To defeat this, you just plain have to be better.  Prepare more then them, think about scenarios in advance, critically evaluate viable alternatives, and put forward an airtight set of logic and facts to counter the Sith's view.

Attribute #3 - Siths are multi-faced.  The Sith shows whatever face they need to show to get what they want.  The Sith will suck up to their boss, giving the boss a carefully framed view of a world in which the Sith does no wrong.  They will badger peers to their face and attack them from behind.  They will threaten or just generally annoy subordinates who want to end the interaction with the Sith as fast as possible.

You can't really defeat this one, it's more of a trait that identifies a Sith.  The best you can do is keep the Sith's boss casually informed of the reality of things (not the Sith's fantasy view), encourage your peers not to put up with the guff and fold against the Sith's pressure, and empower your subordinates to escalate as needed to help the business avoid a bad outcome.  And yes, this is a lot of work.

Sith Mind Tricks Featured in Episode 1

And now for the main feature - the Sith mind tricks and how to defeat them.

Trick #1 - Power Relationships.  A Sith takes advantage of their power relationships in the business.  They bypass their peers and go directly to subordinates to secure commitments that support the Sith's agenda.  It's much easier to drive their agenda with someone beneath them in the organization, to use their power to force commitments and decisions from a subordinate.  If they had to do this with a peer or with peers and their boss, the Sith would have to make a case for what they want to do.  Using implicit control over someone's job security and salary is an effective way to assure the person will agree with you.

A Sith is all stick and no carrot.  No carrots except for themselves of course.

If you're peers with a suspected Sith, take note of who the Sith talks to when you are in and out of the office and if they take the chance to go around you directly to your resources.  Know who those people are and make sure they feel comfortable contacting you any time if someone is asking them to do something they feel is a bad idea.  If you're organizationally subordinate to the Sith and they want you to do something you feel is a bad choice, insist that you defer commitment until after you've had time to review it with your boss.

Trick #2 - Win-lose "information" gathering during meetings.  Similar to Trick #1 but in a public forum, a Sith's power increases when they get the answer they want to hear in a public forum.  The Sith will latch on to meeting points that support their agenda and discard or attack ones that don't.  They will badger and intimidate subordinates in a meeting to make the subordinates verbally support or at least not disagree with the Sith's assertions.  If anyone does disagree, the Sith will accuse them of "not being aggressive enough" or "not being a team player" in front of the group to cow them into silence and make sure the person doesn't dare speak against the Sith again.  The Sith of course makes sure that they either publish the meeting results or the "right" view gets put on the minutes.  As a result of this trick, the Sith has public confirmation that they're view is "right" (the "win") but of course the the people in the meeting and the business suffer (the "lose").

A particularly nasty form of this trick is for the Sith to power up trick #1 by wheeling their boss into discussions to really put some weight behind Sith's view of the world.

This trick is also known as "the elephant in the room" (the obvious fallacy that everyone can see but can't be discussed without being attacked by the Sith), "the emperor has no clothes" (execute anyone who points out the emperor isn't wearing clothes), and "repeat the lie until it considered truth" tricks.  It is bullied group-think.

If the Sith gets the power relationships "right" (for the Sith's agenda), they can talk complete bollocks, buck naked, surrounded by elephants.  This is Sith heaven (and everyone else's view of meeting hell).

How to defeat it?  Own the documentation and communication.  Take advantage of the Sith's laziness to write up minutes.  Make sure the issues and objections get documented.  Point out previous objections when an anticipated issue rears up.  Make sure key observations and decisions are at your fingertips when the inevitable blame fest occurs.

Trick #3 - Divide and conquer "information" gathering.  This trick is much like #2, but the Sith relies on trick #1 to corner people individually to extract the answer that they want.  The Sith quickly identifies who is weaker in the team, who doesn't like to argue, who is less informed but also opinionated, and/or who has an axe to grind with their boss and goes after them to get the answer they want.  They will use whatever technique - sucking up, threatening, or badgering - will work to get their answer played back to them by the hapless target.

Once they have that answer, they use it to go after the others on the delivery team saying that person X on the team said that the Sith's answer is The Answer and use it as a way to get The Answer from others on the team.

This trick can be countered by empowering everyone on the team to give the answer they think is right, not one they're being cowed into.  If that doesn't work, insist that certain team members take a "teamwork view" by empowering them to discuss an answer with key (stronger) peers before they respond with an answer to the Sith.

The Sith are good at praying on the weak and indifferent, defeating one person.  They aren't so good when the team works together and is exquisitely aligned.

Trick #4 - Top down scheduling.  Similar to Tricks #1-3, but focused on delivery deadlines.  This trick has the Sith demanding delivery of something from the delivery team by a certain date.  Why the date?  Maybe it's tied to the Sith's bonus, or a commitment they made to their boss or customer without consulting the delivery team.  Regardless, the Sith will assert a delivery date that the delivery team thinks is impossible.  The delivery team does their best to re-estimate, cut corners.  They offer up scope, resource, quality, risk, and technical balance trade-offs to decrease schedule.  The Sith won't accept any change to these (often because they don't want to take the time to understand them), and may even creep scope some because of new requirements that come up during the trade-off analysis.

Over the course of several meetings, the top down date becomes "The Date" and the Sith actually starts accusing the delivery team of "slipping the schedule" and "not making their dates".  The Sith uses Tricks #1-3 to make sure that the team stops pointing out that the delivery date wasn't their date and they never committed to it.

If scheduling reality starts to interfere with scheduling fantasy, the Sith will allude to legal and contractual disputes if the delivery isn't made.  If that doesn't whip the team into shape, the Sith threatens the team with loss of revenue, the business going Titantic, and everyone losing their jobs if the delivery doesn't happen to the Sith's top-down dates.

Why does the Sith do this?  They believe the best way to manage a team is setting aggressive deadlines and threats.  The Sith thinks they can do a better job with top-of-mind scheduling than a knowledgeable team that has spent a lot of time on project planning.  They want control over all project dimensions to better drive their agenda, even though they don't know jack-schlitz about the details.  The Sith thinks they know better, and don't trust the team anyway.  Most likely the Sith is projecting their own problems onto the team.

What happens as a result?  The Sith's ability to get real information drops to zero.  The Sith doesn't want real information anyway, they want their own view of the world spoken through the mouths of the team so they have someone to take the fall when the project fails to deliver "on time".

And how do you defeat this trick?  The counter-trick again is to control the documentation and communication around key observations and decisions during the project.  Keep key historical points at your fingertips.

Of course, the Sith have a counter-counter trick to this.  When the project ends up failing right where the team said it would, they just accuse members of the team of not taking a strong enough position against their view and convincing them that they're right.  It helps to create key information and decision point milestones in the project, carefully document the project reality at milestone (facts, not interpretation), and widely distribute the milestones.  You need to document and communicate the milestone results before the Sith can.  If you communicate the truth enough times, it usually becomes... truth.

What's the triple counter?  You have two choices.

Choice #1, and probably the easier.  Go find a job where you don't have to work with this person ever again, nor a company or bosses that endorse people like this.

Choice #2, much harder.  Put all the information together, create an airtight view of Sith behaviors and decisions during the project, take it to their boss or your mutual boss and go on the attack.  Recommend you take over decision making authority.  Only do this if you're very confident of winning because if you lose the Sith will own you.

The Epilogue

For a few, there is no hope.

Some Sith are just too far gone.  They are a "nasty piece of work" in an otherwise pleasant place to work.  They've pulled the wool over the eyes of their Boss, their peers don't want to be around them, and their employees are driven through fear.  They are ravaging the organization to improve their personal position.  No matter where you work, there will be at least a few around.  Maybe they're customers, maybe co-workers, or maybe just "friends" of friends socially.  You aren't going to change them, and you may not be in a position to drive them out of your circle of interaction.  Just be careful with them, document your critical communications, and make sure you're well prepared every time you interact.

But for most, there is hope.

Not all Sith mind tricksters are beyond redemption.  Maybe they've dabbled with the dark side in a moment of weakness, frustration or stress.  They're in the gray area, and maybe you can help them out by pulling them back into the light.  I know I've been there before, having just done something regrettable to power through a situation, when a Jedi master has come along and kicked me in the backside and back into the light.  If you can do this for someone, it'll feel damn good and you'll have helped your company out as well.  You owe it to them to at least try.

(There are many more Sith Mind Tricks.  Continue learning how to defeat them in Episode 2 - Attack of the Drones.  Coming soon to a blog near you.)

Synology CS407e NAS Cube Station and file move/copy errors

I've been using the Synology Cube Station CS407e as a home NAS for about a year now, and up until a few weeks ago it worked flawlessly. I'm only using it as a network attached media server, not all the other whizzy features it has.

The setup: I have three different machines at various times coping and and moving files around on the box. Trying to be overly clever, I set up a user "write" on the Synology with write permissions so if I had guests in the house using the media server they by default couldn't clobber any files with the default guest access.

Several weeks ago I started having permissions and file move/copy errors. The errors didn't make sense given the relevant context. I tried relaxing permissions by enabling user guest with full read-write permissions. I tried changing login credentials and permissions from the three different clients I used to copy and move files around. Various brick walls. And yes, I was on the latest firmware Synology had available for the CS407e.

I then noticed that I could ssh into the Synology. As I said, I had just been using it as simple NAS. Viola, after logging into it (btw, admin and root logins have the same password) I had access to chown/chgrp via the Unix like busybox interface and tidied up the permissions.

While that solved the problem, it's worth noting that it appears something about the permissions handling on the Synology is broken. The user and permissions setup via the web interface on the box isn't consistent with the errors happening (I had enabled guest with write permissions, still got client side errors) and what I saw at the filesystem level via the command line. So while repairable (ssh interface to busybox), the repair isn't viable for non-techs.

If you're non-technical and not comfortable working at a Unix command line but you still want to use the Synology, I suggest you do not set up other users. Just stick with user guest with the permissions set wide open.

If you've already dug the hole for yourself as described above, I think you can copy everything someplace else, delete all the offending files/directories, re-create a new directory structure with the guest user, and copy everything back.

Lastly, don't let this one problem put you off from considering a Synology for your home NAS.  I've had reliable use of the Synology CS407e for over a year now (as compared to two different Sans Digital MN2Ls previously - yikes, stay away).  It's quiet, reasonably fast, has a top quality management user interface and it Just Works (assuming you don't monkey with non default users!).

Synology company home page

CS407e product overview

CS407e technical specifications

Download CS407e firmware upgrades

Synology CS407e NAS Cube Station at amazon.com