(Cross-posted and shortened from The Holy Java.)

The recent article Agile will fail GovIT, says corporate lawyer is rather controversial but very valuable. Its value lays not in its claim that agile cannot work in governmental environment, something I quite disagree with, but in its presentation of how (inaccurately) agile is/may be perceived in this environment and of obstacles posed by such an environment to any (and especially to agile) development.

The article reveals a fundamentaly psychological and social issue. It doesn’t question the ability of agile to deliver projects much more successfully than waterfall. The laywer, Alistair Maughan, doesn’t speak at all about projects’ results. He speaks about fear (to bear the consequences of a failure) and constraints present in governmental environment. Thus it’s pointless to argue about benefits of the agile and absurdity of the waterfall approaches. The key to a successful project is to understand those constraints, lift them as much as possible, and create a security structure for both governmental officials and the supplier to be able to work within those constraints safely. We shouldn’t forget that there are also smart people in the government agencies who will gladly accept a methodology that leads to better results as long as they are safe from being denounced on the front side of newspapers as bad public servants when something goes wrong.

That, of course, isn’t easy. But successful agile projects in the public domain prove that it is possible.

An agile approach will never be able to deliver a large, multi-year project within budget, with agreed upon (detailed) features, and on time. Of course a waterfall approach won’t be able to do that either – but, contrary to agile, it can pretend that it can do it. It will fail (or maybe “succeed,” producing something that nobody really wants), but everybody concerned will be safe enough – only tax payers will mourn.

Perhaps the only feasible way to realize an agile project in the public domain is to break the task into multiple projects small enough to be estimated with sufficient reliability (and if the estimates are wrong, such short time frame that they cannot go too wrong) so that the supplier can commit to a fixed price and scope¹. The customer would be then obliged to take over the deliverable and pay for it (to avoid “it’s great, but we need these two more features to make it usable”).

An experienced team, knowing both the domain and technology, can estimate quite reliably in perhaps a three months horizon. So the supplier can make an umbrella contract, which breaks the task into individual 3-months fix-price projects, each starting with a new detailed contract and ending with a working, paid-for deliverable. If the task cannot be broken into such incremental deliverables than it is heading into big troubles and you don’t want to be there anyway.

There is a nice example of how to form a contract so that the customer can feel safe in the blog post Selling Software Projects by Demonstrating Value Early. It also mentions the time & material contract, mistakenly taken as an essential characteristics of “agile” by some, including likely the A. Maughan:

We will work time and materials, it will cost you approximately this much per sprint, and we think it will take about this many sprints but are not really sure. Furthermore, we don’t know exactly how long it is going to take, or what we are going to deliver.

Of course most customers – not only governmental ones – would have problems accepting this.

1) The idea of breaking project down into fixed-price (1-)3 months projects doesn’t come from me but one clever person who I’d need to ask for permission to name.

Conclusion

As I understand it, the main challenge in governmental project is the bureaucratic nature of the customer. More exactly the following implications:

1. Requirement of fixed price, time and scope
2. Paralyzed decision-making process – nobody but the top decision makers really dares to make a decision and they are not available and/or too far away from the problem beeing solved to be able to decide properly and timely
3. Need to “apportion blame,” absence of trust

These factors are very real and every project has to deal with them, being agile or not. Both the agile development methodology and the governmental environment must be adjusted to make a project possible at all.

Development happens in 4D space: time, money, scope, quality. You can ever fix only 3 of these axis (and it cannot be quality for quick&dirty is a contradiction). Pretending that you can fix all of them is a lie. Agile can guarantee fixed price and time but the scope (expressed as a high-level business values and epics)  must vary – to a certain degree – to provide for  inherent complexity of the domain and to provide flexibility for finding out what the customer really needs. It should be noted that detailed up-front specifications lead to the customers trying to add any possible feature they may think of even though 45% of them will never be used and 19% only rarely (Standish Group Study reported at XP2002 by Jim Johnson, taken from Mary Poppendieck’s Lean Software Development slides). Wouldn’t it be better to develop only those ~ 20-35% features whose benefit justifies their cost to get the software at a much lower cost and shorter time?

Without some level of trust (which, of course, cannot just “happen,” but must be gradually built) and a decision structure that actually works, every project will most likely terribly fail. A “watertight contract” and “clear deliverables” are not a solution as the history of failed waterfall projects has shown because only rarely does the customer really know what exactly he needs and mis-trust based relation hinders cooperation and communication, the very essential precondition of a successfull project. “Appropriate remedies” won’t really help anyone, even if the customer gets most money back, he will have wasted many resources. On the other hand, trust is not something you can expect from a bureaucratic ogranization so, as mentioned, you must construct a security structure to get rid of the fear preventing cooperation.

By the way, this reminds me of Mary Poppendieck’s explanation while other US airlines cannot reproduce the success of the lean SouthWest Airlines – they are too much stuck where they are and driven by short-term profits and thus visions – due to their dependency on the short-sighted stock holders – rather than (potentially much higher)  long-term ones.

The culture in government agencies is often little suitable for agile development, but it is possible to a feasible way of introducing agile there by breaking projects into small, few-months ones or perhaps by using a hybrid agile approach, as described in the comments by Dylan Jay.

Follow-Up

I’d highly recommend you to read the original article and at least the following three most interesting comments – the lawyer’s reaction to the comments, an overview of the DSDM method by the DSDM Consortium’s Technical Director Andrew Craddock, and Dylan Jay’s description of a successful hybrid agile project. You should also consider looking at the follow-up post by Nik Silver, Agile government IT can succeed.