Why scrum works??

The evidence indicating that Scrum is a more productive methodology is overwhelming. Companies report substantial growth in productivity and quality when transitioning from waterfall to agile. I am myself a strong proponent of Scrum and use it on a daily basis but I find the root causes of the experienced performance increases blurred.

A plethora of explanations for the effectiveness of Scrum have been offered, the most conspicuous of which is that the procedures, the planning and the artifacts in the Scrum framework constitute a more effective method of execution and that this is the primary contributing factor. Smaller teams doing their own planning and given tools for tracking progress is by this argument more effective than large teams in bureaucratic project formation.

The Hawthorne effect or combinations of the Hawthorne and the Placebo effect has been suggested as explanations for the recorded productivity increases. While the Hawthorne effect is usually quite short lived one could argue that the continuous improving and thus changing nature of the process could prolong or even perpetualise the effect.

If Placebo where to account for the productivity increases the argument would probably be that it is general knowledge that implementing Scrum is usually associated with a productivity increase and also that the implementation process and scrum master training is often littered with success stories told or written by industry experts. The implication being that if implementation does not result in substantial productivity increases, the Scrum is not done “Correct” and Scrum will, as a curious side effect, by logic convention always be successful . This should place strong expectations on the organization implementing scrum and could help making the success prophecies self-fulfilling.

In my opinion, Scrum is not driven by means of Placebo or Hawthorne although I do believe both effects attribute to the success of the methodology. Neither do I think that the methodology is the main contributing factor. My favorite is that the main reason Scrum results in performance increases is the immense positive effect it has on employee motivation. Herzbergs two factor theory claims that some factors, if present, contribute to job satisfaction while the absence of others lead to dissatisfaction. That is, a few basic things (hygiene factors) like fair salary and supervision will cause demotivation when not present but will not by themselves motivate employees while other factors (motivators) such as achievement, responsibility and personal growth will motivate employees. Given that the hygiene factors should be present in most modern well driven companies (even those using waterfall) any motivation based performance gains must be sought in the motivators. I find that the strong overlap between the values of the agile manifesto and Herzbergs motivators are extremely indicative as to the root cause of the experienced performance improvements.

To the best of my knowledge there is no strong research substantiating or quantifying the contributions of the many theories to the reported productivity improvements and I find this quite unsettling. We are faced with a brilliant framework that seems to work but we have no clear understanding of why.

We need this knowledge. Firstly because the scrum cake may not be fully baked yet, adding more chocolate chips may improve the outcome, but we don’t understand which ingredients result in the great taste so we are flying blind. Secondly, because Scrum and Agile are methodologies based on learning and empirical knowledge we should not accept basing it on assumptions.

It should be possible to design experiments that would allow for the quantification of at least some of the contributing factors and I believe that the next improvements of Agile requires this more specific knowledge of the contributing factors. I also think that this knowledge will help to further expand the use of Agile methodologies.