Product development operational reviews

At Zmags we have a quarterly and yearly planning cycle. It works really well, it’s well managed and it ensures that everybody at senior management level is on the same page. Part of the quarterly planning session is operational reviews of all functional areas of the organization. The original framework that we employ for the operational reviews was an adaptation of a sales department operational review kindly provided by the good people at OpenView. I’ve used it for a while and it’s been tweaked a fair bit along the way. I recently had a chance to see the draft operating review framework specifically for product development that Igor Altman at OpenView is creating (and as usual it’s really good).
Seeing Igor’s work got me thinking about how I do operational review presentations and the purpose of them. The way I see it, the purpose of the operational review is to analyze the inner workings and activities of a department or functional area with the purpose of identifying problems and potential improvements.
This is all well and good but the problem here is that at executive and board level only few people have (and should imho have) complete domain knowledge. This means the presenter will be presenting a specific view port into the problem domain and is forced to translate the specifics into commonly understandable terms.
So the tradeoff becomes presenting a too high level view that everybody understands but does not facilitate the identification of improvements or presenting a too detailed view that isn’t understood and causes unnecessary concern and pushes focus towards the KPI’s instead of their meaning.
In addition to the overall purpose of the review I find that the good ones also serve the following purposes:

• Provide executives with an overview of the past and planned operations of the department so that they can plan for upcoming cross departmental activities
• Provide knowledge and ensure that planned operations of the reviewed department is in line with the overall company strategic themes
• Identify areas of possible cross departmental synergy (given the above bullet there should be plenty)
• Instill confidence that you know what you’re doing (if that is the case) and that they don’t need to worry about the things you don’t draw their attention to
  

In general I try to make my review documents substantially shorter and put only a few bullets in writing on every slide. I think it provides a substantial advantage to start my presentation at a slightly higher level and then dive into the underlying KPI’s whenever necessary. In other words, if there really isn’t a quality problem I wouldn’t dive deep into low level quality metrics unless asked and in that case I would have those KPIs in a separate presentation. It (I hope) gives my colleagues the confidence that no matter what KPI they feel like drilling into, I know and have the data and that allows me to fast forward through some of the areas in which I have a strong believe that a lengthy discussion would be futile.
In summary I tend to stick to these basic rules

• Provide reasonably few metrics that highlight problem areas (I provide 5-10)
• Have the underlying KPI’s for each top level metric ready but don’t show them unless necessary (I try to have additional KPI’s ready in a separate presentation or dive right into the reporting systems)
• Prefer meaningful graphs over tables
• Weird domain specific acronyms that I don’t know, makes me feel like a jerk. I assume this is a general thing so I try to avoid them
• Highlight risks
• Listen! – This is your chance to pick the brains of the most expensive coworkers in your company. Don’t waste the opportunity
• Be exceptionally honest (this is your chance to tell them that the multiple-inheritance multi-apartment threaded COM object based architecture that your architect came up with wont be supported in the next version of windows (btw: the version for which the next release is targeted at))
  

Why scrum works??

The evidence indicating that Scrum is a more productive methodology is overwhelming. Companies report substantial growth in productivity and quality when transitioning from waterfall to agile. I am myself a strong proponent of Scrum and use it on a daily basis but I find the root causes of the experienced performance increases blurred.

A plethora of explanations for the effectiveness of Scrum have been offered, the most conspicuous of which is that the procedures, the planning and the artifacts in the Scrum framework constitute a more effective method of execution and that this is the primary contributing factor. Smaller teams doing their own planning and given tools for tracking progress is by this argument more effective than large teams in bureaucratic project formation.

The Hawthorne effect or combinations of the Hawthorne and the Placebo effect has been suggested as explanations for the recorded productivity increases. While the Hawthorne effect is usually quite short lived one could argue that the continuous improving and thus changing nature of the process could prolong or even perpetualise the effect.

If Placebo where to account for the productivity increases the argument would probably be that it is general knowledge that implementing Scrum is usually associated with a productivity increase and also that the implementation process and scrum master training is often littered with success stories told or written by industry experts. The implication being that if implementation does not result in substantial productivity increases, the Scrum is not done “Correct” and Scrum will, as a curious side effect, by logic convention always be successful . This should place strong expectations on the organization implementing scrum and could help making the success prophecies self-fulfilling.

In my opinion, Scrum is not driven by means of Placebo or Hawthorne although I do believe both effects attribute to the success of the methodology. Neither do I think that the methodology is the main contributing factor. My favorite is that the main reason Scrum results in performance increases is the immense positive effect it has on employee motivation. Herzbergs two factor theory claims that some factors, if present, contribute to job satisfaction while the absence of others lead to dissatisfaction. That is, a few basic things (hygiene factors) like fair salary and supervision will cause demotivation when not present but will not by themselves motivate employees while other factors (motivators) such as achievement, responsibility and personal growth will motivate employees. Given that the hygiene factors should be present in most modern well driven companies (even those using waterfall) any motivation based performance gains must be sought in the motivators. I find that the strong overlap between the values of the agile manifesto and Herzbergs motivators are extremely indicative as to the root cause of the experienced performance improvements.

To the best of my knowledge there is no strong research substantiating or quantifying the contributions of the many theories to the reported productivity improvements and I find this quite unsettling. We are faced with a brilliant framework that seems to work but we have no clear understanding of why.

We need this knowledge. Firstly because the scrum cake may not be fully baked yet, adding more chocolate chips may improve the outcome, but we don’t understand which ingredients result in the great taste so we are flying blind. Secondly, because Scrum and Agile are methodologies based on learning and empirical knowledge we should not accept basing it on assumptions.

It should be possible to design experiments that would allow for the quantification of at least some of the contributing factors and I believe that the next improvements of Agile requires this more specific knowledge of the contributing factors. I also think that this knowledge will help to further expand the use of Agile methodologies.