Qualitative metrics and quantitative metrics are complementary approaches to measuring developer productivity.
Qualitative metrics, which are typically derived from surveys, provide a holistic view of productivity that includes both subjective and objective measurements.
Quantitative metrics, on the other hand, provide an objective view of development activity based on data from your engineering systems.
So how do you use qualitative and quantitative metrics together?
Our recommendation – and the approach being built into DX – is to start with qualitative metrics to figure out where to focus. Then to use quantitative metrics to drill in deeper into specific areas.
There are a couple of reasons for this.
First, qualitative metrics provide a much more holistic view from which you are able to identify opportunities, whereas quantitative metrics are typically only available for a narrow set of areas of the development process.
In our recent paper we explain:
Survey data can be collected quickly and, when designed correctly, provide fast and accurate measurements to establish baselines and guide improvement efforts. To augment surveys, organizations should also collect data from systems. Getting end-to-end system data can be difficult, however, requiring instrumentation and normalization of data across disparate tools and teams.
Second, quantitative metrics by themselves lack the context needed to assess whether something is good or not. Google similarly advises its engineering leaders to go to survey data first before looking at logs data for this reason.
We encourage leaders go to the survey data first, because if you just go look at logs data, logs data doesn't really tell you whether it's good or bad. For example, we have an Active Coding Time metric that tracks the time to make a change, but that number is useless by itself. You don't know, is this a good thing? Is it a bad thing? Do we have a problem?
We can focus our attention and efforts by using each type of measurement based on its strengths. To break it down:
- Start with qualitative data to identify your top opportunities
- Once you know what you want to improve, use quantitative metrics to drill-in further
- Track your progress using both qualitative and quantitative metrics
There are exceptions to this approach, of course. For example, if you are looking for an answer to a specific question, it makes sense to go directly to a quantitative metric if one is available.