X: “We need objective metrics on individual developer productivity and performance!”
Me: “Silly question, I know, but what exactly do you mean by objective?”
X: “Um…you know, objective. Unbiased. Facts. Not just one person’s opinion.”
Me: “I ask because a metric like # of pull requests, for example, might be considered objective, but it has low construct and content validity if you're hoping to measure developer performance.”
X: “Construct what? Content what? Why is this so hard!? Engineering isn’t magic!”
Me: “Construct validity is basically how well the measurement tool measures the theoretical concept it claims to measure. Content validity is basically how well the measurement covers the full range of the concept it's supposed to measure.”
X: “OK. But what if # of pull requests basically correlates with someone being good?”
Me: “Imagine you're a manager. Would asking a developer to create as many pull requests as possible be guaranteed to turn them into a high performer? Would it guarantee your team was effective? This is about criterion-related validity actually."
X: “Whenever I have this discussion, it feels like engineers are so elitist and don't want to take responsibility.”
Me: “In a lot of domains, performance is measured by gathering the perspectives of more experienced people and/or people who work with the person. For example…peer reviews, evaluating an athlete, peer critiques/feedback sessions, the list goes on.”
X: “Ugh. Here we go. But what if managers are covering their behinds? What if they aren’t close to the details? Can’t we have something more objective?”
Me: “Got it. It feels like you have a vague sense that the team could be doing more, but you don’t feel like you’re getting the straight story? Hence wanting the metrics?”
X: “I *know* we could be getting more done. But all I seem to get from managers is finger pointing and making excuses. Look, I’m under a lot of pressure here from finance….”
Me: “If you know, then is this a problem?”
X: “I know it. But I need to prove it. I need it to be objective!”
Me: “So you know it subjectively, but you need to prove it objectively? OK. So that’s a different problem. Your goal is to have evidence people can agree on as a way to ask the team to work harder, and take action if they aren’t.”
X: “You got me….”
Me: “Taking a slightly different approach—it sounds like this is about building confidence that we’re working efficiently and having an impact? And improving? I ask because that is a different measurement problem. If your goal doesn’t involve singling out individual productivity, we could probably figure something out. Measuring to improve is different from measuring to prove.”
X: “But….”
