The question on every board agenda

Your teams have AI tools. The licenses are paid. And at some point someone on the board asks the obvious question: is it working? Counting licenses or active users does not answer it. Neither do the vendor claims you read online.

Two papers from DX, the engineering intelligence company, are the best answer we know. Measuring AI code assistants and agents, by Abi Noda and Laura Tacho, sets out the DX AI Measurement Framework. The AI strategy playbook for senior executives, by Justin Reock, turns it into a leadership plan. Here is what they say, in our words, and what it means for you.

One finding should shape every AI budget. The playbook shows that the average gains from AI are positive but modest – and that the averages hide the real story. Broken down by company, some companies see change confidence rise by twenty points with AI, while others see it fall by twenty. Some organizations get much better with AI. Others get worse. The tool is the same; the leadership around it is not.

Five things only leadership can do

Measure teams, never individuals

Metrics such as lines of AI-generated code are easy to game. Use them to judge people and you get compliance instead of results, and the data becomes worthless. DX recommends saying plainly that AI metrics stay out of performance reviews, that the purpose is to understand what helps, and that the data decides where the money goes.

Count agents as part of the team

An agent’s pull request belongs to the team that directs it. That is how the framework counts throughput, and it hints at where the work is going: every developer will increasingly lead a small team of agents, and be measured more like a manager is today.

Balance speed with quality

Speed gains in the first months can turn into maintenance debt later. Track change failure rate, maintainability and confidence alongside throughput. Keep code review mandatory on production paths, and keep your tests and linting running unchanged or stronger.

Make it safe to learn

The playbook puts psychological safety first, and grounds it in Google’s Project Aristotle: people who fear for their jobs don’t experiment. Frame AI as a way to augment your engineers, give them time to learn inside the sprint, and expect a dip before the gains. Pilots are safe-to-fail experiments, not mandates.

Look past code generation

For most organizations, writing code was never the bottleneck. The biggest returns in the playbook’s examples come from code review agents, incident handling, legacy refactoring and documentation – wherever the team’s time is actually stuck. Find the constraint first, then apply AI to it.

Where XALT comes in

XALT is an Atlassian Platinum Solution Partner, for DX too, with specializations in Software Development, Service Management (ITSM) and Cloud Migration. We introduce DX in your engineering organization and set up the measurement the papers describe: the baseline before the rollout, the framework’s three dimensions, and the communication that keeps your teams’ trust.

The measurement then tells you where to go next. It shows which teams already benefit, where the bottleneck sits, and which step on the journey to agentic engineering pays off first: a developer self-service, a DevEx platform, or agents that write the code and the tests and wait for a person’s approval.