The pull request appeared before lunch. A similar change used to take most of a day, so the first conclusion seemed obvious: the AI assistant had made the team dramatically faster.
Then review began. A senior engineer found a missed authorization case. The generated tests covered the happy path but not the failure that mattered. The author revised the change twice, and the release moved to the next morning.
Was the team faster? Possibly. But “the first draft arrived sooner” is not the same claim as “the accepted change required less effort.”
To evaluate engineering effort after LLM adoption, measure the complete path to an accepted outcome. Keep human effort, elapsed delivery time, review and rework, and quality as separate measures. Compare similar work over a defined period, and record how the work mix changed. Do not infer productivity from prompts, generated lines, commits, or an assistant's estimate of time saved.
Do not use raw hours saved or added as an individual productivity score. Task selection, system knowledge, review responsibility, quality risk, and willingness to report rework all affect the total. Use the measures to evaluate a workflow or task family, not to rank engineers.
This guide is for delivery managers and engineering leads running a small internal evaluation. Its worked figures are fictional. They demonstrate the method; they are not a benchmark for a particular model, team, or product.
Download the two-week AI engineering-effort evaluation spreadsheet to follow the worked method with one row per accepted task.
Why the old time total is no longer enough
LLMs can move effort rather than simply remove it. A person may spend less time typing an implementation and more time:
- framing the problem and supplying context;
- comparing generated approaches;
- reviewing code they did not write line by line;
- expanding tests around plausible but incorrect behavior;
- correcting or replacing a weak result;
- coordinating several parallel agent tasks;
- documenting why the accepted result is safe.
All of that is work. Some of it creates more value than manual production did; some is avoidable rework. One undivided time total cannot explain the difference.
The research does not support one universal “AI productivity” percentage. A randomized trial with 96 Google engineers estimated that AI shortened time on one complex enterprise task by about 21%, with a wide confidence interval and explicit limits on generalizing the result. Read the published trial.
In a different randomized study, 16 experienced open-source developers completed 246 tasks in mature repositories and took 19% longer when early-2025 AI tools were allowed. METR stresses that this result describes that setting, not most software work. Its early-2025 study and 2026 methodology update also show why fast-changing tools and task-selection effects make later comparisons difficult.
The useful management conclusion is not that AI always helps or always hurts. It is that your team needs local evidence tied to the work it actually accepts.
Measure four different questions
Do not make one metric carry four meanings.
| Measure | Question it answers | What it does not prove |
|---|---|---|
| Human effort | How much active person-time did the accepted work require? | How quickly it moved through the system |
| Elapsed cycle time | How long passed from the agreed start to acceptance? | How many people or active hours were involved |
| Review and rework | How much effort moved into checking and correction? | Whether that effort was wasteful or necessary |
| Outcome and quality | Did the change meet its acceptance and reliability conditions? | Whether the result was economical |
Add value or customer impact when you can define it credibly. A fast internal prototype, an accepted customer feature, and a low-value report generated because it became cheap are not equivalent outputs.
METR's task-substitution analysis explains why measuring old tasks, new tasks, and value can produce different estimates of AI uplift. Once teams can attempt work they previously would not have chosen, a before-and-after count of tasks no longer compares the same basket.
Divide human effort into five practical categories
Keep the categories broad enough that people will use them consistently.
| Category | Include | Example |
|---|---|---|
| Framing and context | Understanding the request, finding evidence, preparing constraints | Reading the affected service and defining non-negotiable behavior |
| Creation | Writing or generating the first workable implementation | Manual coding, prompting, steering an agent, editing generated output |
| Review and testing | Inspecting, running tests, validating behavior and security | Reviewing a generated migration and exercising rollback |
| Rework | Correcting rejected, incomplete, or misleading output | Replacing a fabricated API call or fixing a missed edge case |
| Coordination | Handoffs, approvals, resolving parallel changes | Reconciling two agent branches or clarifying acceptance with a product owner |
The goal is not minute-by-minute surveillance. For a short pilot, a contributor can record one rounded total per category when the work item reaches acceptance. If a category is difficult to distinguish reliably, combine it with a neighboring category and document the rule.
Count active human work once. If an agent runs for 20 minutes while the engineer reviews another item, do not add 20 minutes to both tasks as human effort. Track model runtime or usage cost separately when it matters financially; it is not another person's labor time.
Classify human review and rework consistently
Review is time spent deciding whether output is acceptable. Rework is time spent changing rejected, incomplete, or unsafe output. Use the same categories in both baseline and AI-assisted weeks:
| Code | Human review or rework category | Count as review when… | Count as rework when… |
|---|---|---|---|
| Correctness | Requirements and behavior | A person checks requirements, logic, data handling, or API use | A person fixes wrong behavior, fabricated APIs, or missed requirements |
| Tests | Tests and edge cases | A person evaluates coverage, failure paths, fixtures, or test meaning | A person adds or repairs tests because the first output was insufficient |
| Security | Security and privacy | A person checks authorization, validation, secrets, or sensitive data | A person corrects a security or privacy weakness found during review |
| Integration | Architecture and maintainability | A person checks compatibility, operability, readability, or documentation | A person restructures output so it integrates safely and remains maintainable |
Record post-acceptance defect correction separately during the chosen follow-up window. A reviewer reading a generated change is review; the author fixing the finding is rework; repairing a related production defect after acceptance is defect correction. Do not log the same interval in more than one category.
Calculate engineering time saved—or added
First calculate active human effort for every accepted task:
Initial human effort = framing + active generation/creation + review + rework + coordination Quality-adjusted human effort = initial human effort + defect-correction effort during the follow-up window
Then compare medians within the same task family, complexity band, acceptance standard, and quality window:
Net human hours saved = baseline median quality-adjusted effort - AI median quality-adjusted effort Relative human-effort change = net human hours saved / baseline median quality-adjusted effort Net elapsed hours saved = baseline median elapsed time - AI median elapsed time
A positive result means time was saved; a negative result means the AI-assisted workflow added time. If the baseline median is in spreadsheet cell B2 and the AI-assisted median is in C2, the copyable formulas are =B2-C2 for net hours saved and =(B2-C2)/B2 for relative change.
Report review, rework, defect count, and defect-correction effort beside the headline. A lower generation time can otherwise conceal a larger assurance cost.
Worked baseline-versus-AI example
Two medium-complexity API validation changes reach the same acceptance standard. The examples are deliberately similar, but they are not statistically meaningful on their own.
| Measure | Baseline task | AI-assisted task | AI-assisted minus baseline |
|---|---|---|---|
| Framing and context | 1.0h | 0.75h | −0.25h |
| Active generation or creation | 3.5h | 1.25h | −2.25h |
| Unattended model runtime | 0h | 0.30h | +0.30h, excluded from human effort |
| Human review | 1.5h | 2.25h | +0.75h |
| Human rework | 0.5h | 1.25h | +0.75h |
| Coordination | 0h | 0.25h | +0.25h |
| Defect-correction effort within 14 days | 0h | 0h | 0h |
| Quality-adjusted human effort | 6.5h | 5.75h | −0.75h |
| Defects or hotfixes within 14 days | 0 | 0 | 0 |
| Total elapsed time to acceptance | 8.0h | 5.0h | −3.0h |
The AI-assisted task used 2.25 fewer active generation/creation hours and 0.25 fewer framing hours, but 1.75 more review, rework, and coordination hours. Quality-adjusted human effort fell by 0.75 hours, not 2.25. Cycle time improved more than effort because the work also waited less.
Using the comparison formula, 6.5h - 5.75h = 0.75h saved, or 0.75 / 6.5 = 11.5% less quality-adjusted human effort. Elapsed time fell by three hours. Both tasks had zero observed defects in the 14-day window, so the example does not hide later correction effort.
That may be a worthwhile result. It may also disappear across the next ten tasks. The table tells the team what to test next; it does not justify an organization-wide forecast.
Download the two-week AI engineering-effort evaluation spreadsheet. It contains the two fictional example rows plus five baseline and five AI-assisted rows for a pilot. Review and rework have separate correctness, test, security, and integration columns; unattended model runtime is kept outside human effort.
Run a two-week evaluation without rebuilding the timesheet system
Before the pilot
Choose one or two task families where “done” can be checked consistently, such as small API changes, test additions, documentation updates, or routine data transformations. Exclude emergency incidents and unfamiliar migrations from the first comparison.
Write down:
- the task family and complexity bands;
- the start and acceptance events;
- the quality guardrails;
- which AI modes are in scope;
- how active human effort is rounded;
- who can see the pilot data and how it may be used.
Do not use the pilot for individual performance ranking. People differ in system knowledge, task choice, review responsibility, and willingness to use experimental tools. Ranking them encourages under-reporting of review and rework—the two categories the evaluation most needs to see.
Week one: establish or clean the baseline
Use recent comparable work if its scope and records are credible. Otherwise, record one week using the new categories before changing the tool policy.
For every included item, retain:
- task family and complexity band;
- original acceptance conditions;
- human effort by category;
- elapsed time to acceptance;
- review outcome and major changes;
- a short quality follow-up window.
If the baseline tasks differ materially from the assisted tasks, report that limitation. Do not “normalize” a risky migration into the same category as a small test update because both closed in one day.
Week two: observe AI-assisted work
Keep the same definition of acceptance. Record the assistance mode at a useful level—such as inline completion, conversational assistant, or delegated coding agent—without collecting prompt counts or confidential prompt text.
Ask contributors for a short note only when the work distribution changed materially:
“Initial generation was quick, but review expanded because the change touched an undocumented authorization path.”
That sentence is more useful than “used 37 prompts.” It explains why the time moved.
End-of-pilot review
Compare medians within each task family and complexity band. Small samples will still be noisy, so show the task count and range rather than publishing a precise percentage alone.
Review in this order:
- Did acceptance and quality remain comparable?
- Did elapsed cycle time change?
- Did total human effort change?
- Which effort categories moved?
- Did the team start doing a different mix of work?
- What friction should be removed before the next pilot?
The answer may be “faster for routine test creation, neutral for product changes, and slower for deeply contextual repository work.” That is more actionable than one company-wide multiplier.
Use Jira time data without overstating what it means
Atlassian documents that Jira worklogs record time spent and support comparison with the original estimate. The remaining estimate can be adjusted as work changes. Jira's time-logging guidance explains those fields and permissions.
Those records can support the human-effort side of the evaluation if the team already logs work consistently. They do not automatically tell you:
- whether an LLM was used;
- which effort category consumed the time;
- when the work became acceptable;
- whether a later defect was related;
- how valuable the outcome was;
- whether the original estimate remained an intact baseline.
For the pilot, use the downloadable worksheet or a deliberately configured reporting method rather than forcing every experimental field into permanent Jira configuration. Keep the accepted issue and worklog references so totals can be reconciled.
If estimates also need review, preserve the original baseline and update the remaining forecast separately. The Jira estimated-versus-actual guide shows why spent effort and current forecast answer different questions.
Do not use raw hours as an individual productivity score
An engineer with more review hours may be protecting the system from a higher-risk change or reviewing other people's output. An engineer reporting rework honestly should not look worse than someone who omits it. Aggregate matched tasks at the team or task-family level, retain quality outcomes, and prohibit the pilot data from being used for compensation, performance ranking, or stack ranking.
Generated lines or lines of code
More code can mean duplicated logic, unnecessary abstraction, or a larger maintenance burden. Less code can represent deletion, reuse, or a task that never needed implementation.
Prompt, token, or agent-run counts
These describe tool interaction or cost. They do not establish effort saved, quality achieved, or customer value.
Commits and pull requests
Task splitting and repository conventions can change counts without changing outcomes. AI may also make it cheaper to open speculative changes that never should be merged.
Acceptance rate alone
A high suggestion-acceptance rate can coexist with later rework. A low rate can reflect deliberate exploration. Treat it as a tool diagnostic, not a business result.
Self-reported time saved alone
Perception matters for developer experience, but it is not the same as observed task time. Use a short experience survey alongside work records rather than converting confidence into saved hours.
The SPACE framework argues that developer productivity cannot be understood through one metric or dimension. DORA's 2025 measurement guidance similarly recommends choosing frameworks for the organizational question and combining AI-specific measures with existing delivery and experience measures.
Turn the result into an operating decision
At the end of the pilot, choose a narrow action:
- expand one AI-assisted task family;
- improve context or testing before expanding;
- require a stronger review path for generated changes;
- stop using the tool for a task family where rework dominates;
- continue measuring because the sample is inconclusive.
DORA's 2025 AI-assisted software development report describes AI as an amplifier of the surrounding organizational system. That is a useful review question: did the tool remove an existing bottleneck, or did it create more work at a weak review, testing, or deployment stage?
Do not turn a pilot result directly into lower estimates for every future task. Update planning assumptions only after repeated comparable evidence, and keep uncertainty visible.
Common questions
Should prompting and reviewing AI output count as work?
Yes, when a person is actively framing, directing, inspecting, testing, or correcting the output. These are human-effort activities. Do not add unattended model runtime as though another employee worked those minutes.
Should AI-assisted work be billed differently?
That depends on the customer agreement and pricing model. Logged human effort, model usage cost, fixed-price value, and billable time are separate concepts. Do not infer the contractual answer from the measurement worksheet.
Should we immediately reduce estimates after a successful pilot?
No. One pilot may reveal a useful task-specific change, but future scope, complexity, model capability, and team familiarity will vary. Preserve the original estimate, maintain the remaining forecast, and update future baselines after repeated evidence.
Can Jira show whether AI improved productivity?
Not by itself. Jira can hold issues, estimates, worklogs, workflow history, and acceptance evidence. Causality requires a comparison design and careful interpretation outside any single field.
Keep the reporting promise modest
TimePillar can be evaluated for rolling native Jira estimates and logged time across related work. That helps reconcile the effort side of a project or epic. It does not identify AI use, calculate causal productivity, or prove that saved hours created more value.
For period review and corrections, use the Jira timesheet and worklog guide. If AI is also drafting the management summary, follow the Jira AI status-update checklist before presenting a productivity conclusion to leadership.
Next steps
- Running the evaluation? Download the two-week AI engineering-effort spreadsheet and begin with one task family.
- Need explainable project totals? Test TimePillar for Jira against the same accepted issue set.