Imagine a weekly report celebrating hundreds of AI drafts and a fully occupied virtual assistant. Customers are still waiting, and the manager spends Friday correcting the drafts. The activity report has missed the result.
Hours still matter for pay, timekeeping, workload and costing. Keep those records. The change is to assess operational performance through useful outcomes as well as time spent.
Choose one workflow, an accountable owner and an acceptance rule before measuring it. An accepted result might be a customer update checked against its sources and approved for release. Define who may accept it; professional or commercial approvals stay with the appropriately authorised person.
The six KPIs below are suggested management definitions. All numerical examples are hypothetical, with no industry targets or claimed Nexus results. Record task type, complexity, AI involvement and review stage so comparisons remain meaningful.
1. Cost per accepted result
Formula: total attributable cost for a defined batch ÷ unique accepted results from that batch.
Include preparation, human checking, correction, software usage and an explicitly allocated share of training and maintenance. Include costs of unsuccessful work in the same batch. Counting only the VA's handling time can hide work transferred to the local reviewer.
Suppose a closed batch costs $1,080 and produces 90 accepted results. The cost is $12 per accepted result. If another 30 drafts were discarded, their preparation costs still belong in the numerator; they add nothing to the denominator.
Document allocation rules and avoid counting a subscription twice. Mark an unfinished batch's cost as provisional. With no accepted results, show “no accepted output” and the cost incurred; the ratio is undefined.
2. First-pass accuracy
Formula: outputs meeting the agreed accuracy criteria at first review ÷ all outputs first reviewed × 100.
If 68 of 80 first-reviewed outputs pass, the rate is 85%. Count each output once. State the checkpoint: a raw AI draft and the VA-checked version are different stages. Blending them hides where quality improves or fails.
Assess substantive correctness against sources, including names, facts and authorised commitments. Classify stylistic preferences separately. For sampled reviews, report sample size and selection method; the result describes that sample.
The OAIC recommends human oversight and a person responsible for verifying personal information obtained through AI. An AI-generated score alone is a weak acceptance basis. OAIC: accuracy and human oversight.
3. Backlog age
Formula for each open item: reporting timestamp minus its original entry timestamp into the measured queue.
Use calendar days for this example. For five open items aged 1, 2, 3, 7 and 12 days, the median is three days and the oldest is 12. Show both, alongside the number of open items. A pleasant median can coexist with a neglected request.
Preserve the entry time when ownership changes or the same unresolved request is reopened. Record customer and external dependencies as reasons alongside the elapsed age. This keeps the customer's delay visible without treating every waiting day as staff effort.
A VA can maintain the records; the queue owner decides priorities and escalations. For solar-specific stage measures, see the solar job dashboard guide.
4. Customer wait time
Formula: timestamp of the first substantive response minus timestamp of the customer's request.
Define a substantive response before reporting: one that addresses the request or gives a supported next action. An automatic receipt message does not qualify under this definition. First response is also separate from final resolution; retain a resolution timestamp for longer cases.
A request received Tuesday at 10 am and answered Wednesday at 2 pm has waited 28 calendar hours. Use timestamps in a consistent time zone and state whether your report uses calendar or business hours.
Report the median for requests receiving their first substantive response in the period, plus the age of still-unanswered requests. Excluding those open requests from view can make performance appear better as the queue grows. Check response quality alongside speed.
5. Rework effort
Formula: active correction and repeat-checking hours ÷ total active workflow hours × 100, with those correction hours included in the total.
If the workflow takes 50 hours, including eight hours fixing preventable defects, rework effort is 16%. Initial checking is ordinary processing; repeat work caused by a defect belongs in rework. Agree that distinction with the team.
Keep new customer instructions and genuine scope changes separately classified. Record the cause of each correction: incorrect source data, an AI omission, unclear instructions or a missed human check. One error can cross several hands, so investigate the chain before assigning blame.
Use this measure beside first-pass accuracy. A small number of complex corrections may consume considerable time even when most outputs pass. Cost per accepted result already includes these hours; do not add them again.
6. Escalation quality
Formula: reviewed escalations meeting the agreed escalation rubric ÷ all escalations reviewed × 100.
An original rubric could require the correct recipient, a clear decision requested, relevant source links, known consequences and timely escalation against your internal rule. If 18 of 20 reviewed escalations meet every criterion, quality is 90%. Record failed criteria for coaching.
Separately sample ordinary cases for issues that should have been escalated but were missed. Otherwise a team can improve its reported score by raising only easy exceptions. Low escalation volume is not automatically a success.
For an AI draft promising an unapproved discount, good escalation preserves the source discrepancy and directs the pricing decision to its owner. The hybrid team responsibility chart helps define that route.
Read the measures together before changing the process
NAIC's implementation guidance recommends acceptance criteria, performance monitoring and regular review of AI systems. These six KPIs support a business discussion; they do not replace broader testing or oversight. NAIC implementation guidance, practice 5.
Compare similar work using stable definitions. Record tool, prompt or process changes alongside the reporting period. The published Generative AI at Work study found effects varied across workers in its customer-support setting; its results are not a universal benchmark for your team. Read the study.
At each review, choose one observed problem, an owner and a follow-up date. Talk to Nexus about VA support to define the workflow and accepted result your team will measure first.
