Why AI Model Benchmarks Fail to Reflect Real Usage Costs

Alibaba recently unveiled its Qwen 3.8-Max model, positioning it as a leader among AI systems. However, independent tests using the VulcanBench framework revealed results were highly sensitive to testing conditions. While Alibaba allowed models up to 12 hours to complete tasks, VulcanBench imposed strict 45–60 minute limits. This discrepancy led to Qwen 3.8-Max delivering average results in standard settings and poor performance under default configurations.

The core issue lies in traditional metrics like cost-per-token, which fail to capture real operational costs. For instance, models may expend excessive tokens on reasoning without reaching a conclusion, resulting in empty responses and full charges for failed attempts. Research by Artificial Analysis found that DeepSeek-V4-Flash used twice the average token count at maximum settings, despite maintaining low costs due to cheap tokens.

Experts advocate shifting to cost-per-success metrics that account for all expenditures, including failed attempts. This is critical for reasoning-heavy models where token or time overruns often cause failures. In Long-Horizon-Terminal-Bench tests, 79% of failures stemmed from exceeding time limits rather than model errors.

Standard request routing approaches also prove inefficient. Escalating to pricier models after failures doesn’t guarantee better results—in some cases, it merely inflates budgets without improving outcomes. Companies like HubSpot and Zendesk have already adopted pay-per-success models to better evaluate AI efficiency.

To improve transparency, tracking failure causes—such as budget overruns or verification errors—is essential. This helps pinpoint which aspects of model performance require optimization. Default settings may also be suboptimal; Qwen 3.8-Max’s default maximum reasoning level underperformed in independent tests.