GPT-5 Benchmarks Reveal 23x Cost Spikes and Mixed Test Scores

Independent evaluations from benchmark testing organizations Artificial Analysis, Vellum, and BenchLM show OpenAI's GPT-5 model series exhibits dramatic cost and token variations based on user settings. Testing across reasoning levels reveals up to a 23-fold difference in token usage and cost between the high and minimal reasoning effort modes.

OpenAI's flagship model series—which includes GPT-5, GPT-5 mini, GPT-5 pro, GPT-5.4 mini, and GPT-5.6 Sol—features an expanded 400,000-token context window and a 128,000-token output limit. The models have been generally available since August 2025 across ChatGPT, ChatGPT Pro, the OpenAI API, and integrations like Slack via Runbear.

Reasoning Effort Drives Massive Token Variance

To manage compute requirements across diverse workloads, OpenAI implemented router-based reasoning effort configurations spanning minimal, low, medium, and high. Setting the reasoning effort to high enables extended internal thinking steps prior to generating answers, delivering superior accuracy on complex tasks at the expense of heavy token consumption.

Conversely, selecting minimal reasoning operates closer to GPT-4.1 intelligence levels while offering significantly higher token efficiency. This allows developers to balance operational expenditure against required technical precision.

Benchmark Performance Hits Highs and Lows

Related: Critical Analysis Challenges AI Hype Surrounding Security Claims and Mathematical Breakthroughs

In technical evaluations, GPT-5 demonstrated strong problem-solving capabilities across mathematical and programming assessments. The model recorded a 100% score on AIME 2025 math challenges, 89.4% on GPQA Diamond, and 74.9% on SWE-bench Verified, marking the highest software engineering score among major foundation models in 2026.

Despite these achievements, performance was not uniform across all evaluation frameworks. On SimpleBench, GPT-5 registered a lower score of 56.7%, prompting discussion among community analysts regarding whether synthetic tests accurately measure practical utility.

It's also a really dumb benchmark. There is very little from this benchmark that would translate into the model being useful in the real world.

Community debate on r/singularity regarding SimpleBench results

How GPT-5 Reasoning Configurations Compare

The system routes prompts dynamically based on user-defined reasoning parameters. High-effort modes allocate extra processing depth to reduce logic errors, whereas minimal settings skip extended processing entirely.

API Pricing Adjustments and Claims

On August 21, 2026, OpenAI lowered API and credit pricing for GPT-5.6 Sol by over 20% for a three-month promotional window. The price reduction aims to encourage wider adoption of the specialized variant among API developers.

OpenAI's system documentation asserts that GPT-5 hallucinates roughly 80% less than GPT-4o. However, this specific reduction claim remains unverified by third-party benchmarking organizations.

Managing Costs and Workloads

Developers using the OpenAI API can control expenditure by specifying the reasoning_effort parameter during call construction. Assigning high reasoning strictly to software engineering or multi-step logic tasks prevents unexpected billing spikes on standard chat interactions.

Model Limitations

While GPT-5 leads on software engineering benchmarks, high-reasoning configurations incur significant latency and token costs. Furthermore, its lower 56.7% score on SimpleBench underlines potential limitations in specific reasoning paradigms.

Sources

• GPT-5.6: Frontier intelligence that scales with your ambition
• GPT-5 Benchmarks and Analysis
• GPT-5 Benchmarks - Vellum
• GPT-5 scores a poor 56.7% on SimpleBench, putting it at ...
• Pricing | OpenAI API
• GPT-5 in 2026: Features, Benchmarks, Pricing, and How to ...
• GPT-5 mini Benchmarks, Pricing & Speed (September 2026)
• OpenAI gave us early access to GPT-5: our independent ...