LLMs Struggle to Predict Token Use in IT Budget Study
Token prediction difficulties in LLMs challenge AI deployment, echoing the rise of cloud cost analytics in 2022.
Key Points
- 1First study on LLMs predicting token usage in agentic mode.
- 2Agentic mode's cost impacts IT budgeting strategies.
- 3Highlights reliance on AI despite unpredictable costs.
What Changed
A new study, conducted by researchers from MIT, Stanford, Google DeepMind, and Microsoft, evaluated the ability of eight Large Language Models (LLMs) to predict token consumption before task execution. This is the first known research to assess LLMs in an agentic context, revealing that agentic modes consume around 1000 times more tokens than chat-based assistants, impacting IT spending heavily. Historically, AI cost studies focused more on hardware or initial software investment, rather than operational costs such as token consumption.
Strategic Implications
This study underscores a critical issue for IT departments: the unpredictability and high cost of deploying LLMs in an agentic mode. Companies planning to use AI extensively will need to adjust budgeting strategies, potentially reducing confidence in AI-driven projects. For tech giants like Google and Microsoft, this insight could influence product development and pricing strategies. Failure to accurately predict costs could shift power toward vendors who can provide more cost-effective solutions.
What Happens Next
Given the high variability in agentic mode costs, companies might adopt more rigorous budget forecasting tools. Expect vendors to develop features for better cost predictability by early 2027, potentially leading to new market entrants offering innovative cost management solutions. Industries heavily reliant on AI will need to reassess deployment strategies, potentially shifting toward more predictable alternatives in the next 12 months.
Second-Order Effects
This finding may spur regulatory discussions on AI usage transparency and cost predictability. As firms like Uber encounter budget overruns in AI deployment, there could be a push for industry standards or guidelines on cost-efficient AI usage. Adjacent markets, like cloud services, could see shifts as firms optimize for token efficiency, possibly impacting server loads and data center demands.
Free Daily Briefing
Top AI intelligence stories delivered each morning.