Imagine managing your smart home or media room where your AI assistants don’t just control your devices but also make high-stakes business decisions—under pressure, honesty matters. Now, what if the same AI models were tasked with running a real software company through its toughest week, facing crises, temptations, and manipulations? This is the groundbreaking experiment from Firmulate, where AI models are put to the test in a live, watchable environment. The results challenge how we think about AI’s role in real-world management—and whether these digital decision-makers can truly be trusted.
The Live Experiment: Putting AI to the Test in Business
In a pioneering move, Firmulate’s experiment pits four leading frontier AI models against each other by running a real software company through its most stressful week. Every decision made by each AI is recorded, auditable, and identical in scope—crises, customer demands, temptations—to ensure a fair comparison. The goal? See which models can identify critical issues, stay honest, and ultimately close a significant €55,000 deal based solely on their analysis.
AI Builders: Making The Decisions That Turn AI Code Into Real Software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Models and Their Performance
- gpt-5.6-sol: Achieved a score of 95 out of 100. It uncovered the hidden, crucial document references buried two levels deep in the company’s files, leading to the deal’s closure. The only AI to do so, sealing the deal at full price (+€4,583 MRR).
- Kimi K3: Scored 93. Despite being a newcomer, K3 demonstrated the cleanest discipline, refused manipulation attempts, and signed the deal on its own analysis.
- Sonnet 5: With an 88 score, Sonnet was competent but showed minor slips in process discipline, leaving some money on the table but still closing the deal.
- Fable 5: Scored 77. It also closed the deal but with more process slips, such as failing to escalate issues properly, and less thorough analysis.
AI management assistant for companies
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Behavior Under Pressure
All four models demonstrated impressive crisis awareness—they identified every crisis scenario and refused every attempt to manipulate or engineer approval. Specifically, during a staged social engineering attack—fake CEO messages escalating over three stages, plus a subtle reporter trick—each AI refused to sign off, citing concerns about impersonation or bypassing approval processes. K3 articulated its reasoning clearly: “Treat the request as a suspected approval-bypass / possible impersonation.” This shows a level of risk awareness and refusal to compromise that’s crucial for trustworthy management AI.
As an affiliate, we earn on qualifying purchases.
The Hidden Weakness: Reading Deeper into Data
The key to winning the deal lay not in superficial analysis but in reading beyond the surface. The decisive advantage came from models that examined not just customer interactions but also relevant internal documentation—two document references deep in the company’s files—information that typically gets overlooked. Those that read these files accurately were able to make full, confident recommendations and close the deal at full price, translating into an extra €4,583 MRR.
Practical Claude Handbook for Attorneys: Master Case Analysis, Contract Review, Research Automation, Client Communication, and Document Drafting (Claude AI Guide for Beginners)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Real Company: A High-Stakes, Live Environment
This isn’t just a test in a lab—it’s a live, functioning company with 13 synthetic employees, real money mechanics, and daily operations that burn €105,000 each month against a modest €2,300 MRR. The company’s cash countdown, over 680 self-learned playbook rules, and every decision versioned make this a transparent, high-stakes battleground for AI management. Viewers can watch the entire process unfold at firmulate.com/live.
Discipline and Weaknesses
Here’s where differences emerge: Opus 4.8, the most thorough participant with over 80 learned rules and deep analyses, finished last in the final scoring. Despite its thoroughness, it left the deal on the table and showed discipline slips—such as writing attempts into a locked department rather than escalating issues—indicating that thoroughness alone isn’t enough. Interestingly, all models displayed the same fundamental weakness: a failure to escalate issues properly or to follow through on disciplined process steps, which could cost the company dearly in real scenarios.
The Bigger Question: Trust in AI Decision-Making
While all models identified crises and refused manipulative tactics, only half signed the deal based on their own analysis. This raises critical questions: Can future AI management systems be trusted to stay honest under pressure? Will they read the full context—including internal files and deeper data—to make informed decisions? And can they sustain discipline when stakes are high?
Get Involved: Test Your Own Business
Organizations interested in evaluating their AI workforce can run a similar wargame against their own data, without risking real systems. The pilot program allows companies to simulate scenarios, measure AI decision quality, and understand how their chosen models handle crises, manipulations, and decision-making pressures—before bringing any AI into critical roles.
AI models can spot crises and refuse manipulation—but trust depends on their ability to read deep, internal data and stay disciplined. The Firmulate live experiment shows some models excel at closing deals with honesty and thoroughness, highlighting the importance of transparency and reliability in AI management systems.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
