Why Confident AI Is a Business Problem

Most conversations about AI in business start in the wrong place. We argue about intelligence, consciousness, and whether machines will “think like humans.” Rafal Lukawiecki’s keynote at ESPC25 in Dublin, Ireland earlier this month, cuts through that quickly. The real question isn’t whether AI will become intelligent. It’s whether we will use powerful, unreliable systems responsibly before we wire them into everything that matters.

Always one of my favorite keynoters, Rafal isn’t anti-AI. In fact, he’s very clear that he uses it daily and benefits from it. AI removes friction. It speeds things up. It makes smart people more productive. He points out that that is exactly why it’s dangerous in a business setting. When something saves time and sounds confident, we stop double-checking. And when we stop double-checking, small errors scale into expensive ones.

His bottom-line message for business leaders is uncomfortable but useful: today’s AI systems are great assistants, but terrible authorities. They generate convincing answers, but they don’t reliably know when they’re wrong, and they’re very good at telling you what you want to hear. That’s fine when you’re drafting an email or summarizing a document. It’s not fine when the output feeds a financial model, a production system, or an autonomous agent.

This leads to his biggest concern: alignment. Alignment is a technical word for a very practical business question: Will this system behave the way we expect when the stakes are high? Not when everything is clean and predictable, but when inputs are messy, incentives conflict, or someone tries to manipulate it. This is one of my primary concerns around autonomous AI, specifically. How can I be hands-off with complex, multi-step processes run by AI?

The uncomfortable truth is that alignment is relative. An AI can be perfectly aligned with a narrow business goal and still cause harm elsewhere. Optimize for speed, engagement, or cost reduction without constraints, and you may get exactly what you asked for… just not what you wanted.

From a business perspective, Rafal’s argument is not that AI is useless, but that LLMs are being mistaken for something they’re not. They’re not reasoning engines. They’re probabilistic systems trained on language. They work brilliantly when the answer already exists or looks similar to something they’ve seen before. They struggle when asked to operate outside that comfort zone, and they don’t reliably warn you when that happens.

That’s why he (like me) is skeptical of the current rush toward fully autonomous “agents.” Once AI systems can take actions instead of just giving suggestions, small misalignments become operational risks. Give an agent access to money, infrastructure, or decision pipelines without hard limits, and you’ve created a single point of failure that operates at machine speed.

So what should businesses actually do? According to Rafal:

• First, treat AI as a tool, not a decision-maker. Use it to accelerate work, not to replace judgment. If an AI produces code, analysis, or plans, assume it needs verification. The time you save upfront should be reinvested in testing and review, not skipped entirely.
• Second, constrain what AI systems can do. Limit their action space. No autonomous transfers. No direct system changes. No silent execution. Humans stay in the loop where consequences are real.
• Third, layer your guardrails. Ethical guidelines, legal constraints, technical controls, audits. No single safeguard is enough on its own.
• Fourth, multiply your testing effort. AI-generated outputs often look correct until they hit edge cases. Testing needs to scale accordingly, especially in regulated or customer-facing systems.
• Fifth, treat inputs as untrusted. Prompts, documents, and data can be manipulated. If your AI reads the internet or internal content, assume someone will try to influence it.
• Sixth, delay high-risk integrations. Just because an agent can be connected to core systems doesn’t mean it should be.
• Finally, resist the hype. AI is not magic. It’s a powerful, immature technology that rewards discipline and punishes shortcuts.

Rafal’s warning isn’t science fiction. It’s a management problem. And like most management problems, it won’t be solved by better tools alone, but by better decisions about how, when, and where we use them.

You can watch Rafal’s entire keynote on the ESPC YouTube site here: