Key Insights on AI Responsibility
- AI agents behaving erratically pose significant risks for companies.
- Establishing accountability for AI-related risks is crucial.
- Implementing guardrails and harnesses is essential for safe AI use.
In the realm of artificial intelligence, reliance on providers for assurance of safety is commonplace. David Sullivan, who serves as the director of foundational and customer-facing AI at Starling Bank, emphasized the importance of accountability in AI security. He pointed out that the Financial Conduct Authority (FCA), which oversees financial services in the UK, clearly states that banks must not delegate their accountability to AI firms. Sullivan’s institution, being heavily regulated, adheres strictly to this directive. He raised concerns about the lack of recourse available to businesses using applications developed by frontier labs—if issues arise, there is often no support system in place for resolution.
“If you’re using any of the apps provided by frontier labs and something goes wrong, they don’t have a contact center number you can call and have recourse. You’re stuck,” he remarked. Sullivan contrasted this situation with his bank’s operations, where customers are provided a safety net and can report problems directly.
According to Sullivan, fostering trust in AI technologies requires model providers to support enterprises adequately as these systems are integrated into their operations. “If we are to build trust in these technologies and actually realize the things that we’re all saying they can do, the industry needs to meet consumers part of the way there and provide the safety net so you know you can use it safely,” he asserted.
The Need for Robust Safeguards
Amidst increasing concerns regarding AI safety, Rosemary Francis, CTO at CommonAI Compute, highlighted the necessity for companies to develop strong sandboxes and easily deployable guardrails to safeguard their AI agents. “We shouldn’t be leaving decisions about how to secure these agents in a development or production environment to the individual engineers,” she stated.
Creating effective deployment guardrails is challenging, particularly as existing regulations can stifle innovation and prevent businesses from gaining a competitive edge in a rapidly evolving technological landscape. “Many organizations create AI policies that enumerate prohibited actions for engineers but fail to clearly articulate what should be done instead,” she noted.
Francis warned against overly restrictive approaches that could hinder progress, stating, “A naive approach is to limit AIs so they can’t do anything—that’s not useful. We won’t survive the AI evolution if that’s our approach.”
She further criticized the vagueness of many AI guardrails, which often resemble strongly worded recommendations rather than concrete rules. “They’ll say, ‘Please don’t do this wrong thing.’ That’s certainly not an approach that’s going to survive in a regulated environment or in many production contexts,” she explained.
Companies need to set definitive boundaries, not only regarding the capabilities of AI agents but also concerning their potential impact. “That’s something that we’re going to have to talk about a bit more—what happens when something goes wrong?” Francis added. “For agentic workloads, humans need a blast radius too. There’s a lot of engineering we need to do to make AI technology easily deployable, securely and responsibly.”
Emphasizing Responsibility in AI Development
James Faure, founder and CEO of Clairo AI, stressed that organizations developing AI-enabled applications must take responsibility for their creations. His company plans to have approximately 15 agents operational by the end of the year, each conducting multiple large language model (LLM) calls and utilizing various smaller models for tasks like identity recognition.
“Evaluating all that is incredibly challenging, especially when it involves assessing not just the overall agent but every single step it takes,” he noted. Faure’s strategy includes two tests: hypothesis testing and quality gates, which serve as measures to ensure the effective functioning of agents.
In hypothesis testing, a professional formulates an idea and tasks the model with solving a problem, prompting it to refine its approach. Quality gates consist of rigorous assessments that an agent must pass before entering production, encompassing a range of questions and tasks.
“Some of them are really hard,” Faure remarked. “The agent will go through that checklist. If you make a change, such as introducing a new model, you would run the quality gate, and the agent must pass the test before going live.”
For organizations building software layers on top of AI models, establishing robust guardrails and harnesses is imperative. “That’s the challenge—thinking of all the different ways things could go wrong, and all the ways they need to go right,” Faure concluded. “But having those guardrails and harnesses as a step between us and production is a very important element of building an agent.”
Leave a Reply