Trust dies in darkness, so let’s take AI out of its ‘black box’

In business, as in life, it is trust that holds everything together — investment, deals, recruitment. Customers must trust cloud service providers because if they do not, they would second guess every technology decision made. A manager must trust an employee because if they do not, they would have to micromanage every task and would have no time for strategic thinking. In these two examples, we see the outcomes of trust clearly. In the former, the cloud customer dares to innovate; in the latter, the manager does the same. The fundamentals are covered, so no need to worry.

But behind every trust relationship are processes. It is precisely because the cloud provider and the employee follow these processes faithfully that they inspire trust. As you read this, across the GCC hundreds if not thousands of AI systems are embedded in enterprises’ day-to-day operations. According to McKinsey research from 2023, in the regional retail sector alone, 75% of companies have adopted AI to augment at least one business function. One can only assume AI is trusted by these companies.

Recommended For You

UAE: New Etihad Rail passenger train station in Fujairah revealed

3 in 10 employees in UAE, GCC are ‘bad hires’; how companies can avoid them

UAE: Now, replace your foreign driving licence online

UAE: Gold may reach $3,000 in Q1 2025; why is it still high time to buy yellow metal?

UAE: Will you pay Dh13,000 for your kid's field trip? Parents debate benefits

 

The ‘black box’

But what is the origin of this trust? AI algorithms are famously non-deterministic. It is, in most cases, difficult if not impossible to trace output back to a branching decision within the system. This gives AI a “black box” feel that could give way to doubt. But of course, if the results send trend lines in the right direction, decision makers are unlikely to be interested in delving into the source of the insights that preluded success. Hence, trust can come before due diligence, where it should be the other way around.

AI is an immensely powerful tool with the potential to upgrade how decisions get made. From cabinet rooms to boardrooms to living rooms, it has a role to play in improving lives. Large language models (LLMs), for example, are not only adept at understanding intent and nuance in human queries; they can organise responses in meaningful ways that aid absorption of knowledge and lead to better insights. It is perhaps unsurprising that generative AI (the family to which LLMs belong) has rapidly taken its place in economic projections. According to one estimate from PwC’s Strategy&, the GCC could generate almost US$10 for every dollar invested in GenAI.

Again, if we look behind the figures, we see potential in LLMs’ ability to connect humans to large data sets that would otherwise be out of reach for non-technical people. But we also see an eagerness to trust because of the powerful results. However, if we take a moment to consider a highly regulated industry like BFSI, we find age-old business processes and workflows defined around consistency and privacy. When audited, a bank must explain exactly what it does with data during each of its workflows. LLMs offer a marvellous interface between raw data, and the rapid insights they can provide are tempting to BFSIs. But when answering the question of what happened to the data during the process, things start to become vague. And trust can take a hit.

Law and order

And when trust takes a hit, there is a knock-on effect of decreased innovation and perhaps even a slowing of economic growth. As such, there is increasing pressure to get the balance right between total trust and the placement of guardrails. Far from being an impediment to highly regulated industries, AI regulation is an attempt to remove it from its black box and make it accountable.

One place this is happening right now is in Europe. The EU Artificial Intelligence Act became law in August 2024. The framework is described by the European Commission as being based around “human rights and fundamental values” and intended as groundwork for development of “an AI ecosystem that benefits everyone”. The law goes so far as to break risk down into four distinct categories and covers every AI use case from “minimal risk” spam filters through “specific transparency risk” chatbots to the “unacceptable risk” of controversial social-scoring models, which is now banned.

The writer is Global SVP & Head of AI Pod, Endava.