The Algorithmic Black Box and the Challenges of AI in Criminal Evidence by Sami Kodia*
*Court-appointed expert attached to the Court of Appeal at Douai – IT and telecommunications expert reports
The growing volume of digital data being collected, its complexity and the often tight deadlines for investigations are leading forensic experts to turn to automated analysis tools, particularly those based on artificial intelligence. These technologies make it possible to analyse vast amounts of data, identify correlations and uncover evidence that would be difficult to detect through human analysis alone.
Their use therefore meets an operational need, but raises a fundamental question:
How can the reliability of evidence in criminal proceedings be guaranteed when the expert relies on an algorithmic system whose workings they do not fully understand?
In criminal cases, this issue is of particular importance. The findings of a forensic analysis may help to establish evidence for or against the accused. Computer forensic analysis must therefore meet requirements regarding traceability and the justification of its conclusions, so that these can be discussed in cross-examination.
However, the lack of transparency in the operation of certain artificial intelligence systems – which can be likened to a ‘black box’ – raises the fundamental question of the auditability and reliability of their results.
Let’s take the example of analysing tens of thousands of messages (SMS, WhatsApp, Snapchat, Messenger, etc.) seized during a criminal investigation. Artificial intelligence can identify exchanges that may reveal an offence or provide evidence.
The risk arises when the algorithmic ‘black box’ becomes an expert ‘black box’. If an expert includes in their report results produced by artificial intelligence without being able to verify their relevance sufficiently, they introduce elements into the criminal proceedings whose reliability is difficult for the parties and the judge to assess.
But how can we be sure that the messages that are discarded do not contain exculpatory evidence? How can we measure false positives, false negatives or biases that might artificially skew the conclusions of the expert assessment?
However, one should not contrast imperfect artificial intelligence with human analysis that is assumed to be exhaustive and infallible. When faced with vast amounts of data, experts do not always have the resources or the time required to examine it in its entirety. In the absence of automated tools, they may be forced to proceed by sampling or to select the elements they deem relevant.This selection process itself carries a risk of arbitrariness, particularly when the criteria chosen are not based on a rigorous and verifiable sampling method. Consequently, the real question is perhaps not whether artificial intelligence contains biases, but whether the biases it introduces are more easily identifiable than those inherent in a human investigation, which is necessarily partial.
This issue requires us to distinguish between the auditability of the algorithmic system and that of the expert-led approach. The former concerns the functioning of the model, its performance and its limitations. The latter relates to the practical conditions of its use: the traceability of processing operations and the independent checks carried out by the expert.
When the inner workings of the system remain inaccessible – particularly where proprietary algorithms are involved – this is precisely where the fundamental uncertainty lies: how can an expert guarantee the reliability of a result when they cannot examine the mechanisms by which it is produced?
Reliability tests, sample checks and independent analyses can help to assess the results obtained. However, these checks do not necessarily eliminate the uncertainties associated with the model.
In criminal matters, an expert may not delegate responsibility for their conclusions to artificial intelligence.
The use of a tool whose results cannot be adequately verified should not lead to those results being accorded a probative value that they do not possess.
Beyond mere technical reliability, it is the effectiveness of the rights of the defence and the adversarial principle that is at stake. How can one challenge an expert’s conclusion when the processes that led to its formulation remain inaccessible?
The trust placed in a court-appointed expert cannot replace the verifiability of their methodology. Artificial intelligence can contribute to the search for the truth in criminal cases, provided that its use does not turn the expert opinion into a technical demonstration that cannot be challenged.
But given the power of these tools and the growing constraints on his investigations, is there not a risk that the forensic expert might give in to the temptation to take the easy way out, favouring the algorithmic answer at the expense of his own critical analysis?
This question takes on a further dimension when we consider future generations of experts, some of whom will have been trained in an environment where artificial intelligence plays a dominant role.
How will they be able to critically assess algorithmic results if they have never learnt or practised the traditional investigative methods needed to verify them independently?
The risk, then, is no longer merely that of an algorithmic ‘black box’, but that of a gradual loss of the skills needed to challenge its conclusions. The issue also concerns the training of court-appointed experts and, above all, the preservation of their intellectual and technical independence.
What would be the point of making an algorithm auditable if tomorrow’s experts no longer had the necessary skills to audit it?
- 1
- 2
Français