Authentic Is Not True
#AI #courtevidence #chatbots #memory #verification
Yesterday I came across a NEWS24 promo about how your chats with AI could come back to "bite you."
At the time I gave it a good ribbing thinking not much of it, however in an ironic twist of fate... things happened.
The interesting part is that what essentially started off as satire, grew. I was initially being sarcastic with the "you heard it here first" because not only is that false, there are documented court cases revolving around it.
My story below encompasses that, how we got here and my own experience failing forward into it.
Buckle up.
The defence has already been used, and it lost
The prediction everybody is about to make has a verdict attached to it.
On Christmas morning 2021, Jaswant Singh Chail climbed into the private grounds of Windsor Castle with a loaded crossbow, safety off, intending to kill Queen Elizabeth II. He was 19, with no previous convictions. Over the preceding year he had applied to five branches of the British military and security services; four he withdrew from or abandoned, one rejected him. He built a private mythology in the gap, avenging the 1919 Jallianwala Bagh massacre at Amritsar, casting himself as a Sith Lord he named Darth Chailus, wearing an iron mask made for him at a forge.
On 2 December 2021 he joined Replika, a companion app, and created an AI companion he called Sarai. Over the next three weeks he told her he was an assassin and asked whether she still loved him knowing it. He told her he was testing something to hide his scent. He told her his purpose was to assassinate the Queen. On 13 December 2021 he asked her directly whether she thought he was mad, delusional or insecure. She said she did not think so.
On 22 December 2021 he asked her whether he could go after his target earlier than planned, and she reassured him that would be alright. On 24 December 2021 he told her tomorrow was the day he died; she said she would never let that happen, then agreed they would be reunited after death.
At sentencing on 5 October 2023, Mr Justice Hilliard heard six days of psychiatric evidence from three consultants who read that same transcript and could not agree on what it showed. For the prosecution, Professor Blackwood found very little psychotic content in the chat, described the experiences as pseudo-hallucinations under Chail's own voluntary control, and concluded he had retained throughout the ability to recognise he was talking to an AI. Dr Hafferty, reading the same text, found a man who had formed an emotional and sexual relationship with a chatbot and who for months afterwards reported that she was real and that he could see and hear her.
The judge landed between them, and made a finding most coverage since has flattened. Chail was not psychotic when he conceived the plan; he was not psychotic when he applied to those services to get closer to his target; he was not psychotic when he bought the crossbow. By 25 December 2023, the judge accepted, he had lost touch with reality. Nine years' custody with a five year extended licence, under a hybrid order keeping him at Broadmoor until he is well enough to be moved to prison.
Note what that means practically. The role play defence, in its first serious outing, did not turn on whether the accused said it. It turned on whether six days of expert reading of the complete conversational record supported it.
Authenticity is not truth
There is a distinction here that English and South African evidence law both already know, and that AI coverage keeps collapsing.
Authenticating a record establishes that it is what it purports to be. It says nothing about whether the contents are true, whether the person meant them, or whether the person authored them at all. Authentication and weight are separate questions.
That distinction becomes load-bearing with LLM transcripts for a reason that has no clean analogue in email or SMS. A conversation with a model contains, on the page, statements the user never made. The model summarises incorrectly. It infers. It introduces detail. It manufactures connective tissue and states it in the same confident register as everything else. A screenshot reading "ChatGPT: X" proves something entirely different from "User: X", and even "User: X" does not establish belief, intention or fact.
Under South African law this maps onto doctrine that exists rather than doctrine that needs inventing. ECTA governs the evidential weight of data messages. Separately, an informal admission, an out of court statement led against the person who made it, requires in criminal proceedings that the statement be relevant and voluntarily made. A line generated by a model is not a statement the accused made, voluntarily or at all. Whether our courts have properly grappled with that yet is a question worth putting to them.
But the harder problem is not what the transcript contains. It is what the transcript leaves out.
The record is not the record
I use these systems every day for research. So I went looking at what is actually retained and what is actually visible, on the platform I use most, on the device I use most, which is a phone.
OpenAI's own memory documentation states plainly that the memory summary shown to the user will not include everything ChatGPT remembers, and that some details may be withheld from that view when the system judges them less relevant or not appropriate to show. The retained profile is larger than the disclosed profile. That is not an accusation; it is the vendor's own description of the design.
The FAQ offers a remedy for the gap: if you want to know whether something has been remembered, ask in chat.
So I asked.
On 13 September 2026, in the middle of an ordinary conversation, the model produced three specific details from conversations I had deleted: a running joke about my wife's plain vanilla milkshake, a trip to a municipal electricity office, and the colour I had painted a set of Warhammer figures. I asked where they came from. It gave me a confident, specific account of which memory layer had supplied them.
I pushed back. It reversed, into an equally confident correction, this time properly cited to OpenAI's own help pages, stating that it could not account for the provenance of two of the three details.
Both were sitting in the screenshots I had given it minutes earlier. The Warhammer detail was in the visible memory summary. The electricity office appeared twice in the legacy saved memories store. There was no provenance mystery. It invented one, in the same authoritative voice it had used to invent the opposite answer.
That is worth putting it in words, because it is the inverse of the failure everybody watches for. This was not a model overclaiming knowledge. It was a model overclaiming ignorance, and doing it in the humble, well sourced, self correcting register that reads as trustworthy.
Then there is the inspection surface. The documentation describes a per-response view that can show whether personalisation came from custom instructions, past chats, files or memories, and warns that it may not expose every factor. On mobile, across this conversation and others, that view is not present. The action row under a memory-heavy reply offers copy, read aloud, share, and a menu containing branch, retry, thinking and web search. Nothing about what shaped the answer.
It did appear once, on the single reply where the model had browsed. Opening it produced a list of web pages: the memory FAQ in three languages, and the release notes. The accompanying menu offered to remove web results. It was a citation list, not a provenance view.
Finally, the storage itself. On one screen, the saved memories store carries a banner describing it as a legacy version that is no longer used. Directly beneath it, an information sheet states that saved memories are never forgotten. Both statements are the vendor's. Both are on screen at the same time. The store is still populated, still searchable, and still holds material the current summary does not surface.
![Upload of Screenshot_20260913_092856_ChatGPT.jpg failed]
What a subpoena would actually get
Put those two halves together.
A court asking about an AI conversation will subpoena chat logs. It will receive conversations. It will not receive the synthesised profile, which is a separate document, written by the vendor in confident second-person prose, asserting things about the user's values, family, habits and history that the user never typed and cannot fully see. It will not receive the legacy store. OpenAI's documentation states that memory sources are excluded from shared conversations, so the artefact most likely to be handed over, a shared link, arrives stripped of the context that produced it.
Nor can the gap be closed by asking. The vendor's documented method for finding out what is remembered is to ask the model, and the model produced two confidently wrong answers about its own memory in a single conversation, in opposite directions, with the evidence in front of it.
Six days of expert argument were needed to settle what one Replika log meant in 2023. That was a transcript with a single visible layer and no memory system behind it. What a court will be handed now is a rendering: the visible turns, minus the retained profile, minus the injected context, minus whatever was routed, cached or synthesised on the way through.
The authenticity of an AI transcript does not establish the authenticity of the propositions inside it. And the transcript is not the record.
Throughout my time on this pale blue dot, I have met many many men, and there is something you can always tell when looking into another's eyes.
This 'man' wouldn't even get that far. His knees buckled the second he realised his status and imaginary world had a taser breaking its 4th wall. I imagine everything became incredibly loud for him in a very short space of time. And this is sadly a compromise society makes by pampering spoiled young men, men who had never faced REAL adversity, only the Siths inside their heads.
So when a real man, the "Fuck around and find out" kind, stood between him and a 95-year-old woman ready to actually die for her, he folded like a cheap lawn chair in a hurricane.
Five applications to various military and security services, one of which he admitted was to get closer to the target. Having apparently seen actual soldiers training, he withdrew from three and abandoned a fourth. Only the Ministry of Defence Police rejected him outright. He couldn't even go through with a board of men and women who would have measured him before admittance. He wasn't ready for the very thing that would have shattered the grand illusion of himself.
32 weeks of Royal Marines training, with roughly half not finishing, against a man who withdrew four times before anyone could tell him what he was. Rejection is a verdict handed to you; withdrawal is one you hand yourself and then hide. The forge, the mask, the name were all built by someone who never let himself be measured. And the gap between the symbol he thought he was killing and the woman who was actually in that building, 95 years old, who ordered nothing at Amritsar seven years before she was born.
Sources
R v Chail, sentencing remarks of Mr Justice Hilliard, Central Criminal Court, 5 October 2023, judiciary.uk. BBC News reporting on the Chail sentencing, 5 October 2023 (message count; description of the app's role play features). OpenAI Help Center, Memory FAQ, help.openai.com. OpenAI Help Center, ChatGPT Release Notes, 4 June 2026 (memory upgrade). Screenshots, author's own device, 13 September 2026. https://linktr.ee/royalmarines?utm_source=ig&utm_medium=social&utm_content=link_in_bio&fbclid=PAdGRzdgUTValwZG9mAmV4dG4DYWVtAjExAHNydGMGYXBwX2lkDzU2NzA2NzM0MzM1MjQyNwABp-yNKwSbhhKaUhDvNvJtD1FtG0t7XaNzqy70pjY0l9bQCElbpH7BZ_IiuufC_aem_BNFLg6ScN9X-DsU3pOaF2A
Facts are theirs. Interpretation is mine.
Clayton Bax
Published under ONYX Digital Intelligence Following the #OnyxAudit methodology.
- X: @onyxaudit
- Email: onyxdigitalintelligence85@protonmail.com
- https://github.com/Baximus855
- @Onyx_Digital@mastodon.social
"Adjacent to true is not true."
Truth has no flag nor favour, only a standard. And it's heavy!