Judicial Prompt Injection: the case that highlights the new legal risks posed by AI

A case uncovered in Brazil set off international alarm bells after lawyers embedded hidden instructions in a court claim in an attempt to influence artificial intelligence systems. The incident not only highlights a new form of abusive litigation, but also reopens the debate on algorithmic governance, the security of AI systems and the need to maintain effective human oversight in legal and administrative settings.

The case that set off the alarms 

What happened in Brazil was, on the surface, a simple manoeuvre, but one of great legal significance. In an employment claim filed with the 3rd Labour Court of Parauapebas, in the state of Pará, the insertion of invisible text – written in white on a white background – was detected. This text was addressed not to the judge or the opposing party, but to an artificial intelligence system involved in the court’s document processing.

The hidden message instructed the AI to respond to the claim ‘superficially’ and not to challenge the accompanying documents, regardless of any subsequent instructions it might receive. In other words, the aim was to alter the system’s behaviour through a covert command embedded within the legal document itself. The court held that this was not a mere formal irregularity, but an act undermining the dignity of justice, and sanctioned the lawyers who had signed the submission.

From a technical point of view, the manoeuvre corresponds to what is now known as ‘prompt injection’: the insertion of malicious or misleading instructions into content that is to be processed by a language model or AI system. Rather than persuading a person, the strategy seeks to influence an automated tool that summarises, classifies, analyses or assists in the review of documents.

That is why the case has been described as one of the first precedents of ‘judicial algorithmic sabotage’ or ‘algorithmic procedural fraud’: a practice aimed at manipulating technological systems that are already, directly or indirectly, part of the management of legal proceedings.

Why does this matter?

For centuries, procedural risks were viewed through a human lens: how to unduly influence judges, witnesses, experts or opposing parties. But the Brazilian case shows that, in the age of artificial intelligence, risks can also be directed against automated systems. And that profoundly changes the legal discourse.

Litigation is beginning to face new threats: hidden instructions, manipulation of document analysis systems, induced biases or alteration of the context of algorithmic processing. It is no longer enough to review the visible content of a legal document; it also matters how that content might be interpreted by an AI.

The impact is not limited to the judicial process. These kinds of risks also affect compliance, cybersecurity, technology governance and the protection of fundamental rights. If an organisation uses AI to review contracts, prioritise claims, shortlist candidates, process files or support decision-making, it must ask itself not only how the system works, but also how it could be manipulated.

In other words: the discussion no longer revolves solely around the accuracy or efficiency of AI, but also around its integrity, resilience and ability to withstand malicious interference.

The new framework approved by Spain

Against this backdrop, Europe continues to move towards more stringent regulatory models. Spain recently approved the referral to the Congress of Deputies of the Draft Organic Law on the Proper Use and Governance of Artificial Intelligence for parliamentary consideration, in line with the European AI Act.

The initiative reflects the direction in which the debate is moving: greater human oversight, algorithmic transparency, governance of AI systems, coordination between supervisory authorities, and a system of penalties for prohibited or high-risk uses.

Rather than going into the details of the Spanish text, what is relevant is the underlying message: AI is no longer being regulated solely as a promise of innovation, but also as a technology that can affect rights, critical processes and public interests if it lacks adequate controls.

The debate for Chile

The question is inevitable: is Chile prepared to tackle the risks of manipulation of AI systems? Today, the country has a National Artificial Intelligence Policy and a draft bill on the use of AI systems currently going through the legislative process, but it still lacks a comprehensive law in force comparable to the European framework.

Even so, the debate is not starting from scratch. Chile has made significant progress with the new Personal Data Protection Act, which strengthens obligations regarding security, accountability, risk assessment and the protection of data subjects’ rights. Furthermore, the right to object to automated decisions and the right to human review and explanation have been incorporated in cases where AI is used. This framework may become particularly relevant when automated systems process personal data, profiles, academic records, biometric data, criminal records or sensitive information for decision-making purposes.

From this perspective, the Brazilian case raises at least four questions for the Chilean context. Firstly, the relationship between AI and data protection: what happens when an automated system makes decisions or draws inferences based on personal information? Secondly, the security and integrity of systems: should organisations implement controls to detect prompt injection, hidden instructions, adversarial manipulation or induced biases?

Thirdly, human oversight: can we speak of genuine control when decisions increasingly depend on automated tools that are opaque or difficult to audit? And fourth, algorithmic compliance: are companies, law firms, courts and public institutions prepared to identify, assess and document the legal risks associated with the use of AI?

The key point is that the risk is no longer theoretical. As AI is integrated into sensitive processes, the requirements for traceability, security, governance and human review cease to be mere aspirational principles and become minimum conditions for responsible use.

Finally, the Brazilian case demonstrates that the risks of artificial intelligence are no longer hypothetical. The discussion is no longer limited to how to use AI to improve efficiency, but also extends to how to oversee, protect, audit and prevent its manipulation.

Whereas the question used to be what artificial intelligence can do for the law, today the question is also what safeguards the law needs in the face of artificial intelligence. And, even more so, in the face of those who seek to exploit it or distort its functioning.

Whilst Europe is moving towards more comprehensive regulatory frameworks and Spain is strengthening its algorithmic governance system, Chile is beginning to face the challenge of adapting its debate on data protection, security and digital transformation to a new generation of technological risks. Judicial prompt injection is just an early warning sign. But it will probably not be the last.