What happens when an automated decision produces significant effects on a person?

The recent case of the Argentinean judge investigated after Google's automated report for storing CSAM (Child Sexual Abuse Material) opens a legal and institutional discussion, but above all a discussion of dependence on technological platforms and principles of personal data protection.

What happened

In 2016, a criminal judge in Tierra del Fuego opened a case for sexual offences against minors. The case file legitimately contained images of said offences and in the exercise of his duties, images constituting offences. Years later, when he backed up his work computer, those files ended up synchronised on his personal Google Drive.

Google's automated system detected the material, suspended her account and reported the discovery to NCMEC (National Center for Missing & Exploited Children), as required by US federal law. This report led to a communication to the Argentine cybercrime prosecutor's office, a search of the judge's home and office was ordered, a criminal case was opened and a devastating media campaign was launched. Months later, he was acquitted, as the courts declared that there was absolutely no criminal conduct.

What Google did 

The detection and mandatory reporting of child sexual abuse material exists to protect real victims. Zero tolerance" acts as a response to a very serious phenomenon where admitting exceptions for "professional reasons" could open a very dangerous door and a commercial platform should not pretend to be a repository of judicial evidence or an arbiter of the legal context of every file it hosts. The problem is apparently not that Google detects and reports, but how it is processed, as a system is able to identify a file, but unable to understand the legal context of that file, qualifying a person's conduct and triggering a criminal prosecution that significantly affected the judge's rights, liberties and reputation.

Thus, the case is extremely delicate on two equally legitimate grounds:

• the absolute obligation to prosecute and report child sexual abuse material;
• and the obligation to protect fundamental rights when there are automated decisions with consequences that produce legal effects on the holder or significantly affect him/her.

In Chile, what rights and guarantees exist in the face of an automated decision?

In our country, according to Law No. 19.628 and its amendments by Law No. 21.719, on Personal Data Protection, for the first time the Chilean legal system expressly recognises the right of the holder to oppose and not be subject to decisions based on automated data processing, including profiling, when such decisions produce legal effects or significantly affect the person.

However, the law indicates that this right to object shall not apply in the following cases:

• When the decision is necessary for the conclusion or execution of a contract between the data subject and the data controller.
• When there is prior and express consent of the data subject in the manner prescribed in Article 12.
• When so stipulated by law, to the extent that the law provides for the use of safeguards to the rights and freedoms of the data subject.

Without prejudice to the foregoing, the Law gives the rightholder, in all cases, including the exceptions mentioned above, the following rights that may be translated into operational safeguards:

• To human intervention
• The right to information and transparency
• The right to obtain an explanation
• The right of the holder to express his or her point of view
• And to request a review of the decision

If this case had occurred in Chile and under the framework of the new law on Personal Data Protection, the relevant question would not be whether the algorithm got it right. It would be whether the data subject had, at some point in the process, the actual conditions to object, if he or she could have done so, or the knowledge about the rights granted to him or her by the law.