The Enterprise AI Failure Casebook

A child protection worker used ChatGPT to write a report for the Children's Court

Victoria's privacy regulator found that a court report drafted with ChatGPT played down the risk to a young child, and ordered the department to block such tools for its child protection staff.

Department of Families, Fairness and Housing (Victoria) · September 24, 2024 · Updated October 31, 2026

A legal officer at the Department of Families, Fairness and Housing (DFFH) in Victoria, Australia, read a child protection report on the day of a Children's Court hearing. The report had gone to the court about a week earlier. The officer noticed unusual language and questioned the adequacy of the risk assessment. Suspecting the report had been drafted with ChatGPT, the officer reported the concern to the Child Protection area.

The case concerned a young child whose parents had been charged with sexual offenses. The charges did not relate to offenses against the child. Child Protection had obtained interim orders removing the child from the parents' care. A caseworker, whom the regulator calls CPW1, wrote the Protection Application Report, which tells the court about the risks to the child and whether the child needs protection. A team manager reviewed and signed off the report. The regulator places its drafting in or around September 2023.

The report mentioned the child's doll in two places. It first described how the father had used the doll for sexual purposes. It later listed the same doll as a notable strength: the parents had given the child "age-appropriate toys." DFFH told the regulator that this wording described the adults in an overly positive light and played down the level of risk to the child. Other passages used American spelling and sentences DFFH called "non-sensical."

The worker admitted using ChatGPT, to save time and to present work more professionally, but denied entering personal information. Colleagues said the worker had demonstrated the tool to them by typing in client names. DFFH withdrew the report, submitted a replacement, told the people affected and, in December 2023, reported the incident to the Office of the Victorian Information Commissioner (OVIC). The mother, father and carer described anxiety and feeling overwhelmed when they were told. The errors did not change the decisions of Child Protection or the court.

OVIC found, on the balance of probabilities, that the worker entered names and other personal information about the mother, father, carer and child into ChatGPT, and so disclosed it to OpenAI. Neither DFFH nor OVIC could check this directly. The department's logs showed who had visited the ChatGPT website, not what they typed. A DFFH review of 796 cases handled by the worker's unit over 12 months found 100 with indicators that ChatGPT may have been used, in court reports, case notes, case plans and risk assessments. Audit logs for July to December 2023 showed that nearly 900 employees, almost 13 percent of a workforce of around 7,000, had visited the ChatGPT website. In April 2024 the department's Chief Information Officer emailed all of them, asking for examples of how they used generative AI. Ten replied.

On September 3, 2024, Rachel Dixon, the Privacy and Data Protection Deputy Commissioner, served a compliance notice. It found serious contraventions of Information Privacy Principle 3.1, on accuracy, and Information Privacy Principle 4.1, on security. It required DFFH to direct child protection staff not to use web-based or external API-based generative AI text tools in their work; to block 15 named tools, including ChatGPT, Claude, Gemini and Copilot, by November 5, 2024 and keep them blocked until November 5, 2026; to scan for new tools and block them; to prevent child protection staff from using Microsoft 365 Copilot; and to report on how well the controls work four times between March 2025 and September 2026. DFFH's Secretary, Peta McCammon, accepted the breach of both principles. Her letter also said the report had not found that staff used generative AI for sensitive work. OVIC's report states that it found the opposite.

What failed

DFFH had no rules on when and how staff should or should not use generative AI tools. Its acceptable use policy and its privacy and security training were general. Its three sessions on generative AI risks in May 2023 went only to managers and leaders, and OVIC found no evidence of any other training for staff before the incident. Nothing on its network stopped staff from opening ChatGPT, and nothing recorded what they entered. OVIC found these controls insufficient for some of the most delicate personal information any government body holds. The department issued generative AI guidance to all staff on October 25, 2023, after the incident. OVIC found that guidance broadly fit for purpose but concluded that in child protection the risks were too great to be managed by policy and guidance alone. The notice required a direction, a technical block and reports on whether the block works.

Sources

1. Office of the Victorian Information Commissioner, Investigation into the use of ChatGPT by a Child Protection worker (report), Paras 1-2, 34-43, 46, 49-54, 57-59, 63, 65, 69, 74-77, 81, 84, 88-94, 97-98, 115, 119-120; executive summary, pp. 7 and 9
2. Office of the Victorian Information Commissioner, Compliance notice to DFFH under section 78 of the Privacy and Data Protection Act 2014, Annexure A, pp. 31-33: background para 1(a)-(e); Specified Actions 1-6
3. Department of Families, Fairness and Housing, Response to the investigation report, September 18, 2024, Annexure B, p. 34
4. Office of the Victorian Information Commissioner, Investigation into the use of ChatGPT by a Child Protection worker (case page), Publication date