The Enterprise AI Failure Casebook

Texas hospitals used a clinical AI sold on a claimed hallucination rate of under 1 in 100,000

Texas settled with Pieces Technologies, whose generative AI summarized patients' condition and treatment for staff at Texas hospitals, over accuracy claims the state alleged were false or misleading; Pieces denied wrongdoing.

Pieces Technologies, Inc. · September 18, 2024 · Updated October 31, 2026

Pieces Technologies, of Irving, Texas, sold generative AI to hospitals. Its website said: "Pieces summarizes, charts, and drafts clinical notes for your doctors and nurses in the [Electronic Health Record]." According to the Texas Attorney General, at least four major Texas hospitals were sending their patients' data to Pieces in real time so that the product could summarize each patient's condition and treatment for hospital staff.

According to the State, Pieces built its own accuracy metrics to sell the product. Many used the word "hallucination", which the State's filing describes as the term commonly used for a generative AI output that is wrong or misleading. The State says Pieces advertised a "critical hallucination rate" and a "severe hallucination rate" of "<.001%" and "<1 per 100,000".

The Attorney General's Consumer Protection Division investigated under the Texas Deceptive Trade Practices Act. The office said its investigation found the metrics were likely inaccurate and may have deceived hospitals about the accuracy and safety of the product. In the court filing, these are the State's allegations: that the claims may have violated the Act because they were false, misleading or deceptive.

Pieces denied any wrongdoing. It said it had stated its hallucination rate accurately. The two sides settled through an Assurance of Voluntary Compliance, which Pieces signed on August 5, 2024 and the State filed in Dallas County district court on August 21, 2024. The agreement says it is not an admission of any violation. It includes no fine or payment to the State, and each side bears its own costs.

Under the agreement, for five years from the date the court approves it, any accuracy figure Pieces uses in marketing must come with a clear statement of what the figure means and how it was calculated. Pieces may instead hire an independent auditor and keep its claims within the auditor's findings. Pieces must also give every current and future customer documentation of the data and models used to train its products, their intended use, their known limitations and risks to patients, the misuses that make wrong outputs more likely, and what a user needs to watch for patterns of inaccuracy.

The Attorney General announced the settlement on September 18, 2024. "Hospitals and other healthcare entities must consider whether AI products are appropriate and train their employees accordingly," he said. Neither the release nor the agreement names the hospitals.

What failed

The hospitals put a generative AI product into clinical work. The vendor marketed it with accuracy figures it had defined and measured itself. Whether the figures were right is disputed: the State alleged they were false or misleading, and Pieces denied it and settled without admitting a violation. The record does not say whether any hospital asked what a "severe hallucination" was, how the rate was measured, or on what data. The settlement's remedies are the questions a buyer could have asked before signing: what each metric means, how it was calculated, who checked it independently, what the product was trained on, where it is known to fail, and what staff need to know to spot a wrong summary. The Attorney General's statement also put a duty on the hospitals: to consider whether an AI product is appropriate and to train their staff.

Sources

1. Texas Attorney General, "Attorney General Ken Paxton Reaches Settlement in First-of-its-Kind Healthcare Generative AI Investigation", press release, September 18, 2024, Paragraphs 1, 2, 3 and 5
2. Texas Attorney General, Petition for Approval and Entry of Assurance of Voluntary Compliance, State of Texas and Pieces Technologies, Inc., Cause No. DC-24-13476, Dallas County, filed August 21, 2024, Assurance paras. 6, 10, 12 to 17, 19, 33 and 34; signature page