Smoke on the water plant: a 2/5 attacker and a city's chlorine levels

Security has a folk theory about who gets to hurt a city through its water supply: it takes a nation-state, a zero-day, months of patient reconnaissance. In March 2016, Verizon’s investigators rated the sophistication of an attack that reached the chemical-dosing controls of a US water utility at 2 out of 5. The folk theory is wrong, and the reason it’s wrong turns out to be the more useful lesson.

One of my Semester-II courses at NFSU — Critical Infrastructure Security — assigned a case study on exactly this incident. Verizon anonymized the utility as “Kemuri Water Company” in their Data Breach Digest; kemuri is Japanese for smoke, a nod to the report’s own title, “Smoke on the Water [plant].” The utility, the county, the exact chemicals in the exact wrong quantities — all masked. What isn’t masked is the architecture, and the architecture is the entire story.

A twenty-eight-year-old machine wearing two hats

Kemuri Water Company supplied and metered water for a population in the hundreds of thousands, across several counties. Like most utilities of its size, it ran two separate worlds on paper: an IT side that billed customers and stored their payment details, and an OT side that ran the actual plant — chemical dosing, valves, flow rates, the machinery that turns river water into something safe to drink.

On the ground, those two worlds were the same machine. KWC’s SCADA platform depended on an IBM AS/400 — a general-purpose business system IBM first shipped in 1988, three years before the web existed and a full decade before SQL injection was even documented as an attack class. By 2016 it had quietly become everything at once: an IT server holding 2.5 million customer records (names, addresses, billing, payment card data), and an OT router with direct lines into hundreds of programmable logic controllers governing valves and chemical dosing. One employee administered it. No backup, no second set of eyes, no architecture review across 28 years of accretion.

It’s worth sitting with that before the attack even starts, because the AS/400 is the case study in miniature. Nobody sat down and designed a machine that bridges customer billing and drinking-water chemistry. It got there the way most catastrophic systems do — one convenient shortcut at a time, over enough years that nobody left who remembered why.

Four steps, no exploit worth naming

Here’s the full attack, and notice what’s absent from it: no malware, no zero-day, nothing Verizon’s own analysts considered technically remarkable.

Step one — the front door. The attackers found KWC’s internet-facing customer payment portal, protected by nothing more than a username and password. They ran a SQL injection against it. Verizon didn’t need to reconstruct anything exotic here; the vulnerability was, in their words, easily identified.

Step two — the password in a text file. On that same payment webserver, investigators found the AS/400’s administrative credentials sitting in a plaintext .ini configuration file — not hashed, not encrypted, not vaulted. The account was QSECOFR, the AS/400’s built-in Security Officer profile: the closest thing that platform has to root. Whoever configured the payment application had hard-coded a connection string to the AS/400 for convenience, and nobody ever came back to fix it.

Step three — the cable. In addition to the credentials, a physical Ethernet cable ran directly from the payment application server to the AS/400. This is the detail that makes the case study, and it earns its own section below.

Step four — the plant. With QSECOFR credentials and a direct path to a system that doubled as a network router, the attackers reached the AS/400’s valve and flow-control applications. Over 60 days they connected four separate times. On at least two of those connections, they changed the setpoints governing chlorine and fluoride dosing, and altered water flow rates.

The cable is the whole lesson

Every other failure in this case study has a name in a standard somewhere — CWE-798 for the hard-coded credential, T0819 for the SQL injection. The cable doesn’t need one, because you don’t need a framework to understand what a length of Ethernet does when it runs straight from an internet-facing web server to a system controlling chemical dosing.

Industrial security has a reference model for exactly this problem: the Purdue Model, which stacks a plant’s systems into levels — internet-facing business systems at the top, physical sensors and actuators at the bottom, and, critically, an Industrial DMZ at the seam between them. The IDMZ’s entire job is to make sure nothing on the business side ever talks directly to anything on the process side. Every request gets mediated, inspected, and — ideally — physically incapable of carrying a live session straight through.

KWC had none of this. Someone — almost certainly the lone administrator, needing quick access with no change-management process to ask permission from — ran a cable between two machines that had no business sharing a segment, let alone a direct physical link. Every logical control anyone might have configured downstream — firewalls, ACLs, VLANs — became irrelevant the moment that cable existed, because Layer 1 doesn’t care about Layer 3 policy. You cannot firewall a patch cable.

This is why the cable is the detail I lead with when I talk about this case, more than the SQL injection or even the chemical dosing. Segmentation isn’t one control among many in OT security — it’s the control that makes every other control’s failure survivable. Miss the WAF and the IDMZ still catches you at the boundary. Miss the boundary and there is no more defense-in-depth; there’s just depth.

What “altering a setpoint” actually means

It’s easy to read “altered chemical dosing setpoints” as an abstraction. It isn’t one. The EPA requires a chlorine residual between 0.2 and 4.0 mg/L in a distribution system. Push it over 4.0 and customers get eye and skin irritation, gastrointestinal distress, in extreme cases chemical burns. Push it under 0.2 and chlorine stops doing its job — E. coli, Legionella, Cryptosporidium, and Giardia are no longer reliably neutralized before the water reaches a tap. Fluoride’s EPA maximum is 4.0 mg/L against a recommended 0.7; chronic overexposure causes skeletal fluorosis and severe dental damage, disproportionately in children.

Verizon’s report includes a detail that should be more alarming than it reads at first pass: the attackers showed “little apparent knowledge of how the flow control system worked.” This wasn’t precision poisoning. It was people turning dials they didn’t understand, on a system that had no idea it needed to stop them.

That second half is the part I keep coming back to. A properly designed treatment plant has a Safety Instrumented System — an independent layer, often with hardwired analog interlocks, that exists specifically to catch unsafe conditions regardless of why they occurred: equipment fault, operator error, or a compromised PLC. It doesn’t care about your threat model. It just refuses to let chlorine leave a safe band. Verizon’s report never mentions an SIS at KWC, and the outcome makes the omission hard to miss: the same PLCs that controlled dosing were the only thing that could have caught a bad dosing command — and they were exactly what had been compromised. Control system and safety system, same box, same point of failure.

The chain, start to finish

PHASE 1 · INITIAL ACCESS      SQL injection on the internet-facing payment portal
   ↓
PHASE 2 · CREDENTIAL ACCESS   QSECOFR admin credentials found in a cleartext .ini file
   ↓
PHASE 3 · LATERAL MOVEMENT    Physical cable: payment server → AS/400 (all logical controls bypassed)
   ↓
PHASE 4 · OT ACCESS           AS/400 doubles as a router into hundreds of PLCs — 4 connections, 60 days
   ↓
PHASE 5 · IMPACT              Chlorine + fluoride dosing setpoints altered; water flow rates changed

ATT&CK for ICS at a glance

Found by accident

KWC didn’t detect this. They hired Verizon’s RISK team for a proactive security assessment — not because anyone suspected a breach, but as routine due diligence. Verizon’s report notes that KWC’s own management was, going in, “adamant that no evidence of unauthorized access had been uncovered.” They were wrong, and confidently wrong, which is its own finding: an organizational culture where a compromise wasn’t just undetected, it was considered implausible.

The investigators found their way in by noticing something operators apparently hadn’t flagged as a security problem: an unexplained pattern of valve and duct movements stretching back 60 days, which they cross-referenced against Verizon’s own threat-intelligence feeds and got hits on IP addresses tied to known hacktivist activity.

Sixty days sounds almost respectable next to the roughly 146-day median dwell time Mandiant reported for enterprise breaches around the same period. It isn’t a fair comparison. IT dwell time measures how long attackers sat on data before someone exfiltrated it or got caught; here, the sixty-day clock was measuring how long attackers had intermittent write access to a city’s chemical dosing. In OT, the consequence doesn’t wait at the end of the dwell time — it starts the moment someone touches the process. And this number is a floor, not a measurement: it’s only how long the intrusion lasted before an unrelated audit happened to catch it. Left alone, there’s no reason to think it would have ended there on its own.

Five years on, an easier version of the same attack

If KWC were a one-off, it would be a curiosity. It isn’t. In February 2021, an operator at the Oldsmar water treatment plant in Florida watched his mouse cursor move on its own, take control of the HMI, and push the sodium hydroxide (lye) setpoint from 100 parts per million to 11,100 — before he grabbed the mouse back and reversed it in real time.

The attacker had gotten in through TeamViewer, on an internet-exposed HMI, using a shared password with no multi-factor authentication. Verizon rated KWC 2 out of 5 in sophistication. Independent analysis rates Oldsmar 1 out of 5 — simpler still. Five years, a body of public reporting on KWC, and a maturing OT security industry, and the attack got easier to pull off, not harder.

Oldsmar had one thing KWC didn’t: a human being paying attention at the exact moment it mattered. That’s not a control you can write into a compliance framework. CISA and the EPA’s own December 2024 joint advisory confirms the pattern never closed — they document continued discovery of internet-exposed HMIs at water utilities via Shodan and Censys, including pro-Russia hacktivists who spent 2024 tampering with setpoints and locking operators out of their own systems by changing administrative passwords.

The threat model nobody budgets for

Ask most people to picture an attack on critical infrastructure and they picture Stuxnet: a nation-state, years of engineering, a payload built to understand and specifically sabotage centrifuge physics. That story is true, and it’s also the wrong mental model for what actually threatens most utilities most of the time.

Doug Wylie, a NexDefense VP quoted in the aftermath of Verizon’s report, put it plainly: what was found at Kemuri Water Company is “likely more typical than unusual.” The skill required to reach a chlorine dosing controller in 2016 was a SQL injection cheat sheet and the patience to poke around for twenty minutes. What made the outcome possible wasn’t the attacker’s capability — it was that every architectural decision downstream had already handed them the plant.

That reframing matters because it changes where the investment should go. If your threat model is “sophisticated nation-state actor,” you build for detection and threat intelligence — SIEMs, IOC feeds, hunting teams. Necessary, but that’s solving KWC’s second-most-important problem. If your actual exposure is “anyone with a SQL injection tutorial and forty minutes,” the highest-leverage money is architectural: kill the hard-coded credentials, build the segmentation, put an independent safety layer between the network and the process. KWC’s own attackers, with “little apparent knowledge” of the system they’d broken into, are the proof that the second model is the one to plan around.

An old lesson, a newer pipe

Verizon’s report is about a US utility, but the failure mode isn’t regional. India’s National Critical Information Infrastructure Protection Centre lists power, transport, telecom, banking, and government as protected sectors — water isn’t one of them. Meanwhile the Jal Jeevan Mission is extending piped water, IoT sensors, and SCADA to a target of 190 million rural households, at a pace and scale most of the utilities it’s connecting have never had to think about network architecture before. None of that is inherently dangerous. What’s dangerous is deploying it without security requirements written into procurement from day one — which is precisely how a 1988 general-purpose server ends up wearing two hats it was never designed to wear.

Takeaways

• Sophistication and consequence don’t correlate in OT. A 2-out-of-5 attacker altered a city’s drinking-water chemistry. The relevant threat model for most utilities isn’t a nation-state; it’s anyone with a SQL injection tutorial and an unsegmented network to walk into.
• You cannot firewall a patch cable. Segmentation isn’t one item on a defense-in-depth checklist — it’s the item that determines whether every other item’s failure is survivable. Once IT and OT share a physical link, the logical controls downstream are theater.
• Safety and security are not the same layer, and one can’t stand in for the other. KWC’s PLCs were both the control system and the only thing that could have caught an unsafe command. An independent Safety Instrumented System exists precisely so a cyber compromise can’t also disable the last line of defense.
• “We’d know if we were breached” is usually false, and confidently false. KWC’s own management denied any compromise right up until an unrelated audit proved otherwise. The absence of alarms is not evidence of safety in an environment with no OT-specific monitoring to raise one.
• Five years bought a simpler attack, not a harder one. Oldsmar (2021) reused KWC’s failure class — internet-facing OT access, weak authentication — and needed less skill to pull off. Publicized lessons don’t patch systems on their own; only budget and architecture do.

Written up from a case study for the Critical Infrastructure Security course (M.Tech Cyber Security, Semester II, NFSU). Primary source: Verizon RISK Team, Data Breach Digest — Scenario 8, “Smoke on the Water [plant],” March 2016. Comparative material from CISA Alert AA21-042A (Oldsmar, 2021) and the EPA/CISA joint fact sheet on internet-exposed HMIs, December 2024.