The check that never checked a file: reading code I didn't write

I needed a tool. A research project on how Linux persistence artifacts survive an attacker’s cleanup script wanted something that would look in a fixed set of locations, record what was there, and flag anything odd. I described what I wanted, an LLM produced 165 lines of Python, and it worked. It ran, it printed alerts, the alerts pointed at real paths, and I used it.

Recently I sat down and read it properly for the first time. Line by line, out loud in my head, asking of each line what it actually does rather than what it looks like it does.

It has fourteen defects. The tool still worked. That combination is the thing worth writing about, so this post is the list, what the defects have in common, and what I would tell anyone about to trust code they generated.

The tool is linux-persistence-collector. Everything below is about my own code.

The one that stung

The tool’s headline check hunts for timestomping, which is an attacker backdating a file to make it look old. Here is how it was written.

for p in PERSISTENCE_PATHS:
    if os.path.isdir(p):
        try:
            st = os.stat(p)
            if st.st_ctime > st.st_mtime + 3600:
                findings["alerts"].append({...})
        except: continue

PERSISTENCE_PATHS is a list of nine locations. Two of them are single files, /etc/passwd and /etc/ld.so.preload. The other seven are directories.

Read the second line again. if os.path.isdir(p) skips both files immediately. And there is no recursion anywhere in the block, so nothing inside those seven directories is examined either.

So the check ran against at most seven objects, and fewer on any machine that does not have all seven directories. Not seven thousand. Seven. The tool’s most important heuristic had never looked at a single file, and /etc/passwd, which is in the path list precisely because it is worth watching, was skipped every time.

I did not notice for a long time, and the reason I did not notice is the interesting part. The check produced alerts. Directory ctime drifts ahead of mtime for ordinary reasons, so every run flagged something, and every flag pointed at a real path with a real number of hours attached.

Had it printed nothing, I would have gone looking for why. Because it printed something plausible, I read the output instead of the code, and the output was consistent with a working check every single time.

What the check was actually supposed to do

The mechanism is worth knowing even if you never touch this tool, because it is one of the few places in forensics where you get a genuine asymmetry to lean on.

Every file on Linux carries several timestamps. Two matter here.

mtime is when the file’s contents last changed.

ctime is when the file’s inode last changed. The inode holds permissions, ownership, link count and the timestamps themselves, so ctime moves when any of those move.

ctime is not creation time, despite what the name suggests. It is change time. This misreading is extremely common and it has a real source: on Windows, ctime genuinely does mean creation time. Python’s own documentation carries the split, describing st_ctime as the time of most recent metadata change on Unix and the time of creation on Windows. Linux does record a birth time, called btime, but it arrived later, it sits behind the statx() syscall, and most tooling still does not show it.

Now the useful part. Under normal use, mtime and ctime move together. Writing to a file changes the contents, which changes the inode, so both land within moments of each other.

An attacker backdating a file runs something like touch -d "2020-01-01" evil.service. That sets mtime to 2020. But the act of setting it is itself an inode change, so ctime updates to right now.

You cannot set ctime directly. There is no syscall for it. utimes() and its relatives move atime and mtime, and every one of them drags ctime forward as a side effect. Avoiding the tell means changing the system clock or writing to the inode through the raw block device, and both of those are a much higher bar with traces of their own.

You can watch it happen on any Linux box.

touch /tmp/demo
stat /tmp/demo                       # Modify and Change agree
touch -d "2020-01-01" /tmp/demo
stat /tmp/demo                       # Modify is 2020, Change is now

That gap between Modify and Change is the whole heuristic. My code looked for it correctly and then applied it to seven directories.

Four failures, one message

except:
    data["error"] = "Could not execute ps/ss"

That block wrapped two external commands, ps auxww and ss -tulpn. Here is what it collapses into one string.

All four wrote Could not execute ps/ss, which tells you nothing about which one happened.

There is a fifth case that is worse, and it is not in the table because it never reaches the except at all. Run ss -tulpn without privilege and it does not fail. It exits cleanly and simply omits the process names. So the tool recorded a partial socket table as though it were a complete one, with no error and no note. In a report, “socket collection was impossible because ss is not present on this host” and “socket collection ran, but every process name is missing” are different sentences with different consequences. My tool produced a confusing message for the first and complete silence for the second.

The Ctrl-C row is the one that should bother you. A bare except: catches KeyboardInterrupt and SystemExit, which are not errors but control flow. I had three of them, and one sat in the function called for every single file while walking /usr/lib. Press Ctrl-C during that walk, Python raises KeyboardInterrupt inside the stat call, the bare except swallows it, and the loop moves calmly on to the next file.

The other cost of a bare except is that it hides your own mistakes. If I had typed st.st_mtim instead of st.st_mtime, that is an AttributeError, the except catches it, and the function returns None for every file on the system. No error message, no output, no clue.

The version I use now names what it expects.

try:
    info = os.lstat(path)
except OSError as error:
    return {"path": path, "error": str(error)}

Two things changed. OSError is specific enough that anything unexpected still reaches me. And the failure is recorded rather than discarded, because “I could not read this file” is itself evidence, particularly if it was readable in last week’s baseline.

The one it got right

Not everything was wrong, and the thing it got right is the hardest call in the file.

st = os.lstat(path)

os.stat() follows symlinks and describes what the link points at. os.lstat() does not follow and describes the link itself.

That distinction is load-bearing here. When systemd enables a service it creates a symlink in a .wants directory pointing at the unit file. An attacker who installs a unit, gets persistence, then deletes the unit file to tidy up leaves the symlink behind unless they also ran systemctl disable. The link now points at nothing, and it still carries the name of what was deleted.

os.stat() on that link raises FileNotFoundError, because the target is gone. os.lstat() succeeds, because the link is sitting right there on disk. Use stat and the tool is blind to its single best piece of evidence.

The generated code chose lstat, correctly, in the function that collects metadata, on line 41. Then it used os.stat in the timestomp check on line 105. Same file, same question, two different answers, sixty lines apart.

That pattern is worth naming. The model knew the idiom. It did not track the consequence across the file. Reviewing generated code means checking consistency between places, not just correctness in one place.

The defect that was not a bug

if f.endswith(".timer") and "research" in f:

That check flags a systemd timer only if its filename contains the word “research”. No attacker names a file that. It matched the units my own experiment planted, which is exactly what it was written to do.

That is not a bug. It is the correct design for a measurement instrument, whose job is to confirm that known artifacts survive a known cleanup procedure. The problem is that nothing said so. The alert type was P3_TIMER_FOUND and the message read “Custom systemd timer detected”, which describes a detector. Anyone reading that output, including me when I came back to it, would believe the tool did something it never did.

The fix was not to the logic. It was to write the limitation into the source where nobody can miss it.

# KNOWN LIMITATION, left in place deliberately.
#
# The next two checks match the word "research", which is the marker our own
# experiment used when planting units. Nobody names real malware that, so
# these will not find an attacker.

A limitation that lives only in your head is one nobody else can act on, and one you will have forgotten in a month.

The flag that ignored you

parser.add_argument("--baseline", help="Capture baseline snapshot")

With no action=, an argparse option takes a value. So --baseline on its own was an error, and --baseline clean.json set args.baseline to the string "clean.json".

The code then only ever tested if args.baseline: and wrote the file to args.output, which defaulted to triage_results.json. Typing --baseline clean.json produced a file called triage_results.json and told you nothing.

A command line that accepts input and then ignores it is worse than one that rejects it, because the user believes they were heard. The fix is one keyword: action="store_true".

What they all have in common

Every defect above is the same shape. Different things were made to look the same.

Four unrelated failures wrote one message. A check over directories looked identical to a check over files. A flag that discards its argument looked identical to one that uses it. A tool that matches your own test markers announced itself as a detector.

That is the thing to hunt for when you read code you did not write. Not “is this line correct”, which generated code usually is. Ask instead: if this went wrong in three different ways, could I tell them apart from the outside?

How to read generated code

That question is easier to ask than to answer by staring at the source. Almost every one of my fourteen defects was a line that looked perfectly reasonable on its own, so reading alone is slow and unreliable. You have to run the thing. Here is the order I use now.

1. Start where you already know the answer

Use a fresh VM you have not touched.

You know in advance what a persistence scan should find on a clean machine: nothing. So every alert it prints is a false positive, and you can go and find the line that produced it.

Run the same tool on a machine whose history you do not know and you lose that. Every alert looks reasonable, because you have nothing to check it against. That is exactly how I used this tool for as long as I did without noticing anything was wrong.

2. Then plant the thing each check is meant to catch

One at a time. Create a dangling symlink. Backdate a file with touch -d. Drop a script into /etc/update-motd.d/.

If the alert appears, that check works. If nothing appears, you have found either a defect or a check that needs some input you have not thought of, and both are worth knowing about.

This would have caught the two worst problems in my tool on the first attempt. Plant a timer and no alert appears, because the timer check only wants filenames containing “research”. Backdate /etc/passwd and no alert appears, because the timestamp check only ever looked at directories.

I found both by reading instead. That worked, but it took an afternoon of close attention, and I only got round to it long after the tool had already done its job. Ten minutes of planting artifacts would have found the same two things on day one. I never did it once.

3. Search for except: before you read anything else

Reading top to bottom will not make these stand out, because a bare except looks exactly like ordinary error handling. Searching for them will.

Then, for each one you find, deliberately cause the failure it is meant to catch and read the message you get back. If several different causes produce the same message, that is your defect. My tool printed Could not execute ps/ss whether ss was missing, whether it had exited non-zero, or whether I had pressed Ctrl-C.

4. Write the limits into the source, not the README

The README is what you tell other people. A comment in the source is what you tell yourself in a year, when you have forgotten which of your checks were real ones.

What is still wrong

The rewrite fixed fourteen things and left three.

The baseline is still never compared against anything. --baseline writes a JSON file that no code in the program reads. Diffing two snapshots is the obvious missing feature and the one that would turn this from a recorder into a tool.

Nine locations is not the whole picture. Kernel modules, initramfs, shell startup files, PAM, and systemd generators are all real persistence locations and none of them are covered.

It reads the live filesystem through the kernel. Anything with enough privilege to install a rootkit has enough privilege to lie to it. For a real incident, image the disk.

The point

I am not arguing against generating code. I generated this, it did its job, and the research it supported is real.

What I got wrong was treating “it runs” as the end of the work. The reading is the work, and it is the part that does not get shorter. A model can produce a lstat call correctly and still leave you with a check that has never examined a file, and nothing in the output will tell you which of those you have.