I was building the inventory of a PHP application with fifteen years in production: a script walked its screens one by one to work out what was actually in there. Audit work. Nothing written, only read.
Mid-afternoon I noticed the laptop fans had been at full tilt for a while. I looked at the container: eight Apache processes with hours of accumulated CPU each, and the container at 794% — eight cores pinned. According to the sensors, the machine had spent three and a half hours between 94 and 96 degrees.
Not one of those eight processes had anyone on the other end. My sweep had closed those connections hours earlier and moved on quite happily.
- 8
- Apache workers spinning at once
- 3 h 30
- at 100% CPU with nobody asking for it
- 96 °C
- machine temperature for the whole stretch
- 3
- safeguards in place that did nothing at all
Three safeguards, and not one of them fired
The first was my own sweep's timeout. Twenty seconds per page: if it did not load, cut and carry on. It worked exactly as written, and that is the problem. A client timeout cuts the connection, not the work. Nobody on the far side finds out that there is no longer anyone waiting.
The second was PHP's disconnect detection. It exists, and it usually works: when the client goes away, the script aborts. But PHP only discovers the client has gone when it tries to write to the output. A loop that prints nothing never reaches that point, so it never finds out. Apache does not kill it either: the connections sat in CLOSE_WAIT and the worker kept spinning with a half-closed socket in its hand.
The third was max_execution_time, PHP's oldest safety net. I set it to 60 seconds, reloaded one of the bad pages and watched it with a stopwatch. At 120 seconds it was still going.
The reason is documented and easy to forget: on Unix that limit does not count time the process spends outside PHP — system calls, stream operations, database queries — and the timer can only fire when the engine gets control back between instructions. A loop that lives inside a native call never gives it that chance. The limit exists, it is configured, and it does not apply where it is needed.
All three are the same thing seen from three angles: limits enforced where it does not hurt. A process that refuses to cooperate can only be stopped from outside the process.
The column that nearly fooled me
My first instinct was to open ps and sort by %CPU. That is the wrong column. ps's %CPU is an average over the entire life of the process, and an Apache worker lives a long time and serves many requests: one that has been idle for three hours and has only just started looping shows a laughably small figure. The case you most want to catch early is the one that looks best.
The column that does not lie is TIME: accumulated CPU time. In a healthy application that reads in seconds. Here it read in hours.
What did work: killing them from outside
Twenty-five lines of bash. Every fifteen seconds it asks the container which processes it has, keeps the apache2 ones carrying more than 60 seconds of accumulated CPU, and kills them.
And what actually matters is not the script, it is where it hooks in. The sweep starts it by itself before the first visit, and refuses to run if it cannot find it:
A safeguard you have to remember to switch on is not a safeguard. It is a mental note, and mental notes get lost at seven in the evening on a Monday.
What is still wrong, and I would rather say it before you find it
The loop is still there. The watchdog is a bandage on the side of whoever is looking, not a fix on the side that is bleeding. Four pages of that application either loop or hang waiting on a machine on the internal network; they are on the sweep's exclusion list and in the client's report. Fixing them is a separate job, with its own budget and its own decision.
The first version of the watchdog was left orphaned. The sweep sent it SIGTERM on exit, but the child process survived. Harmless — the worst it does is watch when nobody asked — but it is exactly the loose end somebody finds three months later without knowing what it is or whether it is safe to kill.
And the uncomfortable part: nothing warned me. No alert, no threshold, no log line. I found out because I rested a hand on the laptop and it was burning. Three and a half hours is precisely how long it took me to touch the case.
What I take from it
When I go into an old system, the question I now bring with me is not only "what does this do?". It is also "in what specific way is this going to bite me while I look at it?". An inventory, a sweep or a diagnostic script is production code for as long as it runs, and the system you are looking at is under no obligation to behave.
None of this appears in the report I hand over. But it is half the work of touching legacy software, and it is the half nobody sees.
The client is under a confidentiality agreement: no name, no code, and nothing about their system beyond what it did to my machine. The watchdog and the sweep are my own tooling, which is why they are shown in full. Stack: PHP on Apache in Docker, sweep in Node with Playwright.