In case anyone wonders about the why: We have some services where we do updates without dropping a client connection by first spawning an additional server, then moving traffic over to the new server, and then the old server will detect that it's idle and its socket has been removed and terminate itself.
At least, that was the theory. In practice, sometimes freak accidents cause libUSB to hang when the process exits. That leads to the old server keeping its file handles open. That means even if log rotation deletes the log file, the disk space cannot be reclaimed. And that means over time the HDD mysteriously fills with deleted files stuck in zombie processes. Of course, one could use killall to terminate all those servers. But that's difficult to coordinate so that you never accidentally terminate one of the new servers, but instead only the old ones.
That's where bombtag comes in. You first mark all running servers as "should exit in 5 minutes" and then you do the usual hot reload. If any server gets stuck, bombtag will forcibly kill it, first with SIGTERM and then with SIGKILL.
For added fancyness, I also relocate the env vars and overwrite argv[0] so that inside the "ps" output, bombtag can show what PID it's waiting for and the current countdown in seconds, e.g.
user 2239536 0.0 0.0 18316 1792 pts/5 TN 23:00 0:00 cat
user 2239542 0.0 0.0 6092 512 pts/5 SN 23:00 0:00 bombtag -p 2239536 -n cat (29s -> TERM)
At least, that was the theory. In practice, sometimes freak accidents cause libUSB to hang when the process exits. That leads to the old server keeping its file handles open. That means even if log rotation deletes the log file, the disk space cannot be reclaimed. And that means over time the HDD mysteriously fills with deleted files stuck in zombie processes. Of course, one could use killall to terminate all those servers. But that's difficult to coordinate so that you never accidentally terminate one of the new servers, but instead only the old ones.
That's where bombtag comes in. You first mark all running servers as "should exit in 5 minutes" and then you do the usual hot reload. If any server gets stuck, bombtag will forcibly kill it, first with SIGTERM and then with SIGKILL.
For added fancyness, I also relocate the env vars and overwrite argv[0] so that inside the "ps" output, bombtag can show what PID it's waiting for and the current countdown in seconds, e.g.