A Dead-Link Checking Batch: Not Treating a Failed Response as Grounds for Deletion
Looking back at content management and dead-link check automation: telling a temporary failure of an external link from a real shutdown, and leaving evidence for a human to judge.
Joohwan Yu · Founder, Full Stack Developer
Toris · Published
Contents
In content management work, I handled a batch that checks for dead links. The goal was to cut down the repetitive work of a person opening links one by one to see if they were alive. At first it looks like sending HTTP requests and collecting the results would be enough, but interpreting failures from external systems was harder than that.
A failed response is different from content being gone
A timeout may just mean the other server is slow. Restricted access may mean it blocks automated requests. And the mere presence of a redirect doesn’t mean the original content can’t be viewed. If you reduce the check result to a simple true or false, the reason for the operator to check again disappears.
Looking back on that experience, I arrived at the standard that automation is good at observation and the final judgment is separate. Linking every failure straight to deletion or unpublishing goes too far. For links outside the service in particular, I can’t control the server’s state, so the uncertainty of the result has to be preserved.
How I would split it now
| Observed result | Next action the operator can take |
|---|---|
| Normal response | Update the last-checked time |
| Timeout or temporary error | Recheck after an interval |
| Access restricted | Separate out as a target for manual browser check |
| Repeated not-found response | Review whether the original has ended |
This table is not a reproduction of the exact implementation of the batch at the time. It is the classification standard I would apply if I built the same feature again. A person can continue the judgment only if there is observed evidence such as the response code, the check time, and the final destination address.
The results of automation also need review
Checking many addresses at once can put a burden on both the other site and our own system. Now I would state the concurrency and the time limit explicitly, and decide where to resume if it stops midway. I would also report separately that the batch itself succeeded and that every link was checked.
The value of this experience for me is not in a number like how many hours automation saved. I can’t attach such a figure if I didn’t measure it separately. Instead, I learned the direction of turning work that people repeatedly opened by hand into a checkable list. Automation was more trustworthy as a tool that steadily gathers the observations needed for judgment than as a tool that rushes to conclusions.
Related post: Scope of change on an operating project
Advertisement