“It happens sometimes” is the hardest class of fault to deal with, and intermittent faults are the ones most likely to be declared fixed when they are not. The temptation is to change several plausible things at once, watch the problem disappear for a fortnight, and move on. It comes back, and now you have three changes and no idea which mattered.
There is a better method, and it is mostly discipline rather than cleverness.
1. Establish what “sometimes” means
Before touching anything, characterise the pattern. Not the cause. The pattern.
- When did it first occur? What changed around that date?
- What time of day, and what day of the week?
- Which users, which devices, which browsers, which locations?
- How often, roughly, per day?
Most intermittent faults stop being intermittent once described precisely. “Sometimes the form fails” becomes “the form fails for users on mobile data during the busiest hour”, which is nearly a diagnosis on its own.
If nobody has this information, the first job is instrumentation, not repair. You cannot debug what you cannot observe.
2. Make it reproducible, or make it visible
If you can reproduce it, the rest is straightforward. Usually you cannot.
The alternative is to make it visible when it next happens: log the request, the user, the timestamp, the inputs and the resulting error, and keep them long enough to compare several occurrences. Three logged occurrences almost always reveal what they have in common.
This step feels like a delay. It is the fastest route, because everything after it is evidence rather than speculation.
3. Change one thing
The rule that gets broken most often, under pressure, with the best intentions.
If you change three things and the fault stops, you have learned nothing except that one of them mattered, and you are now carrying two unnecessary changes with their own risks. If you change one thing and it stops, you know.
Write down each change, the time you made it, and what you expected. Faults with a slow recurrence need a record, because your memory of last Tuesday is not reliable.
4. Distinguish absence from correctness
A fault that occurs twice a week and has not occurred for three days is not fixed. It is quiet.
Decide in advance how long absence has to last before you believe it, based on the observed frequency. Roughly three times the normal interval between occurrences is a reasonable threshold. Say it out loud to whoever is asking, because “it seems fine now” invites the change to be forgotten.
The usual suspects behind intermittent faults
Intermittent faults cluster around a small number of causes:
- Timeouts under load. Fine at normal traffic, fails at peak. Correlates with time of day.
- Caching. Some users get a stale version. Correlates with who, not when, and often disappears for whoever is investigating because they hard-refresh out of habit.
- Scheduled jobs colliding. Something heavy runs at the same time each night and starves everything else.
- Expiring credentials. Not intermittent at all, just periodic. Often monthly or annual, which is why nobody spots the pattern.
- Race conditions. Two things happening close together in the wrong order. Correlates with load and is worst on the fastest connections, which is counterintuitive.
- One server in a pool. If traffic is distributed, a fault on a single node presents as a fraction of requests failing at random.
5. Write down the conclusion
When it is found, record the symptom, the cause, the fix and how it was confirmed. Two paragraphs.
The next occurrence of the same class of problem, possibly years later, will be solved in ten minutes by someone reading that note rather than in three days by someone starting again.
Methodical investigation is the core of our maintenance and troubleshooting work, and the reason we would rather spend a day observing than an afternoon guessing.
Related reading
- Remote or on-site support: what each is genuinely for — Most problems are solved faster remotely. Choosing remote or on-site support well saves both a wasted visit and a wasted afternoon. How to tell them apart.
- WordPress security: what matters and what is theatre — Most WordPress security incidents come from outdated plugins and weak accounts, not clever attacks. What to actually do, and what to stop bothering with.
- Retail and CPG: getting one honest stock figure across channels — Selling in a shop, online and through a marketplace means three systems each confident about retail stock. How to get to one number everyone trusts.