5 bugs your tests won't catch

A green test run tells you the code does what your tests asked it to do. It tells you very little about the things nobody asked.

The five bugs below all pass a normal test run. None of them crash. That is exactly why they get through: tests are good at catching code that breaks, and these don't break. They quietly do the wrong thing.

For each one: what it looks like, why the tests miss it, and the fix.

1. The failure that says "success"

try:
    gateway.charge(order.card, order.amount)
except Exception as e:
    logging.error("charge failed: %s", e)
    return {"status": "success"}

Why tests miss it. Most tests cover the happy path, where the charge goes through. The failure path is rarely tested, and here it returns the same answer as the happy path. The caller, the UI and the person paying all think the payment worked. The only trace is one line in a log nobody is reading.

The fix. Let the caller find out. Either re-raise the error, or return something that is clearly a failure:

try:
    gateway.charge(order.card, order.amount)
except GatewayError as e:
    logging.error("charge failed: %s", e)
    return {"status": "failed", "reason": str(e)}

Then add one test that makes the gateway raise and checks the result is not a success. That single test covers the path that matters most.

2. SQL injection

def search_orders(cursor, term):
    cursor.execute(f"SELECT * FROM orders WHERE note LIKE '%{term}%'")
    return cursor.fetchall()

Why tests miss it. The tests search for normal words like "blue" or "urgent", and the query works. But the input is pasted straight into the SQL, so a single quote in the search box changes the query itself. Someone typing the right string can read or delete data they should never touch.

The fix. Never build SQL with f-strings or string concatenation. Pass the value as a parameter and let the driver handle it:

def search_orders(cursor, term):
    cursor.execute("SELECT * FROM orders WHERE note LIKE %s", (f"%{term}%",))
    return cursor.fetchall()

The placeholder is %s for psycopg and most MySQL drivers, and ? for sqlite3. ORMs do this for you, until someone drops down to raw SQL for one quick query.

3. A secret in the code

const MAILER_API_KEY = "mk_live_7c1e...9f2c";

export function send(to, body) {
  return fetch(MAIL_URL, {
    headers: { Authorization: MAILER_API_KEY },
  });
}

Why tests miss it. Nothing fails. The emails go out and every test passes. The key just sits in every clone, every fork, every build log and every laptop that ever pulled the repo.

The fix. Read it from the environment, and fail loudly if it's missing:

const MAILER_API_KEY = process.env.MAILER_API_KEY;
if (!MAILER_API_KEY) {
  throw new Error("MAILER_API_KEY is not set");
}

Then rotate the old key. Deleting the line isn't enough, because the key is still in your git history.

4. An LLM call inside a loop

for ticket in tickets:
    reply = client.messages.create(
        model="...",
        max_tokens=200,
        messages=[{"role": "user", "content": ticket.text}],
    )
    out.append(reply)

Why tests miss it. The test runs with three tickets, so it's fast and cheap. In production there are three thousand, which means three thousand paid calls, one after another. The bill and the wait both grow with every new ticket, and no test fails along the way.

The fix. Make fewer calls. Put several items in one request and ask for one answer per item:

for i in range(0, len(tickets), 20):
    chunk = tickets[i:i + 20]
    text = "\n\n".join(f"[{t.id}] {t.text}" for t in chunk)
    reply = client.messages.create(
        model="...",
        max_tokens=2000,
        messages=[{"role": "user",
                   "content": "Summarise each ticket in one line. Keep the [id].\n\n" + text}],
    )
    out.append(reply)

Two more things help. If the same text comes up again, cache the answer by a hash of the text. And if you don't need the answers right away, most model providers have a batch API that costs less than live calls.

app = FastAPI()

@app.delete("/orders/{order_id}")
def delete_order(order_id: int):
    db.delete(order_id)
    return {"deleted": order_id}

Why tests miss it. The test calls the route and it works. So does a request from anyone else on the internet. Tests usually check that a route does its job, not who is allowed to call it.

The fix. Add your auth dependency, and check that the order belongs to the user:

@app.delete("/orders/{order_id}")
def delete_order(order_id: int, user: User = Depends(require_user)):
    order = db.get(order_id)
    if order is None or order.owner_id != user.id:
        raise HTTPException(status_code=404)
    db.delete(order_id)
    return {"deleted": order_id}

And add the test that's usually missing: call the route with no token and expect a 401.

Why tests miss all five

Tests check what you expected. These bugs live in what you didn't expect:

the error path (bug 1)
strange input (bug 2)
things that never fail at all (bug 3)
real-world size (bug 4)
who is calling (bug 5)

A few habits catch most of them in review:

For every except or catch, ask what the caller sees when it runs.
Search the diff for SQL built with f-strings, + or template strings.
Search for anything that looks like a key, token or password in quotes.
For every model call, ask how many times it runs with real data.
For every new route, ask who can call it, and whether there's a test for "not logged in".

Checking for them automatically

Review habits work until you're tired, or the diff is 2,000 lines. So I wrote a small free tool that looks for exactly these five in Python and JavaScript/TypeScript: silentbug-lite. It's open source (MIT). It reads your code without running it, sends nothing anywhere and uses no AI, so the same code always gives the same result.

pipx install git+https://github.com/manish-drops/silentbug-lite
silentbug-lite path/to/your-project

It also runs as a GitHub Action on every push and pull request. The setup is in the README.