How we kept a backend fast at 300M+ users: indexes, Redis and RPC failover

What was going wrong?

Two different problems showed up as the user base grew. First, the endpoints that are read far more often than they are written started timing out, and MongoDB's CPU stayed close to 100%. Second, blockchain transactions occasionally failed without a clear error, because the backend talked to the chain through a single RPC provider and that provider sometimes went down.

Neither problem needed a rewrite. Both needed the backend to stop doing work it did not have to do.

How do you find the queries that are slowing everything down?

Start from evidence, not guesses. Turn on MongoDB's slow query logging, then run explain("executionStats") on the worst queries. The two numbers that matter are totalDocsExamined and nReturned: if a query examines 200,000 documents to return 20, it is scanning instead of using an index, and that is where the CPU goes.

db.claims.find({ wallet: w, status: "active" })
  .sort({ createdAt: -1 })
  .limit(20)
  .explain("executionStats")
// Look for: COLLSCAN, or totalDocsExamined far above nReturned

Which indexes fix it?

A compound index that matches the query shape. Order the fields as equality filters first, then the sort field, then range filters. For the query above that means wallet, then status, then createdAt:

db.claims.createIndex({ wallet: 1, status: 1, createdAt: -1 })

With the right index, MongoDB reads only the 20 documents it returns, already sorted. Repeat this for the handful of queries that make up most of the load; you rarely need more than a few new indexes, and every extra index slows down writes, so add them where the profiler says, not everywhere.

How does Redis caching help read-heavy endpoints?

Many endpoints return the same answer to thousands of users for a few seconds or minutes: public stats, configuration, token data. Serving those from Redis instead of MongoDB removes most of the database load. The simplest pattern is cache-aside: read from Redis, and on a miss read from MongoDB and store the result with a short expiry.

async function getStats() {
  const cached = await redis.get("stats:v1")
  if (cached) return JSON.parse(cached)

  const stats = await Stats.aggregate(pipeline)   // the expensive part
  await redis.set("stats:v1", JSON.stringify(stats), { EX: 60 })
  return stats
}

Keep the expiry short for data that changes, version the key (v1) so a format change never serves stale shapes, and never cache per-user private data under a shared key.

How do you stop one RPC node from breaking transactions?

Do not depend on one provider. We integrated 5 RPC providers, picked one at random for each call to spread the load, and on failure retried on the next one, so a dead node never reached the user.

const providers = RPC_URLS.map((url) => new JsonRpcProvider(url))

async function withFailover(call) {
  const order = [...providers].sort(() => Math.random() - 0.5)
  let lastError
  for (const provider of order) {
    try {
      return await call(provider)
    } catch (err) {
      lastError = err   // try the next provider
    }
  }
  throw lastError
}

const balance = await withFailover((p) => p.getBalance(address))

In production, add a timeout per call and log which provider failed, so you can drop a provider that keeps failing.

What were the results?

• API timeouts stopped and response times stayed stable under peak load.
• MongoDB CPU dropped back into a safe operating range.
• Individual RPC node failures no longer caused transaction downtime.
• The backend held through the $185M private sale and growth past 300M users.

What should you check first on any slow backend?

1. Which 5 queries or endpoints take the most total time? Fix those first.
2. Do those queries use an index that matches their filter and sort?
3. Which responses are the same for many users? Cache them with a short expiry.
4. Which external service, if it went down, would take you down with it? Add a second one.