Content discovery at scale without getting rate-limited

Content discovery is brute force by nature: tens or hundreds of thousands of requests to find the handful of paths and parameters that matter. Run that from one box at full tilt and you get rate-limited, WAF-blocked, or simply IP-banned before the interesting results show up. The fix is not "go slower and pray." Crossfyre spreads the work across your own nodes and proxy routes and paces the run for you, so a large authorized scan gets broad coverage with far fewer blocks, and an interruption costs minutes, not the whole run. This post covers that with mach, the Crossfyre content-discovery engine.

Why one box at full speed fails

A defender sees content discovery as a flood of requests from a single source, mostly returning 404, climbing a wordlist. That pattern is trivial to rate-limit or block. Crank concurrency higher and you trip the threshold faster; crank it lower and the run takes days. The single source is the real problem, not the speed alone, and that is the constraint Crossfyre is built to remove.

A baseline mach run

Start with the engine itself. mach substitutes each wordlist entry into the ::FUZZ:: marker and records what comes back. It exposes the usual pacing knobs, including --tasks (concurrent in-flight requests) and --interval (milliseconds each task waits between requests), so you can run it standalone when you want to.

bash
# Single-node path discovery, paced to stay polite.
mach scan \
  --url "https://app.example.com/::FUZZ::" \
  --wordlist-path ./wordlists/raft-large-dirs.txt \
  --tasks 30 \
  --interval 40 \
  --success-status-codes 200,204,301,302,401,403 \
  --headers "User-Agent: Mozilla/5.0"

Tracking the right status codes matters: a target that returns 403 or 401 for protected paths is telling you those paths exist, so do not filter them out. Run mach standalone like this, though, and you are back to one address moving at one speed, which is exactly the wall the rest of this post gets past.

Distribute the work across a node fleet

The single-source problem disappears when the run does not come from a single source. When you launch a content-discovery workflow on Crossfyre, the control plane splits it into discrete operations and publishes them onto a durable, acknowledged work queue. Each node pulls operations and runs the mach daemon against them, so the wordlist is chewed through in parallel across machines with different egress addresses.

• Throughput scales with the fleet. Five nodes mean roughly five times the completed work per minute, without any single node working harder.
• Load spreads across addresses. The target sees traffic arriving from several sources instead of one source hammering, so a large run is far less likely to look like an attack from any one address.
• No babysitting. The platform assigns and paces the work across the fleet for you, so you launch the job and let it run instead of hovering over one box.

Layer proxy chains over egress

Nodes add address diversity at the machine level. Proxy chains add it underneath. Each Crossfyre node can route its outbound traffic through layered proxy chains and isolated VPN tunnels, so scans leave from where you choose and never from the host network. Combine that with a fleet and you get many egress paths carrying the run instead of one.

• Spread, not hide. Splitting egress across proxy routes gives a large run many exit points instead of one, so no single address carries the whole scan.
• Egress control. Because traffic leaves through a controlled namespace, you decide the exit point per node and keep the scan off the host’s own network.
• Composability. Fleet distribution and proxy routes stack: your nodes and their proxy paths together give a large run many independent exit points.

Pacing you do not have to babysit

There is no single correct rate for every target, and finding it by hand means sitting over a run and reacting to it. Crossfyre does that part for you: it paces the run adaptively across your nodes and proxy routes, adjusting as conditions change, so you get strong coverage without translating limits you cannot see into flag values by trial and error.

• Adaptive pacing, handled. The platform manages how hard the run pushes, so you are not tuning numbers live to keep it moving.
• Broad coverage, fewer blocks. Spreading and pacing together mean a large run keeps making progress instead of stalling behind a wall of blocks.
• Sharper wordlists still win. A focused list (right tech stack, right conventions) finds more in fewer requests than a giant generic one, which is the cheapest way to keep a run light.

Resume instead of restarting

Long runs get interrupted: a node is reclaimed, a target starts tarpitting, you pause to adjust. mach persists scan state, so rerunning the same command continues from where it stopped rather than re-walking the wordlist. Pass --fresh-start only when you intend to discard saved state.

Across the fleet this is automatic. Operations live on the durable queue until a node completes them, so a node that drops mid-run has its in-flight work redelivered to a healthy one, and nothing already finished is rescanned. With reserve-then-reconcile billing you are charged for completed work, not for the attempt that died. The full mechanics are in how scans survive a crashed node.

Put it together: a tuned wordlist, work spread across your fleet, egress layered over proxy routes, adaptive pacing handled by the platform, and crash-safe resume. That is how a six-figure content-discovery run finishes without getting your addresses banned halfway through. New to the engines? Start with meet the engines, and see pricing for node and concurrency limits per plan.