Observed requests, classified with care
Edge and origin logs record what actually requested a URL, when it happened, how the server responded, and which route family was involved. We normalize those records into a usable event stream, verify crawler identity using appropriate network checks, and separate confirmed search crawlers from user agents that merely claim a familiar name. Without that verification, a crawl report can mistake scraping, monitoring, or spoofed traffic for search-engine behavior.
We classify requests against the site’s route model: canonical landing pages, product and category routes, faceted combinations, internal search, pagination, obsolete paths, and malformed variants. The result is a view of bot attention by class and outcome. It highlights where useful pages are being revisited, where low-value states dominate, and where server responses indicate a discoverability problem rather than a simple crawling preference.
Prune traps and resolve soft-error clusters
Faceted navigation can create a combinatorial route space from a manageable catalog. Sort order, filter sequence, tracking parameters, and empty result states may each produce a distinct address. We identify which facets have standalone search value and which combinations should remain useful only to visitors. Then we align link generation, canonical behavior, robots directives, and response handling so the intended policy is consistent throughout the platform. Blocking a route in one layer while generating thousands of links to it in another rarely produces a clean result.
Soft 404s often cluster around discontinued inventory, empty categories, expired campaigns, and application states that return a successful status with little meaningful content. We trace those clusters back to their templates and lifecycle rules. The correction may be a relevant replacement, a properly maintained page, or an honest not-found response. The goal is to stop presenting thin or empty states as viable destinations while preserving useful alternatives for visitors and crawlers.
Concentrate internal authority on useful hubs
Internal links shape discovery and communicate which sections matter. We measure how templates connect to commercial hubs, how deep those hubs sit from stable entry points, and whether repeated links point to parameterized or stale variants. Improvements often come from simple structural decisions: reduce dead-end branches, strengthen contextual paths to high-value categories, and make navigation resolve to the canonical form consistently.
We establish a baseline before changing directives or link logic, then compare request distribution and response patterns after release. Crawl activity can shift for reasons outside a single change, so we annotate deployments and compare equivalent periods and route cohorts. The resulting readout separates request reduction from actual improvement: fewer wasteful fetches matter when important pages remain accessible, are recrawled appropriately, and continue to earn visibility and qualified visits.
Continuity and accountability
AI retrieval agents add a new reason to understand request behavior, but they do not justify treating every unfamiliar user agent as a trusted search system. We identify the agent, confirm its published behavior where possible, and analyze its requests separately from conventional search crawlers. This lets teams make informed access decisions, distinguish retrieval activity from indexing, and avoid changing site policy based on noisy labels. The same discipline applies to security, performance, and capacity questions raised by automated traffic.