Find out which fetches your logs can actually establish.
When important pages appear overlooked, delivery logs can provide supporting evidence. ImbaHub can analyze an authorized, bounded sample and distinguish verified crawler traffic from claimed names. We state what the logging layer sees and which conclusions need other evidence.
- Source
Define layer, period and known gaps.
- Verification
Classify senders using a documented method.
- Interpretation
Connect observed responses with a testable technical finding.
Observed fetching is different evidence from indexing or visibility.
Origin has no request, but CDN might have served a cached response. Was the page never fetched?
Origin alone cannot establish that. Review delivery-layer coverage and supporting evidence.
Start with a question and an available source
Choose a task, such as whether a defined language group returns errors during fetches. Bound the host, environment, period and URL group. Establish whether evidence comes from CDN, origin or another layer and whether cache or sampling can hide requests.
Actual findings require access and sufficient fields. Agree secure transfer, limited access and handling of information outside the task. A report should not reproduce entire raw logs or personal query parameters merely to demonstrate a technical issue.
Verify crawler classes before counting
A user-agent name is a claim the sender can choose. Classification needs a current provider method and a trustworthy client address. Behind a proxy, establish where the address field originates; arbitrary client-supplied headers are not trusted evidence.
Separate verified, unverified and unknown traffic. A failed DNS lookup alone does not justify calling the sender an impersonator. Record verification method and time, and keep different crawler categories separate even when the same company operates them.
| Question | Evidence | Limit |
|---|---|---|
| Who fetched? | Verified sender category | User-agent alone is insufficient |
| Which address? | Host, path and relevant method | Logs may normalise the URL |
| Which response? | Status at the observed delivery layer | Origin and CDN can see different responses |
| How often? | Count within the bounded sample | Not total traffic without coverage evidence |
| What does it mean? | Technical pattern and next check | No automatic indexing conclusion |
Connect patterns with repeatable checks
A fictional finding might show repeated error responses to a verified crawler in a specified language branch. The next step is to check affected URLs and delivery rules, rather than assume weak content. Keep the request, its response and later search engine decisions separate.
Compare the relevant group with a control and record configuration changes or missing intervals. Many requests to one URL can have several causes. A small sample without a fetch does not establish that the page has never been fetched.
Receive a bounded report with next steps
The delivery describes log source, coverage, verification method, aggregate observations and a prioritized technical check. Each finding has sample evidence and explicit uncertainty. CDN, hosting or application changes are scoped separately.
Scope and pricing follow the available evidence and question. Authorized search tools can add indexing status where access exists. A log response proves neither rankings, AI citations nor later customer outcomes.
Sources and further reading
Questions, answered
Can you analyze without our logs?
We can define the question and data requirements, but cannot report actual fetching patterns without sufficient evidence.
Does a successful Googlebot fetch mean the page is indexed?
No. It establishes the verified request and observed response; indexing status needs separate support.