Establish which address field is trustworthy
Document whether the log comes from CDN, load balancer or origin and how the client address is forwarded. A proxy address may identify an intermediary. Do not treat a client-controlled header as the verified sender address without checking the trust chain.
Keep only the information needed for the agreed analysis and protect the limited raw sample. If an export lacks a trustworthy client address, summarise user-agents as claims rather than upgrading them to verified identity.
Use the current method for the correct category
Follow the provider's current verification instructions, such as confirmed DNS checks or current published network ranges for the relevant category. For DNS verification, a plausible reverse lookup name is insufficient; forward resolution must also agree with the original address.
Record method, check time and outcome. Frozen historical lists are not a permanent source of truth. Keep common crawlers separate from special or user-triggered fetchers where the provider distinguishes them. An unknown result can remain unknown.
| Evidence | Supported classification | Next question |
|---|---|---|
| User-agent only | Claimed crawler name | Is a trusted client address available? |
| Complete current verification | Verified sender category | Which crawler type and check time? |
| Technical lookup failure | Unknown or unverified | Were the method and lookup available? |
| Origin logs only | Observation in one delivery layer | Which cached responses are invisible? |
| Incomplete period | Bounded sample | Which intervals are missing? |
Describe coverage before calculating percentages
State period, time zone, host, logging layer and any sampling procedure. Check whether errors, different methods and cache hits are included. Do not combine two exports in a way that double-counts the same request without understanding their relationship.
Count requests in the observed sample and name the denominator. A higher error share can follow a changed export or traffic group. A missing URL in a small origin export shows only that it is absent there, rather than never fetched by a crawler.
Make the finding possible to review again
Choose a bounded pattern, such as errors for one language group, and retain aggregate evidence plus a protected example for the responsible technician. State alternative explanations and the additional check that separates them.
After an agreed technical change, compare a relevant new sample under the same definitions. A verified successful fetch is not proof of indexing. Authorised search tools and actual content need separate checks; logs cannot support promises about rankings or AI answers.