imbahub
Menu +
EN
Talk to us
IMBAHUB / Knowledge & guides

Verify the sender and the sample before counting crawler visits.

A log entry claiming to be Googlebot or Bingbot is not a completed identity check. A sample also does not automatically include every fetch. Use this guide to classify senders and describe what the available evidence actually covers.

Three categories instead of an assumed bot count
  1. Verified

    The correct method supports the sender category.

  2. Unverified

    The name lacks sufficient verification evidence.

  3. Unknown

    A required field or lookup result is unavailable.

Unknown must not be counted as certainly genuine or certainly false.

The export has user-agents but no trusted client IP. Are all Googlebot rows verified?

No. Report them as claimed identities and state what evidence verification needs.

Establish which address field is trustworthy

Document whether the log comes from CDN, load balancer or origin and how the client address is forwarded. A proxy address may identify an intermediary. Do not treat a client-controlled header as the verified sender address without checking the trust chain.

Keep only the information needed for the agreed analysis and protect the limited raw sample. If an export lacks a trustworthy client address, summarise user-agents as claims rather than upgrading them to verified identity.

Use the current method for the correct category

Follow the provider's current verification instructions, such as confirmed DNS checks or current published network ranges for the relevant category. For DNS verification, a plausible reverse lookup name is insufficient; forward resolution must also agree with the original address.

Record method, check time and outcome. Frozen historical lists are not a permanent source of truth. Keep common crawlers separate from special or user-triggered fetchers where the provider distinguishes them. An unknown result can remain unknown.

EvidenceSupported classificationNext question
User-agent onlyClaimed crawler nameIs a trusted client address available?
Complete current verificationVerified sender categoryWhich crawler type and check time?
Technical lookup failureUnknown or unverifiedWere the method and lookup available?
Origin logs onlyObservation in one delivery layerWhich cached responses are invisible?
Incomplete periodBounded sampleWhich intervals are missing?

Describe coverage before calculating percentages

State period, time zone, host, logging layer and any sampling procedure. Check whether errors, different methods and cache hits are included. Do not combine two exports in a way that double-counts the same request without understanding their relationship.

Count requests in the observed sample and name the denominator. A higher error share can follow a changed export or traffic group. A missing URL in a small origin export shows only that it is absent there, rather than never fetched by a crawler.

Make the finding possible to review again

Choose a bounded pattern, such as errors for one language group, and retain aggregate evidence plus a protected example for the responsible technician. State alternative explanations and the additional check that separates them.

After an agreed technical change, compare a relevant new sample under the same definitions. A verified successful fetch is not proof of indexing. Authorised search tools and actual content need separate checks; logs cannot support promises about rankings or AI answers.

Sources and further reading

Questions, answered

Is a bot name in the user-agent reliable?

No. The sender can choose the string. Verification needs a trusted address and a current provider method.

Does a failed DNS lookup prove a fake bot?

Not by itself. Record uncertainty and check the method and technical conditions.

IMBAHUB

What would you like to make possible?

Put logging layer, time gaps and verification method at the top of your next crawl report.

Talk to us
Product interface set to English in an isolated environment with fictional English demo content.

Product interface set to English in an isolated environment with fictional English demo content.