The public internet leaves records in systems built for accountability, operations, research, and interoperability. Security practitioners have used these records for years because each one reveals a different layer: certificates, pages, registrations, delegations, routes, measurements, or history. Access models differ: some sources are open, some are governed by agreements or limited to noncommercial research, and others require commercial licensing. None is self-explanatory.
The productive question is not “Which dataset has every company?” It is “Which observable event would support this specific claim?” A certificate log can show certificate material. A BGP collector can show a route announcement seen by a peer. A crawl can show a page captured at a time. None of them is a universal organization registry.
Certificate transparency logs: certificate events
Certificate Transparency creates auditable append-only logs for publicly trusted certificate material. Domain names in logged certificates make the logs valuable for discovering hostnames and monitoring unexpected issuance. Chrome's log lists identify the logs Chrome currently recognizes and their states.
- Good for: certificate issuance monitoring, hostname discovery, historical evidence tied to a logged object.
- Weak for: current service availability, legal ownership, user activity, internal-only names, and names never placed on a publicly trusted certificate.
- Time means: a certificate or log event, depending on the field. It does not automatically mean service launch.
Common Crawl: sampled web observations
Common Crawl publishes recurring web crawl archives, URL indexes, metadata, and extracted data. Its WARC files preserve request and response material at enormous scale. The official index lets researchers find captures before retrieving full archive records.
- Good for: link graphs, public page text, technology traces, language research, and changes across available crawl releases.
- Weak for: complete site coverage, authenticated pages, authoritative current state, and precise traffic or audience claims.
- Time means: a crawler fetched a particular URL in a particular crawl. Absence from a crawl is not evidence that the page did not exist.
RDAP: registration records with policy boundaries
ICANN describes RDAP as the standardized successor to WHOIS for domain-name registration data. The protocol returns structured objects rather than ad hoc text. RFC 9083 defines events, entities, notices, remarks, status values, and links in the JSON response model.
- Good for: registrar, status, nameserver, event, and available registration-entity context returned by the authoritative RDAP service.
- Weak for: discovering the human or company behind privacy-protected data, current website operation, and control of every service below the domain.
- Policy boundary: responses can be redacted, differentiated by requester, or accompanied by notices and terms. Structured does not mean unrestricted.
TLD zone files: delegated names, under agreements
ICANN's Centralized Zone Data Service provides a workflow for requesting access to participating generic top-level-domain zone files. A zone can reveal delegated domain names and DNS records published at that layer. It does not enumerate every hostname below those domains, and a delegation does not prove that a website or company is active.
CZDS is a useful reminder that “publicly queryable” and “open for any redistribution” are different. Access requests, credentials, and zone-data agreements govern use. A product that derives statistics or candidate names from zone files still needs to honor those agreements and document which transformations it publishes.
BGP collectors: routes as seen from particular peers
Route Views and RIPE NCC's Routing Information Service collect BGP routing tables and updates from participating networks. Their archives let researchers study which autonomous system originated a prefix, how paths changed, and which events were visible from collector peers.
- Good for: routing history, origin and path observations, leak or hijack analysis, topology research, and change detection.
- Weak for: legal ownership, packet-level reachability from every user, physical infrastructure location, and routes hidden from the collector vantage points.
- Time means: a collector received a table or update from a peer. BGP is a distributed view, not a single global ledger.
RIPE Atlas: active measurements from known vantage points
RIPE Atlas uses volunteer-hosted probes and anchors to run active DNS, ping, traceroute, TLS, HTTP, and related measurements. Unlike a passive archive, it can ask a new question from selected probes. The answer is still conditional on probe placement, connectivity, resolver choice, target, protocol, and measurement time.
- Good for: latency and reachability comparisons, DNS behavior, path changes, outages, and geographically distributed checks.
- Weak for: a representative statement about every user or network without a sampling design.
- Operational boundary: active measurement consumes shared infrastructure and reaches targets. Follow the platform rules and design the smallest measurement that answers the question.
Internet-wide scans: dated service measurements
scans.io is the Stanford Internet Research Data Repository, hosted by the Stanford Empirical Security Research Group. It archives research datasets about Internet hosts, services, and websites, and restricts its data to noncommercial use. Rapid7 Sonar Data is a separate commercial offering built from Project Sonar's Internet-wide scans.
Rapid7 limits Sonar Data to legitimate cybersecurity purposes under commercial licensing. It prohibits offensive use, marketing or lead generation, and bulk redistribution. Those conditions make it unsuitable as a general sales-data source. Separately, any scan result says only that a method observed a response from an address at a time. Shared hosting, anycast, proxies, NAT, and cloud reassignment complicate organization joins, and downloading an existing measurement does not authorize follow-up scanning or exploitation.
The Wayback index: available historical captures
The Internet Archive's CDX interface exposes metadata for available Wayback captures, including timestamps, original URLs, status codes, media types, and digests. It is useful for locating a historical page or establishing that the archive captured a response near a date.
The archive is not a complete, continuous recording of the web. A site can be missed, excluded, blocked, or captured without every asset. A timestamp means the archive has a capture for that URL, not that the page first appeared then or remained unchanged until the next capture.
Joins create claims, not just rows
The most consequential error usually happens after collection. A researcher joins a hostname to a domain, a domain to an IP, an IP to a prefix, a prefix to an autonomous system, and that system to a company. Every edge can change over time and can be many-to-many. The final company label looks clean because the uncertainty was discarded during joins.
Preserve at least four things with every derived record: the source, the source observation time, the retrieval or publication time, and the transformation version. Keep the raw identifier beside the friendly organization name. Record whether an edge is authoritative, company-controlled, measured, or inferred. When two datasets use different clocks, do not collapse them into one “date added” field.
Make public data easier without erasing its boundaries
Democratizing access does not mean presenting every row as a lead or hiding uncertainty behind a score. A useful public-data product can make a difficult source searchable, snapshot changes, preserve provenance, explain field semantics, link primary documentation, and expose enough method for another person to reproduce the result.
It should also publish what it does not know. Coverage can be incomplete. Access may be governed. An active measurement can affect other systems. A public label can be ambiguous. A missing record is rarely proof of absence. Those constraints are not reasons to keep the datasets obscure. They are the terms of using them well.