How to build an ecommerce store database

Start with the record, not the crawler

Before collecting anything, define the finished record. At minimum, a store record needs a canonical domain, platform, detected technologies or apps, currency and country where observed, a source URL, a last-seen date, and a verification status. Public business contact fields can be added when a store publishes them, but they should not define whether the business is a store. This schema determines which observations the pipeline must retain and which candidate outcomes can be audited later.

Also define the counting units. Candidate domains, live websites, verified stores, and stores with email addresses are different populations. Mixing them produces impressive but misleading totals. OpenData illustrates the distinction in its 15 Nov 2026 snapshot: 1,554,041 domains were tracked, while 1,504,744 met its verified-store rule. Its public reports use the narrower verified count for the sellable database.

Discover seeds from public signals

Seed discovery should begin with publicly available domain signals rather than a purchased list whose provenance cannot be explained. Useful sources include public domain datasets and indexes, DNS-oriented discovery, hosted-platform subdomain patterns, and certificate-transparency logs that reveal hostnames when certificates are issued. Each source finds a different slice: platform subdomains expose hosted storefronts, certificate logs can reveal newly configured shops and custom domains, and broad domain sources provide candidates that still require extensive screening.

A seed is only a hypothesis. The pipeline should record where it came from, when it was observed, and whether different discovery routes found the same domain. That provenance helps measure source quality and prevents one noisy feed from silently dominating the database. It also allows an operator to stop using a route that generates mostly parked pages, corporate sites, or unreachable hosts.

Screen DNS before spending fetch capacity

DNS screening separates candidates that have usable address records from names that are absent, malformed, or not currently configured. It is a cheap filter, not a store test. A domain can resolve and still be a blog, software service, parked page, or company brochure. Conversely, temporary DNS failures should not always be treated as permanent evidence that a business has closed; retry rules and failure timestamps matter.

Record the screening outcome in a durable bucket. Candidates might be ready to fetch, temporarily unresolved, permanently absent, or excluded by a clear policy rule. Keeping counts for each bucket makes losses visible and supports later investigation. A pipeline that reports only successful stores cannot show whether discovery is getting worse, DNS data is incomplete, or a processing stage is discarding valid candidates.

• Publicly available domain indexes and datasets
• Hosted-platform subdomain patterns
• Certificate-transparency logs
• DNS records and changes observed over time
• Referrals from already observed public storefronts

Fetch storefronts politely and preserve the outcome

Fetching turns a domain candidate into page evidence. Requests should be bounded, timed out when a site does not respond, and paced so a small number of aggressive targets cannot consume the entire run. Do not bypass access controls or disguise the purpose of automated collection. Some sites will block automated requests, return an error, redirect to another domain, or present an interstitial. Each result is data about observability, but it is not proof that no store exists.

Store the final URL, response outcome, fetch time, redirect relationship, and the page evidence needed for classification. A later technology detector or reviewer must be able to explain the result without guessing which version of the site was observed. Failed fetches should remain in explicit buckets such as blocked, unavailable, timed out, dead, or redirected. Those buckets guide retries and prevent a pipeline from repeatedly spending the same effort on the same unproductive hosts.

Detect technology from storefront evidence

Technology detection looks for reliable markers in the fetched page and its observable resources: platform-specific markup, script or asset patterns, structured storefront elements, application signatures, and combinations that distinguish a commerce system from a generic content site. No single marker should be treated as infallible. Themes, embedded widgets, migrations, and shared hosting components can create weak or conflicting signals, so strong rules combine evidence and retain the detected technology list rather than only a yes-or-no flag.

Verification begins when the evidence supports the store definition. OpenData requires a fetched storefront and at least one detected ecommerce technology. That threshold does not prove revenue, traffic, or business quality, and the database should not pretend otherwise. It establishes the narrower and more defensible claim that the observed site operates as an ecommerce store. Other technologies and apps can enrich the record, but they should remain separate from the basic verification decision.

• Blocked or access-limited
• Dead or persistently unreachable
• Redirected to another canonical domain
• Fetched but not a store
• Fetched and eligible for technology verification

Deduplicate to one store and one record

The same shop commonly appears under several addresses. The www and non-www hosts may resolve together, an old domain may redirect to a new one, and a hosted-platform subdomain may point to the same storefront as a custom domain. If each alias becomes a separate record, the database overstates its size and buyers receive duplicates. A canonicalisation stage should follow redirect evidence, normalise host variants, and attach aliases to one surviving store record without erasing their history.

Deduplication must be conservative. Similar names do not prove that two domains are the same business, and regional storefronts may intentionally operate as separate stores. Use observed redirects and storefront relationships rather than name similarity alone. When records merge, preserve source URLs and observation dates from the contributing evidence so the final count remains explainable. OpenData's operating rule is one store, one record; its catalog structure reflects that in segments such as stores by country and stores by technology.

Extract only public business contact facts

Contact enrichment should begin only after a store has been verified. Look on the store's own public pages for contact information it publishes for customers and business enquiries. Email collection should be limited to addresses the business publishes for contact, not private inboxes or addresses constructed from a person's name. A source URL should accompany the contact field so its origin can be checked. If an address cannot be attributed to a public business page, leave the field empty rather than filling it from a broker or an inference.

Coverage will therefore be incomplete, and that is preferable to false completeness. In OpenData's 15 Nov 2026 snapshot, 205,297 verified stores had a public business email and 111,859 had a store-published business phone. Those counts are useful precisely because they are subsets, not assumptions applied to all 1,504,744 verified stores. Removal requests should be honoured promptly; OpenData states that removals are honoured within 48 hours.

Refresh for store decay and re-check before packaging

Store data decays because businesses close, domains expire, platforms change, and contact pages move. A newly verified record begins ageing immediately. Maintain a refresh queue based on last-seen date and risk: important or previously volatile records can be revisited sooner, while packaging should trigger a fresh check for the stores selected for delivery. When a re-check contradicts the old record, update the status instead of protecting the historical count.

OpenData refreshes its index monthly and re-checks stores before records are exported. Buyers can test the result in the free checker, which allows 30 full-record unlocks per month, or inspect the free sample files before moving to pricing. A sustainable database is therefore not a one-time crawl. It is a loop of discovery, screening, fetching, detection, deduplication, contact checks, and scheduled re-observation, with honest counts retained for stores, non-stores, blocked pages, and dead domains at every pass.

Questions, answered straight

Next steps