Open data sources: where to find usable datasets

Government portals: the biggest free datasets

Government catalogues are the natural starting point. data.gov indexes United States federal datasets; data.gov.uk does the same for UK public bodies; data.europa.eu aggregates the national portals of Europe into one search; and national statistics offices publish the underlying demographic and economic series directly. Coverage is broad and the price is zero.

The catch is consistency. One agency publishes a clean, documented API; the next offers a spreadsheet that was last updated years ago. Before building on any government dataset, check its update date and whether the publisher still maintains it.

• What it holds: statistics, budgets, registers, transport, environment, health
• Formats: CSV, JSON, GeoJSON, spreadsheets and APIs
• Licence: US federal data is public domain; the UK uses the OGL; the EU is mostly CC BY — check each dataset
• Best use: market context, demographics and official registers

International organisations: World Bank, UN Data, WHO

For data that crosses borders, international organisations are the reliable layer. World Bank Open Data publishes development indicators — population, GDP, trade, education, health — for economies worldwide, with one of the best APIs in the field. UN Data aggregates the statistics of the UN system, and the World Health Organization publishes global health data, from disease surveillance to health-system capacity.

Because definitions are standardised across countries, this is the safest source for comparing markets. The trade-off is granularity: you get national and regional series, not street-level or company-level detail.

• What it holds: development indicators, health statistics, population and economic series
• Formats: CSV and JSON downloads, plus well-documented APIs
• Licence: generally CC BY-style terms — attribution required
• Best use: cross-country comparisons and macro context

Geographic data: OpenStreetMap

OpenStreetMap is the community-built map of the world, maintained by volunteers and companies in the open. It is often more current than commercial maps in fast-changing areas, and it is the standard open source for roads, buildings, addresses and points of interest.

Its licence is the thing to understand before you build. ODbL allows commercial use, but if you publish a database derived from OpenStreetMap data, that database must remain under ODbL — the share-alike catch. Finished works, such as an app screen or a printed map, need only attribution. Official boundary and address data from government portals makes a good complement where you need legal-grade geography.

• What it holds: roads, buildings, addresses, points of interest, boundaries
• Formats: full planet file, regional extracts (e.g. Geofabrik), and APIs
• Licence: ODbL — attribution plus share-alike on derived databases
• Best use: store locators, delivery coverage, location analytics

Web corpora: Common Crawl

Common Crawl is a non-profit that crawls the public web and gives the archive away. Each crawl contains billions of pages, stored as WARC files on Amazon S3, and anyone can download them — you pay only for your own storage and compute. It is the raw material behind a great deal of web research and behind many datasets built from public pages, including technology detection of the kind our store reports summarise.

Be clear about what you are getting: raw HTML, exactly as crawled. Deduplication, extraction, cleaning and verification are all your job, and the free crawl does not transfer any rights in the underlying content — it is a research corpus, not a licence to republish pages.

• What it holds: archived copies of billions of public web pages
• Formats: WARC archives with index files, hosted on Amazon S3
• Licence: the crawl is free to use; the underlying page content still belongs to its owners
• Best use: research, technology detection and analysing the web at scale

Reference data: Wikidata

Wikidata is the structured knowledge base behind Wikipedia: millions of entities — companies, places, products, people — each with a stable identifier and machine-readable facts. It is published under CC0, the most permissive terms available, so it can be merged into commercial data without attribution obligations.

Its best use is as connective tissue. Match your own records to Wikidata identifiers and you can inherit classifications, locations, founding dates and relationships, and keep datasets from different sources consistent with each other.

• What it holds: structured facts about people, places, organisations, products and concepts
• Formats: SPARQL endpoint, APIs and full dumps
• Licence: CC0 — no attribution legally required
• Best use: enriching and linking your own records to known entities

Commercial-ready public business data

Some public data is valuable but impractical to assemble yourself. Verified business data is the clearest case: building it means fetching millions of storefronts, detecting what each one runs, discarding the dead and the duplicates, and repeating the exercise continuously as stores open, close and change platforms. That pipeline is the product.

OpenData is that product for ecommerce. It holds 1,504,744 verified stores (snapshot 15 Nov 2026) — verified means the storefront was fetched and at least one ecommerce technology detected — including 205,297 stores with a public business email and 111,859 with a store-published business phone. Every record carries its source URL and last-observed date; only public business facts and published business emails are kept, and removal requests are honoured within 48 hours and permanently suppressed.

You can slice it by technology or app, inspect records in the free checker (30 full-record unlocks a month on a free account), or take a free sample file per industry first. Records are priced per row with PayPal checkout and instant export — full records $4.00 per 1,000 rows, platform- or technology-only records $1.00 per 1,000 rows, filtered by app, technology or country as you need — and the index is refreshed monthly.

Choosing a source: five checks before you build

Whichever source you pick, run these five checks first. Ten minutes here prevents the two classic failures: building on data you have no right to use, and building on data that quietly stopped being true.

• Licence — is reuse, including commercial reuse, actually permitted?
• Freshness — when was it last updated, and is it still maintained?
• Format — is there a complete download or API, or only web pages?
• Coverage — what is missing, and does the publisher say so?
• Personal data — public does not mean publishable; personal data needs a lawful basis, so prefer business facts and published business contacts

Questions, answered straight

Next steps