Web data collection sits at the intersection of technology, contract and privacy law. The same technique can be responsible in one project and inappropriate in another, depending on the source, the data and the purpose. This guide sets out the standard we apply to our own work. It is intended to help buyers of web data ask better questions of any provider, including us. It is not legal advice, and projects involving personal data or restricted sources should be reviewed by counsel.
Five principles
- Public data only. We collect information that is available to any visitor without logging in, unless the client has its own lawful entitlement to access the source and instructs us accordingly.
- Know the source. Each source's terms and technical signals are reviewed before collection begins.
- Be considerate. Request rates are set so that collection does not burden the source's infrastructure.
- Minimise personal data. If a field about an individual is not needed for the purpose, it is not collected.
- Know the purpose. We understand what the client will do with the data and decline uses that could cause harm.
Before collection: the project assessment
Every project begins with a short assessment. It is proportionate: a price monitoring project on public retail pages needs less review than a project that touches professional profiles or reviews written by individuals.
| Question | Why it matters | Typical outcome |
|---|---|---|
| What is the business purpose? | Purpose determines what data is necessary and which laws apply | Documented in the statement of work |
| Is the data publicly accessible without login? | Access controls signal that the owner has restricted the data | Proceed, or require evidence of the client's entitlement |
| What do the source's terms say? | Terms may restrict automated access or specific uses | Proceed, adjust scope, or decline |
| Does robots.txt address the relevant paths? | It is a widely recognised signal of the operator's preferences | Factored into scope and rate decisions |
| Will any personal data be collected? | Privacy laws apply to personal data even when it is public | Fields removed, masked or justified |
| Are special categories involved? | Health, religion, ethnicity and similar data carry higher risk | Excluded by default |
| Which jurisdictions are involved? | Source location, data subjects and client location all matter | Legal frameworks identified |
Reviewing terms and technical signals
Terms of use vary widely. Some prohibit automated access outright, some restrict commercial reuse, and some are silent. We record what each source's terms say, consider it alongside the purpose and the nature of the data, and adjust or decline where appropriate. We also consider robots.txt directives. They are not a legal instrument in every jurisdiction, but they express the operator's preferences and a responsible collector takes them into account.
We do not bypass logins, paywalls or other access controls unless the client holds a lawful entitlement to that access, such as its own account on a supplier portal, and has instructed us to act on its behalf within the terms of that entitlement.
During collection: operating controls
- Rate limits per source, set conservatively and reduced at peak hours for smaller sites.
- Caching and incremental collection so unchanged pages are not requested repeatedly.
- Collection limited to the pages and fields in scope, rather than whole-site crawling.
- Monitoring for error responses and automatic back-off when a source shows signs of strain.
- No attempts to disguise collection in ways designed to defeat a source's legitimate protections.
- Logs retained so that collection activity can be reviewed if questions arise.
Responsible collection is not a single rule. It is a set of small decisions, made before and during each project, that together keep the work proportionate.
Personal data
Public availability does not remove privacy obligations. Laws such as the GDPR apply to personal data regardless of where it was found. Our default is to design personal data out of datasets. A review analysis rarely needs reviewer names. A property listing analysis rarely needs agent phone numbers. Where personal data is genuinely needed, for example business contact details for a lawful B2B purpose, we document the purpose, the legal basis the client relies on and the retention period, and we collect only the fields required.
Personal data checklist
- List every field that could identify an individual, directly or in combination.
- Remove any field not required for the stated purpose.
- Pseudonymise or aggregate where individual-level data is not needed.
- Exclude special category data unless there is a clear lawful basis and a specific agreement.
- Agree retention and deletion with the client before delivery.
- Confirm the client's responsibilities as controller of the delivered data.
Law by region
Projects are scoped against the frameworks that apply to the source, the data subjects and the client. The table below lists the main frameworks we consider. It is a starting point, not a complete legal analysis.
| Region | Frameworks we consider | Points to check |
|---|---|---|
| Europe and UK | GDPR, UK GDPR | Lawful basis, transparency, data minimisation, international transfers |
| Middle East and GCC | UAE PDPL, Saudi PDPL | Scope of personal data, cross-border transfer conditions |
| Asia-Pacific | Singapore PDPA, Malaysia PDPA, Indonesia PDP Law, Australia Privacy Act | Consent and exceptions, purpose limitation, overseas disclosure |
| North America | CCPA/CPRA, PIPEDA | Notice, opt-out rights, business purpose limits |
| Latin America | LGPD, Mexico LFPDPPP, Colombia Law 1581, Chile Law 19.628 and its 2024 reform | Legal basis, data subject rights, registration or transfer requirements |
Beyond privacy, projects may raise questions of contract, database rights, copyright and computer misuse law. These depend heavily on jurisdiction and facts, which is why the assessment is done per project rather than once for all.
After collection: delivery and use
Responsibility continues after delivery. Clients are responsible for their own lawful use of the data, and our agreements say so. We also vet clients and use cases at the outset. We decline projects intended to profile or track individuals, to harass or discriminate, to circumvent security, or to support any unlawful activity.
- Datasets delivered only to agreed destinations, with access limited to named users.
- Retention periods set in the statement of work, with deletion on request.
- A contact route for site operators and individuals who wish to raise a concern or request removal.
- Periodic review of long-running projects in case sources, terms or laws have changed.
Questions to ask any provider
- How do you decide whether a source is appropriate?
- Do you review terms and robots.txt, and what happens when they restrict collection?
- How are request rates set and monitored?
- What personal data will the dataset contain, and why?
- Which use cases do you decline?
- How can a site operator contact you with a concern?
Clear answers to these questions are a good sign that a provider has thought about responsibility as part of its operating model. Our own answers are set out in our responsible data policy, and we are happy to discuss them for any specific project.