Email scraping and robots.txt: the rules a responsible crawler follows

The answer depends on method and use

There is no single rule for “scraping emails.” The law tends to focus on specific collection methods and on what the sender does with the address. In the United States, CAN-SPAM regulates how commercial email is sent and treats harvesting and dictionary attacks as aggravating factors. In Canada, CASL generally requires consent and allows implied consent from a published business address only under conditions, while PIPEDA separately restricts address-harvesting software. [1][2][3][4]

Site terms, provider acceptable-use policies, and the recipient’s own published wishes are separate gates. A method can be lawful in one place, prohibited by a website’s terms, and still unacceptable to the mailbox provider you rely on. Nothing here is legal advice; get qualified counsel for your actual jurisdictions and data flows.

Robots.txt is a courtesy protocol, not consent

The Robots Exclusion Protocol, standardized as RFC 9309, lets a site publish rules about which paths automated crawlers may access. The standard is explicit that these rules are not a form of access authorization. Crawlers are asked to follow them; nothing technically forces compliance. [5]

That cuts both ways. Ignoring robots.txt marks a crawler that does not respect site operators and gives them every reason to block it. But honoring robots.txt does not create permission to email anyone. A path that is open to crawlers says nothing about whether the person listed there wants commercial messages.

RFC 9309 also expects crawlers to identify themselves: the product token a crawler matches in robots.txt should appear in its HTTP User-Agent. Pair that with a contact address and a policy page, so a site operator can tell you what went wrong and you can fix it. [5]

Rules a responsible B2B crawler follows

None of these steps make an unsolicited email welcome. They make collection explainable, limited, and easy to challenge, which is the minimum a site operator or recipient should be able to expect.

• Start from a named company and its own domain. Crawl first-party pages you have a specific reason to read, not the open web.
• Identify the crawler with a stable user agent, a policy link, and a monitored contact address.
• Fetch robots.txt, obey it, and refresh it; RFC 9309 says a cached copy should not be used for more than 24 hours unless the file is unreachable.
• Stay on the same domain, fetch a small number of pages, delay between requests, and bound response size and time.
• Refuse credentials, logged-in areas, private network destinations, and anything that looks like an access control.
• Extract only the business-contact evidence the campaign needs, and discard page bodies.
• Record provenance—the exact source URL, method, time, and a hash of the evidence—so an inclusion can be explained or corrected later.
• Skip any page that says it does not want unsolicited commercial email, and keep no address from it.
• Expire uncontacted records on a schedule and honor correction, deletion, and objection requests.

Sources [5]

United States: CAN-SPAM singles out harvesting

CAN-SPAM is usually described as an opt-out law. The FTC’s guide centers on accurate headers and subject lines, a valid postal address, a clear way to opt out, and honoring opt-outs within 10 business days, and it says business-to-business email is not exempt. [1]

The statute adds aggravated violations when an address was obtained by automated means from a website or online service that posted a notice saying it would not give, sell, or transfer addresses for email purposes, or by automated means that generate addresses from permutations of names, letters, or numbers—the “dictionary attack.” The FTC lists address harvesting and dictionary attacks among the practices that can bring additional penalties. [2][1]

Pattern guessing deserves particular care. Generating many address permutations and sending to see which survive resembles the dictionary-attack language even when the targets are businesses. If you infer an address at all, limit it to one candidate for a named person found in first-party evidence, and verify it before use.

Canada: implied consent from publication has conditions

CASL generally prohibits sending a commercial electronic message without the recipient’s consent, and it applies when a computer system in Canada is used to send or access the message. Consent can be implied when a person has conspicuously published their address, but only if the publication is not accompanied by a statement that they do not want unsolicited commercial messages and the message is relevant to their business role or duties. Whoever relies on consent has to prove it. [3]

That turns a crawler detail into a legal one: a “no unsolicited commercial email” notice on the page removes the conspicuous-publication route. Separately, PIPEDA prohibits collecting an electronic address with a computer program designed or marketed primarily for generating or searching for, and collecting, electronic addresses, and prohibits using addresses collected that way. The exceptions that normally allow collection or use without consent do not apply to either. [3][4]

A general-purpose “email extractor” is the kind of tool that language appears aimed at. A bounded crawler that reads a named company’s own team page for a specific role is a different activity, but how PIPEDA’s wording applies to a particular tool is a question for counsel, not a blog post.

How Actually Agentic’s crawler is bounded

Actually Agentic’s crawler identifies itself as ActuallyAgenticBot/1.0 with a link to its sourcing policy and a privacy contact. It starts only from company seeds the customer’s agent supplies, fetches a small configured number of public HTML pages on that company’s own domain—eight by default—follows robots.txt, delays requests, bounds size and time, and rejects credentials, private network addresses, cross-domain navigation, and unsafe redirects. [6]

It skips any page that asks not to receive unsolicited commercial email and keeps no address from it, retains no page bodies, and records the source URL, extraction method, confidence, timestamps, and a SHA-256 evidence hash. Pattern inference is off by default. Uncontacted records expire—after 90 days by default—and anyone can ask where a contact came from, correct it, or object. [6]

Actually Agentic is prelaunch and is not a contact database. The service is limited to organizations and recipients in the United States and Canada, and the customer remains the sender responsible for the lawful basis of each campaign. [7]