Crawler information
You are most likely reading this because you found this address in your server logs. This page explains who we are, exactly what we request, how often, and how to stop us. Excluding your store takes one email and needs no explanation.
Every request we send carries this exact user agent string:
In robots.txt our token is SmartikaDataBot. All of our traffic originates from a single server hosted by Hetzner in Nuremberg, Germany, at 167.233.243.68. If you see our user agent from any other address, it is not us and we would be grateful if you told us at abuse@smartikadata.com.
| Service | Smartika Data |
|---|---|
| Legal entity | SmartikaGlobal OÜ |
| Registry code | 17359643 |
| Registered office | Sepapaja tn 6, Tallinn, Estonia |
| Jurisdiction | Estonia, European Union |
| Contact | hello@smartikadata.com |
| Exclusion and abuse | abuse@smartikadata.com |
We maintain a historical archive of publicly listed product information so that changes over time remain observable after the original pages have changed. We record what was publicly visible on a given date. We sell historical records derived from what was publicly listed, not copies of your pages, and we do not present our archive as an authoritative record of your catalogue.
robots.txtWe use no account, no API key, and no circumvention of any technical protection measure. We do not attempt to disguise our identity as a browser, we do not rotate addresses, and we do not solve human verification challenges. If a field we collect happens to contain a person's name, it is discarded before it is written to our database.
| Per store | At most one request per week |
|---|---|
| Between requests | At least 5 seconds to the same host |
| On HTTP 429 or 430 | We stop immediately for that host and do not retry for 24 hours |
| After 3 consecutive failures | Your store is suspended from our list and reviewed by a person |
robots.txt | Read before we ever request your store, and re-read every 30 days |
One request per week is a smaller load on your server than a single human visitor. We mention this as a fact about our footprint, not as an argument for why you should tolerate it. If you would rather we did not visit at all, that decision is yours and the next section is all you need.
Write to abuse@smartikadata.com and name your domain. That is the whole procedure.
You do not need to give a reason, and we will not ask for one, argue, or try to change your mind. Your exclusion is applied within 48 hours, it is permanent, and it is recorded with the date it was requested and the date it was applied.
You may instead block us in robots.txt, which we honour and which takes effect on our next read:
User-agent: SmartikaDataBot Disallow: /
Email is the faster route. A robots.txt rule is respected from our next scheduled read of that file, whereas an email is acted on within 48 hours.
When a store enters our list, we fetch its published terms of use and store the text with the date we read it. We search that text for language prohibiting automated access, including the terms scrape, crawl, automated, robot, data mining, extract and bot. Where such a prohibition is found, the store is excluded automatically and the reason is recorded. Ambiguous cases are held for human review and are not requested until that review is done.
We state plainly that this is an automated reading of your terms and not a substitute for your explicit decision. If our check reached the wrong conclusion about your store, the email address above corrects it within 48 hours.
We collect no personal data from stores. Our lawful basis for processing business contact information held elsewhere in our operations is legitimate interest, notice is given at first contact, and erasure requests are applied within 30 days. Our supervisory authority is the Estonian Data Protection Inspectorate.
Our full Terms of Use and Disclaimer set out what we provide, the limits of the archive, and the conditions under which the data may be used.