How a product page is built.
Every value comes from that product’s own website. Each one carries the page it was read from and the day we last saw it there.
Before we read anything
A product is on Stakkd because the owner added it, or approved a request to add it. Scanning a site never creates a listing, and nothing becomes public until the owner publishes it.
Before a page is fetched we read that host’s own robots.txt, per host, not per company, because a subdomain serves its own and usually a stricter one. Where we cannot read it, we do not read the host. We keep the file we were served, the time, and the name we gave ourselves, so a decision about permission can be shown rather than asserted.
Team, careers and about-the-team pages are excluded before the fetch plan is drawn, not filtered afterwards.
What we take off a page
Most values are read by code against the page itself: a pattern that either matches or does not, and can be tested. Some are matched against lists we hold, so that two vendors writing the same thing differently come out the same. A few need reading comprehension, and for those a model proposes a value which code then checks appears word for word on the page. A value that does not appear is discarded.
Vendor prose is stored exactly as written. We never edit a quote, and we never summarise a claim into something they did not say.
What a missing value means
They state it.
Some of it is stated, and we can tell we did not get all of it. The page says so.
We read the page where it would be, and it is not there.
The page where it would be did not load, or is not one we may read.
The last two read the same: “Not found on the 41 pages we checked, 2 September 2026.” A product page shows what we hold. What is not on it is not a claim about the vendor. It is something we did not find on the pages we read, on the day we read them. We say how many pages that was, at the top of every listing.
A field a page has no room for is not dropped: the MCP server answers it, and each listing says how many more fields it holds than it shows.
Nothing is overwritten
A value that changes is written as a new record and the old one is marked as no longer current. Nothing is edited in place and nothing is deleted, so what we published and when can always be shown, and that is what a correction has to be argued from.
Where we got something wrong, we withdraw the value as our own misreading rather than recording it as a change on the vendor’s side.
If we have it wrong
A vendor can stop our crawler at any time by changing their own robots.txt, which takes effect without asking us and cannot be faked by anyone else. Removing a published listing goes through an ownership check, so a competitor cannot delete a rival.