Two rows in the prospect list were the same business, and every useful column was generated
A scraper produced a shortlist of fabrication businesses, each row carrying a name, a phone number, a website, and then several columns that were the actual point of the exercise: what this business probably struggles with, a score for how good a fit they are, and a suggested opening pitch.
Two rows were wrong in ways that made the rest of the file suspect.
One row’s scraped website content belonged to a different company in the list — the crawler had followed something and attributed another firm’s pages to this one, so every derived column for that row described somebody else’s business. Another row’s “website” was a page on a third-party business directory, not the company’s own site at all, so everything inferred from it was inferred from a directory template.
Neither of those looks wrong in a spreadsheet. Both rows are complete. Both have plausible content in every column.
The separation that fixed it
The clean-up was not a de-duplication pass. It was a reclassification of every column into two categories, applied to the whole file.
Observed columns are ones where the value came from somewhere with a chain of custody. The business name. The phone number. What their own site says they manufacture. Whether their site looks recently built or a decade old. Those can be wrong, but when they are wrong you can find out by looking.
Generated columns are the pain points, the fit scores and the suggested pitch. Those are a model’s inference from the observed columns, which means they inherit every error in the observed columns and add their own. If the observed columns describe the wrong company, the generated ones describe the wrong company confidently and in complete sentences.
The generated columns were not deleted. They were relabelled as unverified hypotheses and excluded from any decision. Each row got a data-quality flag saying what was trusted about it. Two rows were dropped entirely with a note saying why, rather than being silently removed, because a dropped row with a reason is information and a dropped row is just a shorter file.
That is the whole technique, and it is the thing I would apply to any model-enriched dataset anywhere in a product. A generated field is a hypothesis wearing the same font as a fact. If your interface renders both the same way, the people using it will not maintain the distinction, because nothing in front of them supports maintaining it.
The consequence for anything that scores or prioritises is direct. A score derived from generated fields, presented as a number, sorted descending, is a confident ordering of a set of guesses. It will be acted on as though it were measured.
Replacing the generated columns with calls
The generated columns claimed to know what these businesses struggle with. The replacement was to ring them and ask, and the structure of that call is the part worth keeping.
The product is not mentioned until well into the conversation. Everything before that is one question, asked as openly as possible, followed by silence: walk me through the last quotation you made — an enquiry comes in, then what.
No suggested answers. Not “do you use a spreadsheet”. The answer might be a spreadsheet, or paper, or a messaging app, or a rate-per-kilogram rule applied from memory, and which of those it is changes what you are building. Offering the options destroys the finding, because people accept a plausible option rather than describing their actual process.
The single most important thing recorded from each call is not what the pain is. It is whether they volunteered it or only agreed to it once prompted. Prompted agreement is worth nothing — everybody politely concurs that the thing you just described sounds annoying. Unprompted description of a problem, in their own words, before you have said what you sell, is the only signal that survives.
The structure has an explicit failure condition written into it, which is the part most discovery processes leave out. If most of these businesses quote in ten minutes from a rule of thumb and feel no pain about it, the premise is wrong, and no amount of product work fixes a premise. Writing that down before making the calls is what makes it possible to actually conclude it afterwards.
What the observed columns did say
The observed fields, once the generated ones were set aside, carried one finding that mattered for the product rather than for the sales process.
Size does not predict digital maturity in this sector. One of the larger and longer-established businesses in the list — decades old, more than a hundred staff, serving institutional customers — had a website that was an unmodified agency template. Another, founded in the seventies, had a site that had plainly not been touched in years. The most digitally current site in the set belonged to one of the smaller operations.
That is an observable fact from public information, and it contradicts the assumption that a substantial manufacturer has substantial software.
For product decisions it points in a specific direction. The person preparing quotations is very often the owner or the owner’s son rather than a dedicated estimator, working from a phone as much as a desk, with no tolerance for configuration screens. It supports things this product ended up doing: hiding material pickers unless a user explicitly asks for them, showing only dimension fields that genuinely need an entry, replacing free-text unit fields with fixed lists, and putting reordering controls on mobile because reordering on mobile is not an edge case for this buyer.
None of that came from the generated pain-point column. All of it came from observed facts plus listening.
The rule
In any dataset built by scraping and enrichment, mark every column as observed or generated, and never let a generated column reach a decision without a human check. Keep the flag visible in whatever interface displays the data, because the distinction survives only as long as it is rendered.
Then go and verify the generated columns directly, with the product unmentioned, and write down in advance what result would tell you the premise is wrong. Ten conversations will tell you more than any amount of enrichment, and unlike the enrichment, they can say no.