Turning a five-query research run into 297 ranked, structured business records — automatically
Manual local-market research is repetitive in a way that doesn't reward care: search, open each listing, check the website, copy fields into a spreadsheet, notice duplicates two hundred rows later. I built a modular Python engine that does that acquisition and enrichment work end to end — collecting public local-business listings, enriching each one from its own website, deduplicating, and scoring the result so a reviewer starts with the businesses worth calling, not an alphabetical dump.
Working technical demonstration · Quality hardening
Why the project mattered
Built for prospect-research workflows: a team needs to know which local businesses are active, whether they have a real web presence, and which ones are worth prioritizing — with a GUI for a one-off research pass and a CLI for repeatable, automatable jobs. It's the acquisition-mechanics counterpart to the other two data platforms in this portfolio: where the ATS Intelligence Engine solves normalization across ten APIs and Signal Intelligence solves evidence lineage across many source families, this project is about the raw mechanics of collecting and enriching public listing data reliably at volume.
My role and ownership
- System architecture, GUI, CLI and checkpointing
- The concurrent acquisition layer
- The sitemap/BFS enrichment crawler that walks each company's own website
- Normalization, deduplication and scoring logic
- Multi-format export: CSV, JSON, styled XLSX
Personal contribution
Full solo ownership of the architecture, acquisition, enrichment, scoring and export pipeline.
The operational pain
Listing pages expose data in inconsistent, semi-structured form. Company websites hide contact information in different places — footer, contact page, a link buried three clicks deep in a menu. Repeated searches produce duplicate entries that quietly inflate a dataset. And a raw list of business names tells you nothing about which ones are actually reachable or worth a call, which is the only question that matters commercially once the research is done.
Before
- Repeated manual searches and profile opening
- Inconsistent contact, location and reputation fields
- No systematic enrichment from company websites
- Duplicate records across related queries
- No transparent way to prioritize prospects
After
- Concurrent acquisition with isolated worker contexts
- Normalized 38-field business records
- Bounded website crawl using sitemaps, JSON-LD and contact-page prioritization
- Deterministic deduplication and streaming export
- Separate data, contact and outreach-priority scores
How the system works
Acquisition runs on Playwright/Patchright for dynamic listing pages, orchestrated asynchronously with isolated browser contexts so multiple queries run concurrently without interfering with each other. A separate aiohttp crawler handles each business's own website — faster than a full browser when static HTML is enough to find a footer email address, falling back to fuller rendering when it isn't. Every record normalizes into a 38-field schema, gets deduplicated against everything already collected in the run, and is scored into a tier based on review count, rating and phone/website presence — so the output a reviewer opens is a prioritized shortlist, not a flat alphabetical table.
Records stream to CSV as they're collected rather than buffering in memory until the run finishes, with JSON and styled XLSX export layered on top for whichever format the next step in the workflow needs. A selector-health monitor compares field-fill rates run over run, so if a listing site changes its markup, that shows up as a measurable, visible drop in coverage instead of silently corrupting the dataset with empty or malformed fields nobody notices until a client does.
Architecture and decisions
Acquisition and intelligence are deliberately separate layers: extraction gathers evidence, and scoring/classification live in a dedicated module downstream — which means the business logic that decides "is this prospect worth a call" is inspectable and testable on its own, not buried inside scraping code. Every long research run checkpoints and streams results incrementally, so an interruption costs minutes of re-work, not the whole run.
Separate acquisition from intelligence
Extraction gathers evidence while scoring and classification live in a dedicated layer, making the business logic inspectable and independently testable.
Bound website enrichment
Sitemap discovery and contact-page relevance improve coverage, while page caps and early stopping prevent wasteful, unbounded full-site crawling.
Stream results and preserve progress
Incremental writes and checkpoints reduce the cost of interruptions and make long research runs safe to leave running unattended.
Expose both a GUI and a CLI
The GUI supports a human operator running a one-off pass; the CLI supports repeatable jobs, scheduling and technical inspection of exactly what ran.
Evidence that can be inspected.
A supplied 297-record Casablanca demonstration run across five queries, shown exactly as it was captured.
Value and honest current state
Operational impact
The supplied Casablanca run turned five search queries into 297 unique, normalized business records automatically: 95.6% with a working phone number, 98.0% with coordinates, 49.2% with a website, 14.1% with an email — the kind of coverage that turns a research task measured in days into one measured in minutes.
Verified evidence
Current limitations
The 297-record run is one supplied demonstration, not a claim about performance at every scale or category. Auditing that dataset surfaced three real data-quality issues worth fixing before the next run: 137 rows carry an incorrect category from a selector fallback, 289 descriptions duplicate the address field, and roughly 188 secondary-phone values just repeat the primary number. None of that is hidden — the selector-health monitor exists precisely to catch this class of problem, and these three are its first real findings.
Need structured market data, not a raw export?
Acquisition, enrichment, deduplication and scoring, built for a workflow that needs the result prioritized, not just collected.