Phi Consulting
Talk to Phi

Custom Website Data Extraction

A Python scraper that extracts clean, structured data from any public website. You control the selectors, fields, and output format.

OutcomeFeeds pipeline-ready prospect or market data directly into your CRM, enrichment layer, or scoring system.

AdvancedTime to value: 1 to 2 weeks6 tools

How it flows

How it flows.

Source
Target Website
Process
scraper.py
Agent
Selenium
Agent
Chrome Browser
Store
Page HTML
Process
BeautifulSoup
Store
Parsed Elements
Process
Pandas
Output
JSON File
Output
CSV File
Channel
CRM / Enrichment
Drag to pan · scroll to zoom

    Build it, step by step

    Build it, step by step.

    1. 01

      Install Python and dependencies

      Set up your local environment with Python and the five libraries required for browser control, parsing, and export.

      • Verify Python is installed and in your PATH with python --version
      • Install selenium for browser automation
      • Install webdriver-manager to handle Chrome driver updates
      • Install beautifulsoup4 for HTML parsing
      • Install requests for HTTP calls
      • Install pandas for structured data output

    2. 02

      Save the scraper script

      Copy the Python scraper code and save it locally as scraper.py.

      • Open the scraper template linked in the workflow documentation
      • Copy the full Python code block
      • Save it as scraper.py in a dedicated folder
      • Open scraper.py in VS Code or Cursor

    3. 03

      Point the script at your target site

      Replace the placeholder URL with the actual website you want to scrape.

      • Locate the line url = "YOUR_TARGET_WEBSITE_URL_HERE" in scraper.py
      • Replace the placeholder with the real URL
      • Save the file

    4. 04

      Run the scraper

      Execute the script from the command line. It will open Chrome, load the page, extract data, and write output files.

      • Open Terminal (Mac) or Command Prompt (Windows)
      • Navigate to the folder containing scraper.py
      • Run python scraper.py
      • The script opens Chrome in automated mode
      • Loads the target URL and waits for page elements
      • Extracts fields using defined selectors
      • Writes results to .json and .csv files in the same folder

    5. 05

      Review the output files

      Open the CSV or JSON to confirm data quality and field completeness.

      • Locate the .json file for structured, nested output
      • Locate the .csv file for spreadsheet use
      • Open the CSV in Excel, Google Sheets, or Clay
      • Spot-check rows for missing fields or malformed data

    6. 06

      Adjust selectors and fields

      Modify XPath or CSS selectors to target the exact elements you need. Use browser dev tools and Claude to refine.

      • Use Chrome DevTools to inspect page structure and copy selectors
      • Update selectors in scraper.py if initial extraction is incomplete
      • Add or remove data fields (name, email, phone, link) as needed
      • Share HTML snippets or selector errors with Claude for refinement
      • Re-run the script after each change to test accuracy

    7. 07

      Handle failures and iterate

      If elements fail to load or selectors break, increase wait times, update selectors, or add rate-limiting delays.

      • Increase wait times if dynamic content loads slowly
      • Update selectors when site layouts change
      • Add random delays between requests to reduce bot detection
      • Re-run the script after each adjustment
      • Log errors and feed them back to Claude for troubleshooting

    The stack

    The stack.

    Related workflows

    Related workflows.

    Revenue WorkflowVisitor De-AnonymizationCaptures anonymous website traffic, unmasks company and contact data, scores fit, and triggers tiered outbound motion based on intent strength.
    Revenue WorkflowTarget Account Build and ScorePull closed deal data from your CRM, backtest to define your real ICP, build and score a broad TAM, then orchestrate outbound and signal tracking that pushes qualified accounts back into pipeline.
    Revenue WorkflowSignal ActivationThis workflow ingests intent signals from CRM, product, review sites, partner networks, technographic feeds, and social channels. It cleans, scores, and routes them into your CRM and activation tools so reps act on qualified opportunities fast.

    Talk to us: Want this workflow run for you?

    We wire it into your stack, then run it.

    Talk to Phi