Python Web Scraping Engineer - London

Salary: £30,000 - 60,000 per year

Requirements

• We require expert-level Python web scraping skills.
• We require direct, hands-on experience scraping Google at scale.
• We require deep understanding of anti-bot and bot-detection systems.
• We require knowledge of browser and network fingerprinting.
• We require familiarity with CAPTCHA systems and mitigation techniques.
• We require strong understanding of HTTP, TLS, cookies, headers, redirects, and browser networking behaviour.
• We require experience with Playwright, Selenium, Puppeteer, or equivalent automation frameworks.
• We require the ability to design asynchronous and concurrent scraping architectures.
• We require proven experience running scraping systems at scale in cloud environments.
• We require excellent debugging skills and the ability to reason about complex failure modes.
• We require strong communication skills and the ability to explain complex technical behaviour clearly.
• We require UK-based candidates with the unrestricted right to work in the UK; we do not offer sponsorship.

Responsibilities

• We design and operate large-scale scraping systems handling 10+ million requests per day, primarily targeting Google and Google-like platforms.
• We build robust scrapers for dynamic, JavaScript-heavy environments using browser automation and hybrid approaches.
• We continuously adapt to changes in markup, request flows, ranking logic, and anti-automation mechanisms.
• We engineer extraction pipelines with a strong emphasis on correctness, consistency, and observability.
• We implement and maintain advanced anti-bot evasion strategies, including proxy and request routing, browser and headless fingerprinting, and CAPTCHA handling.
• We monitor system health, detect anomalies early, and debug complex production issues.
• We optimise performance, cost, and latency across large-scale scraping infrastructure.
• We collaborate closely with data engineers, data scientists, and product teams to ensure scraped data is reliable and usable.
• We produce clear documentation and operational runbooks to support long-term maintainability.

Technologies

• AI
• Cloud
• HTTP
• Support
• JavaScript
• Network
• Playwright
• Python
• Selenium
• Web
• Fabric

More

We are hiring a Senior Python Web Scraping Engineer for a fully remote role offering up to £70k. This is a specialist position focused on building and operating high-volume, highly resilient scraping systems with a sustained emphasis on scraping Google at scale. The role sits at the intersection of data engineering, reverse engineering, and large-scale systems reliability, and is best suited to engineers who thrive in hostile, rapidly changing environments and actively use AI-assisted coding practices to improve productivity, reasoning, and system design. We are looking for UK-based candidates with the unrestricted right to work in the UK, as we do not offer sponsorship. We offer the opportunity to work closely with data engineers, data scientists, and product teams in a technically challenging environment where trusted data delivery is critical.

last updated 19 week of 2026