Research Crawling Engineer
True Talent Solutions Partners · Remote
ABOUT THE ROLE
Join a data infrastructure team building large-scale web crawling systems and datasets for AI research and model development. This role focuses on reliable data acquisition across diverse, changing web environments.
WHAT YOU'LL DO
- Build and maintain web crawlers across social, travel, and multilingual websites.
- Design high-throughput, fault-tolerant systems that collect millions to billions of URLs per day.
- Adapt crawling approaches to anti-bot systems, rate limits, and dynamic or JavaScript-heavy sites.
- Develop pipelines to clean, deduplicate, filter, and normalize web data at TB-to-PB scale.
- Build and maintain datasets for research and model training in collaboration with research teams.
- Monitor crawl performance, coverage, and data quality, and adjust systems as websites change.
- Optimize crawling infrastructure for cost, latency, and reliability across cloud or bare-metal environments.
WHAT WE'RE LOOKING FOR
- At least 2 years of experience building web crawlers or large-scale data pipelines; typically 4 or more years, or 2 or more years with a PhD.
- Hands-on experience building and operating web crawlers at scale, including data pipelines handling TB-scale workloads.
- Strong proficiency in Go, Rust, Python, Java, or C++, with experience in distributed systems or parallel processing.
- Deep knowledge of HTTP, networking, and browser behavior, including experience working with anti-bot systems, rate limits, and JavaScript-heavy sites.
- Experience with headless browsers, proxy systems, IP rotation, and request orchestration is valuable.
- Familiarity with data quality evaluation, large-scale data processing, and cloud or bare-metal infrastructure.
- Experience with NLP pipelines, machine learning dataset curation, or AI research data is a plus.
- A degree in computer science, engineering, or a related technical field is helpful; a PhD is a plus.
COMPENSATION & BENEFITS
Salary range: $160,000 to $250,000 annually. Visa sponsorship is not available.
LOCATION
Remote.