Staff engineer to build web-scale crawling and content acquisition infrastructure.
Design and operate distributed web crawling systems that fetch and refresh content at internet scale for an AI-native search platform. You'll architect URL discovery, scheduling, and ingestion workflows handling billions of URLs while optimizing coverage, freshness, and resource efficiency. This role requires deep expertise in distributed systems, Go or C++, and production-grade infrastructure experience; you'll collaborate with indexing and ML teams to ensure content quality for downstream AI systems.
Membership is €29/month, cancel anytime: every rate, every original listing link, and a daily alert for roles matching your filters.
Found at a specialist agency · listed 17 October 2026 · InsideJobs links you to the original posting.