Senior Collections Data Engineer
New
I
iCOUNTERCybersecurity, threat intelligence
US-RemoteFull-TimeSenior
Salary140,000 - 180,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years of experience as a Software Engineer or Data Engineer
- Required Skills
- PythonApache KafkaSpark
Requirements
- Have 5+ years of experience as a Software Engineer or Data Engineer, focused on large-scale data collection and processing.
- Have deep experience collecting data at scale from open and dark web sources, including scraping, crawling, API integration, Tor-based collection, and anti-detection or anti-bot techniques.
- Have strong experience building and operating data pipelines with Airflow, Dagster, Prefect, Spark, Kafka, Flink, or equivalent.
- Have advanced data parsing skills, including transforming unstructured web data into structured intelligence.
- Have expert-level Python proficiency, including async programming, concurrency, and performance optimization.
- Have hands-on experience designing agentic workflows with LangChain, CrewAI, AutoGen, LlamaIndex, or similar frameworks, and integrating LLMs into production data systems.
- Have a track record of building and maintaining production APIs with FastAPI, Flask, or similar.
- Have a strong understanding of cybersecurity concepts and threat intelligence data types, including IOCs, TTPs, malware samples, phishing kits, and underground forums.
- Be comfortable working in a fast-paced, ambiguous environment with high-stakes data quality requirements.
- Experience with cloud platforms; knowledge of modern data stack tools such as dbt, Snowflake, ClickHouse, or DuckDB; familiarity with Docker, Kubernetes, and infrastructure-as-code; or prior cybersecurity, threat intelligence, or intelligence collection experience are relevant.
- A background check is required for employment.
Responsibilities
- Design and scale high-volume data collection systems for open-web, deep-web, and dark-web sources, including Tor, I2P, and other anonymity networks.
- Build and maintain fault-tolerant batch and streaming pipelines for data ingestion, transformation, enrichment, and storage.
- Develop parsing and normalization logic for unstructured, semi-structured, and rapidly evolving threat data.
- Design, build, and maintain agentic workflows using autonomous, LLM-powered agents for collection and enrichment tasks.
- Architect and operate media pipelines to ingest, process, store, and enrich collected images, videos, screenshots, and documents.
- Design and build internal and external APIs that expose collected intelligence to downstream security teams and platforms.
- Write production-grade Python for data engineering and automation.
- Collaborate with threat intelligence analysts, detection engineers, and security researchers to turn intelligence requirements into reliable data systems.
- Monitor, troubleshoot, and optimize collection performance, data quality, and latency at scale.
- Track emerging threat actor TTPs, data sources, and evasion techniques to improve collection coverage.
View Full Description & ApplyYou'll be redirected to the employer's site
