- Design and scale high-volume data collection systems for open-web, deep-web, and dark-web sources, including Tor, I2P, and other anonymity networks.
- Build and maintain fault-tolerant batch and streaming pipelines for data ingestion, transformation, enrichment, and storage.
- Develop parsing and normalization logic for unstructured, semi-structured, and evolving threat data formats.
- Design, build, and maintain autonomous, LLM-powered agentic workflows for multi-step collection and enrichment tasks.
- Architect and operate media pipelines to ingest, process, store, and enrich images, videos, screenshots, and documents.
- Design and build internal and external APIs for downstream security teams and platforms.
- Write production-grade Python for data engineering and automation.
- Collaborate with threat intelligence analysts, detection engineers, and security researchers to translate intelligence requirements into data systems.
- Monitor, troubleshoot, and optimize collection performance, data quality, and latency.
PythonCybersecurity