Principal Engineer - Data Pipeline - US Remote

View full description

Requirements:

• 10+ years of software development experience in building large-scale Data Analytics solutions.
• Proficiency in Golang (preferred) or Python.
• Proficiency with common telemetry frameworks such as Prometheus and Open Telemetry.
• Experience with big data pipeline technologies including Logstash, FluentD, Kafka, Spark, Hadoop, Storm, and Flink.
• Domain knowledge in networking, including OSI stack, BGP, IP, TCP, UDP, HTTP, DNS, Linux network stack, SDN, and network automation.
• Experience with machine learning libraries like Scikit-learn, Pytorch, Tensorflow, or Keras is a plus.
• Experience with Cloud Platforms is advantageous.
• BS in Computer Science, Software Engineering, Machine Learning, Mathematics, Statistics, or equivalent.

Responsibilities:

• Responsible for designing and architecting the data pipeline infrastructure needed for network analytics including data collection, storage, and processing.
• Lead and coordinate with engineers to identify and collect relevant data from network devices such as routers, switches, and firewalls.
• Develop algorithms to transform and cleanse raw network data, involving parsing, normalizing, aggregating, and enriching data.
• Integrate the data pipeline with various analytics tools and platforms.
• Optimize the data pipeline to handle large volumes of network data efficiently.
• Implement data quality checks and validation mechanisms.
• Set up monitoring tools to monitor the health and performance of the data pipeline.
• Work closely with cross-functional teams.