Hadoop Big Data Developer
New
100% Remote (Continental United States)Full-TimeMiddle
Salary100,000 - 150,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years
- Required Skills
- PythonSQLETLHadoopKafkaAirflowSparkDistributed Systems
Requirements
- Bachelor’s degree in Computer Science, Engineering, or a related technical discipline.
- Five or more years of professional experience designing and operating big-data pipelines on Hadoop.
- Strong hands-on expertise with Apache Spark (Scala, Python, or Java) in production environments.
- Solid experience with Hive, HDFS, Sqoop, HBase, and the broader Hadoop ecosystem.
- Hands-on experience with streaming data platforms such as Kafka, Spark Streaming, or Flink.
- Strong SQL skills and experience working with both relational and NoSQL data stores.
- Experience with workflow orchestration tools such as Airflow or Oozie.
- Solid understanding of distributed systems concepts, including partitioning, replication, and fault tolerance.
- Strong scripting skills in Python or Shell.
- Excellent troubleshooting, debugging, and documentation skills.
Responsibilities
- Design, develop, and operate end-to-end big-data pipelines on Hadoop, ingesting data from a diverse mix of relational, file-based, streaming, and API-driven sources.
- Build robust ETL/ELT workflows using Apache Spark, Hive, Pig, and Sqoop, with strong attention to data quality, idempotency, error handling, and recoverability.
- Develop high-throughput streaming data pipelines using Kafka, Spark Streaming, or Flink, and integrate them with downstream analytical and operational systems.
- Optimize Spark and MapReduce jobs through careful tuning of partitioning, memory, serialization, and skew handling to meet demanding SLAs at minimal cost.
- Design and maintain data models and storage layouts on HDFS, Hive, HBase, and modern lakehouse formats to balance flexibility and performance.
- Build robust monitoring, alerting, and logging strategies for big-data pipelines, including job-level SLAs and proactive failure detection.
- Automate pipeline orchestration using Airflow, Oozie, or similar workflow engines, with clean dependency management and clear ownership boundaries.
- Lead performance reviews and architecture audits of existing pipelines, proposing concrete refactoring and optimization initiatives.
View Full Description & ApplyYou'll be redirected to the employer's site
