Big Data Application Developer
New
J
JobgetherData Engineering
Based in United StatesFull-TimeSenior
Salary100,000 - 150,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years
- Required Skills
- PythonSQLHadoopJavaKafkaAirflowSparkScala
Requirements
- Bachelor’s degree in Computer Science, Engineering, or a related technical field.
- 5+ years of professional experience designing and operating big data pipelines.
- Strong hands-on experience with Apache Spark in production environments using Scala, Python, or Java.
- Solid expertise with Hadoop ecosystem technologies, including Hive, HDFS, Sqoop, and HBase.
- Experience with streaming platforms such as Kafka, Spark Streaming, or Flink.
- Strong SQL skills and experience working with relational and NoSQL databases.
- Experience with workflow orchestration tools such as Airflow or Oozie.
- Strong understanding of distributed systems architecture and data processing principles.
- Proficiency in Python or Shell scripting.
- Excellent troubleshooting, debugging, analytical, and documentation skills.
Responsibilities
- Design, develop, and operate large-scale data processing pipelines within Hadoop-based ecosystems.
- Build and maintain data ingestion, transformation, and analytics workflows supporting enterprise applications.
- Develop high-performance processing solutions using Apache Spark with Scala, Python, or Java.
- Work with Hadoop technologies including Hive, HDFS, Sqoop, HBase, and related distributed data platforms.
- Implement and support streaming data solutions using technologies such as Kafka, Spark Streaming, or Flink.
- Optimize data processing performance through effective use of distributed computing concepts, including partitioning, replication, and fault tolerance.
- Develop and maintain workflow automation using orchestration tools such as Airflow or Oozie.
- Write efficient SQL queries and work with relational and NoSQL databases.
- Create scripts and automation tools using Python or Shell.
- Troubleshoot complex data platform issues, perform debugging, and maintain technical documentation.
View Full Description & ApplyYou'll be redirected to the employer's site
