Data Engineer
New
P
PulsePointHealthcare adtech
Location: WorldwideFull-TimeSenior
Salary90,000 - 150,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years of software engineering experience.
- Required Skills
- PythonSQLHadoopKafkaRDBMSSparkLinux
Requirements
- Hold a BA/BS degree in Computer Science or a related field.
- Have 5+ years of software engineering experience.
- Bring strong expertise in Spark; Spark Streaming is highly desirable.
- Be proficient in Linux.
- Be fluent in Python.
- Have a strong understanding of RDBMS and SQL.
- Be willing to participate in a 24×7 on-call rotation.
- Scala or Java experience is a strong plus.
- Knowledge of distributed production systems such as Hadoop is a plus.
- Exposure to cloud migration is a plus.
Responsibilities
- Install, maintain, and monitor Kafka, Hadoop, Presto, and RDBMS systems.
- Ingest, validate, and process internal and third-party data.
- Create, maintain, and monitor Hive, SQL, and Presto data flows for consistency, accuracy, and lag time.
- Maintain and enhance the framework for jobs, primarily aggregate jobs in Hive.
- Build Kafka consumers using Spark Streaming for near-real-time aggregation.
- Train developers and analysts on data tools.
- Evaluate, select, and implement new tools.
- Handle backups, retention, high availability, and capacity planning.
- Review and approve DDL for databases, Hive framework jobs, and Spark Streaming.
- Participate in a 24×7 on-call rotation for production support.
View Full Description & ApplyYou'll be redirected to the employer's site
