Data Platform Engineer
New
Anywhere in the U.S.Full-TimeMiddle
Salary100000 - 150000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 3+ years
- Required Skills
- AWSDockerPythonElasticSearchGCPKafkaKubernetesAirflowGoRedisSparkRESTful APIsBigQuery
Requirements
- 3+ years of experience in Platform Engineering
- Proven track record of deploying and maintaining scalable Data platforms in high-traffic production environments
- Proficient in streaming architectures (Kafka/Flink/PubSub)
- Experience building low-latency services to serve stream ingestion and processing for model inference in <100ms
- Proficient with Containerization, Docker, Kubernetes and cluster-level management
- Deep experience building a platform for managing the full Data lifecycle, including setting up a data exploration environment
- Expert in coding with Python and Go
- Deep experience with Cloud services, preferred with GCP services (BigQuery, Cloud Functions, GKE) or AWS equivalents
- Excellent communication skills
- Stakeholder management
- Outstanding problem-solving skills
- Extensive experience in Big data technologies like Spark, Flink, Kafka or Kinesis, Argo/Airflow, Polaris, OpenMetadata, Iceberg, Lakehouse, Redis, Elasticsearch, and Databases
- Experience with building REST APIs
- Experience with package management and building libraries
- Key contributor to projects through the entire development lifecycle from concept to release
Responsibilities
- Design and build the Data platform for Batch and Streaming use cases
- Build and maintain a platform with cutting-edge technologies
- Enable data users by building data catalog and data lineage capabilities
- Contribute to design and enforce robust data security architectures and controls
- Build platform for deploying low-latency services to pipe data for streaming or near real time use cases
- Power real-time decisions across the platform, from dynamic oddsmaking and risk analysis to smart deposit defaults
- Champion best practices for model deployment, monitoring, and CI/CD for Data pipeline deployment
- Enable complete observability for batch and streaming data platform and ensure the availability of 99.99%
View Full Description & ApplyYou'll be redirected to the employer's site
