Senior Data Engineer (AI Consumer Intelligence Platform)
New
K
Kratos GrowthConsumer Intelligence
New York, Remote, New York, Country code: US, 4+ hour overlap with U.S. Eastern Time Zone desiredFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years
- Required Skills
- PythonSQLKubernetesAzureCI/CDDatabricksNLPLLMPySpark
Requirements
- 5+ years building and maintaining production ETL/ELT data pipelines
- 2+ years working with text/NLP data (tokenization, embeddings, multilingual processing)
- 3+ years of data products shipped to production that serve active business users
- 4+ years of Python experience in production environments
- 2+ years of PySpark experience
- 3+ years of SQL experience including complex queries and performance optimization
- 1+ years production use of Databricks (notebooks, Delta Lake, job scheduling)
- 2+ years with Azure or equivalent AWS/GCP experience
- Experience deploying containerized applications to Kubernetes
- Bachelor's degree in Computer Science, Data Science, Engineering, or related quantitative field
Responsibilities
- Design, build, and maintain production data pipelines processing 10M+ text records daily across multiple languages
- Architect scalable NLP data infrastructure using PySpark, Databricks, and Azure services
- Integrate Large Language Model APIs into production pipelines for text analysis and enrichment
- Establish DataOps standards including CI/CD, testing frameworks, and deployment automation
- Implement observability and alerting for pipeline health, data quality, and system performance
- Collaborate with data scientists to productionize ML models and NLP systems
- Define data governance frameworks and quality SLAs for enterprise client delivery
- Mentor team members and contribute to technical hiring as the team scales
View Full Description & ApplyYou'll be redirected to the employer's site
