Senior Data Engineer
New
O
OportunFinancial services
Remote - India; the role must be performed from within India.Full-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 6+ years of experience in data engineering
- Required Skills
- AWSPythonSQLSparkCI/CDDatabricksPySpark
Requirements
- Have 6+ years of experience in data engineering, focused on data architecture, data pipelines, data platforms, and database or lakehouse management.
- Hold a bachelor's or master's degree in Computer Science, Data Science, Engineering, or a related field, or have equivalent practical experience.
- Have strong proficiency in Python/PySpark and SQL.
- Bring hands-on experience with Databricks, Apache Spark, Spark SQL, and Delta Lake.
- Have experience with workflow orchestration, CI/CD, Git-based workflows, automated testing, and production end-to-end data integrations or products.
- Have experience with data quality, observability, alerting, and lineage.
- Have experience with cloud platforms and services, especially AWS.
- Have experience with secure access controls, secrets management, encryption, auditability, and sensitive member data.
- Be familiar with Agile ways of working.
- Financial services, lending, marketing technology, customer communications, regulated environments, Unity Catalog or comparable governance tools, and Java or Scala are nice to have.
- Experience using AI tools or agents for engineering productivity and delivery is a plus.
Responsibilities
- Design and implement scalable, secure data architectures for member communications, including data models, contracts, lineage, and ownership.
- Lead complex data engineering initiatives from requirements and design through production delivery, coordinating engineers and stakeholders and managing dependencies, risks, and trade-offs.
- Develop and optimize production data pipelines and integrations using Databricks, Apache Spark, PySpark, Spark SQL, Delta Lake, and SQL.
- Build data-loading, reconciliation, retry, idempotency, recovery, and deployment patterns, and operate data lakehouse and operational data assets.
- Establish data-quality and governance practices, including validation, monitoring, alerts, dashboards, runbooks, and lineage.
- Partner with Privacy, Compliance, Risk, and Security on PII, preferences, consent, suppression, retention, and access controls.
- Provide technical leadership through design and code reviews, mentor junior engineers, investigate production issues, identify root causes, and implement durable fixes.
- Partner with cross-functional stakeholders to translate needs and technical trade-offs into solutions, and drive release readiness through testing, CI/CD, rollout planning, and post-release measurement.
- Monitor pipeline performance, quality, timeliness, cost, and reliability, and explore practical applications of AI tools.
View Full Description & ApplyYou'll be redirected to the employer's site
