Data Engineer – AI/BI
New
Remote-friendly work arrangement based in IndiaFull-Time
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Required Skills
- PythonSQLMLFlowSparkScalaDatabricks
Requirements
- Strong hands-on experience with Databricks, Delta Lake, and Apache Spark in building scalable data engineering solutions.
- Proficiency in SQL and Python (or Scala) for data transformation, pipeline development, and workflow automation.
- Experience implementing Medallion Architecture and working with large-scale cloud data platforms.
- Proven ability to build and manage ELT pipelines, data ingestion frameworks, and orchestration workflows.
- Experience integrating enterprise systems such as Salesforce, PeopleSoft, or similar platforms.
- Understanding of data governance, metadata management, lineage tracking, and data quality best practices.
- Familiarity with cloud environments such as Azure or AWS, including storage and security concepts.
- Experience with BI/AI enablement tools such as Databricks AI/BI, dashboards, or semantic modeling is highly desirable.
- Strong analytical thinking, problem-solving skills, and ability to work in a fast-paced engineering environment.
Responsibilities
- Design, build, and optimize end-to-end data pipelines on Databricks using Apache Spark and Delta Lake, following Medallion Architecture (bronze, silver, gold) principles.
- Develop and operationalize AI/BI solutions including dashboards, semantic models, and Databricks Genie experiences for self-service analytics.
- Integrate enterprise data sources such as PeopleSoft, Salesforce, and D2L using APIs, JDBC, and other ingestion frameworks.
- Implement scalable ELT workflows with orchestration, monitoring, schema evolution, and automated error handling.
- Establish data quality frameworks, validation rules, and observability mechanisms to ensure accuracy, reliability, and performance.
- Implement data governance, lineage, and metadata management using tools such as Unity Catalog or equivalent solutions.
- Support AI/ML teams by delivering curated datasets, feature stores, and ML-ready data pipelines using tools like MLflow.
- Ensure data security, privacy, and compliance through access controls, encryption, masking, and governance best practices.
- Maintain technical documentation, architecture diagrams, and enablement materials for platform adoption and scalability.
View Full Description & ApplyYou'll be redirected to the employer's site
