Data Engineer

Posted about 11 hours agoViewed

💎 Seniority level: Middle, 4+ Years

📍 Location: India

🔍 Industry: Experience Management

🏢 Company: Experience.com👥 101-250💰 $14,575,000 Series A about 6 years agoCustomer Service Consumer Information Services Consulting SaaS Analytics Quality Assurance Information Technology Software

⏳ Experience: 4+ Years

🪄 Skills: PythonSQLElasticSearchETLGCPMongoDBAirflow

Requirements:

4+ years of experience with PySpark and SQL for building scalable ETL pipelines.
Strong proficiency in Python programming.
Knowledge of GCP Data Analytics ecosystem (BigQuery, PySpark, SQL, etc.).
Experience with Airflow/Composer for workflow orchestration.
Experience with in-memory applications, database design, and data integration.
Strong analytical thinking and problem-solving abilities.

Responsibilities:

Design, build, and maintain scalable ETL/ELT pipelines using PySpark and SQL.
Work on data extraction, transformation, and loading processes from multiple sources into data warehouses such as BigQuery.
Leverage GCP data analytics tools (BigQuery, DataProc, Cloud Functions, etc.) to process and analyze data.
Optimize data workflows for benchmarking, performance, and tuning to ensure efficiency and reliability.
Collaborate with engineering and analytics teams to develop data integration solutions that meet business needs.
Ensure the accuracy and quality of data by implementing strong in-memory applications and database designs.
Implement monitoring and alerting for pipelines and workflows to ensure data consistency and issue resolution.

Apply

Related Jobs

Apply

🔥 Senior Data Engineer

Posted 8 days ago

📍 South Africa, Mauritius, Kenya, Nigeria

🔍 Technology, Marketplaces

BSc degree in Computer Science, Information Systems, Engineering, or related technical field or equivalent work experience.
3+ years related work experience.
Minimum of 2 years experience building and optimizing ‘big data’ data pipelines, architectures and maintaining data sets.
Experienced in Python.
Experienced in SQL (PostgreSQL, MS SQL).
Experienced in using cloud services: AWS, Azure or GCP.
Proficiency in version control, CI/CD and GitHub.
Understanding/experience in Glue and PySpark highly desirable.
Experience in managing data life cycle.
Proficiency in manipulating, processing and architecting large disconnected data sets for analytical requirements.
Ability to maintain and optimise processes supporting data transformation, data structures, metadata, dependency and workload management.
Good understanding of data management principles - data quality assurance and governance.
Strong analytical skills related to working with unstructured datasets.
Understanding of message queuing, stream processing, and highly scalable ‘big data’ datastores.
Strong attention to detail.
Good communication and interpersonal skills.

Suggest efficiencies and execute on implementation of internal process improvements in automating manual processes.
Implement enhancements and new features across data systems.
Improve streamline processes within data systems with support from Senior Data Engineer.
Test CI/CD process for optimal data pipelines.
Assemble large, complex data sets that meet functional / non-functional business requirements.
Highly efficient in ETL processes.
Develop and conduct unit tests on data pipelines as well as ensuring data consistency.
Develop and maintain automated monitoring solutions.
Support reporting and analytics infrastructure.
Maintain data quality and data governance as well as upkeep of overall maintenance of data infrastructure systems.
Maintain data warehouse and data lake metadata, data catalogue, and user documentation for internal business users.
Ensure best practice is implemented and maintained on database.

AWSPostgreSQLPythonSQLETLGitCI/CD

Posted 8 days ago

Apply

🔥 Sr Data Engineer

Posted about 1 month ago

📍 India

🔍 SaaS-based Global Employment Platform

Extensive experience in developing data and analytics applications in geographically distributed teams.
Hands-on experience in using modern architectures and frameworks, structured, semi-structured and unstructured data, and programming with Python.
Hands-on SQL knowledge and experience with relational databases such as MySQL, PostgreSQL, and others.
Hands-on ETL knowledge and experience.
Knowledge of commercial data platforms (Databricks, Snowflake) or cloud data warehouses (Redshift, BigQuery).
Knowledge of data catalog and MDM tooling (Atlan, Alation, Informatica, Collibra).
CICD pipeline for continuous deployment (CloudFormation template).
Knowledge of how machine learning/A.I. workloads are implemented in batch and streaming, including the preparing of datasets, training models, and using pre-trained models.
Exposure to software engineering processes that can be applied to Data Ecosystems.
Excellent analytical and troubleshooting skills.
Excellent communication skills.
Excellent English (both verbal and written).
BS. in Computer Science or equivalent.

Design and develop our best-in-class cloud platform, working on all parts of the code stack from front-end, REST and asynchronous APIs, back-end application logic, SQL/NoSQL databases and integrations with external systems.
Develop solutions across the data and analytics stack from ETL and Streaming data.
Design and develop reusable libraries.
Enhance strong processes in Data Ecosystem.
Write unit and integration tests.

PostgreSQLPythonSQLETLMachine LearningMySQLSnowflakeData engineering

Posted about 1 month ago

Apply

🔥 Data Engineer

Posted about 1 month ago

📍 India

🔍 Artificial intelligence and aquaculture

🏢 Company: Nacre Capital👥 10-100 Venture Capital

Proficiency in English (spoken and written).
Strong experience in Python and SQL.
Hands-on experience with data processing in Apache Airflow.
Experience working with databases, including MongoDB (NoSQL) and relational databases.
Understanding of data modeling, ETL processes, and data warehousing concepts.
Experience with cloud platforms like AWS, GCP, or Azure.

Develop, maintain, and optimize data pipelines and workflows to support ML research and model development.
Design and implement scalable data architectures for handling large datasets used in ML models.
Collaborate closely with ML researchers and data scientists to understand data requirements and ensure data availability and quality.
Prepare and transform data for ML experiments using databases and data integration processes.
Utilize MongoDB and other NoSQL databases to manage unstructured and semi-structured data.
Write efficient, reliable, and maintainable code in Python and SQL for data processing tasks.
Implement data validation and monitoring systems to ensure data integrity and performance.
Support the deployment of ML models by integrating data solutions into production environments.
Ensure the scalability and performance of data systems through rigorous testing and optimization.

AWSPythonSQLApache AirflowETLGCPMongoDBAzure

Posted about 1 month ago

Apply

🔥 Senior Staff Data Engineer

Posted about 2 months ago

📍 India

🔍 Fintech

🏢 Company: Oportun👥 1001-5000💰 $235,000,000 Post-IPO Debt 3 months ago🫂 Last layoff about 1 year agoDebit Cards Consumer Lending Financial Services FinTech

Requires 12+ years of related experience in data engineering.
Bachelor's degree in Computer Science; or a Master's degree with equivalent experience.
Extensive experience in developing end-to-end data engineering infrastructure for large-scale applications.
Proven leadership in guiding technical teams to innovative outcomes.
Mastery of data engineering architecture, frameworks, and processing including Hadoop, Databricks, Python, and PySpark.
Comprehension of software engineering principles and collaborative development workflows.
Experience with cloud platforms (AWS, Azure, GCP) and cloud-native services.
Track record of integrating DevOps practices and CI/CD pipelines.
Strong problem-solving skills and communication abilities.

Set strategic vision and implement cutting-edge data infrastructure roadmap.
Provide technical leadership, mentoring, and guidance to data engineers.
Translate model requirements into optimized data pipelines with data quality assurance.
Establish best practices for model versioning, experiment tracking, and evaluation.
Engineer automated CI/CD pipelines for seamless deployment and monitoring.
Define and refine performance benchmarks, optimizing data infrastructure.
Collaborate with various teams to design, develop, and test major software and data systems components.
Drive strong data engineering practices and remain at the forefront of industry trends.

AWSPythonGCPGitHadoopAzureData engineeringCI/CDDevOps

Posted about 2 months ago

Apply

🔥 Data Engineer II

Posted 2 months ago

📍 India

🧭 Regular Full-Time

🔍 Healthcare information

🏢 Company: Precision AQ

Bachelor’s degree in engineering (B.E.) or Technology (B. Tech.), preferably in computer science or related fields.
Minimum 2 years’ experience with complex data ecosystems focusing on data analysis.
1+ years’ experience with cloud-based infrastructure (AWS, Azure) and cloud database management (e.g., Redshift).
Experience with Stored procedures for ETL automation.
Fluency with SQL, data warehousing, data analysis, and cloud technologies.
Experience with code versioning tools like Git and CI/CD setups, preferably Azure DevOps.
Experience in Life Sciences.
Experience leading technical discussions.

Translate business requirements into technical requirements.
Partner with SMEs to develop data queries and tools.
Manage monthly data operations processes for multiple products.
Contribute to DevOps platforms for code management.
Take ownership of existing solutions and support enhancements.
Build data ETLs for data management.
Manage CI/CD pipelines for dashboards and datasets.
Conduct risk-based code reviews for development solutions.
Work with the Product Team on application enhancements.
Interact with various stakeholders including end users and vendors.
Automate data loading and validation tasks.

AWSSQLData AnalysisETLGitAzureCI/CDDevOpsData management

Posted 2 months ago

Apply

🔥 Data Engineer

Posted 2 months ago

📍 India

🏢 Company: Unison Consulting Pte Ltd

Minimum 6+ years of Data Ingestion, Integration, ETL, or security engineering experience.
Extensive knowledge of AWS, Azure, GCP.
Strong understanding of Data Management or Data Engineering.
Experienced in Agile methods and Atlassian stack (e.g., JIRA).
Ability to develop roadmaps for data-centric products.
Experience with monitoring frameworks and observability products.
Expertise in SIEM solutions and cloud-based data sources.
Familiarity with security monitoring solutions like Splunk and Datadog.
Experience in DevSecOps/IRE and agile environments.
Expertise in scripting languages (PowerShell, Python, Bash).
Experience with Docker, Kubernetes, Ansible, or Terraform.
Related security certifications (e.g., CISSP, CCSP).
Experience with Linux/Ubuntu/Mac systems.
Experience in creating dashboards and troubleshooting connectivity issues.

Define and manage data models, schemas, metadata, and security rules.
Design, create, deploy, and manage databases and data structures on-premise and in the cloud.
Identify and mitigate potential security risks.
Ensure compliance with data privacy laws and regulations.
Conduct risk assessments and take appropriate actions to mitigate data security risks.
Train and educate stakeholders about data management.
Collaborate with IT team members and stakeholders to secure data architectures.

AWSDockerPythonAgileBashData AnalysisETLGCPKubernetesJiraAzureData engineeringData StructuresCollaborationLinuxTerraformCompliance

Posted 2 months ago

Apply

🔥 Sr Data Engineer

Posted 3 months ago

📍 India

🔍 SaaS-based global employment platform

🏢 Company: G-P

Extensive experience in developing data and analytics applications in geographically distributed teams.
Hands-on experience in using modern architectures and frameworks, structured, semi-structured and unstructured data, and programming with Python.
Hands-on SQL knowledge and experience with relational databases such as MySQL, PostgreSQL, and others.
Hands-on ETL knowledge and experience.
Knowledge of commercial data platforms (Databricks, Snowflake) or cloud data warehouses (Redshift, BigQuery).
Knowledge of data catalog and MDM tooling (Atlan, Alation, Informatica, Collibra).
CICD pipeline for continuous deployment (CloudFormation template).
Knowledge of how machine learning / A.I. workloads are implemented in batch and streaming, including the preparing of datasets, training models, and using pre-trained models.
Exposure to software engineering processes that can be applied to Data Ecosystems.
Excellent analytical and troubleshooting skills.
Excellent communication skills.
Excellent English (both verbal and written).
BS. in Computer Science or equivalent.

Design and develop our best-in-class cloud platform, working on all parts of the code stack from front-end, REST and asynchronous APIs, back-end application logic, SQL/NoSQL databases and integrations with external systems.
Develop solutions across the data and analytics stack from ETL and Streaming data.
Design and develop reusable libraries.
Enhance strong processes in Data Ecosystem.
Write unit and integration tests.

PostgreSQLPythonSQLAgileETLMachine LearningMySQLSnowflakeNosqlCommunication Skills

Posted 3 months ago

Apply

🔥 Principal Data Engineer

Posted 3 months ago

📍 Bangalore

🧭 Full-Time

🔍 Digital engineering and modernization

🏢 Company: Encora👥 10001-10001💰 $200,000,000 Private over 5 years agoBig Data Cloud Computing Software

Bachelor’s or Master’s degree in Computer Science, Statistics, Mathematics, or a related field.
10+ years of experience in data science, with a proven track record.
Experience with Databricks and proficiency in programming languages such as Python and Pyspark.
Extensive experience with machine learning frameworks like TensorFlow, PyTorch, and scikit-learn.
Strong knowledge of statistical analysis, data mining, predictive modeling, and machine learning algorithms.
Experience with big data technologies such as Hadoop and Spark.
Familiarity with cloud platforms including AWS, GCP, and Azure.

Design, develop, and deploy advanced data science models and algorithms.
Ensure models are scalable, maintainable, and optimized for performance.
Lead and mentor a team of data scientists and engineers.
Work closely with product managers, engineers, and business stakeholders.
Conduct exploratory data analysis to uncover trends and insights.
Lead data science projects from conception to completion.

AWSDockerLeadershipProject ManagementPythonSQLData AnalysisData MiningHadoopKubernetesMachine LearningPyTorchAlgorithmsData engineeringData scienceSparkTensorflowAnalytical SkillsCollaboration

Posted 3 months ago

Apply

🔥 Lead Data Engineer

Posted 3 months ago

📍 India

🧭 Full-Time

🔍 Digital engineering and modernization

🏢 Company: Encora👥 10001-10001💰 $200,000,000 Private over 5 years agoBig Data Cloud Computing Software

7-10 years of strong development experience performing ETL and/or data pipeline implementations.
Expert in programming languages, preferably Python.
Expert in delivering end-to-end analytic solutions using AWS services (EMR, Airflow, S3, Athena, Kinesis, Redshift).
Experience in batch technologies like Hadoop, Hive, Athena, Presto.
Strong SQL skills, including query optimization, schema design, complex analytics.
Expert in data modeling and metadata management like Glue Catalog etc.
Experience in deployment tools like GitHub actions, Jenkins, AWS Code Pipeline etc.
Experience in data quality tools like Deque or Great Expectations is Nice To Have.

Collaborate and partner with Business Analyst teams located in US and EMEA regions.
Interface across our Business Analyst and Data Science teams.
Play a key role in integrating new data sources into our data & analytical ecosystem over AWS cloud.
Implement data lake solutions while addressing common data concerns, such as data quality, data governance.
Set the standard for technical excellence as we move / build our data ecosystem into the cloud.
Understand their common data problems and deliver scalable solutions.

AWSLeadershipPythonSQLData AnalysisETLHadoopJenkinsAirflowData science

Posted 3 months ago

Apply

🔥 Data Engineer

Posted 3 months ago

📍 India

🧭 Full-Time

🔍 Healthcare technology

🏢 Company: Reveal Health Tech

Bachelor’s degree in Computer Science, Engineering, or related field.
Proficiency in data modeling and using AWS services.
Proven experience as a Data Engineer or similar role.
Strong knowledge of AWS services related to data engineering.
Experience designing and building data pipelines.
Understanding of data lake and warehousing best practices.
Proficiency in programming languages for data manipulation (e.g., Python, SQL).
Experience with relational databases like PostgreSQL.
Understanding of database systems and performance optimization.
Experience implementing data quality checks.
Familiarity with Cloudformation and CI/CD for deployment.
Experience with data quality validation tools.
Ability to work independently and collaboratively.

Designing and implementing scalable, robust, and maintainable data pipelines using AWS services.
Implementing data quality checks and validation processes.
Designing data models and architectures for data processing workflows.
Utilizing validation tools to automate processes.
Collaborating with cross-functional teams to meet data requirements.
Monitoring and optimizing data pipeline performance.
Debugging and resolving data-related issues.
Documenting data pipeline architectures and processes.
Staying updated with new technologies in data engineering.

AWSPostgreSQLPythonSQLDynamoDBMachine LearningData engineeringServerlessCI/CD

Posted 3 months ago

Apply

Data Engineer

Requirements:

Responsibilities:

Related Jobs

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities