Data Engineer

Posted about 1 month agoViewed

View full description

📍 Location: India

🔍 Industry: Artificial intelligence and aquaculture

🏢 Company: Nacre Capital👥 10-100 Venture Capital

🗣️ Languages: English

🪄 Skills: AWSPythonSQLApache AirflowETLGCPMongoDBAzure

Requirements:

Proficiency in English (spoken and written).
Strong experience in Python and SQL.
Hands-on experience with data processing in Apache Airflow.
Experience working with databases, including MongoDB (NoSQL) and relational databases.
Understanding of data modeling, ETL processes, and data warehousing concepts.
Experience with cloud platforms like AWS, GCP, or Azure.

Responsibilities:

Develop, maintain, and optimize data pipelines and workflows to support ML research and model development.
Design and implement scalable data architectures for handling large datasets used in ML models.
Collaborate closely with ML researchers and data scientists to understand data requirements and ensure data availability and quality.
Prepare and transform data for ML experiments using databases and data integration processes.
Utilize MongoDB and other NoSQL databases to manage unstructured and semi-structured data.
Write efficient, reliable, and maintainable code in Python and SQL for data processing tasks.
Implement data validation and monitoring systems to ensure data integrity and performance.
Support the deployment of ML models by integrating data solutions into production environments.
Ensure the scalability and performance of data systems through rigorous testing and optimization.

Apply

Related Jobs

Apply

🔥 Data Engineer

Posted about 16 hours ago

📍 India

🧭 Full-Time

🔍 Experience Management

🏢 Company: Experience.com👥 101-250💰 $14,575,000 Series A about 6 years agoCustomer Service Consumer Information Services Consulting SaaS Analytics Quality Assurance Information Technology Software

4+ years of experience with PySpark and SQL for building scalable ETL pipelines.
Strong proficiency in Python programming.
Knowledge of GCP Data Analytics ecosystem (BigQuery, PySpark, SQL, etc.).
Experience with Airflow/Composer for workflow orchestration.
Experience with in-memory applications, database design, and data integration.
Strong analytical thinking and problem-solving abilities.

Design, build, and maintain scalable ETL/ELT pipelines using PySpark and SQL.
Work on data extraction, transformation, and loading processes from multiple sources into data warehouses such as BigQuery.
Leverage GCP data analytics tools (BigQuery, DataProc, Cloud Functions, etc.) to process and analyze data.
Optimize data workflows for benchmarking, performance, and tuning to ensure efficiency and reliability.
Collaborate with engineering and analytics teams to develop data integration solutions that meet business needs.
Ensure the accuracy and quality of data by implementing strong in-memory applications and database designs.
Implement monitoring and alerting for pipelines and workflows to ensure data consistency and issue resolution.

PythonSQLElasticSearchETLGCPMongoDBAirflow

Posted about 16 hours ago

Apply

🔥 Senior Data Engineer

Posted 9 days ago

📍 South Africa, Mauritius, Kenya, Nigeria

🔍 Technology, Marketplaces

BSc degree in Computer Science, Information Systems, Engineering, or related technical field or equivalent work experience.
3+ years related work experience.
Minimum of 2 years experience building and optimizing ‘big data’ data pipelines, architectures and maintaining data sets.
Experienced in Python.
Experienced in SQL (PostgreSQL, MS SQL).
Experienced in using cloud services: AWS, Azure or GCP.
Proficiency in version control, CI/CD and GitHub.
Understanding/experience in Glue and PySpark highly desirable.
Experience in managing data life cycle.
Proficiency in manipulating, processing and architecting large disconnected data sets for analytical requirements.
Ability to maintain and optimise processes supporting data transformation, data structures, metadata, dependency and workload management.
Good understanding of data management principles - data quality assurance and governance.
Strong analytical skills related to working with unstructured datasets.
Understanding of message queuing, stream processing, and highly scalable ‘big data’ datastores.
Strong attention to detail.
Good communication and interpersonal skills.

Suggest efficiencies and execute on implementation of internal process improvements in automating manual processes.
Implement enhancements and new features across data systems.
Improve streamline processes within data systems with support from Senior Data Engineer.
Test CI/CD process for optimal data pipelines.
Assemble large, complex data sets that meet functional / non-functional business requirements.
Highly efficient in ETL processes.
Develop and conduct unit tests on data pipelines as well as ensuring data consistency.
Develop and maintain automated monitoring solutions.
Support reporting and analytics infrastructure.
Maintain data quality and data governance as well as upkeep of overall maintenance of data infrastructure systems.
Maintain data warehouse and data lake metadata, data catalogue, and user documentation for internal business users.
Ensure best practice is implemented and maintained on database.

AWSPostgreSQLPythonSQLETLGitCI/CD

Posted 9 days ago

Apply

🔥 Sr Data Engineer

Posted about 1 month ago

📍 India

🔍 SaaS-based Global Employment Platform

Extensive experience in developing data and analytics applications in geographically distributed teams.
Hands-on experience in using modern architectures and frameworks, structured, semi-structured and unstructured data, and programming with Python.
Hands-on SQL knowledge and experience with relational databases such as MySQL, PostgreSQL, and others.
Hands-on ETL knowledge and experience.
Knowledge of commercial data platforms (Databricks, Snowflake) or cloud data warehouses (Redshift, BigQuery).
Knowledge of data catalog and MDM tooling (Atlan, Alation, Informatica, Collibra).
CICD pipeline for continuous deployment (CloudFormation template).
Knowledge of how machine learning/A.I. workloads are implemented in batch and streaming, including the preparing of datasets, training models, and using pre-trained models.
Exposure to software engineering processes that can be applied to Data Ecosystems.
Excellent analytical and troubleshooting skills.
Excellent communication skills.
Excellent English (both verbal and written).
BS. in Computer Science or equivalent.

Design and develop our best-in-class cloud platform, working on all parts of the code stack from front-end, REST and asynchronous APIs, back-end application logic, SQL/NoSQL databases and integrations with external systems.
Develop solutions across the data and analytics stack from ETL and Streaming data.
Design and develop reusable libraries.
Enhance strong processes in Data Ecosystem.
Write unit and integration tests.

PostgreSQLPythonSQLETLMachine LearningMySQLSnowflakeData engineering

Posted about 1 month ago

Apply

🔥 Data Engineer II

Posted 2 months ago

📍 India

🧭 Regular Full-Time

🔍 Healthcare information

🏢 Company: Precision AQ

Bachelor’s degree in engineering (B.E.) or Technology (B. Tech.), preferably in computer science or related fields.
Minimum 2 years’ experience with complex data ecosystems focusing on data analysis.
1+ years’ experience with cloud-based infrastructure (AWS, Azure) and cloud database management (e.g., Redshift).
Experience with Stored procedures for ETL automation.
Fluency with SQL, data warehousing, data analysis, and cloud technologies.
Experience with code versioning tools like Git and CI/CD setups, preferably Azure DevOps.
Experience in Life Sciences.
Experience leading technical discussions.

Translate business requirements into technical requirements.
Partner with SMEs to develop data queries and tools.
Manage monthly data operations processes for multiple products.
Contribute to DevOps platforms for code management.
Take ownership of existing solutions and support enhancements.
Build data ETLs for data management.
Manage CI/CD pipelines for dashboards and datasets.
Conduct risk-based code reviews for development solutions.
Work with the Product Team on application enhancements.
Interact with various stakeholders including end users and vendors.
Automate data loading and validation tasks.

AWSSQLData AnalysisETLGitAzureCI/CDDevOpsData management

Posted 2 months ago

Apply

🔥 Data Engineer

Posted 2 months ago

📍 India

🏢 Company: Unison Consulting Pte Ltd

Minimum 6+ years of Data Ingestion, Integration, ETL, or security engineering experience.
Extensive knowledge of AWS, Azure, GCP.
Strong understanding of Data Management or Data Engineering.
Experienced in Agile methods and Atlassian stack (e.g., JIRA).
Ability to develop roadmaps for data-centric products.
Experience with monitoring frameworks and observability products.
Expertise in SIEM solutions and cloud-based data sources.
Familiarity with security monitoring solutions like Splunk and Datadog.
Experience in DevSecOps/IRE and agile environments.
Expertise in scripting languages (PowerShell, Python, Bash).
Experience with Docker, Kubernetes, Ansible, or Terraform.
Related security certifications (e.g., CISSP, CCSP).
Experience with Linux/Ubuntu/Mac systems.
Experience in creating dashboards and troubleshooting connectivity issues.

Define and manage data models, schemas, metadata, and security rules.
Design, create, deploy, and manage databases and data structures on-premise and in the cloud.
Identify and mitigate potential security risks.
Ensure compliance with data privacy laws and regulations.
Conduct risk assessments and take appropriate actions to mitigate data security risks.
Train and educate stakeholders about data management.
Collaborate with IT team members and stakeholders to secure data architectures.

AWSDockerPythonAgileBashData AnalysisETLGCPKubernetesJiraAzureData engineeringData StructuresCollaborationLinuxTerraformCompliance

Posted 2 months ago

Apply

🔥 MResult - CDP-Data Engineer

Posted 3 months ago

📍 India

🧭 Full-Time

🔍 Data & Analytics, AI

🏢 Company: Nexthire

Strong experience with Data Factory, Azure, Microsoft Dynamics 365, and Customer Data Platform (CDP).
Experience in MS Customer Insights Data.
Experience in MS customer insights journey.
Experience with Azure, Data Verse, and Power Platform.
Overall experience of 8-10 years.
At least one full project experience on CDP or 3 years in CDP-related support.

Design, develop, and implement solutions using Microsoft Customer Data Platform (CDP) to manage and analyze customer data.
Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
Integrate CDP with various data sources and ensure seamless data flow and accuracy.
Develop and maintain data pipelines, ensuring data is collected, processed, and stored efficiently.
Create and manage customer profiles, segments, and audiences within the CDP.
Implement data governance and security best practices to protect customer data.
Monitor and optimize the performance of the CDP infrastructure.
Provide technical support and troubleshooting for CDP-related issues.
Stay updated with the latest trends and advancements in CDP technology and best practices.

Microsoft DynamicsAzure

Posted 3 months ago

Apply

🔥 Sr Data Engineer

Posted 3 months ago

📍 India

🔍 SaaS-based global employment platform

🏢 Company: G-P

Extensive experience in developing data and analytics applications in geographically distributed teams.
Hands-on experience in using modern architectures and frameworks, structured, semi-structured and unstructured data, and programming with Python.
Hands-on SQL knowledge and experience with relational databases such as MySQL, PostgreSQL, and others.
Hands-on ETL knowledge and experience.
Knowledge of commercial data platforms (Databricks, Snowflake) or cloud data warehouses (Redshift, BigQuery).
Knowledge of data catalog and MDM tooling (Atlan, Alation, Informatica, Collibra).
CICD pipeline for continuous deployment (CloudFormation template).
Knowledge of how machine learning / A.I. workloads are implemented in batch and streaming, including the preparing of datasets, training models, and using pre-trained models.
Exposure to software engineering processes that can be applied to Data Ecosystems.
Excellent analytical and troubleshooting skills.
Excellent communication skills.
Excellent English (both verbal and written).
BS. in Computer Science or equivalent.

Design and develop our best-in-class cloud platform, working on all parts of the code stack from front-end, REST and asynchronous APIs, back-end application logic, SQL/NoSQL databases and integrations with external systems.
Develop solutions across the data and analytics stack from ETL and Streaming data.
Design and develop reusable libraries.
Enhance strong processes in Data Ecosystem.
Write unit and integration tests.

PostgreSQLPythonSQLAgileETLMachine LearningMySQLSnowflakeNosqlCommunication Skills

Posted 3 months ago

Apply

🔥 Principal Data Engineer

Posted 3 months ago

📍 Bangalore

🧭 Full-Time

🔍 Digital engineering and modernization

🏢 Company: Encora👥 10001-10001💰 $200,000,000 Private over 5 years agoBig Data Cloud Computing Software

Bachelor’s or Master’s degree in Computer Science, Statistics, Mathematics, or a related field.
10+ years of experience in data science, with a proven track record.
Experience with Databricks and proficiency in programming languages such as Python and Pyspark.
Extensive experience with machine learning frameworks like TensorFlow, PyTorch, and scikit-learn.
Strong knowledge of statistical analysis, data mining, predictive modeling, and machine learning algorithms.
Experience with big data technologies such as Hadoop and Spark.
Familiarity with cloud platforms including AWS, GCP, and Azure.

Design, develop, and deploy advanced data science models and algorithms.
Ensure models are scalable, maintainable, and optimized for performance.
Lead and mentor a team of data scientists and engineers.
Work closely with product managers, engineers, and business stakeholders.
Conduct exploratory data analysis to uncover trends and insights.
Lead data science projects from conception to completion.

AWSDockerLeadershipProject ManagementPythonSQLData AnalysisData MiningHadoopKubernetesMachine LearningPyTorchAlgorithmsData engineeringData scienceSparkTensorflowAnalytical SkillsCollaboration

Posted 3 months ago

Apply

🔥 Lead Data Engineer

Posted 3 months ago

📍 India

🧭 Full-Time

🔍 Digital engineering and modernization

🏢 Company: Encora👥 10001-10001💰 $200,000,000 Private over 5 years agoBig Data Cloud Computing Software

7-10 years of strong development experience performing ETL and/or data pipeline implementations.
Expert in programming languages, preferably Python.
Expert in delivering end-to-end analytic solutions using AWS services (EMR, Airflow, S3, Athena, Kinesis, Redshift).
Experience in batch technologies like Hadoop, Hive, Athena, Presto.
Strong SQL skills, including query optimization, schema design, complex analytics.
Expert in data modeling and metadata management like Glue Catalog etc.
Experience in deployment tools like GitHub actions, Jenkins, AWS Code Pipeline etc.
Experience in data quality tools like Deque or Great Expectations is Nice To Have.

Collaborate and partner with Business Analyst teams located in US and EMEA regions.
Interface across our Business Analyst and Data Science teams.
Play a key role in integrating new data sources into our data & analytical ecosystem over AWS cloud.
Implement data lake solutions while addressing common data concerns, such as data quality, data governance.
Set the standard for technical excellence as we move / build our data ecosystem into the cloud.
Understand their common data problems and deliver scalable solutions.

AWSLeadershipPythonSQLData AnalysisETLHadoopJenkinsAirflowData science

Posted 3 months ago

Apply

🔥 GCP Data Engineer

Posted 3 months ago

📍 India

🧭 Full-Time

🔍 Digital engineering and modernization

🏢 Company: Encora👥 10001-10001💰 $200,000,000 Private over 5 years agoBig Data Cloud Computing Software

At least 6 years of experience in data engineering.
Strong proficiency in GCP services.
Experience with Big Query for data handling.
Familiarity with Data Flow for processing data streams.

Design and implement scalable data solutions based on GCP.
Utilize Big Query for efficient data analysis and reporting.
Employ Data Flow for stream processing and data integration.

GCPData engineering

Posted 3 months ago

Apply

Data Engineer

Requirements:

Responsibilities:

Related Jobs

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities