Data Engineer

Posted 3 months agoViewed

View full description

📍 Location: India

🔍 Industry: Artificial intelligence and aquaculture

🏢 Company: Nacre Capital👥 10-100 Venture Capital

🗣️ Languages: English

🪄 Skills: AWSPythonSQLApache AirflowETLGCPMongoDBAzure

Requirements:

Proficiency in English (spoken and written).
Strong experience in Python and SQL.
Hands-on experience with data processing in Apache Airflow.
Experience working with databases, including MongoDB (NoSQL) and relational databases.
Understanding of data modeling, ETL processes, and data warehousing concepts.
Experience with cloud platforms like AWS, GCP, or Azure.

Responsibilities:

Develop, maintain, and optimize data pipelines and workflows to support ML research and model development.
Design and implement scalable data architectures for handling large datasets used in ML models.
Collaborate closely with ML researchers and data scientists to understand data requirements and ensure data availability and quality.
Prepare and transform data for ML experiments using databases and data integration processes.
Utilize MongoDB and other NoSQL databases to manage unstructured and semi-structured data.
Write efficient, reliable, and maintainable code in Python and SQL for data processing tasks.
Implement data validation and monitoring systems to ensure data integrity and performance.
Support the deployment of ML models by integrating data solutions into production environments.
Ensure the scalability and performance of data systems through rigorous testing and optimization.

Apply

Related Jobs

Apply

🔥 Sr Data Engineer

Posted 11 days ago

📍 US, Europe and India

🔍 Software Development

🔧 Requirements

Extensive experience in developing data and analytics applications in geographically distributed teams
Hands-on experience in using modern architectures and frameworks, structured, semi-structured and unstructured data, and programming with Python
Hands-on SQL knowledge and experience with relational databases such as MySQL, PostgreSQL, and others
Hands-on ETL knowledge and experience
Knowledge of commercial data platforms (Databricks, Snowflake) or cloud data warehouses (Redshift, BigQuery)
Knowledge of data catalog and MDM tooling (Atlan, Alation, Informatica, Collibra)
CICD pipeline for continuous deployment (CloudFormation template)
Knowledge of how machine learning / A.I. workloads are implemented in batch and streaming, including the preparing of datasets, training models, and using pre-trained models
Exposure to software engineering processes that can be applied to Data Ecosystems

💡 Responsibilities

Design and develop our best-in-class cloud platform, working on all parts of the code stack from front-end, REST and asynchronous APIs, back-end application logic, SQL/NoSQL databases and integrations with external systems
Develop solutions across the data and analytics stack from ETL and Streaming data
Design and develop reusable libraries
Enhance strong processes in Data Ecosystem
Write unit and integration tests

PythonSQLApache AirflowCloud ComputingETLMachine LearningSnowflakeAlgorithmsApache KafkaData engineeringData StructuresCommunication SkillsAnalytical SkillsCI/CDRESTful APIsDevOpsMicroservicesExcellent communication skillsData visualizationData modelingData analyticsData management

Posted 11 days ago

Apply

🔥 Data Engineer - Tableau

Posted 21 days ago

📍 India

🧭 Full-Time

🔍 Technology Services

🏢 Company: computer_aid

🔧 Requirements

1+ years of working knowledge in Tableau.
3+ years of experience with AWS Data services like S3, Glue, Lake Formation, EMR, Kinesis, RDS, DMS, and Redshift.
3+ years of experience building Data Warehouses on Snowflake, Redshift, HANA, Teradata, Exasol, etc.
3+ years of experience in building Delta Lakes using technologies like Apache or Databricks.
3+ years of experience working with ETL tools and technologies.
3+ years of experience in any programming language (Python, R, Scala, Java).
Bachelor's degree in computer science, information technology, data science, data analytics, or related field.
Experience working on Agile projects and familiarity with Agile methodology.
4+ years of experience operating on AWS Cloud building Data Lake architectures.
Strong RDBMS and data modeling skills.
AWS cloud certification is a plus.
Strong communication skills, both written and spoken.

💡 Responsibilities

Design and develop data lakes and manage data flows integrating various information sources.
Code and manage delta lake implementations on S3 using Databricks or Apache Hoodie.
Triage, debug, and fix technical issues related to Data Lakes.
Design and develop data warehouses for scalable solutions.
Evaluate various data models like Star, Snowflake, and Flattened.
Design data access patterns for OLTP and OLAP transactions.
Coordinate with business and technical teams throughout the software development life cycle.
Participate in major technical and architectural decision-making.

AWSPythonSQLETLSnowflakeTableau

Posted 21 days ago

Apply

🔥 Senior Data Engineer

Posted about 1 month ago

📍 South Africa, Mauritius, Kenya, Nigeria

🔍 Technology, Marketplaces

🔧 Requirements

BSc degree in Computer Science, Information Systems, Engineering, or related technical field or equivalent work experience.
3+ years related work experience.
Minimum of 2 years experience building and optimizing ‘big data’ data pipelines, architectures and maintaining data sets.
Experienced in Python.
Experienced in SQL (PostgreSQL, MS SQL).
Experienced in using cloud services: AWS, Azure or GCP.
Proficiency in version control, CI/CD and GitHub.
Understanding/experience in Glue and PySpark highly desirable.
Experience in managing data life cycle.
Proficiency in manipulating, processing and architecting large disconnected data sets for analytical requirements.
Ability to maintain and optimise processes supporting data transformation, data structures, metadata, dependency and workload management.
Good understanding of data management principles - data quality assurance and governance.
Strong analytical skills related to working with unstructured datasets.
Understanding of message queuing, stream processing, and highly scalable ‘big data’ datastores.
Strong attention to detail.
Good communication and interpersonal skills.

💡 Responsibilities

Suggest efficiencies and execute on implementation of internal process improvements in automating manual processes.
Implement enhancements and new features across data systems.
Improve streamline processes within data systems with support from Senior Data Engineer.
Test CI/CD process for optimal data pipelines.
Assemble large, complex data sets that meet functional / non-functional business requirements.
Highly efficient in ETL processes.
Develop and conduct unit tests on data pipelines as well as ensuring data consistency.
Develop and maintain automated monitoring solutions.
Support reporting and analytics infrastructure.
Maintain data quality and data governance as well as upkeep of overall maintenance of data infrastructure systems.
Maintain data warehouse and data lake metadata, data catalogue, and user documentation for internal business users.
Ensure best practice is implemented and maintained on database.

AWSPostgreSQLPythonSQLETLGitCI/CD

Posted about 1 month ago

Apply

🔥 Data Engineer

Posted about 1 month ago

📍 India

🔍 Data and cloud engineering services

🏢 Company: Enable Data Incorporated

🔧 Requirements

Bachelor's degree in Computer Science, Engineering, or a related field.
4+ years of experience with Big Data technologies like Azure/AWS.
Experience with Databricks and Snowflake, with Python scripting and SQL knowledge.
Strong knowledge and experience with Microsoft Azure cloud platform.
Proficiency in SQL and experience with SQL-based database systems.
Experience with batch and data streaming.
Experience using Azure data services such as Azure SQL Database, Azure Data Lake, and Azure Blob Storage.
Strong analytical and problem-solving skills.
Good understanding of data engineering principles and best practices.
Experience with programming languages such as Python or Scala.
Relevant certifications in Azure data services or data engineering are a plus.

💡 Responsibilities

Design, develop, and implement scalable and reliable data solutions on the Azure/AWS platform.
Collaborate with cross-functional teams to gather and analyze data requirements.
Design and implement data ingestion pipelines to collect data from various sources.
Perform data integration and transformation activities, ensuring data quality and consistency.
Implement data storage and retrieval mechanisms using Azure services.
Monitor data pipelines and troubleshoot issues.
Implement data quality measures and governance practices.
Collaborate with data scientists and analysts to support their data needs.

AWSPythonSQLETLMicrosoft AzureSnowflakeData engineering

Posted about 1 month ago

Apply

🔥 Staff Data Engineer - R12645

Posted 3 months ago

📍 India

🔍 Fintech

🏢 Company: Oportun👥 1001-5000💰 $235,000,000 Post-IPO Debt 4 months ago🫂 Last layoff over 1 year agoDebit Cards Consumer Lending Financial Services FinTech

🔧 Requirements

Bachelor's or Master's degree in Computer Science, Data Science, or a related field.
10+ years of experience in data engineering focused on architecture, ETL, and database management.
Proficiency in programming languages like Python/PySpark, Java/Scala.
Expertise in big data technologies such as Hadoop, Spark, Kafka.
In-depth knowledge of SQL and experience with various database technologies.
Familiarity with cloud platforms and their data services.
Strong leadership, problem-solving, and decision-making skills.

💡 Responsibilities

Provide technical leadership and mentorship to the data engineering team.
Lead the design and implementation of complex data architectures.
Design and develop efficient and scalable data pipelines.
Oversee design and management of databases and ETL processes.
Manage complex data engineering projects ensuring timely delivery.
Establish data quality standards and monitoring processes.
Collaborate with stakeholders for data solutions.

AWSPostgreSQLPythonSQLApache AirflowCloud ComputingETLGCPHadoopJavaJenkinsKafkaMySQLAzureData engineeringNosqlSparkScala

Posted 3 months ago

Apply

🔥 Data Engineer

Posted 4 months ago

📍 India

🏢 Company: Unison Consulting Pte Ltd

🔧 Requirements

Minimum 6+ years of Data Ingestion, Integration, ETL, or security engineering experience.
Extensive knowledge of AWS, Azure, GCP.
Strong understanding of Data Management or Data Engineering.
Experienced in Agile methods and Atlassian stack (e.g., JIRA).
Ability to develop roadmaps for data-centric products.
Experience with monitoring frameworks and observability products.
Expertise in SIEM solutions and cloud-based data sources.
Familiarity with security monitoring solutions like Splunk and Datadog.
Experience in DevSecOps/IRE and agile environments.
Expertise in scripting languages (PowerShell, Python, Bash).
Experience with Docker, Kubernetes, Ansible, or Terraform.
Related security certifications (e.g., CISSP, CCSP).
Experience with Linux/Ubuntu/Mac systems.
Experience in creating dashboards and troubleshooting connectivity issues.

💡 Responsibilities

Define and manage data models, schemas, metadata, and security rules.
Design, create, deploy, and manage databases and data structures on-premise and in the cloud.
Identify and mitigate potential security risks.
Ensure compliance with data privacy laws and regulations.
Conduct risk assessments and take appropriate actions to mitigate data security risks.
Train and educate stakeholders about data management.
Collaborate with IT team members and stakeholders to secure data architectures.

AWSDockerPythonAgileBashData AnalysisETLGCPKubernetesJiraAzureData engineeringData StructuresCollaborationLinuxTerraformCompliance

Posted 4 months ago

Apply

🔥 MResult - CDP-Data Engineer

Posted 4 months ago

📍 India

🧭 Full-Time

🔍 Data & Analytics, AI

🏢 Company: Nexthire

🔧 Requirements

Strong experience with Data Factory, Azure, Microsoft Dynamics 365, and Customer Data Platform (CDP).
Experience in MS Customer Insights Data.
Experience in MS customer insights journey.
Experience with Azure, Data Verse, and Power Platform.
Overall experience of 8-10 years.
At least one full project experience on CDP or 3 years in CDP-related support.

💡 Responsibilities

Design, develop, and implement solutions using Microsoft Customer Data Platform (CDP) to manage and analyze customer data.
Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
Integrate CDP with various data sources and ensure seamless data flow and accuracy.
Develop and maintain data pipelines, ensuring data is collected, processed, and stored efficiently.
Create and manage customer profiles, segments, and audiences within the CDP.
Implement data governance and security best practices to protect customer data.
Monitor and optimize the performance of the CDP infrastructure.
Provide technical support and troubleshooting for CDP-related issues.
Stay updated with the latest trends and advancements in CDP technology and best practices.

Microsoft DynamicsAzure

Posted 4 months ago

Apply

🔥 Sr Data Engineer

Posted 4 months ago

📍 India

🔍 SaaS-based global employment platform

🏢 Company: G-P

🔧 Requirements

Extensive experience in developing data and analytics applications in geographically distributed teams.
Hands-on experience in using modern architectures and frameworks, structured, semi-structured and unstructured data, and programming with Python.
Hands-on SQL knowledge and experience with relational databases such as MySQL, PostgreSQL, and others.
Hands-on ETL knowledge and experience.
Knowledge of commercial data platforms (Databricks, Snowflake) or cloud data warehouses (Redshift, BigQuery).
Knowledge of data catalog and MDM tooling (Atlan, Alation, Informatica, Collibra).
CICD pipeline for continuous deployment (CloudFormation template).
Knowledge of how machine learning / A.I. workloads are implemented in batch and streaming, including the preparing of datasets, training models, and using pre-trained models.
Exposure to software engineering processes that can be applied to Data Ecosystems.
Excellent analytical and troubleshooting skills.
Excellent communication skills.
Excellent English (both verbal and written).
BS. in Computer Science or equivalent.

💡 Responsibilities

Design and develop our best-in-class cloud platform, working on all parts of the code stack from front-end, REST and asynchronous APIs, back-end application logic, SQL/NoSQL databases and integrations with external systems.
Develop solutions across the data and analytics stack from ETL and Streaming data.
Design and develop reusable libraries.
Enhance strong processes in Data Ecosystem.
Write unit and integration tests.

PostgreSQLPythonSQLAgileETLMachine LearningMySQLSnowflakeNosqlCommunication Skills

Posted 4 months ago

Apply

🔥 Data Engineer (Bengaluru)

Posted 5 months ago

📍 India

🔍 Data Technology

🏢 Company: Demyst👥 51-100💰 about 2 years agoBig Data Financial Services Broadcasting Data Integration Analytics Information Technology FinTech Software

🔧 Requirements

Bachelor's in Computer Science, Data Science, Engineering or similar technical discipline (or commensurate work experience); Master's degree preferred.
1-3 years of Python programming (with Pandas experience).
Experience with CSV, JSON, parquet, Avro, and other common formats.
Data cleaning and structuring (ETL experience).
Knowledge of API (REST and SOAP), HTTP protocols, API Security and best practices.
Experience with SQL, Git, and Airflow.
Strong written and oral communication skills.
Excellent attention to detail.
Ability to learn and adapt quickly.
Willingness to align to US working hours.

💡 Responsibilities

Collaborate with internal project managers, sales directors, account managers, and clients’ stakeholders to identify requirements and build external data-driven solutions.
Perform data appends, extracts, and analyses to deliver curated datasets and insights to clients to help achieve their business objectives.
Understand and keep current with external data landscapes such as consumer, business, and property data.
Engage in projects involving entity detection, record linking, and data modelling projects.
Design scalable code blocks using Demyst’s APIs/SDKs that can be leveraged across production projects.
Govern releases, change management and maintenance of production solutions in close coordination with clients' IT teams.

AWSPythonSQLETLGitSnowflakeSoftware ArchitectureAirflowData sciencePandasCommunication SkillsJSON

Posted 5 months ago

Apply