Data Engineer

Posted 4 months agoViewed

View full description

📍 Location: India

🔍 Industry: Artificial intelligence and aquaculture

🏢 Company: Nacre Capital👥 10-100 Venture Capital

🗣️ Languages: English

🪄 Skills: AWSPythonSQLApache AirflowETLGCPMongoDBAzure

Requirements:

Proficiency in English (spoken and written).
Strong experience in Python and SQL.
Hands-on experience with data processing in Apache Airflow.
Experience working with databases, including MongoDB (NoSQL) and relational databases.
Understanding of data modeling, ETL processes, and data warehousing concepts.
Experience with cloud platforms like AWS, GCP, or Azure.

Responsibilities:

Develop, maintain, and optimize data pipelines and workflows to support ML research and model development.
Design and implement scalable data architectures for handling large datasets used in ML models.
Collaborate closely with ML researchers and data scientists to understand data requirements and ensure data availability and quality.
Prepare and transform data for ML experiments using databases and data integration processes.
Utilize MongoDB and other NoSQL databases to manage unstructured and semi-structured data.
Write efficient, reliable, and maintainable code in Python and SQL for data processing tasks.
Implement data validation and monitoring systems to ensure data integrity and performance.
Support the deployment of ML models by integrating data solutions into production environments.
Ensure the scalability and performance of data systems through rigorous testing and optimization.

Apply

Related Jobs

Apply

🔥 Software Engineer -Data Engineer (Geo) - (Remote - India)

Posted 1 day ago

📍 India

🔍 Software Development

🏢 Company: Jobgether👥 11-50💰 $1,493,585 Seed about 2 years agoInternet

🔧 Requirements

Hands-on experience with GIS, location-based data ingestion pipelines, and AWS services such as EC2, S3, and Lambda.
Proficiency in Python or Java for orchestration of data pipelines.
Strong experience in writing analytical queries using SQL.
Familiarity with Airflow, Docker, and version control with Git.

💡 Responsibilities

Design and maintain data ingestion pipelines that integrate and process large datasets from multiple sources.
Build infrastructure for ETL (Extract, Transform, Load) processes, utilizing AWS technologies such as EC2, S3, EMR, and Lambda.
Collaborate with Product, Analytics, and Client Services teams to resolve data-related technical issues and ensure data infrastructure needs are met.
Write and optimize SQL queries to extract and analyze data effectively.
Participate in code reviews, ensure quality control, and test applications before deployment.
Contribute to the improvement of the location-based platform by proposing and implementing innovative solutions.

AWSDockerPythonSQLCloud ComputingETLGitJavaAirflowData engineering

Posted 1 day ago

Apply

🔥 Senior Data Engineer

Posted 5 days ago

📍 Worldwide

🔍 Hospitality

🏢 Company: Lighthouse

🔧 Requirements

4+ years of professional experience using Python, Java, or Scala for data processing (Python preferred)
You stay up-to-date with industry trends, emerging technologies, and best practices in data engineering.
Improve, manage, and teach standards for code maintainability and performance in code submitted and reviewed
Ship large features independently, generate architecture recommendations and have the ability to implement them
Great communication: Regularly achieve consensus amongst teams
Familiarity with GCP, Kubernetes (GKE preferred), CI/CD tools (Gitlab CI preferred), familiarity with the concept of Lambda Architecture.
Experience with Apache Beam or Apache Spark for distributed data processing or event sourcing technologies like Apache Kafka.
Familiarity with monitoring tools like Grafana & Prometheus.

💡 Responsibilities

Design and develop scalable, reliable data pipelines using the Google Cloud stack.
Optimise data pipelines for performance and scalability.
Implement and maintain data governance frameworks, ensuring data accuracy, consistency, and compliance.
Monitor and troubleshoot data pipeline issues, implementing proactive measures for reliability and performance.
Collaborate with the DevOps team to automate deployments and improve developer experience on the data front.
Work with data science and analytics teams to enable them to bring their research to production grade data solutions, using technologies like airflow, dbt or MLflow (but not limited to)
As a part of a platform team, you will communicate effectively with teams across the entire engineering organisation, to provide them with reliable foundational data models and data tools.
Mentor and provide technical guidance to other engineers working with data.

PythonSQLApache AirflowETLGCPKubernetesApache KafkaData engineeringCI/CDMentoringTerraformScalaData modeling

Posted 5 days ago

Apply

🔥 Senior Data Engineer - Data Services

Posted 6 days ago

📍 Worldwide

🧭 Full-Time

🔧 Requirements

NOT STATED

💡 Responsibilities

Own the design and implementation of cross-domain data models that support key business metrics and use cases.
Partner with analysts and data engineers to translate business logic into performant, well-documented dbt models.
Champion best practices in testing, documentation, CI/CD, and version control, and guide others in applying them.
Act as a technical mentor to other analytics engineers, supporting their development and reviewing their code.
Collaborate with central data platform and embedded teams to improve data quality, metric consistency, and lineage tracking.
Drive alignment on model architecture across domains—ensuring models are reusable, auditable, and trusted.
Identify and lead initiatives to reduce technical debt and modernise legacy reporting pipelines.
Contribute to the long-term vision of analytics engineering at Pleo and help shape our roadmap for scalability and impact.

SQLData AnalysisETLData engineeringCI/CDMentoringDocumentationData visualizationData modelingData analyticsData management

Posted 6 days ago

Apply

🔥 Data Engineer

Posted 8 days ago

📍 Worldwide

🧭 Full-Time

💸 145000.0 - 160000.0 USD per year

🔧 Requirements

Proficiency in managing MongoDB databases, including performance tuning and maintenance.
Experience with cloud-based data warehousing, particularly using BigQuery.
Familiarity with DBT for data transformation and modeling.
Exposure to tools like Segment for data collection and integration.
Basic knowledge of integrating third-party data sources to build a comprehensive data ecosystem.

💡 Responsibilities

Overseeing our production MongoDB database to ensure optimal performance, reliability, and security.
Assisting in the management and optimization of data pipelines into BigQuery, ensuring data is organized and accessible for downstream users.
Utilizing DBT to transform raw data into structured formats, making it useful for analysis and reporting.
Collaborating on the integration of data from Segment and various third-party sources to create a unified, clean data ecosystem.
Working closely with BI, Marketing, and Data Science teams to understand data requirements and ensure our infrastructure meets their needs.
Participating in code reviews, learning new tools, and contributing to the refinement of data processes and best practices.

SQLETLMongoDBData engineeringData modeling

Posted 8 days ago

Apply

🔥 Software Engineer -Data Engineer (Geo)

Posted 9 days ago

📍 India

🧭 Full-Time

🔍 Software Development

🏢 Company: GroundTruth Careers

🔧 Requirements

Experience with GIS, POI/Location data ingestion pipeline.
Experience with AWS Stack used for Data engineering EC2, S3, EMR, ECS, Lambda, and Step functions
Hands on experience with Python/Java for orchestration of data pipelines
Experience in writing analytical queries using SQL
Experience in Airflow
Experience in Docker

💡 Responsibilities

Create and maintain various ingestion pipelines for the GroundTruth platform.
Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL, GIS and AWS ‘big data’ technologies.
Work with stakeholders including the Product, Analytics and Client Services teams to assist with data-related technical issues and support their data infrastructure needs.
Prepare detailed specifications and low-level design.
Participate in code reviews.
Test the product in controlled, real situations before going live.
Maintain the application once it is live.
Contribute ideas to improve the location platform.

AWSDockerPythonSQLApache AirflowGitData engineeringSoftware Engineering

Posted 9 days ago

Apply

🔥 Data Engineer (Web Scraping)

Posted 10 days ago

📍 India

🔍 Market Research and Analytics

🏢 Company: YipitData (Alternative)

🔧 Requirements

4+ years of experience with web scraping frameworks (e.g., Selenium, Playwright, or Puppeteer).
Strong understanding of HTTP, RESTful APIs, HTML parsing, browser rendering, and TLS/SSL mechanics.
Expertise in advanced fingerprinting and evasion strategies (e.g., browser fingerprint spoofing, request signature manipulation).
Deep experience managing cookies, headers, session states, and proxy rotations, including the deployment of both residential and data center proxies.
Experience with logging, metrics, and alerting to ensure high availability.
Troubleshooting skills to optimize scraper performance for efficiency, reliability, and scalability.

💡 Responsibilities

Refactor and Maintain Web Scrapers
Implement Advanced Scraping Techniques
Collaborate with Cross-Functional Teams
Monitor and Troubleshoot
Drive Continuous Improvement

PythonHTMLCSSJavascriptSeleniumRESTful APIs

Posted 10 days ago

Apply

🔥 Data Engineer Lead

Posted 15 days ago

📍 India

🏢 Company: ge_externalsite

🔧 Requirements

Expertise in ETL tools like Informatica PowerCenter, Informatica Cloud ( IICS )
Extensive experience in data engineering, with a focus on Azure cloud platform.
Proficiency in Azure services like Azure Data Factory, Azure Databricks, Azure SQL Data Warehouse, and Azure Stream Analytics.
Strong programming skills in languages such as Python, SQL, or Scala for data manipulation and transformation.
Experience with big data technologies like Hadoop, Spark, or Hive is a plus.
Familiarity with data visualization tools like Power BI or Tableau.
Knowledge of data warehousing concepts, data lakes, and real-time data processing.

💡 Responsibilities

Design and Implement robust ETL workflows using Informatica to extract, transform and load data from diverse sources.
Design data solutions, including data lakes, data warehouses, and real-time data processing systems, leveraging Azure services like Azure Data Lake Storage, Azure Synapse Analytics
Lead a team of azure data engineers in designing, building, and maintaining scalable and efficient data pipelines on Azure cloud platform, utilizing services like Azure Synapse, Azure Data Factory, Azure Databricks, Azure SQL Database, and others.
Integrate diverse data sources into Azure-based solutions.
Ensure smooth and efficient ETL processes, real-time data ingestion, and data transformation.
Implement data integration best practices.
Design and implement data models and schemas to support business requirements.
Ensure data accuracy, consistency, and reliability.
Optimize data structures for performance and scalability.
Implement robust data security measures, including encryption, access control, and data masking.
Ensure compliance with data privacy regulations and company policies.
Monitor and optimize data pipelines and queries for performance and efficiency.
Implement caching, partitioning, and indexing strategies.
Troubleshoot and resolve performance issues.
Establish and enforce data quality standards.
Implement data governance practices, metadata management, and data lineage tracking.
Ensure data quality through validation and cleansing processes.
Collaborate with business stakeholders to understand data requirements and deliver.
Create and maintain comprehensive technical documentation, including system architecture, design documents, and deployment procedures.
Ensure knowledge sharing within the team.
Implement Lean daily management and Lean continuous improvement concepts in Application development and operations.

LeadershipProject ManagementPythonSQLApache HadoopETLMicrosoft Power BICross-functional Team LeadershipTableauAzureData engineeringSparkCommunication SkillsCI/CDProblem SolvingDocumentationAdaptabilityScalaData visualizationTeam managementData modeling

Posted 15 days ago

Apply

🔥 Senior Data Engineer

Posted 20 days ago

📍 Canada, United Kingdom, India

🧭 Full-Time

🔍 Software Development

🏢 Company: Loopio Inc.

🔧 Requirements

5+ years of experience in data engineering in a high-growth agile software development environment
Strong understanding of database concepts, modeling, SQL, query optimization
Ability to learn fast and translate data into actionable results
Experience developing in Python and Pyspark
Hands-on experience with the AWS services (RDS, S3, Redshift, Glue, Quicksight, Athena, ECS)
Strong understanding of relational databases (RDS, MySQL) and NoSQL
Experience with ETL & Data warehousing, building fact & dimensional data models
Experience with data processing frameworks such as Spark / Databricks
Experience in developing Big Data solutions (migration, storage, processing)
Experience with CI/CD tools (Jenkins) and pipeline orchestration tools (Databricks Jobs, Airflow)
Experience working with data visualization and BI platforms (Quicksight, Tableau, Sisense, etc)
Experience working with Clickstream data (Amplitude, Pendo, etc)
Experience building and supporting large-scale systems in a production environment
Strong communication, collaboration, and analytical skills
Demonstrated ability to work with a high degree of ambiguity, and leadership within a team (mentorship, ownership, innovation)
Ability to clearly communicate technical roadmap, challenges, and mitigation

💡 Responsibilities

Be responsible for building, evolving and scaling data platforms and ETL pipelines, with an eye towards the growth of our business and the reliability of our data
Promote data-driven decision-making across the organization through data expertise
Build advanced automation tooling tooling for data orchestration, evaluation, testing, monitoring, administration, and data operations.
Integrate various data sources into our Data lake, including clickstream, relational, and unstructured data
Developing and maintaining a feature store for use in analytics & modeling
Partner with data scientists to create predictive models to help drive insights and decisions, both in Loopio’s product and internal teams (RevOps, Marketing, CX)
Work closely with stakeholders within and across teams to understand the data needs of the business and produce processes that enable a better product and support data-driven decision-making
Build scalable data pipelines using Databricks, and AWS (Redshift, S3, RDS), and other cloud technologies
Build and support Loopio’s data warehouse (Redshift) and data lake (Databricks delta lake)
Orchestrate pipelines using workflow frameworks/tooling

AWSPythonSQLData AnalysisETLJenkinsMachine LearningAirflowData engineeringNosqlSparkCommunication SkillsAnalytical SkillsCollaborationCI/CDData visualizationData modeling

Posted 20 days ago

Apply

🔥 Senior Data Engineer

Posted 29 days ago

📍 India

🏢 Company: BlackStone eIT👥 251-500 Augmented Reality Robotics Analytics Project Management

🔧 Requirements

5+ years of experience in data engineering or a similar role.
Proficiency in SQL and experience with relational databases.
Hands-on experience with data pipeline tools and ETL processes.
Familiarity with big data technologies (e.g., Hadoop, Spark) is a plus.
Experience with cloud-based data solutions is an advantage.

💡 Responsibilities

Design, implement, and maintain scalable data architectures and pipelines.
Develop ETL processes to facilitate smooth data transfer and transformation from various sources.
Collaborate with data scientists and analysts to fulfill data needs for analytics and reporting.
Optimize database performance and maintain data integrity across systems.
Conduct data quality checks and resolve discrepancies.
Mentor junior data engineers and provide technical guidance.
Stay current with emerging technologies and best practices in data engineering.

SQLApache HadoopCloud ComputingData AnalysisETLData engineeringRDBMSSparkData visualizationData modelingData management

Posted 29 days ago

Apply

🔥 Data Engineer Azure (replacement)

Posted about 1 month ago

📍 India

🧭 Contract

🏢 Company: DATAMAXIS

🔧 Requirements

7 years of hands-on experience in designing and developing distributed data pipelines.
5 years of hands-on experience in Azure data service technologies.
5 years of hands-on experience in Python, SQL, Object oriented programming, ETL and unit testing
Experience with data integration with APIs, Web services, Queues
Experience with Azure DevOps and CI/CD as well as agile tools and processes including JIRA, confluence.

💡 Responsibilities

Understand requirements and engage with team to design and deliver projects.
Design and implement data lake house projects within azure.
Design and develop application lifecycle utilizing Microsoft Azure technologies
Participate in design and planning and necessary documentation
Participate in Agile ceremonies including daily standups, scrum, retrospectives, demos, code reviews.
Hands on with Python/SQL development and Azure data pipelines
Engage with team to develop and deliver cross functional products
Document and maintain project artifacts.
Maintain comprehensive knowledge of industry standards, methodologies, processes, and best practices.
Complete training as required for Privacy, Code of Conduct etc.
Promptly report any known or suspected loss, theft or unauthorized disclosure or use of PI to the General Counsel/Chief Compliance Officer or Chief Information Officer.
Adhere to the company's compliance program.
Safeguard the company's intellectual property, information, and assets.
Other duties as assigned.

PythonSQLAgileETLAzureData engineeringREST APICI/CDDevOpsData modeling

Posted about 1 month ago

Apply