Data Engineer

Posted 7 months agoViewed

💎 Seniority level: Senior, Minimum 6+ years

📍 Location: India

⏳ Experience: Minimum 6+ years

🪄 Skills: AWSDockerPythonAgileBashData AnalysisETLGCPKubernetesJiraAzureData engineeringData StructuresCollaborationLinuxTerraformCompliance

Requirements:

Minimum 6+ years of Data Ingestion, Integration, ETL, or security engineering experience.
Extensive knowledge of AWS, Azure, GCP.
Strong understanding of Data Management or Data Engineering.
Experienced in Agile methods and Atlassian stack (e.g., JIRA).
Ability to develop roadmaps for data-centric products.
Experience with monitoring frameworks and observability products.
Expertise in SIEM solutions and cloud-based data sources.
Familiarity with security monitoring solutions like Splunk and Datadog.
Experience in DevSecOps/IRE and agile environments.
Expertise in scripting languages (PowerShell, Python, Bash).
Experience with Docker, Kubernetes, Ansible, or Terraform.
Related security certifications (e.g., CISSP, CCSP).
Experience with Linux/Ubuntu/Mac systems.
Experience in creating dashboards and troubleshooting connectivity issues.

Responsibilities:

Define and manage data models, schemas, metadata, and security rules.
Design, create, deploy, and manage databases and data structures on-premise and in the cloud.
Identify and mitigate potential security risks.
Ensure compliance with data privacy laws and regulations.
Conduct risk assessments and take appropriate actions to mitigate data security risks.
Train and educate stakeholders about data management.
Collaborate with IT team members and stakeholders to secure data architectures.

Apply

Related Jobs

Apply

🔥 Senior Data Engineer

Posted about 12 hours ago

📍 India

🧭 Full-Time

🔧 Requirements

Hands-on experience in implementing, supporting, and administering modern cloud-based data solutions (Google BigQuery, AWS Redshift, Azure Synapse, Snowflake, etc.).
Strong programming skills in SQL, Java, and Python.
Experience in configuring and managing data pipelines using Apache Airflow, Informatica, Talend, SAP BODS or API-based extraction.
Expertise in real-time data processing frameworks.
Strong understanding of Git and CI/CD for automated deployment and version control.
Experience with Infrastructure-as-Code tools like Terraform for cloud resource management.
Good stakeholder management skills to collaborate effectively across teams.
Solid understanding of SAP ERP data and processes to integrate enterprise data sources.
Exposure to data visualization and front-end tools (Tableau, Looker, etc).

💡 Responsibilities

Design and Develop Data Pipelines: Create data pipelines to extract data from various sources, transform it into a standardized format, and load it into a centralized data repository.
Build and Maintain Data Infrastructure: Design, implement, and manage data warehouses, data lakes, and other data storage solutions.
Ensure Data Quality and Integrity: Develop data validation, cleansing, and normalization processes to ensure data accuracy and consistency.
Collaborate with Data Analysts and Business Process Owners: Work with data analysts and business process owners to understand their data requirements and provide data support for their projects.
Optimize Data Systems for Performance: Continuously monitor and optimize data systems for performance, scalability, and reliability.
Develop and Maintain Data Governance Policies: Create and enforce data governance policies to ensure data security, compliance, and regulatory requirements.

AWSPythonSQLApache AirflowCloud ComputingETLGitJavaSAPSnowflakeData engineeringCommunication SkillsCI/CDRESTful APIsTerraformData visualizationStakeholder managementData modelingEnglish communication

Posted about 12 hours ago

Apply

🔥 Software Engineer IV - Data Engineer & Full stack

Posted 3 days ago

📍 INDIA

🧭 Full-Time

🔍 Software Development

🏢 Company: ext_apac

🔧 Requirements

10+ years of experience in software testing or software engineering
3+ years in non-functional automation & performance testing
3+ years in Public Cloud based engineering
Experience with React components, hooks, and state management.
Strong skills in modern JavaScript and TypeScript.
Competence in HTML5 and CSS3, including responsive design.
Familiarity with Redux, Context API, or similar libraries.
Knowledge of Webpack, Babel, npm/yarn, and testing libraries like Jest.
Expertise in server-side development using Node.js.
Ability to design and consume APIs.
Experience in Agile, Scrum, or Kanban environments.
Basic understanding for effective collaboration with designers.
Experience building and optimizing ‘big data’ data pipelines, architectures and data sets.
Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
Strong analytic skills related to working with structured and unstructured datasets.
Build processes supporting data transformation, data structures, metadata, dependency and workload management.
A successful history of manipulating, processing and extracting value from large disconnected datasets.
Working knowledge of message queuing, stream processing, and highly scalable ‘big data’ data stores.
Strong project management and organizational skills.
Experience supporting and working with cross-functional teams in a dynamic environment.
Experience with ETL,and big data integration services: Confluent Kafka, BigQuery, Data Bricks, Data Factory, etc.
Experience with relational SQL and NoSQL databases, including DataBricks, BigQuery, Azure Data Warehouse, etc.
Experience with stream-processing systems: kSQL, Flink SQL, dbtLabs, DataBricks, Spark-Streaming, etc.
Experience with object-oriented, functional and scripting languages: Python, Java, C#, Scala, etc.
Experience with Dev Ops tools: CI & Dev Ops: GitHub, GitHub Actions, Jenkins, JIRA, Chef, Sonar
Experience with Testing tools: PractiTest, NUnit, Selenium, BlazeMeter
Strong computer science fundamentals: OOP, design patters, data structures & algorithms
Experience with unit testing, integration testing, performance testing and user acceptance testing.
Azure or GCP Public Cloud Technologies
In-depth knowledge of end-to-end systems development life cycles (including agile, iterative, and other modern approaches to software development)
Outstanding verbal and written communication skills to technical and non-technical audiences of various levels in the organization (e.g., executive, management, individual contributors)
Ability to estimate work effort for project sub-plans or small projects and ensure projects are successfully completed
Quality assurance mindset
Positive outlook, strong work ethic, and responsive to internal and external customers and contacts
Willingly and successfully fulfills the role of teacher, mentor and coach
Requires in-dsepth knowledge of networking, computing platform, storage, database, security, middleware, network and systems management, and related infrastructure

💡 Responsibilities

Work with stakeholders throughout the organization to identify opportunities for leveraging company data to drive business solutions.
Mine and analyze data from different NCR data sources to drive optimization of operations, and improve customer experience.
Assess the effectiveness and accuracy of new data sources and data gathering techniques.
Develop custom data models and algorithms to apply to data sets.
Use predictive modeling to increase and optimize customer experiences, cost savings, actionable insights and other business outcomes.
Develop company A/B testing framework and test model quality.
Collaborate with different functional teams to implement models and monitor outcomes.
Develop processes and tools to monitor and analyze model performance and data accuracy.
Be part of an Agile team, participate in all Agile ceremonies & activities and be accountable for the sprint deliverable
Create and maintain optimal data delivery architecture
Assemble large, complex data sets that meet functional / non-functional business requirements.
Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using Azure and GCP ‘big data’ technologies.
Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics.
Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data delivery needs.
Keep our data separated and secure across national boundaries through multiple data centers and cloud regions.
Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.
Work with data and analytics experts to strive for greater functionality in our data systems.

Backend DevelopmentDockerGraphQLNode.jsPythonSQLAgileData AnalysisData MiningETLFrontend DevelopmentFull Stack DevelopmentGCPHTMLCSSJavaJavascriptKubernetesMachine LearningReact.jsSpring BootTypeScriptC#AlgorithmsApache KafkaAzureData engineeringData scienceData StructuresReduxReactSparkCI/CDRESTful APIsDevOpsMicroservicesJSONScalaData modeling

Posted 3 days ago

Apply

🔥 Job Application for Data Engineer II at ShipBob, Inc. - Greenhouse

Posted 6 days ago

📍 India

🔍 Supply Chain and Fulfillment Technology

🏢 Company: ShipBob, Inc.

🔧 Requirements

3+ years of experience in data engineering or a related field.
Strong SQL skills and a solid understanding of data querying and manipulation.
Experience with performance tuning and optimization.
Exposure to data modeling concepts and warehouse structures.
Familiarity with Azure SQL and data pipeline tools (e.g., Azure Data Factory).
Basic scripting experience (e.g., PowerShell or Python).
Clear and effective communication skills.
Eagerness to work in a fast-paced, collaborative environment.

💡 Responsibilities

Support the development and maintenance of data pipelines.
Write and optimize SQL queries for data transformation, analysis, and reporting.
Assist with performance tuning and basic database administration tasks.
Collaborate on the creation of Power BI visualizations and reports.
Contribute to database and data warehouse modeling under guidance.
Work on defined tasks and actively seek clarification when needed.
Communicate status, blockers, and progress effectively within the team.
Demonstrate curiosity, energy, and a willingness to learn and grow.
Live by ShipBob’s core values and actively contribute to a positive team culture.
Accept and incorporate feedback to improve performance.
Other duties/responsibilities as assigned.

SQLData engineeringData modelingScripting

Posted 6 days ago

Apply

🔥 Lead Data Engineer(Snowflake/DBT)

Posted 9 days ago

📍 UK, India, Germany

🧭 Full-Time

🔍 Fintech

🏢 Company: Careers at Tide

🔧 Requirements

4+ years of extensive development experience using snowflake or similar data warehouse technology
Working experience with dbt and other technologies of the modern data stack, such as Snowflake, Apache Airflow, Fivetran, AWS, git ,Looker
Experience in agile processes, such as SCRUM
Extensive experience in writing advanced SQL statements and performance tuning them
Experience in Data Ingestion techniques using custom or SAAS tool like fivetran
Experience in data modelling and can optimise existing/new data models
Experience in data mining, data warehouse solutions, and ETL, and using databases in a business environment with large-scale, complex datasets
Experience architecting analytical databases (in Data Mesh architecture) is added advantage
Experience working in agile cross-functional delivery team
High development standards, especially for code quality, code reviews, unit testing, continuous integration and deployment
Strong technical documentation skills and the ability to be clear and precise with business users
Business-level of English and good communication skills
Basic understanding of various systems across the AWS platform ( Good to have )
Preferably, you have worked in a digitally native company, ideally fintech

💡 Responsibilities

Developing end to end ETL/ELT Pipeline working with Data Analysts of business Function.
Designing, developing, and implementing scalable, automated processes for data extraction, processing, and analysis in a Data Mesh architecture
Mentoring Fother Junior Engineers in the Team
Be a “go-to” expert for data technologies and solutions
Ability to provide on the ground troubleshooting and diagnosis to architecture and design challenges
Troubleshooting and resolving technical issues as they arise
Looking for ways of improving both what and how data pipelines are delivered by the department
Translating business requirements into technical requirements, such as entities that need to be modelled, DBT models that need to be build, timings, tests and reports
Owning the delivery of data models and reports end to end
Perform exploratory data analysis in order to identify data quality issues early in the process and implement tests to ensure prevent them in the future
Working with Data Analysts to ensure that all data feeds are optimised and available at the required times. This can include Change Capture, Change Data Control and other “delta loading” approaches
Discovering, transforming, testing, deploying and documenting data sources
Applying, help defining, and championing data warehouse governance: data quality, testing, coding best practices, and peer review
Building Looker Dashboard for use cases if required

AWSPythonSQLAgileApache AirflowBusiness IntelligenceData AnalysisData MiningETLGitSCRUMSnowflakeData engineeringTroubleshootingData visualizationData modeling

Posted 9 days ago

Apply

🔥 Senior Data Engineer

Posted 9 days ago

📍 Worldwide

🧭 Full-Time

💸 140000.0 - 175000.0 USD per year

🔍 Software Development

🏢 Company: Figment👥 11-50 Hospitality Travel Accommodations Art

🔧 Requirements

Extensive experience with data engineering, including building and managing data pipelines and ETL processes.
Proficiency in the Python programming language and SQL.
Experience developing highly concurrent and performant applications ensuring scalability and efficient resource utilization in distributed or multi-threaded systems.
Experience implementing robust microservices following best practices in error handling, logging, and testing for production-grade systems.
Experience with using CI/CD pipelines for automated data infrastructure provisioning and application deployment.
Experience with the data orchestration tool Dagster or Airflow.
Experience designing and orchestrating complex DAGs to manage dependencies, triggers, and retries for data workflows, ensuring reliable and efficient pipeline execution.
Experience with the data transformation tool DBT.
Experience designing and implementing complex data transformations using advanced DBT models, materializations, and configurations to streamline data workflows and improve performance.
Experience optimizing and troubleshoot DBT pipelines for scale, ensuring that transformations run efficiently in production environments, handling large datasets without issues.
Experience with cloud data warehousing platforms (e.g. Snowflake)
Experience architecting and optimizing Snowflake environments for performance, including designing partitioning strategies, clustering keys, and storage optimizations for cost-effective scaling.
Has an understanding of security and governance policies within Snowflake, including data encryption, access control, and audit logging to meet compliance and security best practices.

💡 Responsibilities

Implement and maintain reliable data pipelines and data storage solutions.
Implement data modeling and integrate technologies according to project needs.
Manage specific data pipelines and oversees the technical aspects of data operations
Ensure data processes are optimized and align with business requirements
Identify areas for process improvements and suggests tools and technologies to enhance efficiency
Continuously improve data infrastructure automation, ensuring reliable and efficient data processing.
Develop and maintain data pipelines and ETL processes using technologies such as Dagster and DBT to ensure efficient data flow and processing.
Automate data ingestion, transformation, and loading processes to support blockchain data analytics and reporting.
Utilize Snowflake data warehousing solutions to manage and optimize data storage and retrieval.
Collaborate with Engineering Leadership and Product teams to articulate data strategies and progress.
Promote best practices in data engineering, cloud infrastructure, networking, and security.

PythonSQLCloud ComputingETLSnowflakeData engineeringCI/CDRESTful APIsMicroservicesData modeling

Posted 9 days ago

Apply

🔥 Lead Data Engineer

Posted 9 days ago

📍 United States, Latin America, India

🧭 Full-Time

🔍 Software Development

🏢 Company: phData👥 501-1000💰 $2,499,997 Seed about 7 years agoInformation Services Analytics Information Technology

🔧 Requirements

8+ years as a hands-on Data Engineer designing and implementing data solutions
Programming expertise in Java, Python and/or Scala
Core cloud data platforms including Snowflake, AWS, Azure, Databricks and GCP
SQL and the ability to write, debug, and optimize SQL queries
Client-facing written and verbal communication skills and experience
4-year Bachelor's degree in Computer Science or a related field

💡 Responsibilities

Designing and implementing data solutions
Team lead, and/or mentorship of other engineers
Develop end-to-end technical solutions into production — and to help ensure performance, security, scalability, and robust data integration.
Create and deliver detailed presentations
Detailed solution documentation (e.g. including POCS and roadmaps, sequence diagrams, class hierarchies, logical system views, etc.)

AWSPythonSoftware DevelopmentSQLCloud ComputingETLGCPJavaKafkaSnowflakeAirflowAzureData engineeringSparkCommunication SkillsMentoringClient relationship managementScalaData visualizationData modeling

Posted 9 days ago

Apply

🔥 Data Engineer Azure (replacement)

Posted 9 days ago

📍 India

🧭 Contract

🏢 Company: DATAMAXIS

🔧 Requirements

7 years of hands-on experience in designing and developing distributed data pipelines.
5 years of hands-on experience in Azure data service technologies.
5 years of hands-on experience in Python, SQL, Object oriented programming, ETL and unit testing
Experience with data integration with APIs, Web services, Queues
Experience with Azure DevOps and CI/CD as well as agile tools and processes including JIRA, confluence.

💡 Responsibilities

Understand requirements and engage with team to design and deliver projects.
Design and implement data lake house projects within azure.
Design and develop application lifecycle utilizing Microsoft Azure technologies
Participate in design and planning and necessary documentation
Participate in Agile ceremonies including daily standups, scrum, retrospectives, demos, code reviews.
Hands on with Python/SQL development and Azure data pipelines
Engage with team to develop and deliver cross functional products
Document and maintain project artifacts.
Maintain comprehensive knowledge of industry standards, methodologies, processes, and best practices.
Complete training as required for Privacy, Code of Conduct etc.
Promptly report any known or suspected loss, theft or unauthorized disclosure or use of PI to the General Counsel/Chief Compliance Officer or Chief Information Officer.
Adhere to the company's compliance program.
Safeguard the company's intellectual property, information, and assets.
Other duties as assigned.

PythonSQLAgileETLAzureData engineeringCommunication SkillsCI/CDRESTful APIsDevOpsData modeling

Posted 9 days ago

Apply

🔥 Data Engineer, Contract

Posted 26 days ago

📍 India

🧭 Contract

🔍 Construction

🏢 Company: Kojo👥 101-250💰 $39,000,000 Series C over 2 years agoLogistics Construction SaaS Supply Chain Management Commercial Software Procurement

🔧 Requirements

Strong understanding of data warehouse concepts and dimensional modeling
Proficiency with PostgreSQL and Amazon Redshift
Experience with AWS Data Migration Service and Change Data Capture methodologies
Hands-on experience with ELT/ETL tools, particularly Stitch
Proficiency in Python programming for data transformation and pipeline development
Experience with dbt for data transformations and testing
Familiarity with workflow orchestration tools, specifically Dagster
Knowledge of Apache Superset for data visualization
Understanding of containerization concepts, Kubernetes, and AWS EKS
Strong SQL skills for complex data manipulations and optimizations
Ability to implement and maintain efficient data pipeline architectures

💡 Responsibilities

Manage ELT processes using Stitch, including configuration, monitoring, and troubleshooting
Maintain and update AWS Data Migration Service (DMS) tasks for data replication from PostgreSQL to Redshift, with emphasis on Change Data Capture (CDC) implementations
Design, implement, and optimize data models within our Redshift data warehouse following best practices
Develop and maintain data transformation pipelines using Python, dbt, AWS Lambda, and Dagster
Create and maintain data marts for various business domains
Support containerized applications running on AWS EKS (Kubernetes)
Collaborate with engineering team to ensure data quality and accessibility

PostgreSQLPythonSQLAWS EKSETLKubernetesData engineeringData visualizationData modeling

Posted 26 days ago

Apply

🔥 Senior Data Engineer

Posted about 1 month ago

📍 India

🧭 Full-Time

🔍 Market Research & Analytics

🏢 Company: YipitData (Alternative)

🔧 Requirements

5+ years of proven experience in data engineering, particularly in systems with high uptime requirements.
Eager to learn basic application development using Python frameworks and Databricks to automate analytical and data entry workflows
Possess strong communication skills, responsiveness, attention to detail, a team-oriented mindset, and the ability to collaborate effectively with both technical and non-technical stakeholders.
Show a track record of excellent problem-solving and debugging abilities, maintaining reliable codebases, and architecting efficient data processes.
Are proficient in Python, Spark, Docker, AWS, and database technologies. (Experience with Pandas, Plotly Dash, Databricks, or REST APIs is a plus but not required.)

💡 Responsibilities

Own and maintain core data pipelines that power strategic internal and external analytics products.
Build lightweight data applications and tools on top of these pipelines using Python to streamline data refinement, transformation, and processing workflows.
Drive reliability, efficiency, and performance improvements across the data platform.
Diagnose and resolve technical issues in data applications and platform services, including web application performance, optimizing SQL, Pandas, and PySpark queries, and interacting with REST APIs.
Partner with analysts, product teams, and engineering stakeholders to understand data requirements and translate them into scalable solutions.
Identify and implement process improvements to streamline support workflows, reduce repetitive tasks, and improve application and data platform efficiency.

AWSDockerPythonSQLETLGitData engineeringREST APIPandasSparkData modelingDebugging

Posted about 1 month ago

Apply

🔥 Software Engineer -Data Engineer (Geo)

Posted 2 months ago

📍 India

🧭 Full-Time

🔍 Advertising

🏢 Company: GroundTruth Careers

🔧 Requirements

Experience with GIS, POI/Location data ingestion pipeline.
Experience with AWS Stack used for Data engineering EC2, S3, EMR, ECS, Lambda, and Step functions
Hands on experience with Python/Java for orchestration of data pipelines
Experience in writing analytical queries using SQL
Experience in Airflow
Experience in Docker

💡 Responsibilities

Create and maintain various ingestion pipelines for the GroundTruth platform.
Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL, GIS and AWS ‘big data’ technologies.
Work with stakeholders including the Product, Analytics and Client Services teams to assist with data-related technical issues and support their data infrastructure needs.
Prepare detailed specifications and low-level design.
Participate in code reviews.
Test the product in controlled, real situations before going live.
Maintain the application once it is live.
Contribute ideas to improve the location platform.

AWSDockerPythonSQLApache AirflowGitData engineeringSoftware Engineering

Posted 2 months ago

Apply