Senior Data Engineer

Posted about 1 month agoViewed

View full description

💎 Seniority level: Senior, 5+ years

📍 Location: India, IST

🔍 Industry: Market Research & Analytics

🏢 Company: YipitData (Alternative)

🗣️ Languages: English

⏳ Experience: 5+ years

🪄 Skills: AWSDockerPythonSQLETLGitData engineeringREST APIPandasSparkData modelingDebugging

Requirements:

5+ years of proven experience in data engineering, particularly in systems with high uptime requirements.
Eager to learn basic application development using Python frameworks and Databricks to automate analytical and data entry workflows
Possess strong communication skills, responsiveness, attention to detail, a team-oriented mindset, and the ability to collaborate effectively with both technical and non-technical stakeholders.
Show a track record of excellent problem-solving and debugging abilities, maintaining reliable codebases, and architecting efficient data processes.
Are proficient in Python, Spark, Docker, AWS, and database technologies. (Experience with Pandas, Plotly Dash, Databricks, or REST APIs is a plus but not required.)

Responsibilities:

Own and maintain core data pipelines that power strategic internal and external analytics products.
Build lightweight data applications and tools on top of these pipelines using Python to streamline data refinement, transformation, and processing workflows.
Drive reliability, efficiency, and performance improvements across the data platform.
Diagnose and resolve technical issues in data applications and platform services, including web application performance, optimizing SQL, Pandas, and PySpark queries, and interacting with REST APIs.
Partner with analysts, product teams, and engineering stakeholders to understand data requirements and translate them into scalable solutions.
Identify and implement process improvements to streamline support workflows, reduce repetitive tasks, and improve application and data platform efficiency.

Apply

Related Jobs

Apply

🔥 Senior Data Engineer

Posted about 15 hours ago

📍 India

🧭 Full-Time

🔧 Requirements

Hands-on experience in implementing, supporting, and administering modern cloud-based data solutions (Google BigQuery, AWS Redshift, Azure Synapse, Snowflake, etc.).
Strong programming skills in SQL, Java, and Python.
Experience in configuring and managing data pipelines using Apache Airflow, Informatica, Talend, SAP BODS or API-based extraction.
Expertise in real-time data processing frameworks.
Strong understanding of Git and CI/CD for automated deployment and version control.
Experience with Infrastructure-as-Code tools like Terraform for cloud resource management.
Good stakeholder management skills to collaborate effectively across teams.
Solid understanding of SAP ERP data and processes to integrate enterprise data sources.
Exposure to data visualization and front-end tools (Tableau, Looker, etc).

💡 Responsibilities

Design and Develop Data Pipelines: Create data pipelines to extract data from various sources, transform it into a standardized format, and load it into a centralized data repository.
Build and Maintain Data Infrastructure: Design, implement, and manage data warehouses, data lakes, and other data storage solutions.
Ensure Data Quality and Integrity: Develop data validation, cleansing, and normalization processes to ensure data accuracy and consistency.
Collaborate with Data Analysts and Business Process Owners: Work with data analysts and business process owners to understand their data requirements and provide data support for their projects.
Optimize Data Systems for Performance: Continuously monitor and optimize data systems for performance, scalability, and reliability.
Develop and Maintain Data Governance Policies: Create and enforce data governance policies to ensure data security, compliance, and regulatory requirements.

AWSPythonSQLApache AirflowCloud ComputingETLGitJavaSAPSnowflakeData engineeringCommunication SkillsCI/CDRESTful APIsTerraformData visualizationStakeholder managementData modelingEnglish communication

Posted about 15 hours ago

Apply

🔥 Senior Data Engineer

Posted 9 days ago

📍 Worldwide

🧭 Full-Time

💸 140000.0 - 175000.0 USD per year

🔍 Software Development

🏢 Company: Figment👥 11-50 Hospitality Travel Accommodations Art

🔧 Requirements

Extensive experience with data engineering, including building and managing data pipelines and ETL processes.
Proficiency in the Python programming language and SQL.
Experience developing highly concurrent and performant applications ensuring scalability and efficient resource utilization in distributed or multi-threaded systems.
Experience implementing robust microservices following best practices in error handling, logging, and testing for production-grade systems.
Experience with using CI/CD pipelines for automated data infrastructure provisioning and application deployment.
Experience with the data orchestration tool Dagster or Airflow.
Experience designing and orchestrating complex DAGs to manage dependencies, triggers, and retries for data workflows, ensuring reliable and efficient pipeline execution.
Experience with the data transformation tool DBT.
Experience designing and implementing complex data transformations using advanced DBT models, materializations, and configurations to streamline data workflows and improve performance.
Experience optimizing and troubleshoot DBT pipelines for scale, ensuring that transformations run efficiently in production environments, handling large datasets without issues.
Experience with cloud data warehousing platforms (e.g. Snowflake)
Experience architecting and optimizing Snowflake environments for performance, including designing partitioning strategies, clustering keys, and storage optimizations for cost-effective scaling.
Has an understanding of security and governance policies within Snowflake, including data encryption, access control, and audit logging to meet compliance and security best practices.

💡 Responsibilities

Implement and maintain reliable data pipelines and data storage solutions.
Implement data modeling and integrate technologies according to project needs.
Manage specific data pipelines and oversees the technical aspects of data operations
Ensure data processes are optimized and align with business requirements
Identify areas for process improvements and suggests tools and technologies to enhance efficiency
Continuously improve data infrastructure automation, ensuring reliable and efficient data processing.
Develop and maintain data pipelines and ETL processes using technologies such as Dagster and DBT to ensure efficient data flow and processing.
Automate data ingestion, transformation, and loading processes to support blockchain data analytics and reporting.
Utilize Snowflake data warehousing solutions to manage and optimize data storage and retrieval.
Collaborate with Engineering Leadership and Product teams to articulate data strategies and progress.
Promote best practices in data engineering, cloud infrastructure, networking, and security.

PythonSQLCloud ComputingETLSnowflakeData engineeringCI/CDRESTful APIsMicroservicesData modeling

Posted 9 days ago

Apply

🔥 Senior Data Engineer

Posted 2 months ago

📍 Canada, United Kingdom, India

🧭 Full-Time

🔍 Software Development

🏢 Company: Loopio Inc.

🔧 Requirements

5+ years of experience in data engineering, with hands-on experience in ETL, data warehousing, and cloud-based data platforms.
Strong proficiency in AWS (S3, RDS, Redshift, Glue).
Deep understanding of database design, query optimization, and data modeling.
Expertise in distributed data processing frameworks (Spark, Databricks, Presto).
Strong Python and PySpark development skills.
Experience with CI/CD pipelines (Jenkins) and data workflow automation (Airflow, Databricks Jobs).
Proven ability to build and optimize large-scale data infrastructure, ensuring low-latency access, efficient storage, and high reliability.

💡 Responsibilities

Design, build, and optimize data pipelines to support high-scale data ingestion, transformation, and storage.
Develop efficient ETL workflows using Databricks, Redshift, and AWS cloud technologies.
Develop and maintain a feature store to support predictive models, AI-driven insights, and data science applications.
Enable seamless integration of structured (RDS, MySQL) and unstructured (clickstream, logs) data into the data lake / data warehouse / data mart layers.
Build robust automation tooling for data orchestration, testing, monitoring, and administration.
Leverage workflow orchestration tools (Airflow, Databricks Workflows) to streamline data operations.
Architect scalable data solutions leveraging AWS services (S3, Redshift, Lambda, RDS, DMS) and distributed computing frameworks (Spark, Presto).
Optimize query performance and ensure reliability in high-volume environments.
Partner with Data Scientists, ML Engineers, and business stakeholders to create AI-powered insights, drive product improvements, and enable data-driven decision-making.

AWSPythonSQLData AnalysisETLJenkinsMachine LearningAirflowData engineeringNosqlSparkCommunication SkillsAnalytical SkillsCollaborationCI/CDData visualizationData modeling

Posted 2 months ago

Apply

🔥 Senior Data Engineer

Posted 3 months ago

📍 Worldwide

🧭 Full-Time

💸 167471.0 USD per year

🔍 Software Development

🏢 Company: Float.com

🔧 Requirements

Expertise in ML, expert systems, and advanced algorithms (e.g., pattern matching, optimization) with applied experience in Scheduling, Recommendations, or Personalization.
Proficient in Python or Java and comfortable with SQL and Javascript/Typescript.
Experience with large-scale data pipelines and stream processing (e.g., Kafka, Debezium, Flink).
Skilled in data integration, cleaning, and validation.
Familiar with vector and graph databases (e.g., Neo4j).

💡 Responsibilities

Lead technical viability discussions:
Resource Recommendation Engine:
Analyse existing data:
Evaluate our data streaming pipeline:
Lead technical discussions related to optimization, pattern detection, and AI, serving as the primary point of contact for these areas within Float.
Develop and implement advanced algorithms to enhance the Resource Recommendation Engine and other product features, initially focused on pattern detection and optimization.
Design, implement, and maintain our streaming data architecture to support real-time data processing and analytics, ensuring data integrity and reliability.
Establish best practices and standards for optimization, AI, and data engineering development within the organization.
Mentor and train team members on optimization, AI, and data engineering concepts and techniques, fostering a culture of continuous learning and innovation.
Stay updated with the latest trends and related technologies, and proactively identify opportunities to incorporate them into Float's solutions.

PythonSQLKafkaMachine LearningAlgorithmsData engineering

Posted 3 months ago

Apply

🔥 Senior Data Engineer

Posted 4 months ago

📍 Worldwide

🔍 Event Technology

🔧 Requirements

NOT STATED

💡 Responsibilities

NOT STATED

AWSDockerPostgreSQLPythonSQLApache AirflowCloud ComputingData AnalysisETLKubernetesAlgorithmsApache KafkaData engineeringData StructuresCI/CDRESTful APIsMicroservicesData visualizationData modeling

Posted 4 months ago

Apply

🔥 Senior Data Engineer

Posted 7 months ago

📍 India

🔍 Automotive

🏢 Company: Salvo Software

🔧 Requirements

5+ years experience in programming and development of aftermarket scan tools.
5+ years of experience managing and integrating OEM automotive data into automotive aftermarket scan tool products.
Experience with ODX ISO 22901 (Open Diagnostic eXchange).
Understanding of various data formats supplied by OEMs.
Technical knowledge required to parse, modify and integrate that data into a unified format.
Disciplined practice for thorough and clear documentation.

💡 Responsibilities

Oversee the management and modification of OEM diagnostic data for the engineering team.
Advise the engineering team on best practices for handling diagnostic data.
Understand how to integrate data from various sources into a scan tool device.

Software DevelopmentSQLData AnalysisEmbedded SystemsData engineeringCommunication SkillsAnalytical SkillsProblem SolvingAgile methodologiesRESTful APIsDocumentationJSONData modelingScriptingData management

Posted 7 months ago

Apply