Remoote.app

Work from anywhere. Start here.

Free resume builderCompaniesLog in
Apply

Senior Data Engineer

Posted about 15 hours agoViewed

View full description

šŸ’Ž Seniority level: Senior

šŸ“ Location: India

šŸ—£ļø Languages: English

šŸŖ„ Skills: AWSPythonSQLApache AirflowCloud ComputingETLGitJavaSAPSnowflakeData engineeringCommunication SkillsCI/CDRESTful APIsTerraformData visualizationStakeholder managementData modelingEnglish communication

Requirements:
  • Hands-on experience in implementing, supporting, and administering modern cloud-based data solutions (Google BigQuery, AWS Redshift, Azure Synapse, Snowflake, etc.).
  • Strong programming skills in SQL, Java, and Python.
  • Experience in configuring and managing data pipelines using Apache Airflow, Informatica, Talend, SAP BODS or API-based extraction.
  • Expertise in real-time data processing frameworks.
  • Strong understanding of Git and CI/CD for automated deployment and version control.
  • Experience with Infrastructure-as-Code tools like Terraform for cloud resource management.
  • Good stakeholder management skills to collaborate effectively across teams.
  • Solid understanding of SAP ERP data and processes to integrate enterprise data sources.
  • Exposure to data visualization and front-end tools (Tableau, Looker, etc).
Responsibilities:
  • Design and Develop Data Pipelines: Create data pipelines to extract data from various sources, transform it into a standardized format, and load it into a centralized data repository.
  • Build and Maintain Data Infrastructure: Design, implement, and manage data warehouses, data lakes, and other data storage solutions.
  • Ensure Data Quality and Integrity: Develop data validation, cleansing, and normalization processes to ensure data accuracy and consistency.
  • Collaborate with Data Analysts and Business Process Owners: Work with data analysts and business process owners to understand their data requirements and provide data support for their projects.
  • Optimize Data Systems for Performance: Continuously monitor and optimize data systems for performance, scalability, and reliability.
  • Develop and Maintain Data Governance Policies: Create and enforce data governance policies to ensure data security, compliance, and regulatory requirements.
Apply

Related Jobs

Apply
šŸ”„ Senior Data Engineer
Posted 9 days ago

šŸ“ Worldwide

šŸ§­ Full-Time

šŸ’ø 140000.0 - 175000.0 USD per year

šŸ” Software Development

šŸ¢ Company: FigmentšŸ‘„ 11-50HospitalityTravel AccommodationsArt

  • Extensive experience with data engineering, including building and managing data pipelines and ETL processes.
  • Proficiency in the Python programming language and SQL.
  • Experience developing highly concurrent and performant applications ensuring scalability and efficient resource utilization in distributed or multi-threaded systems.
  • Experience implementing robust microservices following best practices in error handling, logging, and testing for production-grade systems.
  • Experience with using CI/CD pipelines for automated data infrastructure provisioning and application deployment.
  • Experience with the data orchestration tool Dagster or Airflow.
  • Experience designing and orchestrating complex DAGs to manage dependencies, triggers, and retries for data workflows, ensuring reliable and efficient pipeline execution.
  • Experience with the data transformation tool DBT.
  • Experience designing and implementing complex data transformations using advanced DBT models, materializations, and configurations to streamline data workflows and improve performance.
  • Experience optimizing and troubleshoot DBT pipelines for scale, ensuring that transformations run efficiently in production environments, handling large datasets without issues.
  • Experience with cloud data warehousing platforms (e.g. Snowflake)
  • Experience architecting and optimizing Snowflake environments for performance, including designing partitioning strategies, clustering keys, and storage optimizations for cost-effective scaling.
  • Has an understanding of security and governance policies within Snowflake, including data encryption, access control, and audit logging to meet compliance and security best practices.
  • Implement and maintain reliable data pipelines and data storage solutions.
  • Implement data modeling and integrate technologies according to project needs.
  • Manage specific data pipelines and oversees the technical aspects of data operations
  • Ensure data processes are optimized and align with business requirements
  • Identify areas for process improvements and suggests tools and technologies to enhance efficiency
  • Continuously improve data infrastructure automation, ensuring reliable and efficient data processing.
  • Develop and maintain data pipelines and ETL processes using technologies such as Dagster and DBT to ensure efficient data flow and processing.
  • Automate data ingestion, transformation, and loading processes to support blockchain data analytics and reporting.
  • Utilize Snowflake data warehousing solutions to manage and optimize data storage and retrieval.
  • Collaborate with Engineering Leadership and Product teams to articulate data strategies and progress.
  • Promote best practices in data engineering, cloud infrastructure, networking, and security.

PythonSQLCloud ComputingETLSnowflakeData engineeringCI/CDRESTful APIsMicroservicesData modeling

Posted 9 days ago
Apply
Apply
šŸ”„ Senior Data Engineer
Posted about 1 month ago

šŸ“ India

šŸ§­ Full-Time

šŸ” Market Research & Analytics

šŸ¢ Company: YipitData (Alternative)

  • 5+ years of proven experience in data engineering, particularly in systems with high uptime requirements.
  • Eager to learn basic application development using Python frameworks and Databricks to automate analytical and data entry workflows
  • Possess strong communication skills, responsiveness, attention to detail, a team-oriented mindset, and the ability to collaborate effectively with both technical and non-technical stakeholders.
  • Show a track record of excellent problem-solving and debugging abilities, maintaining reliable codebases, and architecting efficient data processes.
  • Are proficient in Python, Spark, Docker, AWS, and database technologies. (Experience with Pandas, Plotly Dash, Databricks, or REST APIs is a plus but not required.)
  • Own and maintain core data pipelines that power strategic internal and external analytics products.
  • Build lightweight data applications and tools on top of these pipelines using Python to streamline data refinement, transformation, and processing workflows.
  • Drive reliability, efficiency, and performance improvements across the data platform.
  • Diagnose and resolve technical issues in data applications and platform services, including web application performance, optimizing SQL, Pandas, and PySpark queries, and interacting with REST APIs.
  • Partner with analysts, product teams, and engineering stakeholders to understand data requirements and translate them into scalable solutions.
  • Identify and implement process improvements to streamline support workflows, reduce repetitive tasks, and improve application and data platform efficiency.

AWSDockerPythonSQLETLGitData engineeringREST APIPandasSparkData modelingDebugging

Posted about 1 month ago
Apply
Apply
šŸ”„ Senior Data Engineer
Posted 2 months ago

šŸ“ Canada, United Kingdom, India

šŸ§­ Full-Time

šŸ” Software Development

šŸ¢ Company: Loopio Inc.

  • 5+ years of experience in data engineering, with hands-on experience in ETL, data warehousing, and cloud-based data platforms.
  • Strong proficiency in AWS (S3, RDS, Redshift, Glue).
  • Deep understanding of database design, query optimization, and data modeling.
  • Expertise in distributed data processing frameworks (Spark, Databricks, Presto).
  • Strong Python and PySpark development skills.
  • Experience with CI/CD pipelines (Jenkins) and data workflow automation (Airflow, Databricks Jobs).
  • Proven ability to build and optimize large-scale data infrastructure, ensuring low-latency access, efficient storage, and high reliability.
  • Design, build, and optimize data pipelines to support high-scale data ingestion, transformation, and storage.
  • Develop efficient ETL workflows using Databricks, Redshift, and AWS cloud technologies.
  • Develop and maintain a feature store to support predictive models, AI-driven insights, and data science applications.
  • Enable seamless integration of structured (RDS, MySQL) and unstructured (clickstream, logs) data into the data lake / data warehouse / data mart layers.
  • Build robust automation tooling for data orchestration, testing, monitoring, and administration.
  • Leverage workflow orchestration tools (Airflow, Databricks Workflows) to streamline data operations.
  • Architect scalable data solutions leveraging AWS services (S3, Redshift, Lambda, RDS, DMS) and distributed computing frameworks (Spark, Presto).
  • Optimize query performance and ensure reliability in high-volume environments.
  • Partner with Data Scientists, ML Engineers, and business stakeholders to create AI-powered insights, drive product improvements, and enable data-driven decision-making.

AWSPythonSQLData AnalysisETLJenkinsMachine LearningAirflowData engineeringNosqlSparkCommunication SkillsAnalytical SkillsCollaborationCI/CDData visualizationData modeling

Posted 2 months ago
Apply
Apply
šŸ”„ Senior Data Engineer
Posted 3 months ago

šŸ“ Worldwide

šŸ§­ Full-Time

šŸ’ø 167471.0 USD per year

šŸ” Software Development

šŸ¢ Company: Float.com

  • Expertise in ML, expert systems, and advanced algorithms (e.g., pattern matching, optimization) with applied experience in Scheduling, Recommendations, or Personalization.
  • Proficient in Python or Java and comfortable with SQL and Javascript/Typescript.
  • Experience with large-scale data pipelines and stream processing (e.g., Kafka, Debezium, Flink).
  • Skilled in data integration, cleaning, and validation.
  • Familiar with vector and graph databases (e.g., Neo4j).
  • Lead technical viability discussions:
  • Resource Recommendation Engine:
  • Analyse existing data:
  • Evaluate our data streaming pipeline:
  • Lead technical discussions related to optimization, pattern detection, and AI, serving as the primary point of contact for these areas within Float.
  • Develop and implement advanced algorithms to enhance the Resource Recommendation Engine and other product features, initially focused on pattern detection and optimization.
  • Design, implement, and maintain our streaming data architecture to support real-time data processing and analytics, ensuring data integrity and reliability.
  • Establish best practices and standards for optimization, AI, and data engineering development within the organization.
  • Mentor and train team members on optimization, AI, and data engineering concepts and techniques, fostering a culture of continuous learning and innovation.
  • Stay updated with the latest trends and related technologies, and proactively identify opportunities to incorporate them into Float's solutions.

PythonSQLKafkaMachine LearningAlgorithmsData engineering

Posted 3 months ago
Apply
Apply
šŸ”„ Senior Data Engineer
Posted 4 months ago

šŸ“ Worldwide

šŸ” Event Technology

NOT STATED
NOT STATED

AWSDockerPostgreSQLPythonSQLApache AirflowCloud ComputingData AnalysisETLKubernetesAlgorithmsApache KafkaData engineeringData StructuresCI/CDRESTful APIsMicroservicesData visualizationData modeling

Posted 4 months ago
Apply
Apply
šŸ”„ Senior Data Engineer
Posted 7 months ago

šŸ“ India

šŸ” Automotive

šŸ¢ Company: Salvo Software

  • 5+ years experience in programming and development of aftermarket scan tools.
  • 5+ years of experience managing and integrating OEM automotive data into automotive aftermarket scan tool products.
  • Experience with ODX ISO 22901 (Open Diagnostic eXchange).
  • Understanding of various data formats supplied by OEMs.
  • Technical knowledge required to parse, modify and integrate that data into a unified format.
  • Disciplined practice for thorough and clear documentation.
  • Oversee the management and modification of OEM diagnostic data for the engineering team.
  • Advise the engineering team on best practices for handling diagnostic data.
  • Understand how to integrate data from various sources into a scan tool device.

Software DevelopmentSQLData AnalysisEmbedded SystemsData engineeringCommunication SkillsAnalytical SkillsProblem SolvingAgile methodologiesRESTful APIsDocumentationJSONData modelingScriptingData management

Posted 7 months ago
Apply