Data Engineer

Posted 2024-11-07

View full description

📍 Location: United States

🔍 Industry: Consulting and Technology

🏢 Company: BCC-NIH

🗣️ Languages: English

🪄 Skills: PythonSQLAgileBashKubernetesSCRUMJira

Requirements:

Bachelor's degree in a STEM field (Engineering, Computer Science, Mathematics, Physics) or equivalent industry experience in bioinformatics.
Experience in large and complex data operations environments including relational databases and SQL.
Proficiency in scripting languages like Bash and Python.
Familiarity with LINUX/UNIX systems and troubleshooting operational pipelines.
Excellent interpersonal skills and team collaboration.

Responsibilities:

Support data engineering efforts at the National Institutes of Health (NIH) by managing and optimizing data pipelines.
Troubleshoot operational pipelines to resolve priority issues and implement effective solutions.

Apply

Related Jobs

Apply

🔥 Senior Data Engineer

Posted 2024-11-22

📍 Ontario

🔍 Customer engagement platform

🏢 Company: Braze

5+ years of hands-on experience in data engineering, cloud data warehouses, and ETL development.
Proven expertise in designing and optimizing data pipelines and architectures.
Strong proficiency in advanced SQL and data modeling techniques.
A track record of leading impactful data projects from conception to deployment.
Effective collaboration skills with cross-functional teams and stakeholders.
In-depth understanding of technical architecture and data flow in a cloud-based environment.
Ability to mentor and guide junior team members on best practices for data engineering and development.
Passion for building scalable data solutions that enhance customer experiences and drive business growth.
Strong analytical and problem-solving skills, with a keen eye for detail and accuracy.
Extensive experience working with and aggregating large event-level data.
Familiarity with data governance principles and ensuring compliance with industry regulations.
Preferable experience with Kubernetes for container orchestration and Airflow for workflow management.

Lead the design, implementation, and monitoring of scalable data pipelines and architectures using tools like Snowflake and dbt.
Develop and maintain robust ETL processes to ensure high-quality data ingestion, transformation, and storage.
Collaborate closely with data scientists, analysts, and other engineers to design and implement data solutions that drive customer engagement and retention.
Optimize and manage data flows and integrations across various platforms and applications.
Ensure data quality, consistency, and governance by implementing best practices and monitoring systems.
Work extensively with large-scale event-level data, aggregating and processing it to support business intelligence and analytics.
Implement and maintain data products using advanced techniques and tools.
Collaborate with cross-functional teams including engineering, product management, sales, marketing, and customer success to deliver valuable data solutions.
Continuously evaluate and integrate new data technologies and tools to enhance our data infrastructure and capabilities.

SQLBusiness IntelligenceETLSnowflakeData engineeringCollaborationCompliance

Posted 2024-11-22

Apply

🔥 Senior Data Engineer

Posted 2024-11-22

📍 United States of America

💸 90000 - 154000 USD per year

🏢 Company: VSPVisionCareers

Bachelor’s degree in computer science, data science, statistics, economics or related area.
Excellent written and verbal communication skills.
6+ years of experience in development teams focusing on analytics.
6+ years of hands-on experience in data preparation and SQL.
Knowledge of data architectures like event-driven architecture and real-time data.
Familiarity with DataOps practices and multiple data integration platforms.

Design, build, and optimize data pipelines for analytics.
Collaborate with multi-disciplinary teams for data integration.
Analyze data requirements to develop scalable pipeline solutions.
Profile data for accuracy and completeness in data gathering.
Drive automation of data tasks to enhance productivity.
Participate in architecture and design reviews.

AWSSQLAgileETLKafkaOracleSCRUMSnowflakeApache KafkaData scienceData StructuresCommunication SkillsCollaboration

Posted 2024-11-22

Apply

🔥 Data Engineer (Remote in US)

Posted 2024-11-21

📍 United States

🔍 Consumer insights

Strong PL/SQL and SQL development skills.
Proficient in Python and Java.
3-5 years of experience in Data Engineering with Oracle and MS SQL.
Experience with cloud services like Snowflake and Azure.
Familiar with data orchestration tools such as Azure Data Factory and DataBricks.
Understanding of data privacy regulations.

Design, implement and maintain scalable data pipelines and architecture.
Unit test and document solutions that meet product quality standards.
Identify and resolve performance bottlenecks in data processing workflows.
Implement data quality checks to ensure accuracy and consistency.
Collaborate with cross-functional teams to address data needs.

PythonSQLETLGitJavaOracleQASnowflakeJiraAzureData engineeringCI/CD

Posted 2024-11-21

Apply

🔥 Staff Data Engineer, Ads

Posted 2024-11-21

📍 US

🧭 Full-Time

💸 206700 - 289400 USD per year

🔍 Social media / Online community

MS or PhD in a quantitative discipline: engineering, statistics, operations research, computer science, informatics, applied mathematics, economics, etc.
7+ years of experience with large-scale ETL systems, building clean, maintainable, object-oriented code (Python preferred).
Strong programming proficiency in Python, SQL, Spark, Scala.
Experience with data modeling, ETL concepts, and manipulating large structured and unstructured data.
Experience with data workflows (e.g., Airflow) and data visualization tools (e.g., Looker, Tableau).
Deep understanding of technical and functional designs for relational and MPP databases.
Proven track record of collaboration and excellent communication skills.
Experience in mentoring junior data scientists and analytics engineers.

Act as the analytics engineering lead within Ads DS team and contribute to data science data quality and automation initiatives.
Ensure high-quality data through ETLs, reporting dashboards, and data aggregations for business tracking and ML model development.
Develop and maintain robust data pipelines and workflows for data ingestion, processing, and transformation.
Create user-friendly tools for internal use across Data Science and cross-functional teams.
Lead efforts to build a data-driven culture by enabling data self-service.
Provide mentorship and coaching to data analysts and act as a thought partner for data teams.

LeadershipPythonSQLData AnalysisETLTableauStrategyAirflowData analysisData engineeringData scienceSparkCommunication SkillsCollaborationMentoringCoaching

Posted 2024-11-21

Apply

🔥 Data Engineer (Remote in US)

Posted 2024-11-21

📍 US

🔍 Consumer insights

Strong PL/SQL and SQL development skills.
Proficient in multiple data engineering languages such as Python and Java.
Minimum 3-5 years of experience in Data engineering with Oracle and MS SQL.
Experience with data warehousing concepts and cloud-based technologies like Snowflake.
Experience with cloud platforms such as Azure.
Knowledge of data orchestration tools like Azure Data Factory and DataBricks workflows.
Understanding of data privacy regulations and best practices.
Experience working with remote teams.
Familiarity with tools like Git, Jira.
Bachelor's degree in Computer Science or Computer Engineering.

Design, implement and maintain scalable pipelines and architecture to collect, process, and store data from various sources.
Unit test and document solutions that meet product quality standards prior to release to QA.
Identify and resolve performance bottlenecks in pipelines to ensure efficient data delivery.
Implement data quality checks and validation processes.
Work with Data Architect to implement best practices for data governance, quality, and security.
Collaborate with cross-functional teams to address data needs.

PythonSQLETLGitJavaOracleQASnowflakeJiraAzureData engineeringCI/CD

Posted 2024-11-21

Apply

🔥 QA Data Engineer

Posted 2024-11-17

📍 United States

🧭 Full-Time

💸 95000 - 110000 USD per year

🔍 Healthcare

🏢 Company: Wider Circle

Degree in Computer Science, Information Systems, or equivalent education or work experience.
3+ Years experience with AWS or similar technologies (S3, Redshift, RDS, EMR).
3+ Years strong abilities with SQL and Python.
Experience building test automation suites for test and production environments.
Experience using APIs for data extraction and updating.
Experience with Git and version control.

Develop and maintain data quality and accuracy dashboards, and scorecards to track data quality and model performance.
Develop, maintain, and enhance a comprehensive data quality framework that defines data standards, quality and accuracy expectations, and validation processes.
Enhance data quality through rapid testing, feedback, and insights.
Partner with Engineering & Product to predict data quality issues and production flaws.
Conceptualize data architecture (visually) and implement practically into logical structures.
Perform testing of data after ingesting and database loading.
Manage internal SLAs for data quality and frequency.
Provide expert support for solving complex problems of data integration across multiple data sets.
Update and evolve data ecosystem to streamline processes for maximum efficiency.

AWSPythonSQLGitProduct DevelopmentData engineering

Posted 2024-11-17

Apply

🔥 AWS / Azure Cloud Data Engineer (3931)

Posted 2024-11-15

📍 United States

🧭 Full-Time

💸 124300 - 186500 USD per year

🔍 Technology / Cloud Services

🏢 Company: SMX

Two + years of experience in relevant fields.
Expertise in complex SQL.
Knowledge of AWS technologies.
Solid understanding of RDBMS concepts, including Postgres, RedShift, and SQL Server.
Experience with logical data modeling and database/query optimization.
Familiarity with AWS data migration tools (DMS).
Scripting knowledge, especially in Python and Lambda.
Experience with version control and CI/CD tools such as Git, TFS, and Azure DevOps.
Awareness of network authentication and authorization protocols (Kerberos, SAML/OAUTH).
Some knowledge of networks.
Ability to obtain and maintain a Public Trust clearance; US Citizenship is required.
Strong collaboration and communication skills.

Assist Data Architect and customer in collecting requirements and documenting tasks for the data loading platform, focusing on performance and data quality.
Implement data loading and quality control activities based on project requirements and customer tickets.
Create CI/CD pipelines for infrastructure and data loading related to data warehouse maintenance.
Code and implement unique data migration requirements using AWS technologies like DMS and Lambda/Python.
Resolve identity and access management issues for various data sets in Postgres, Redshift, and SQL Server.

AWSPythonSQLETLGitOAuthAzurePostgresRDBMSCI/CDDevOps

Posted 2024-11-15

Apply

🔥 Junior Data Engineer (V)

Posted 2024-11-15

📍 Latin America, United States, Canada

🔍 Life insurance

The ideal candidate will be independent and a great communicator.
Attention to detail is critical.
Must possess problem-solving skills.

Develop and maintain enterprise data and analytics systems for a US client.
Optimize performance by building and supporting decision-making tools.
Collaborate closely with software engineering, AI/ML, Cybersecurity, and DevOps/SysOps teams.
Support end-to-end data pipelines using Python or Scala.
Participate in Agile framework-related tasks.

PythonSoftware DevelopmentAgileData engineeringDevOpsAttention to detail

Posted 2024-11-15

Apply

🔥 Principal Data Engineer

Posted 2024-11-15

📍 United States

🧭 Full-Time

💸 210000 - 220000 USD per year

🔍 Healthcare

10+ years of experience in data engineering with a strong background in building and scaling data architectures.
Advanced knowledge of SQL, relational databases, and big data tools like Spark and Kafka.
Proficient in cloud-based data warehousing and services, especially Snowflake and AWS.
Understanding of AI/ML workflows.
Experience in service-oriented and event-based architecture with strong API development skills.
Strong communication and project management skills.

Lead the Design and Implementation using modern data architecture principles.
Scale Data Platform for optimal data extraction, transformation, and loading.
Design and build scalable AI and ML platforms.
Collaborate with Executive, Product, Clinical, Data, and Design teams.
Build and optimize complex data pipelines.
Create and maintain data tools and pipelines for analytics and data innovation.
Provide technical leadership and mentorship to the data engineering team.

AWSLeadershipProject ManagementPythonSQLJavaKafkaSnowflakeC++AirflowData engineeringSparkCommunication SkillsProblem SolvingOrganizational skills

Posted 2024-11-15

Apply

🔥 Data Engineer, Blockchain

Posted 2024-11-15

📍 North America, South America, Europe

💸 100000 - 500000 USD per year

🔍 Web3, blockchain

🏢 Company: Edge & Node

A self-motivated, team member with keen attention to detail.
Proactive collaboration with team members and a willingness to adapt to a growing environment.
Familiarity and experience with Rust, particularly focusing on data transformation and ingestion.
A strong understanding of blockchain data structures and ingestion interfaces.
Experience in real-time data handling, including knowledge of reorg handling.
Familiarity with blockchain clients like Geth and Reth is a plus.
Adaptability to a dynamic and fully-remote work environment.
Rigorous approach to software development that reflects a commitment to excellence.

Develop and maintain data ingestion adapters for various blockchain networks and web3 protocols.
Implement data ingestion strategies for both historical and recent data.
Apply strategies for handling block reorgs.
Optimize the latency of block ingestion at the chain head.
Write interfaces with file storage protocols such as IPFS and Arweave.
Collaborate with upstream data sources, such as chain clients and tracing frameworks, and monitor the latest upstream developments.
Perform data quality checks, cross-checking data across multiple sources and investigating any discrepancies that arise.

Software DevelopmentBlockchainData StructuresRustCollaborationAttention to detail

Posted 2024-11-15

Apply

Data Engineer

Requirements:

Responsibilities:

Related Jobs

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities