Senior Data Engineer
New
I
IDT CorporationBusiness Intelligence
Only accepting applicants from LATAM.Full-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Languages
- English
- Experience
- 8+ years
- Required Skills
- PythonSQLAgileGitHadoopKafkaSnowflakeSparkRedshift
Requirements
- 8+ years of experience as a Data Engineer.
- Demonstrated experience in utilizing Python for data engineering tasks, including transformation, advanced data manipulation, and large-scale data processing.
- Hands-on experience with big data technologies including Apache Spark, Hadoop, and Kafka for distributed processing and real-time data ingestion.
- Experience designing complex data pipelines extracting data from RDBMS, JSON, API, and Flat file sources.
- Demonstrated skills in SQL and PLSQL programming, with advanced mastery in Business Intelligence and data warehouse methodologies.
- Hands-on experience in one or more relational database systems and cloud-based database services such as Snowflake/Redshift.
- Understanding of software engineering principles and skills working on Unix/Linux/Windows Operating systems.
- Experience with Agile methodologies.
- Proficiency in version control systems, with experience in managing code repositories, branching, merging, and collaborating within a distributed development environment.
- Effective oral and written English communication skills.
Responsibilities
- Design, develop, and maintain scalable data pipelines to support ingestion, transformation, and delivery into centralized feature stores, model-training workflows, and real-time inference services.
- Build and optimize workflows for extracting, storing, and retrieving semantic representations of unstructured data to enable advanced search and retrieval patterns.
- Architect and implement lightweight analytics and dashboarding solutions that deliver natural language query experience and AI-backed insights.
- Define and execute processes for managing prompt engineering techniques, orchestration flows, and model fine-tuning routines to power conversational interfaces.
- Oversee vector data stores and develop efficient indexing methodologies to support retrieval-augmented generation (RAG) workflows.
- Partner with data stakeholders to gather requirements for language-model initiatives and translate into scalable solutions.
- Create and maintain comprehensive documentation for all data processes, workflows and model deployment routines.
View Full Description & ApplyYou'll be redirected to the employer's site
