Senior Data Engineer, AI Systems

View full description

Requirements:

• 5+ years in data engineering with a track record of developing and scaling data-driven products
• Expertise in building data pipelines, architectures, and datasets, with experience in AWS cloud services (EC2, EMR, RDS, Redshift).
• Proficient in big data tools (Hadoop, Spark, Kafka) and machine learning frameworks (TensorFlow, PyTorch).
• 3+ years experience with Python
• Deep knowledge of SQL and NoSQL databases, workflow management tools (Azkaban, Luigi, Airflow), and an understanding of the machine learning model deployment cycle.
• Top candidates will have experience or working knowledge of vector databases and advanced Retrieval-Augmented Generation (RAG) systems (Langchain, Pinecone, OpenAI/ChatGPT).

Responsibilities:

• Architect and implement highly scalable advanced Retrieval-Augmented Generation (RAG) data pipelines to support the aggregation and analysis of legislative bills, social media posts, documents, and testimonial content.
• Design robust data pipelines that enable real-time processing and analysis of vast datasets to inform public affairs strategies and decision-making.
• Design and implement data cleansing and transformation pipelines to power the AI product.
• Lead cloud-based deployments, primarily in AWS, ensuring optimal performance, security, and cost-efficiency.
• Innovate and iterate on data architecture to support Quorum Copilot's dynamic needs, ensuring the product remains at the cutting edge of AI-driven public affairs tools.
• Drive build vs buy, tool selection, and solution trade-off analysis using standard engineering principles.