Principal Data Engineer - AI (REMOTE)

10+ years of software/data engineering experience At least 4 years in technical leadership roles Proven track record building data platforms for production systems at scale Deep expertise in traditional data engineering (Spark, Airflow, data lakes) Deep expertise in ML-specific infrastructure (feature stores, model serving) Experience with vector databases (Pinecone, Weaviate, Qdrant, Milvus, pgvector, Opensearch, ElasticSearch) Demonstrated experience with LLM applications, RAG architectures, and semantic search Understanding of Kubernetes, cloud-native architectures, and infrastructure-as-code principles Strong understanding of data requirements for AI/ML systems Hands-on experience building knowledge bases and semantic search systems Experience with embedding models for code and technical documentation Knowledge of time-series data processing Understanding of graph databases

Define and drive the technical vision for data platforms supporting AI-powered features Lead the design of data pipelines for ML model training datasets Architect vector search and RAG systems Build data infrastructure for semantic search and indexing Establish frameworks for collecting, processing, and analyzing infrastructure configuration data Design data pipelines for Crossplane-specific data Create infrastructure for indexing and searching Upbound Marketplace content Develop metrics and monitoring for AI features Design data systems for AI agents for infrastructure provisioning & operations Create feature engineering platforms Implement data infrastructure for training predictive models Drive the development of knowledge graph representations