AI Architect (Generative AI & Databricks)
E
Entrada AIGenerative AI, data engineering
100% Remote: Full flexibility to work from anywhere in Poland/ Romania / GreeceFull-TimeSenior
Salary200 - 250 PLN per month net b2b
Apply NowOpens the employer's application page
Job Details
- Languages
- En C1
- Experience
- 7+ years of experience in data engineering, data architecture, or applied machine learning systems; 4+ years of hands-on experience designing and deploying solutions on the Databricks Lakehouse Platform.
- Required Skills
- PythonPySpark
Requirements
- Bachelor’s or Master’s degree in Computer Science, Artificial Intelligence, Data Engineering, or a related field.
- 7+ years of experience in data engineering, data architecture, or applied machine learning systems.
- 4+ years of hands-on experience designing and deploying solutions on the Databricks Lakehouse Platform, including Unity Catalog, Delta Lake, Databricks Model Serving, and Databricks SQL.
- Experience building structured extraction pipelines using function calling, Pydantic validation, and custom guardrail or citation-verification systems.
- Experience parsing semi-structured and unstructured business documents using Python and PySpark libraries such as PyMuPDF, pdfplumber, OpenPyXL, and BeautifulSoup4.
- Hands-on experience developing interactive data applications with Streamlit or Dash and deploying containerized apps inside Databricks Apps.
- Understanding of Microsoft Graph API, OAuth 2.0 client credential flows, Microsoft Entra ID (Azure AD), and Databricks Secret Scopes.
- Experience with data modeling, governance, and auditability in Unity Catalog, including Volumes, Delta Tables, ACLs, and row- or column-level security.
- Experience or working familiarity with downstream enterprise ERP integration patterns.
- Strong problem-solving and analytical skills for designing resilient, production-ready systems.
- Excellent communication, consulting, and client-facing skills, including explaining GenAI concepts to technical teams and executive stakeholders.
Responsibilities
- Design and oversee end-to-end GenAI Lakehouse architecture on Databricks to transform unstructured emails and attachments into verified, ERP-ready tabular data.
- Architect secure, incremental ingestion pipelines using Microsoft Graph API and PySpark to process PDFs, Excel workbooks, and HTML bodies.
- Build deterministic extraction workflows on Databricks Model Serving using Claude or GPT and strict Pydantic schemas.
- Implement a three-layer anti-hallucination verification engine using schema constraints, source citations, and PySpark string assertions.
- Deploy a Human-in-the-Loop review application on Databricks Apps using Streamlit or Dash, with reconciliation, manual overrides, and secure authentication.
- Model Gold-layer pricing staging tables and create automated batch feeds for downstream ERP systems using Databricks SQL Warehouses and REST APIs.
- Establish Unity Catalog governance, data lineage, error-quarantine routing, and system security practices across client engineering teams.
- Create and maintain architectural documentation, data models, and interface specifications, and mentor client teams.
View Full Description & ApplyYou'll be redirected to the employer's site
