Data/AI Security Architect - Databricks

Contract Duration: 6 months Extension: Yes Number of Positions: 1 Schedule Hours: 9am-5pm Monday-Friday; standard 37.5 hrs/week Reason: extra support for a new team Story Behind the Need Business group: Cloud & Platform Engineering -Ensuring the successful implementation of Azure Databricks by guiding the project through all phases required to build, configure, and secure the platform. This includes providing direction on best?practice architecture, enforcing security controls, and overseeing the delivery of a stable, compliant, and fully operational Databricks environment. We are seeking a Senior Data and AI Security Architect to serve as trusted security advisor to the teams that own and operate Azure Databricks data platform, data pipeline infrastructure, and AI/ML infrastructure. This is a consulting and advisory role to provide expert guidance, security architecture recommendations, and risk assessments. You will be the primary security voice in platform design reviews and architecture decisions — translating security requirements into clear, actionable guidance that engineering teams can execute. Success in this role means building strong working relationships with platform owners, influencing without authority, and making security a natural part of how the teams build and operate data and AI systems. Utilize architecture patterns to suggest the most adequate utilization of technical platforms in support of the holistic AI solution security architecture design Job Responsibilities:  Security Advisory & Architecture Guidance Serve as the dedicated security advisor to the Azure Databricks platform team, data engineering, and AI/ML teams, providing expert recommendations on workspace configuration, networking, access controls, and data governance. Participate in architecture and design reviews, offering security risk assessments and prescriptive guidance — with clear rationale — that the platform team can prioritize and implement. Translate organizational security policies, compliance requirements, and threat intelligence into specific, actionable controls for the data and AI platform. Advise on Unity Catalog governance design, including data classification schemas, access control policies, row/column-level security patterns, and audit strategy. Guide the platform team on secure networking patterns including Private Link, VNet injection, and network segmentation. Provide security advisory support for Azure services adjacent to Databricks (ADLS Gen2, Azure Key Vault, Azure Data Factory, Event Hubs) as they intersect with the platform. AI & Machine Learning Security Advisory Advise on security considerations throughout the ML lifecycle, including training data integrity, model registry access controls, and inference endpoint security. Assess AI-specific risks and recommend appropriate mitigations for the organization's AI workloads. Guide teams on secure patterns for integrating third-party LLMs and foundation model APIs, including authentication, data handling, and output filtering controls. Stay current on the evolving AI threat landscape and proactively brief platform and security leadership on emerging risks relevant to the organization's tooling. Perform threat modeling sessions with the platform team using structured methodologies (STRIDE, MITRE ATT&CK) to identify risks early in the design process. Review and assess proposed configurations, policies, and architectural decisions against security best practices, providing documented findings and prioritized recommendations. Provide input on AI governance frameworks aligned with NIST AI RMF, OWASP LLM Top 10, and applicable regulations, translating requirements into actionable guidance. Data Pipeline Security Advise on end-to-end security for data ingestion, transformation, and delivery pipelines. Assess and provide guidance on secrets and credential management within pipeline orchestration tools, recommending patterns that eliminate hardcoded credentials and reduce blast radius. Recommend security controls for pipeline code and configuration, including supply chain risk for third-party connectors, libraries, and ingestion plugins. Advise on data lineage and provenance tracking from a security and auditability perspective, ensuring sensitive data flows are visible, logged, and governed throughout the pipeline. Guide teams on securing streaming and event-driven architectures, including authentication, authorization, and schema validation as a security boundary. Review pipeline CI/CI processes and provide recommendations for integrating security scanning without disrupting engineering velocity. MLOps Security Advise on security controls across the full MLOps lifecycle – from data preparation and feature engineering through model training, validation, registration, deployment, and monitoring. Provide guidance on securing MLflow and the Databricks Model Registry, including access controls, artifact registry, model signing, and audit logging for model promotion workflows. Recommend security patterns for model serving endpoints and inference APIs. Assess risks in feature stores and shared feature pipelines, advising on access controls, data freshness integrity, and preventing cross-team data contamination. Advise on CI/CD pipeline security for model deployment workflows, including separation of duties between data scientists, ML engineers, and productions environments. Governance, Compliance & Stakeholder Engagement Represent the security function in platform team planning and roadmap discussions, ensuring security requirements are surfaced and scoped early rather than retrofitted. Provide security input for audit and compliance activities as they relate to the Databricks platform and AI workloads. Develop and maintain security reference architectures and advisory documents that platform teams and leadership can reference. Build trusted relationships with platform owners, data engineers, and ML engineers — acting as a collaborative partner rather than a gate. Communicate security risk clearly to both technical and non-technical audiences, including senior leadership and compliance stakeholders. Candidate Requirements/Must Have Skills: 10+ years in information security, with at least 4 years in cloud security architecture in Azure environments. Deep knowledge of Azure Databricks security concepts: Unity Catalog, workspace isolation, cluster policies, secrets management, network controls, and audit logging — including practical experience assessing or advising on these areas. Strong understanding of Azure security services. Practical knowledge of AI/ML security risks and established frameworks (NIST AI RMF, OWASP LLM Top 10, MITRE ATLAS). Demonstrated experience in a consulting, advisory, or embedded security architect role — influencing implementation. Nice-To-Have Skills: Certifications: AZ-500 (Azure Security Engineer), Databricks Certified Data Engineer Professional, CISSP, CCSP, or equivalent Experience conducting threat modeling and risk assessments for complex cloud data environments Prior background in data engineering or ML engineering — enabling more credible and practical security conversations with platform teams Familiarity cloud security posture management (CSPM) platforms Experience advising on multi-cloud security (Azure or GCP) Soft Skills Required: Excellent written and verbal communication skills, with a track record of producing clear security guidance documents and presenting risk to diverse audiences. Ability to build collaborative relationships with engineering teams and navigate organizational dynamics to drive security outcomes. Education: Degree in Engineering or Computer Science Certifications: AZ-500 (Azure Security Engineer), Databricks Certified Data Engineer Professional, CISSP, CCSP, or equivalent Best VS. Average Candidate: The best candidate is someone with experience with security in Databricks. Candidate Review & Selection 1 round – Hiring manager and Director – Online – 60 mins – Technical assessment – experience check and cultural fit. Hiring Manager’s availability to interview: ASAP