Data Engineering Architect (Databricks)
What is the role?
Every reliable AI system stands on a reliable data platform. As a Data Engineering Architect at Tensorplay, you’ll own the technical design of Databricks lakehouse platforms for our enterprise clients — shaping how data is ingested, modelled, governed, and served to analytics, ML, and GenAI workloads. You’ll lead client architecture discussions, set engineering standards, and guide data engineers from design through to production.
About you
You’ve designed and delivered large-scale data platforms and you think in terms of the whole system — cost, performance, governance, and the people who will maintain it. You know Databricks deeply, from Delta Lake internals to Unity Catalog and workflow orchestration, and you can explain trade-offs clearly to both engineers and business stakeholders. You care about data quality as much as data volume.
Your Responsibilities
- Architect end-to-end lakehouse platforms on Databricks using Delta Lake and medallion (bronze/silver/gold) patterns
- Define data governance, security, and lineage with Unity Catalog across workspaces and environments
- Design batch and streaming ingestion with Auto Loader, Structured Streaming, and Delta Live Tables / Lakeflow
- Lead migrations from legacy warehouses and Hadoop/on-prem platforms to the Databricks lakehouse
- Build data foundations for ML and GenAI — feature tables, vector search, and RAG-ready document pipelines
- Set standards for CI/CD, testing, and infrastructure-as-code using Databricks Asset Bundles and Terraform
- Optimise cluster, SQL warehouse, and job costs and performance across client environments
- Mentor data engineers and review designs and code across delivery teams
Requirements
- 8+ years in data engineering, with at least 3 years architecting solutions on Databricks
- Expert knowledge of Apache Spark (PySpark and Spark SQL), Delta Lake, and Unity Catalog
- Strong data modelling skills — dimensional modelling, data vault, or lakehouse-native patterns
- Hands-on experience with at least one major cloud (Azure, AWS, or GCP) and its data services
- Experience designing streaming and incremental pipelines at scale
- Proven track record leading platform migrations or greenfield data platform builds
- Excellent communication skills and experience working directly with enterprise clients
Nice to Have
- Databricks Certified Data Engineer Professional or Solutions Architect certification
- Experience with MLflow, Mosaic AI, Databricks Vector Search, or Model Serving
- Familiarity with dbt, Airflow, Kafka, or Fivetran in a lakehouse context
- Background in regulated industries such as finance, healthcare, or insurance
We Offer
- Competitive salary with performance-based bonuses
- Fully remote, async-first work culture
- Databricks certification and training budget
- Architectural ownership across diverse enterprise data and AI projects
- Work directly with founders and senior engineers on high-impact engagements