Databricks SC Cleared Data Engineer

Salary: £42,000 - 72,500 per year

Requirements

• Proven commercial experience as a Databricks Data Engineer
• Strong PySpark and SQL capability
• Hands-on delivery within AWS, Azure or GCP
• Experience with Delta Lake, data lakes and data warehouses
• Strong understanding of data modelling and scalable pipeline design
• Experience with Git, CI/CD and agile ways of working
• Active SC clearance (mandatory)
• Previous UK government or public sector contracts (desirable)
• Experience with Airflow, Azure Data Factory, AWS Glue or Kafka (desirable)
• Exposure to Snowflake or hybrid data architectures (desirable)
• Knowledge of secure, regulated data environments (desirable)

Responsibilities

• Design, build and maintain Databricks data pipelines using Apache Spark (PySpark & Spark SQL)
• Implement Delta Lake / Lakehouse architectures (Bronze, Silver, Gold)
• Develop scalable ELT/ETL workflows for batch and streaming data
• Optimise performance, reliability and cost across cloud data platforms
• Apply data quality, governance and security controls aligned to government standards
• Collaborate with architects, analysts and platform teams in an agile delivery environment
• Contribute to CI/CD pipelines and automated testing practices

Technologies

• Airflow
• AWS
• AWS Glue
• Azure
• CI/CD
• Cloud
• Databricks
• ETL
• GCP
• Git
• Kafka
• PySpark
• SQL
• Security
• Snowflake
• Spark

More

We are recruiting a contract Databricks Data Engineer for a large-scale UK Government programme delivering critical, cloud-native data platforms. This is a long-term engagement within a high-profile public sector transformation, working with sensitive, mission-critical data at national scale. You must already hold active SC clearance to be considered. We offer competitive market rates and a hybrid/remote working environment.

last updated 18 week of 2026