Data Engineering Lead – AWS & AI - Manchester - #2151258

An established financial services organisation is looking for a hands-on Data Engineering Lead to take ownership of its existing data estate and deliver a modern platform supporting Business Intelligence, Data Science and Applied AI.

This is an influential position combining technical leadership, platform ownership and line management. You’ll modernise the organisation’s data capabilities, introduce scalable batch and real-time processing, and eliminate manual or fragile processes through intelligent automation.

You’ll lead a small team of Data Engineers while remaining closely involved in architecture, technology selection and engineering delivery.

• Own the reliability, security and evolution of the organisation’s data platform.
• Modernise an existing SQL Server, SSIS/SSDT and Power BI environment.
• Develop scalable batch and real-time data-processing capabilities.
• Lead the design of a Customer Data Platform.
• Build and evolve a modern AWS lakehouse architecture.
• Create governed semantic models, consistent metrics and clear data definitions.
• Enable secure, self-service data analysis through modern AI tools.
• Automate manual, repetitive and fragile data processes.
• Support BI, Data Science, Machine Learning and Applied AI teams through a unified platform.
• Own data governance, platform security, monitoring and operating costs.
• Manage, develop and help grow the Data Engineering team.
• Work closely with technical leaders and senior stakeholders across the organisation.

You’ll be an experienced Data Engineer or Data Engineering Lead who has built and operated complete data platforms—not simply individual components.

You’ll combine strong technical knowledge with the confidence to make architectural decisions, lead change and determine when existing technology should be evolved or replaced.

Your experience should include:

• Batch processing, data warehousing, ELT and orchestration.
• Modern streaming technologies such as Flink or comparable platforms.
• Production AWS data engineering.
• AWS services including S3, Lambda, Glue, Athena, Kinesis, RDS, IAM and CloudWatch.
• Lakehouse architectures and open table formats such as Iceberg.
• Medallion-style data layering and dimensional modelling.
• Strong SQL and Python development skills.
• Infrastructure as code using Terraform.
• CI/CD, automated testing and modern engineering practices.
• SQL Server, SSIS/SSDT and Power BI.
• Semantic models, shared metrics and data glossaries.
• Data governance, retention, lineage, classification and auditability.
• Managing, mentoring or developing Data Engineers.

You should also be comfortable using advanced AI coding assistants, such as Claude Code or equivalent tools, to accelerate development across code, infrastructure, testing and documentation while maintaining appropriate quality and security controls.

• Financial services or another regulated environment.
• PySpark for large-scale batch or streaming transformations.
• Migrating SQL Server and SSIS estates to lakehouse or metadata-driven architectures.
• Customer Data Platforms or event-driven systems.
• Data-quality tools such as dbt tests or Great Expectations.
• MCP servers or governed natural-language data interfaces.
• Financial crime, fraud or regulatory-reporting data.
• Power BI administration and semantic modelling.

This is an opportunity to take genuine ownership of a business-critical data platform and shape how data and AI are used across an organisation.

You’ll have the freedom to introduce modern technologies, develop a talented engineering team and build a platform capable of supporting future analytics, AI products and strategic decision-making.

Location: Manchester

Working pattern: Hybrid