Lead Data Engineer (Azure/Databricks) at IntelliAM

Location: Sheffield (Full Time)

Reporting to:  VP of Engineering

IntelliAM is an intelligent asset management company specialising in industrial IoT data. We ingest high-frequency factory and sensor data, apply machine learning techniques, and present high value actionable insights to analysts and customers. 

We are building a cutting-edge UNS aligned Industrial IoT data platform with a highly scalable architecture to handle our next phase of growth. We require a delivery-focused technical leader to own the data engineering domain, re-architect and build our pipelines from the ground up, and mentor a small but talented team.  

You will not be maintaining a legacy system; you will be architecting the new one. You will take ownership of the data flows within the platform, designing a robust architecture that supports both real-time operational views and deep historical analysis. 

• Core: Azure, MQTT, Databricks, Python, SQL, dbt
• Storage & Serving: Delta Lake, Postgres, TimescaleDB
• Modelling: MLflow
• Visualisation: Grafana 

• Re-platforming: Own the redesign and rebuild of the data transformation layer (post-event bus) in Databricks. Move from ad-hoc scripts to software engineering standards (CI/CD, testing, modular code).
• Modelling Implementation: Support the implementation of the "Unified Name Space" (UNS) across the data estate, to include schema and path standardisation; machine hierarchies and semantic relationships; and methodologies for validation and processing of data against contracts.
• Enablement: Establish a stable data serving layer for Grafana and Analytics (via Databricks/MLflow), unblocking the Data Science team.
• Mentorship: Upskill the existing team in good practices (dbt, git workflows, SQL optimisation, data modelling). 

• Pragmatic Architect: Capable of making trade-offs between "academic perfection" and "business value". They must align architecture to the 80/20 rule.
• Technical Authority: Comfortable discussing and constructively challenging architectural decisions with the Head of Software and VP of Engineering. Must be able to justify changes to the core architecture and challenge existing thinking.
• Mentor vs. Manager: Willing to sit and pair-program with junior engineers. They should measure their success by the team's outcomes, not simply their own. A full “Team Manager” is not required; this isn’t a pastoral-focused hire.
• Business Translator: Able to explain to non-technical stakeholders why changes are required, and what their impacts may be. Similarly, the ability to provide a bridge between technical approaches and limitations, and business requirements. 

• Advanced Data Engineering: Proven experience building data platforms on Azure. You know Databricks inside out – not just how to write a notebook, but how to architect a Lakehouse, manage clusters, and optimise costs.
• Data Modelling Proficiency: You understand that data engineering is more than moving JSON blobs. You have strong opinions on schema design, data standardisation, and ideally have worked with graph data or hierarchical models.
• Code Quality: You treat data pipelines as software. You use git, CI/CD, and automated testing. You can teach these practices to a junior engineer.
• Communication: You can explain to a business stakeholder why architectural decisions matter for their bottom line, and you can debate architectural trade-offs with Software Engineers. 

• Industrial / IoT Context: Highly desirable. You should be comfortable with the chaotic nature of sensor data, time-series continuity, and the physical reality of the machines we are modelling.
• Alternatively, experience in high-volume data systems that are subject to similar challenges (duplicated or missing data, spiky load, variable schemas, etc.) would be beneficial. 

• Autonomy: You have the support of leadership to make architectural decisions. You aren't fighting for permission to fix the tech debt; you are being hired specifically to fix it.
• Impact: Your work directly enables the Data Science team to stop cleaning data and start building better predictive models.
• Modern Stack: We are moving toward a cutting-edge IIoT stack (UNS + Digital Twins), offering you the chance to work on industry-leading patterns. 

• 5 weeks paid annual leave, plus UK bank holidays
• Tax efficient stock options (subject to scheme rules and eligibility)
• Workplace pension with employer contributions, in line with UK auto-enrolment legislation (subject to eligibility and scheme terms)
• Salary Sacrifice EV Scheme (subject to eligibility)
• Training and professional development opportunities 

This is a full-time role, based at our Sheffield City Centre office. We welcome requests for flexible working and will consider them in line with UK flexible working legislation. 

If your experience does not fully align with every requirement, but you believe you have transferrable skills that would allow you to succeed in the role, we encourage you to apply and highlight this in your application. 

We are an equal opportunities employer and are committed to creating an inclusive workplace. We welcome applications from all suitably qualified candidates regardless of age, disability, gender reassignment, marriage or civil partnership, pregnancy or maternity, race, religion or belief, sex, or sexual orientation. 

Applicants must have the right to work in the UK. 
  
We will process your personal data as part of the recruitment process in accordance with applicable data protection laws.  Please refer to our Recruitment Privacy Notice for further information on how we collect, use, and store candidate information.