Senior Data Engineer

Senior Data Engineer

You will be working on some of the largest and most exciting digital projects currently in the UK. These projects will impact every aspect of our society and every corner of the country.

Your daily tasks will be:

• Developing complex data processing pipelines in a cloud or on-premise environment, ensuring solutions are robust and scalable;
• Providing technical leadership and setting strategic direction to ensure the products that are developed conform to data standards and strategic principals;
• Leading, managing and delivering all aspects of code development activities for data ingest routines for unstructured data;
• Implementing and maintaining coding and tooling standards and ensuring adherence by the wider community;

Leading, managing and delivering all aspects of development activities to meet customer needs;

• Promoting the development of reusable, robust and scalable products e.g. code, citizen centric data models, exploitation products;
• Working with other technologists, including Lead Data Engineers and Data Architects to investigate and develop solutions.

You will have:

• Extensive proven experience of big data engineering techniques and concepts using the Hadoop Stack (Cloudera/EMR), including processing and storage using HDFS, Spark, Hive and Impala;
• Expert level experience in the development of complex data routines and patterns utilising the Cloudera distribution of Hadoop;
• Experience of design and implementation of data storage, including HDFS, relational and NoSQL;
• Experience of developing/utilising programming and query languages e.g. SQL, Java, Scala;
• Monitoring performance and advising on any required infrastructure or changes;
• Prioritise and role-model continuous self-learning and development, and identify areas individuals and teams need to develop in order to achieve future objectives.

What are we offering?

• A competitive base salary - see bandings on advertisement
• Civil service contribution pension
• Flexi-time and alternative working patterns
• 25 days annual holiday
• 5 days leave for training and development

The Department is committed to promoting flexible ways of working, whilst enabling the business to operate at maximum efficiency and will expect colleagues to work from a blend of locations, including some time working at home, as required. The corporate hub location will be the designated place of work and any remote or home working arrangement does not constitute a change to your designated place of work or contractual Terms and Conditions.

This vacancy is using Success Profiles, and will assess your Behaviours, Experience and Technical skills.

Stage 1: Application & Sift

Stage 2 - Video Interview

Successful candidates must pass a disclosure and barring security check.

Our total offer to successful candidates will be based on an assessment of your skills and experience as demonstrated at interview.

People working with government assets must complete basic personnel security standard checks.