SUMMARY:

As a Data Engineer, you will be responsible for designing, developing, maintaining, and optimising a data pipeline infrastructure using a proprietary data platform , which is based on Databricks. You will collaborate with cross-functional teams to design and implement scalable data solutions, ensuring efficient data ingestion, transformation, storage, and analysis.

WHAT WILL YOU BE DOING:

Data Engineering Ownership: End-to-end creation and upkeep of robust data workflows and components, leveraging tools such as Databricks. This includes designing early-stage prototypes and deploying large-scale data acquisition, handling, and storage strategies.
ETL/ELT Workflow Management: Construct and refine data ingestion pipelines to seamlessly integrate varied data sources into the central platform. Ensure data consistency through validation, cleaning, and enrichment routines.
Stakeholder Collaboration: Partner with commercial teams and data stakeholders to gather and refine specifications for analytical products and visual reporting needs.
Architectural Design & Data Modelling: Engage with analysts and data experts to define efficient data models and architectural plans. Support dashboard and report development, while ensuring optimal data structuring for performance.
Pipeline Performance & Reliability: Continuously assess and fine-tune data workflows, addressing system inefficiencies, integration problems, and data fidelity concerns.
Quality Assurance in Data: Define expectations for data integrity and collaborate with QA to automate validation checks. Utilise monitoring tools to surface and track data quality metrics.
Data Controls & Compliance: Apply internal governance standards and safeguard sensitive data through access rules, encryption protocols, and retention strategies, aligning with organisational security frameworks.
Technical Enablement & Knowledge Sharing: Work closely with interdisciplinary teams to ensure data needs are met, while thoroughly documenting processes and solutions. Convey technical ideas in a way that’s accessible across departments.
Innovation & Process Evolution: Keep informed on developments within the data ecosystem. Advocate for and implement improvements in efficiency, tooling, and automation. Participate in internal knowledge communities.
Agile Delivery Engagement: Contribute to delivery efforts by actively participating in sprint planning, stand-ups, and backlog refinement. Help define and deliver technical tasks, aligning work with CI/CD best practices.

WHAT WE ARE LOOKING FOR:

Approximately 4 years of experience in data engineering for mid-level roles
6 or more years of experience for senior-level data engineering positions
Between 1 to 4 years of experience building data products using platforms such as Databricks, Spark, Cloudera, or HortonWorks
Skilled in Python (especially PySpark), Scala, and SQL
Experienced in designing and implementing scalable data pipelines for high-volume environments
Solid understanding of ELT/ETL practices and data integration strategies
Capable of writing robust production code with automated testing
Familiar with CI/CD tools such as GitHub Actions and Jenkins for deploying code
Hands-on experience with distributed data processing using Apache Spark
Proficient with cloud services including AWS, Azure, or Google Cloud and tools like S3, Glue, Lambda, Redshift, and BigQuery
Good knowledge of data modelling, relational databases, and SQL performance optimisation
Strong analytical and problem-solving abilities, with attention to troubleshooting details
Effective communicator with experience working in collaborative, cross-functional teams
Basic understanding of machine learning concepts such as classification, regression, A/B testing, and experimental design

AWESOME BUT NOT REQUIRED:

Understanding of the UK media landscape, including over-the-top (OTT) and traditional broadcast advertising
Familiarity with concepts and trends in digital advertising and marketing analytics
Experience applying statistical approaches such as regression and classification, as well as designing and analysing A/B and other controlled experiments
Awareness of modern data architecture methodologies including Data Mesh, enterprise-level data frameworks, and business intelligence architecture
Sol

Senior Data Engineer AWS & GCP at Vigil