Senior Data Engineer

KPIKPI Digital helps clients reach their business objectives by identifying required capabilities and by providing the proper data and analytics strategy and foundation allowing handling of all data challenges including enterprise-wide AI scaling. We specialize in Data & Analytics from Strategy, Architecture, Governance, Modern D&A Platform, MDM and the entire D&A Value Chain (Data Ingestion, Data Pipelines, Lakehouse, AI Modeling, DO/OR, BI, Deployment and Monitoring).

Performance Management and Digital Experience are two additional practices that complement the D&A practice. In addition to the Consulting Services, we offer Managed Services, Education and Training.

The Role

We are looking for a Data Engineer that combines Data Engineering, platform architecture and management expertise. The Data Engineer will:

• Provide consulting services to our customers as the KPI Digital Lead Data Engineer
• Define and maintain the KPI Digital Modern D&A Platform Architecture. This D&A Platform Architecture must:
• Support real-time LakeHouse Architecture on Azure, AWS and GCP
• Data feeds from business applications and sensors, IoT, PLC, Scada
• Any type of workload from API on Insights, BI, AI ML/DL, NLP, DO/OR, Execution of ML models in real-time, batches, micro-batches or on edge
• Data Ingestion in batch, micro-batches and streaming
• Real-time dashboard
• Respect highest security standards
• Scalable, elastic and performant
• Maintain AND Improve Accelerators to speed up the creation of Data Pipelines on major cloud providers
• Participate and support the AI Pre-Sales activities
• Build a Data Engineering team to meet the expected customers growth
• Create Data Engineering content (articles, white papers and so on)
• Speak in conferences, webinars to promote KPI Digital brand

The Ideal Candidate Is

• Goal oriented and possesses strong decision-making and problem-solving skills
• Able to build a team of diverse talents to meet the customers expectations
• Expected to work in close collaboration with the AI Team Lead, the Data Governance Lead and the VP of D&A
• Able to devise training programs around the consulting mandates for his/her team to grow
• Communicates and explains the plan, templates, approaches to his/her team and other teams
• Can communicate clearly in both languages (French and English)
• Excellent written and verbal communication skills for coordinating across teams
• Ability to discern fundamental and key aspects worth ‘fighting for’ versus the less ‘important’ ones
• Work with our customers’ stakeholders to identify opportunities for leveraging their existing D&A platform toward a modern real-time LakeHouse
• Coordinate with different functional teams to implement all D&A assets with proper monitoring
• Keep up to date with various modern D&A platform services and trends

Specific Skills

• Has implemented Lambda Architecture (may also have implemented KAPPA)
• Understands and did implement how to move data from Event Hub (or similar) to a raw zone, then an integrated zone and then to a consumption zone with replication to a Serving Layer
• Has been implementing the main 3 types of slowly changing dimensions in this technology or older ones
• Has Data Engineering using DataVault
• Has used Delta Lake
• Knows how to stream data in the Data Lake using Databricks
• Is very solid with Spark and can tune the environment using multiple nodes
• Can optimize the use of Azure resources to optimize costs and usage
• Has some Azure administrator and cloud architecture experience
• Can define the access rights management solution
• Is an adept of CI-CD and can implement Azure DevOps
• Can define real-time and batch orchestration
• Has used these software: Chef, Jenkins, Terraform, Confluence, GIT, SVN, GitHub, BitBucket, Jira
• Has the title implied, the candidate is a data engineer meaning he is a full-fledged experienced developer very solid in both Python and Spark
• Databricks is also a requirement
• Has used noSQL technologies and Azure Data Lake Store (gen 2)
• Understand asynchronous architecture using ‘queue mechanisms’ such as Event Hub/Kafka
• Learns and masters new technologies and techniques rapidly and continuously
• Able to understand rapidly the patterns and directions taken by the D&A Practice Lead
• Able to zero-in on ambiguous concepts and made them crystal clear rapidly
• Pragmatic but not to the expense of faulty designs/concepts
• Must be comfortable working with a wide range of stakeholders and functional teams
• A drive to learn and master new technologies and techniques

We’re looking for someone with 15 years of ETL/ELT and Data Pipelines experience and 7 years as a solution architect. Minimum of 5 years in Senior Management role. Experience working with data architecture. Experience with ADLS gen 2, S3, Spark, Databricks, Synapse Analytics, Snowflake, PowerBI, AWS stack: Kinesis Data/Stream/Firehose, S3, Lake Formation, RedShift, Tableau.

What will make you successful here?

• A desire to be accountable for owning problems from design to implementation
• A bias for action and pragmatic solutions
• A low ego and humility; an ability to gain trust through effective communication
• A talent for operating effectively at various levels of abstraction
• An aptness to distill complex, technical topics articulately to a non-technical audience
• Adaptability, flexibility, and an ability to navigate ambiguity

Benefits

• The opportunity to join and grow an ever-expanding professional network of high-profile clients and reputable colleagues
• Permanent, full-time job (40 hours a week)
• Flexible schedule
• Hybrid (work from home and/or office)
• Competitive salary
• Attractive group insurance plan
• Retirement savings plan
• Small company culture
• Free use of gym within building
• Casual work environment: jeans-friendly!