At a Glance
- Tasks: Build AI-powered solutions and optimize data pipelines for real-world industrial applications.
- Company: Join a forward-thinking organization in the Oil & Gas sector.
- Benefits: Competitive salary, flexible work options, and opportunities for professional growth.
- Other info: Exciting chance to work on high-impact projects and mentor others in a dynamic environment.
- Why this job: Make a real impact by transforming traditional processes with cutting-edge AI technology.
- Qualifications: Senior-level experience with Azure Databricks, SQL, Python, and strong engineering background.
The predicted salary is between 123750 - 151250 $ per year.
We are looking for a Senior Databricks Data Engineer to join our team. This role is ideal for someone who enjoys building end-to-end solutions and wants to work on real-world industrial use cases that come from an Engineering background, and has experience in Oil & Gas.
In this position, you’ll contribute to a variety of impactful client projects, including:
- Building AI-powered anomaly detection systems for operational and industrial data
- Modernizing asset management workflows
- Developing natural language interfaces for querying enterprise and operational data
- Implementing Azure Databricks foundations and pipelines with best practices
- Creating scalable data pipelines and ML workflows in cloud environments
- Enabling real-time and batch data processing for analytics and AI use cases
- Working through complex and challenging data conditions, including incremental processing, late-arriving or changing records, complex business and engineering rules, and reconciliation of results across processing runs
The ideal candidate combines strong data engineering experience with an engineering or applied-science background. Direct experience with industrial time-series data, scientific measurements, telemetry, financial reconciliation, or other datasets where accuracy, traceability, and incremental recalculation are critical.
Key Responsibilities
- Design, build, and optimize pipelines for sensor and related operational data.
- Develop complex transformation and calculation logic based on engineering requirements.
- Implement robust incremental-processing patterns for high-volume and continuously changing datasets.
- Design, build, and deploy end-to-end AI/ML solutions in production environments.
- Develop robust backend systems and APIs to support AI-driven applications.
- Build and maintain data pipelines and feature engineering workflows.
- Implement and operationalize machine learning models (training, deployment, monitoring).
- Work with modern AI tooling (LLMs, agents, orchestration frameworks).
- Collaborate with clients to translate business problems into technical solutions.
- Contribute to architecture decisions and best practices across projects.
- Mentor client team members and contribute to internal capability building.
- Work directly with engineering and operational SMEs to understand physical processes and translate their knowledge into technical requirements.
- Make engineering calculations and data transformations explainable, traceable, testable, and auditable.
- Document data lineage, calculation logic, assumptions, dependencies, and exception-handling rules.
Ideal Background
- Senior-level experience designing and developing production data pipelines with Azure Databricks including strong experience with complex SQL, Python, Spark, or comparable data-processing technologies.
- Complex Excel and CSV integration experience.
- Experience with Databricks dashboards and Genie.
- Strong requirements gathering experience with Field Engineers and Technical SMEs.
- Demonstrated experience with incremental processing, change detection, reconciliation, and pipeline design.
- Experience handling time-series, telemetry, sensor, operational, scientific or industrial data.
- Medallion architecture and modelling.
- Strong analytical and investigative skills, with the patience to work through detailed logic and difficult data-quality problems.
- Degree or professional background in petroleum, reservoir, chemical, mechanical, geological, geophysical, or another relevant engineering or applied-science discipline is strongly preferred.
- Experience in upstream oil and gas, thermal operations, SAGD, well surveillance, production engineering, or subsurface data would be a significant asset.
This role presents an exciting opportunity to work on practical, high-impact AI use cases - not just prototypes, shape how AI is applied to client environments, and change the game on traditional processes and platforms. Come join a growing organization helping clients take a new, lean and value-driven approach to data and engineering!
Senior Databricks Data Engineer in Calgary employer: Data Elephant
Join a leading boutique consulting firm in Calgary that prioritizes employee growth and collaboration. As an Azure Databricks Engineer, you'll work on impactful projects with a talented team, benefiting from a culture that values innovation, client success, and professional development. Enjoy competitive compensation, flexible work arrangements, and the opportunity to enhance your skills in a dynamic environment focused on modern data solutions.
