At a Glance
- Tasks: Design and develop data pipelines, optimize data flow, and automate tasks using cutting-edge cloud technologies.
- Company: Join a dynamic team focused on transforming data management for the Government of Ontario.
- Benefits: Competitive salary, health benefits, and opportunities for professional growth.
- Other info: Onsite role with a collaborative environment and strong focus on career advancement.
- Why this job: Make a real impact by working on innovative projects that support Ontario's economy.
- Qualifications: 8+ years in data engineering with expertise in cloud platforms and data pipeline development.
The predicted salary is between 121500 - 148500 $ per year.
To respond to the priorities of the Government of Ontario through the passage of the Bill 5, Protect Ontario by Unleashing Our Economy Act 2025, the resource may be partially assigned to Bill 5 related projects or initiatives on an as needed basis. This position is listed as "Onsite" unless otherwise instructed by the Client; the resource is expected to work 7.25 hours per calendar day between the standard working hours of 8:00-5:00 PM (excluding lunch breaks) every Monday to Friday inclusive at the identified OPS office location.
Responsibilities:
- Designing and developing data pipelines from source to end user.
- Optimizing data pipelines.
General Skills:
- Experience with Cloud data platforms, data management and data exchange tools and technologies.
- Experience with commercial and open-source data and database development and management, specializing in data storage setting and managing cloud Data As a Service (DaaS), Database as a Service (DBaaS), Data Warehouses as a Service (DWaaS), and other storage platforms (both in the cloud and on-premise).
- Experience with data pipeline and workflow development, orchestration, deployment, and automation, specializing in programming and pipelines to create and manage the data flow and movement of data.
- Familiarity with different programming languages and ability to integrate with various platforms to create data pipelines, automate tasks, and write scripts.
- Experience in DataOps principles, best practices, and implementation, as well as Agile project development and deployment.
- Experience with Continuous Integration/Continuous Development/Deployment (CI/CD) and data provisioning automation.
- Experience with digital product, data analysis, data exchange, data provisioning, and data security.
- Extensive experience in designing, developing, and implementing data conversion and migration of Very Large Data (VLD) of Online Analytical Processing (OLAP) and Online Transaction Processing (OLTP) environments to Cloud Software-as-a-Service (SaaS), Platform-as-a-Service (PaaS), and Infrastructure-as-a-Service (IaaS) environments.
- Experience in design, development, and implementation of fact/dimension model, data mapping, data warehouse, data lake, and data lakehouse for enterprise.
- Experience managing Cloud Data services for project delivery, including storage, repositories, Data Lake, Data Lakehouse, key vault, virtual machine, disk, etc.
- Experience with structured, semi-structured, unstructured data collection, ingestion, provisioning, and exchange technological development of enterprise data warehouse and data lake and data lakehouse solutions and operational support.
- Experience with DataOps performance monitoring and tuning.
- Excellent analytical, problem-solving, and decision-making skills; verbal and written communication skills; presentation skills; interpersonal and negotiation skills.
- A team player with a track record for meeting deadlines.
Experience and Skill Set Requirements:
- Mandatory Skills and Experience (8+ years experience):
- Design, build, automate, and optimize complex data ETL/ELT processes using Databricks, Delta Live Tables (DLT), and equivalent.
- Development of unified data platforms using Databricks and Unity catalog or its equivalent.
- Relational databases (Oracle, MySQL, SQL Server), data modelling (relational & dimensional), advanced SQL, query optimization, data replication, administration.
- Advanced SQL skills (PL/SQL, TSQL).
- Big Data processing frameworks (PySpark).
- Proficient in creating Azure Data Factory pipelines for copy activity and custom data pipelines in Databricks with CI/CD (Azure DevOps Git Integration).
- Data migration and data integration across different platforms (Oracle to Azure Data Lake, Oracle to Databricks, Oracle to Microsoft Fabric).
- Desirable Skills and Experience:
- Data lake design and development with focus on extracting, transforming data coming from various data sources and loading into Lakehouse in medallion architecture.
- Advanced skills and hands-on experience with Oracle databases, Azure SQL Server, and Azure Data Factory.
- Cloud technologies (Azure, Google, AWS).
Senior DataOps Cloud Data Engineer in Toronto employer: The Code Crackers
As a Senior Scrum Master, you will thrive in a dynamic and collaborative work environment that prioritizes employee growth and development. Our company offers comprehensive training programs, mentorship opportunities, and a strong commitment to Agile methodologies, ensuring that you can lead teams effectively while fostering a culture of continuous improvement. Located in a vibrant area, we provide a supportive atmosphere where innovation is encouraged, making it an excellent place for professionals seeking meaningful and rewarding employment.
