At a Glance
- Tasks: Design and maintain scalable data pipelines using Python, PySpark, and cloud technologies.
- Company: Join a leading tech company focused on innovative data solutions.
- Benefits: Competitive salary, flexible work options, and opportunities for professional growth.
- Other info: Dynamic work environment with a focus on innovation and career advancement.
- Why this job: Make an impact by working with cutting-edge data technologies and collaborating with talented teams.
- Qualifications: Bachelor’s or Master’s in Computer Science or related field; strong Python and SQL skills.
The predicted salary is between 121500 - 148500 $ per year.
The Data Engineer will design, develop, and maintain scalable data pipelines and workflows using Python, Py Spark, and cloud technologies.
This role involves managing data storage solutions, orchestrating ETL processes, and deploying containerized applications to support enterprise data initiatives.
Key Responsibilities
- Develop, construct, test, and maintain scalable data pipelines and ETL/ELT workflows using Python, Py Spark, and SQL.
- Manage and optimize data storage solutions in Snowflake and Data Lakes.
- Integrate data from multiple sources while ensuring data quality, integrity, and security.
- Design and implement automated workflows using Azure Data Factory or similar orchestration tools.
- Deploy and manage containerized applications and services on Open Shift.
- Collaborate with Data Scientists, Analysts, and stakeholders to understand data requirements.
- Monitor and troubleshoot data pipelines, resolving performance bottlenecks.
- Maintain documentation of data architecture, workflows, and best practices.
- Ensure compliance with data governance and security policies.
- Required Qualifications
- Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related field.
- Minimum 10 years of overall experience in data engineering or similar roles.
- Strong proficiency in Python and Py Spark.
- Expertise in SQL and experience with Snowflake Data Warehouse.
- Hands-on experience with data orchestration tools such as Azure Data Factory.
- Knowledge of Data Lakes architecture and management.
- Experience deploying and managing containerized applications on Open Shift or Kubernetes.
- Familiarity with cloud platforms (Azure, AWS, GCP).
- Understanding of data security, governance, and compliance best practices.
- Strong problem-solving skills and attention to detail.
- Excellent communication and teamwork abilities.
- Preferred Qualifications
- Experience with big data tools (e. g., Kafka, Hadoop).
- Knowledge of CI/CD pipelines and Dev Ops practices.
- Familiarity with additional cloud data services and tools.
- #J-18808-Ljbffr
Data Engineer Python/PySpark/OpenShift in Mississauga employer: Compunnel Inc.
Join a forward-thinking company in Toronto that prioritizes innovation and employee development. As a Digital Banking Engineering Manager, you'll thrive in a collaborative work culture that values continuous improvement and offers ample opportunities for professional growth. With a focus on modern banking solutions, this role not only allows you to lead a talented team but also empowers you to make a meaningful impact in the digital banking landscape.
