At a Glance
- Tasks: Lead data science for AI evaluations, developing methods and frameworks to measure quality and performance.
- Company: Join RBC's innovative AI Group focused on building trusted AI capabilities.
- Benefits: Enjoy a comprehensive rewards program, coaching opportunities, and a collaborative work environment.
- Other info: Work in a dynamic team at our Montréal office with excellent career growth potential.
- Why this job: Make a real impact in AI by shaping evaluation standards and mentoring future data scientists.
- Qualifications: 8+ years in data science with strong skills in AI evaluation and technical leadership.
The predicted salary is between 124200 - 151800 $ per year.
RBC's AI Group is building trusted AI capabilities for the enterprise. The Staff Data Scientist, AI Evaluations will lead the data science work for measuring model and agent quality, safety, risk, and performance. This includes developing evaluation datasets, scoring methods, human evaluation protocols, benchmarks, and measurement frameworks, while providing technical guidance and mentorship to other data scientists.
What you’ll do
- Design model and agent evaluation methods, including datasets, rubrics, LLM-as-judge methods, deterministic scorers, human evaluation, and measurement frameworks.
- Define standards that translate model risk, responsible AI, product quality, safety, and business expectations into measurable criteria and repeatable methods.
- Own evaluation dataset and scorecard lifecycles, including sourcing, curation, validation, quality checks, versioning, lineage, and ongoing improvement.
- Develop scalable evaluation methods for generative AI and agentic systems at task, workflow, trajectory, and runtime levels.
- Partner with research, engineering, product, risk, governance, and business teams to embed evaluations into development, release, certification, monitoring, and recertification.
- Establish human evaluation and review protocols, and improve scorer accuracy, calibration, robustness, failure-mode coverage, and explainability.
- Provide technical leadership, communicate findings, and mentor junior data scientists.
What you’ll need
- At least 8 years of experience in data science, applied machine learning, AI evaluation, ML quality, or a related technical field, including technical leadership and mentorship.
- Experience designing evaluation frameworks for machine learning, generative AI, or agentic AI systems.
- Practical experience with methods such as LLM-as-judge, deterministic scoring, human evaluation, hallucination and factuality assessment, safety evaluation, or benchmarking.
- Strong foundation in data science, statistics, machine learning, experimentation, data curation, Python, SQL, and modern AI/ML practices.
- Ability to translate governance, model risk, responsible AI, and business requirements into measurable controls and decision-ready evidence.
- Strong communication and stakeholder management skills.
- Experience evaluating agentic systems, financial services or regulated AI, and evaluation or observability tooling is an asset.
- A relevant master's or Ph.D. is also an asset.
What RBC offers
- A comprehensive total rewards program, development and coaching opportunities, challenging work, and a collaborative team.
Position details
- Location: 6666 Rue St-Urbain, Floor 3, Montréal, Quebec
- Hours: 37.5 per week
- Employment type: Full-time, regular, salaried
- Application deadline: Applications accepted until 11:59 PM on the day before October 31, 2026.
Staff Data Scientist, AI Evaluations Platform in Montréal employer: Skillhub Solutions Inc.
Scotiabank is an exceptional employer that fosters a dynamic and inclusive work culture, where innovation thrives and employees are empowered to grow. As a Technical Lead in the GWMS Digital Client Services team, you will benefit from comprehensive professional development opportunities, competitive compensation, and a collaborative environment that values your contributions. Located in a vibrant city, Scotiabank offers unique advantages such as access to cutting-edge technology and a commitment to employee well-being, making it a rewarding place to advance your career.
