At a Glance
- Tasks: Build deterministic evaluation gates and fact-reconciliation layers for AI systems.
- Company: Join 15Rock, a cutting-edge financial intelligence firm in downtown Toronto.
- Benefits: Enjoy a hybrid work environment with competitive pay and growth opportunities.
- Other info: Work with a small, senior team and have direct access to leadership.
- Why this job: Make a real impact by ensuring AI accuracy in financial research.
- Qualifications: Experience shipping LLM systems and strong Python skills required.
The predicted salary is between 110250 - 134750 $ per year.
Applied AI Engineer — Evaluation & Reliability 15Rock is a financial intelligence firm.
Our research goes in front of institutional investors and gets read line by line — by people looking for the one number that doesn't hold up.
Every figure has to reconcile across every table and sentence; every claim has to trace to a live source.
We build that research with a multi-agent pipeline on large language models.
LLMs write well and count badly: left alone, they produce clean prose that contradicts the table three paragraphs up.
Our answer isn't "prompt harder." It's to route every number through deterministic computation and gate every report behind checks that pass or fail — never a model grading its own work.
You'd build that layer.
What you'll build Deterministic evaluation gates that block a report when a defect is present — measured against thresholds set before the run, not a gut feel.
A fact-reconciliation layer that verifies a figure agrees with itself everywhere it appears.
Citation-integrity checks: every reference resolves, every quote is real, every source is live.
Golden-set and regression harnesses that catch quality drift before it ships.
The path from research to production: hardening a validated approach into the pipeline, behind CI, with the traces to prove it works.
What we're looking for You've shipped LLM systems to production — not demos — where being wrong had a cost.
You've built the evaluations that kept them honest: you think in precision and recall, and you define "good" before you measure it.
Strong Python; you can own backend services and data pipelines yourself.
You reason about the numbers, not just the tokens — enough quantitative or financial literacy to know when a figure is wrong, or the appetite to learn it fast.
Self-directing.
You'd rather write a one-page memo than wait for a ticket.
Bonus: model-risk or evaluation-infrastructure work in a regulated setting; Lang Graph, Duck DB, CI you owned; a finance or quantitative background.
This role is not Prompt engineering, a pure research seat, or a place where "the model said so" is an answer.
How we work A small, senior team with direct access to the founder and research lead.
Async by default — we meet on blockers, not to read status aloud.
We run on hypotheses and measurable outcomes.
Details Downtown Toronto, on-site / hybrid.
Must be authorized to work in Canada.
Early-stage financial intelligence firm, recognized by the Globe and Mail Changemaker Award and selected for Google for Startups.
15Rock is an equal-opportunity employer.
#J-18808-Ljbffr
Applied AI Engineer in Toronto employer: 15Rock, Ltd.
At 15Rock, we pride ourselves on being an exceptional employer that fosters a culture of innovation and precision in the financial intelligence sector. Our small, senior team offers direct access to leadership, promoting a collaborative environment where your contributions are valued and impactful. Located in the vibrant heart of downtown Toronto, we provide unique opportunities for professional growth and development, ensuring that every team member can thrive while working on cutting-edge AI solutions.
