Senior AI Engineer for Observability

Designs, builds, and maintains systems that track, evaluate, and debug non-deterministic AI models and agent workflows. Trace LLM calls, monitor token usage, detect drift, set up automated evaluations, and govern autonomous agent behaviors in production. Designs and develops machine learning (ML) and deep learning systems. Runs machine learning tests and experiments. Trains and retrain systems to prevent drift and optimize results.

Qualifications:

• Bachelor's Degree in Computer Science, Engineering or other relevant domain
• 7-10 years experience in Site Reliability Engineering
• Experience in Artificial Intelligence Observability Engineering, facilitating the ability to Automate Detect, Investigate, and Remediate
• Exposure to Platform Engineering, building AI capabilities
• Highly proficient in automation, monitoring, observability, buildouts, AI engineering, understand model architectures, evaluation metrics, token usage, and how to spot concept or data drift
• Experience using Python for custom evaluations and SQL or LogQL/PromQL for querying telemetry, logs, and metrics data.
• Provision, configure, and standardize observability pipelines, logging sidecars, and centralized dashboards using Prometheus, Grafana, Datadog to track SLIs/SLOs, handle alert fatigue, and run root cause analysis.
• Exposure to heavily regulated industry organizations

Agent & LLM Tracing

• Implement OpenTelemetry-based frameworks to capture step-by-step agent trajectories, spans, prompts, tool calls, and intermediate outputs.
• Design automated evaluation pipelines for relevance, groundedness, hallucination detection, and prompt adherence using LLM-as-a-judge or custom code.
• Track model drift, data quality degradation, latency spikes, and error rates across production pipelines.
• Build telemetry and sampling strategies to control high data and inference costs, managing log retention and cardinality.
• Define operational boundaries for autonomous AI agents, setting rules for when they can self-remediate.
• Reduce alert fatigue by replacing static thresholds with dynamic, intelligent anomaly detection and building runbooks for AI failures.

Salary: $80,000.00 - $175,000.00 Pay Type: Salaried.

BMO Financial Group’s total compensation package will vary based on the pay type of the position and may include performance-based incentives, discretionary bonuses, as well as other perks and rewards.

BMO also offers health insurance, tuition reimbursement, accident and life insurance, and retirement savings plans.

BMO is committed to an inclusive, equitable and accessible workplace. By learning from each other’s differences, we gain strength through our people and our perspectives. Accommodations are available on request for candidates taking part in all aspects of the selection process.