Build AI observability systems at BMO in a hybrid role, using Python, telemetry, and monitoring to evaluate production models and agents. This senior engineering role focuses on artificial intelligence observability and platform capabilities. You will trace LLM calls, monitor token usage, detect drift, and govern autonomous agent behaviors. The work also includes machine learning systems, experiments, algorithms, and specialized technical support. Key Responsibilities:
Design systems tracking, evaluating, and debugging non-deterministic AI models Implement OpenTelemetry tracing for agent trajectories, prompts, and tool calls Build automated evaluations for relevance, groundedness, and hallucinations Monitor drift, latency, errors, token usage, and data quality Define guardrails and self-remediation boundaries for autonomous agents Requirements:
Bachelor's degree in Computer Science, Engineering, or relevant domain Seven to ten years of Site Reliability Engineering experience Artificial intelligence observability engineering and automation experience Python plus SQL, LogQL, or PromQL experience Prometheus, Grafana, or Datadog observability pipeline experience Apply senior AI engineering expertise across observability, evaluation, tracing, monitoring, and agent governance at BMO.
#J-18808-Ljbffr
