At a Glance
- Tasks: Manage and optimize GPU clusters, ensuring top performance and reliability.
- Company: Join a leading tech company focused on cutting-edge infrastructure.
- Benefits: Competitive salary, equity options, and opportunities for professional growth.
- Other info: Collaborative environment with a focus on innovation and career advancement.
- Why this job: Be hands-on with advanced technology and make a significant impact in a dynamic team.
- Qualifications: 5+ years in large compute clusters and proficiency in Go or Python.
Own the physical reality of the platform in one of our seven regions. You bring up new GPU racks, validate InfiniBand fabric end-to-end, and keep the cluster running at the SLA. This is a hands-on role; you will see the hardware.
Type: Full-time · IC4-IC6
Stack: Linux, InfiniBand (NDR / XDR), NCCL / RCCL, Kubernetes (host-level), Terraform, Prometheus / Grafana, Go or Python.
The team consists of six engineers across the seven regions. Each region has a primary and a secondary; you will be one of those for your region. The team coordinates daily, deploys weekly, and rotates a global pager. Reports to the head of cluster engineering. Primary on a single region; rotates secondary for one neighboring region.
What you'll do:
- Bring up new B200 / B300 / MI300X racks: cabling, ToR config, NCCL/RCCL all-reduce validation, MFU baseline tests.
- Drive InfiniBand fabric to spec - NDR / XDR depending on the rack - and chase residual bit-error budget down to zero.
- Run capacity planning across seven regions: forecast demand, model power and thermal headroom, work with procurement on lead times.
- Own the regional incident response. P1 incidents page within fifteen minutes; resolution target is four hours.
- Build and maintain the bring-up runbook so the second hire after you can do their first rack solo.
- Carry the global pager about one week per six, alongside runtime and customer engineering.
The bar:
- Five-plus years operating large compute clusters — supercomputing centers, hyperscaler infra, or HPC at a national lab count.
- Deep InfiniBand and Ethernet RoCE experience: subnet manager tuning, fabric debugging, lossless networking.
- Comfort writing Go or Python for tooling. We are not strict about which.
- Calm under load. You will be the named person on a $50M-ARR account when something goes wrong.
Nice to have, not required:
- Experience with NVIDIA Bright / Base Command Manager.
- Bare-metal provisioning systems: Tinkerbell, MAAS, Razor, or in-house equivalents.
Compensation:
Base: $210,000 – $340,000 USD (US Cologix regions) / equivalent in CA. Equity: Meaningful early-stage equity, refreshed on tenure milestones.
Notes:
On-site pay differential at Cologix regions outside SF / NYC / Bay Area is +5-10% to compensate for travel.
Equal opportunity:
We hire on the work. Race, gender, age, nationality, religion, sexual orientation, disability, and veteran status do not factor into our decisions. We sponsor visas for senior roles in the US, UK, and EU — bring it up on the manager call.
Cluster Site Reliability Engineer in Winnipeg employer: iFrame Corporation
Join a dynamic team at Cologix, where we prioritise innovation and collaboration in the field of Site Reliability Engineering. Our work culture fosters continuous learning and growth, providing employees with opportunities to engage with cutting-edge technology while ensuring the reliability of advanced compute clusters. Located in vibrant regions, we offer a supportive environment that values hands-on involvement and encourages impactful contributions to our computing services.
