Core responsibilities
- Version models, prompts and evaluation sets
- Monitor errors, cost, throughput and latency
- Roll out gradually and define incident procedures
Technical skills
- Serving and caching
- Regression testing and telemetry
- SLOs, secrets and trace redaction
Behavioural skills
- Coordinate trade-offs with product teams
- Write actionable incident procedures
Typical deliverable
A canary rollout with token budgets, alerts and rollback to a stable release.
Possible assessment methods
Use an incident trace to distinguish saturation from a quality regression and propose a mitigation plan.
