About this role
Build monitoring infrastructure for production models, covering both traditional ML and GenAI/LLM systems, including automated pipelines, dashboards, and alerting. Define and track model health metrics - for ML: accuracy, precision/recall, AUC, calibration, feature and prediction drift. For GenAI: output quality, hallucination/grounding checks, relevance, latency, token/cost usage, and guardrail adherence. Detect and diagnose issues such as data drift, concept drift, performance decay, and data-quality breaks, then triage and escalate to the appropriate model owners. Establish thresholds and alerting that balance early detection with alert fatigue, and document expected behavior and remediation runbooks. Partner with data scientists and ML engineers to integrate monitoring into the model deployment lifecycle (CI/CD, MLOps/LLMOps). Support model governance and compliance by producing monitoring evidence, audit-ready reporting, and documentation aligned with enterprise model risk management standards. Analyze production outcomes against business KPIs to surface opportunities for model improvement or retraining. Communicate findings clearly to both technical and non-technical stakeholders through reporting and periodic model health reviews. Experience building dashboards and reports (e.g., Streamlit, Tableau, or similar). Experience with model monitoring / observability tooling Experience with A/B testing or experiment design to test impact of solutions Bachelor's Degree plus a minimum 3 years, typically 4 or more years of experience, or equivalent, is required. Mathematics, Economics, Statistics or other quantitative field are preferred fields of study. Advanced knowledge of data sources, tools, statistical principles and methodologies, and techniques. Advanced proficiency in Excel (VBA, macros, scripts, formulas, data visualization, etc.), PowerPoint, and statistical software packages (SAS, Emblem). Must have good planning, analytical, decision-making and communication skills. Solid understanding of business to improve business outcomes.