About this role
About the Role
Testing AI systems is a fundamentally different problem than testing traditional software. Outputs are non-deterministic. "Correct" is often a spectrum. And the failure modes—hallucinations, drift, prompt injection—don't show up in unit tests. We need an engineer who understands this and can build the testing strategies, evaluation frameworks, and quality infrastructure to keep our agents reliable in production.
As an AI Quality Engineer, you'll design how we test intelligent agents, agentic workflows, and Foundation Layer capabilities. This is not a manual QA role—you'll write code, build evaluation pipelines, and create automated testing frameworks that run in CI/CD. You'll define what "quality" means for AI systems at AGS and build the systems to measure it.
You'll work across every solution the team builds, which means you'll have broad visibility into the architecture and deep understanding of how our agents behave in the real world. If you're an engineer who cares about quality and wants to solve testing problems that most teams haven't figured out yet, this is the role.
Responsibilities
Testing Strategy & Design
Define testing strategies for AI agents, conversational interfaces, and agentic workflowsDesign behavioral test suites for non-deterministic outputs—where "correct" isn't binaryBuild evaluation frameworks that measure groundedness, factuality, relevance, and task completionIdentify failure modes specific to AI systems: hallucinations, prompt injection, context window limitations, driftDevelop testing approaches for each architecture pattern: RAG, function calling, human-in-the-loop, autonomous workflows
Test Automation & Infrastructure
Build automated evaluation pipelines that run as part of CI/CDCreate test harnesses for LLM-based systems—mocking, fixtures, and reproducible test scenariosDevelop regression suites that detect quality degradation when prompts, models, or data changeBuild monitoring and alerting for production agent quality (accuracy, latency, error rates)Maintain test infrastructure: test data management, environment setup, reporting dashboards
Evaluation & Metrics
Define quality metrics for each solution—what to measure and what thresholds matterBuild and maintain evaluation datasets (ground truth, reference outputs, edge case collections)Conduct systematic prompt evaluation when prompts or models changeTrack quality trends over time and identify when re-evaluation is neededReport quality metrics to the team and stakeholders in clear, actionable terms
Collaboration & Quality Culture
Partner with AI Solutions Engineers to define testability requirements during designWork with AI Solutions Analysts to translate acceptance criteria into test scenariosReview solution designs from a quality and testability perspectiveAdvocate for quality practices across the team—testing isn't an afterthought, it's part of deliveryContribute to incident response by diagnosing quality failures and building regression tests Qualifications
Required
3–7 years of software engineering or quality engineering experienceStrong programming skills in Python and/or TypeScript—you write test code, not just test casesExperience designing and building automated test frameworksUnderstanding of AI/ML systems—you know why testing LLM outputs is different from testing deterministic codeExperience with CI/CD pipelines and integrating automated tests into build processesAbility to reason about non-deterministic systems and design meaningful quality metricsStrong analytical skills—you can look at agent outputs and determine whether they're good enough
Preferred
Experience testing AI/ML applications, conversational interfaces, or chatbotsBackground in LLM evaluation: prompt testing, groundedness scoring, factuality checkingFamiliarity with evaluation frameworks (DeepEval, Ragas, custom evaluation pipelines)Experience with Microsoft Power Platform (Power Automate, Copilot Studio) testingBackground in Azure services and cloud-based test infrastructureExperience with load testing and performance testing for API-based systemsFamiliarity with staffing, HR tech, or workforce management domains
Technology Stack
Languages: Python, TypeScriptPlatforms: Azure (Container Apps, Functions, AI Services), Microsoft 365Testing: pytest, evaluation frameworks (DeepEval, Ragas, custom), load testing toolsAI/ML: LLM evaluation, prompt testing, RAG evaluation, behavioral testingData: REST APIs, Dataverse, SQLTools: Git, GitHub, CI/CD pipelines, Docker, monitoring/alerting (Application Insights)We don't expect expertise in everything. AI quality engineering is a new discipline—we expect strong engineering fundamentals and the ability to figure out new problems.
What We're NOT Looking For
Manual testers who write test cases in spreadsheetsQA professionals who treat testing as a gate at the end of development rather than a practice woven into itPeople who expect deterministic pass/fail for every test—AI quality requires nuanceEngineers who test to the spec but don't think about how real users will break things
What Makes You Stand Out
You've tested a system where "correct" was hard to define—and found a way to measure it anywayYou write test code that's as clean and maintainable as production codeYou think about edge cases that nobody else considersYou can explain why a particular quality metric matters and what threshold makes senseYou've built test automation that actually caught regressions before they hit productionYou're comfortable saying "this isn't good enough" and backing it up with data
What We're Building
The AI Engineering team delivers intelligent solutions for AGS's global clients:
Intelligent Agents — Conversational AI that helps hiring managers, recruiters, and internal teams get work done fasterAgentic Workflows — Automated processes where AI executes tasks with human oversightFoundation Capabilities — Reusable AI services that power multiple solutionsYou'll make sure these systems work reliably—not just at launch, but as models change, data evolves, and usage scales.
Career Growth
AI quality engineering is an emerging discipline with no ceiling. Growth paths include:
Depth — Become the team's authority on AI evaluation and testing methodology, influencing quality standards across the organizationBreadth — Move into a Senior or Lead AI Solutions Engineer role, bringing your quality mindset to architecture and deliverySpecialization — Build expertise in areas like LLM security testing, AI safety, or evaluation research As a workplace, we focus on relationships – with each other, our clients and our candidates - in fact serving others is one of our core values. We support open communication and recognize that giving constructive criticism can be even harder than receiving it. We appreciate the fearless and the passionate, who force us to be better. Everything we do sits on a pillar of diversity - diverse perspectives, backgrounds and ideas drive innovation and make us successful.
See what it’s like to work at AGS by searching #LifeAtAGS on any social network.