About this role
Trilon is building a supercharged, technology-enabled future for our people and partners. The QA Engineer plays a critical role in that mission by ensuring every AI-powered tool shipped by their product pod meets a high bar for quality, reliability, and real-world usability. This is an embedded role within a product pod, acting as the end-to-end owner of quality across everything the team delivers. You are responsible for validating both traditional software behavior and the more complex, non-deterministic outputs of AI systems, ensuring that every solution performs as expected in real-world engineering workflows. You work closely with the Lead Engineer and Applied AI Engineer to define acceptance criteria, design test strategies, and build automated test frameworks that catch issues early and prevent regressions over time. You are accountable for production readiness and serve as the final quality gate before tools are released to operating company engineers. This role requires strong test automation skills, hands-on experience testing AI-enabled applications, and the judgment to define what “good” looks like in an AEC context. You are comfortable writing production-quality test code, evaluating AI output quality, and clearly communicating risks and tradeoffs to both technical and non-technical stakeholders. Quality Ownership and Test Strategy Own the full quality lifecycle for your product pod, from test planning through production sign-off Define acceptance criteria in partnership with engineers and product managers Design comprehensive test strategies that cover functionality, performance, and reliability Establish and maintain quality standards for all deliverables Test Automation and Frameworks Design, build, and maintain automated test suites for APIs, services, and user interfaces Develop regression tests to detect issues as systems evolve Integrate automated testing into CI/CD pipelines to ensure continuous validation Write clean, maintainable test code that scales with the product AI Output Evaluation Design and implement testing approaches for AI-generated outputs, including prompt evaluation and RAG validation Define evaluation criteria and benchmarks for accuracy, consistency, and usefulness Monitor for quality drift as models, prompts, or data sources change Partner with engineers to improve output reliability and performance Defect Detection and Resolution Identify, document, and track bugs and quality issues across the stack Work closely with engineers to troubleshoot and resolve defects early in the development cycle Prioritize issues based on impact and risk to end users Release Readiness and Quality Gatekeeping Validate that all features meet acceptance criteria and quality standards before release Assess production readiness, including performance, reliability, and edge cases Serve as the final sign-off authority for releases within the pod Collaboration and Continuous Improvement Partner with Lead Engineers, Applied AI Engineers, and Product Managers to improve development and testing practices Engage with operating company users to understand real-world expectations and feedback Continuously improve testing methodologies, tools, and frameworks