About this role
While technology is the heart of our business, a global and diverse culture is the heart of our success. We love our people and we take pride in catering them to a culture built on transparency, diversity, integrity, learning and growth.
If working in an environment that encourages you to innovate and excel, not just in professional but personal life, interests you- you would enjoy your career with Quantiphi! Role: Lead/Associate Lead – QA + MLOps & Generative AI Experience: 10+ years Location: Mumbai/Bangalore (Hybrid)
Key Responsibilities: AI/ML & GenAI Testing Strategy (AWS Ecosystem) Define testing approaches for AI systems built on AWS services such as:
• Amazon SageMaker
• Amazon Bedrock
• AWS Lambda
• Amazon API Gateway
• Amazon Kinesis
• AWS Glue
• Amazon S3
• Amazon CloudWatch
Design validation frameworks covering:
• Model accuracy & performance validation
• Data drift & concept drift detection
• Hallucination detection for LLMs
• Prompt robustness testing
• RAG validation (retrieval accuracy + grounding)
• Bias & fairness validation
• Safety & toxicity testing
MLOps Quality Engineering (AWS-Centric) Validate the end-to-end ML lifecycle including:
• Data ingestion & feature pipelines
• Model training & hyperparameter tuning
• Model versioning & registry
• Deployment validation
• Canary & blue/green release validation
Work with AWS-native services such as:
• SageMaker Pipelines
• SageMaker Model Monitor
• SageMaker Feature Store
• Bedrock model evaluation workflows
• CloudWatch-based observability
Implement CI/CD quality gates for ML pipelines integrated with AWS DevOps tools.
GenAI & Agentic AI Testing Define quality engineering approaches for:
• LLM-based applications using Amazon Bedrock
• Prompt engineering validation
• Multi-agent orchestration testing
• Chatbot & Voice bot conversational testing
• Intent classification validation
• Conversation drift & fallback validation
• API contract validation for LLM integrations
Build reusable evaluation harnesses for:
• BLEU / ROUGE scoring
• Embedding similarity scoring
• Response consistency
• Safety scoring frameworks
Framework & Capability Development
• Design reusable AI testing accelerators
• Create AWS-aligned AI test automation frameworks (Python-first)
• Develop synthetic data generation strategies
• Establish AI quality scorecards
• Build an internal AI QA Center of Excellence
Client Engagement & Leadership
• Lead AI/ML quality strategy workshops
• Perform AI risk & readiness assessments
• Present quality architecture to CXOs
• Drive QA transformation programs
• Mentor QA teams on AWS-based AI testing
• Own delivery for AI testing engagements end-to-end
Must have skills: Testing Expertise
• 8–12+ years in Quality Engineering
• Strong test strategy, automation & governance experience
• Experience leading QA transformation initiatives
• Experience building frameworks from scratch AI/ML & GenAI Expertise
• Deep understanding of ML lifecycle
• Experience testing ML models (NLP preferred)
• Hands-on experience validating LLM applications
• Strong understanding of:
• Prompt engineering
• RAG architecture
• Embeddings
• Bias & explainability AWS AI/ML Expertise
• Hands-on experience with:
• Amazon SageMaker (training, deployment, monitoring)
• Amazon Bedrock (LLM integration & evaluation)
• S3-based data pipelines
• AWS IAM (security validation)
• CloudWatch monitoring
• Lambda & API Gateway integrations
• AWS CI/CD (CodePipeline / CodeBuild preferred)
Understanding of:
• Infrastructure as Code (Terraform / CloudFormation)
• Observability in AI systems
• Cost monitoring for ML workloads
Technical Skills
• Python (mandatory)
• Experience with ML libraries (Scikit-learn, TensorFlow, PyTorch)
• Experience with LLM frameworks (LangChain, etc.)
• API & automation testing frameworks
• Git-based workflows
• Leadership & Communication
• Strong client-facing communication
• Experience leading QA teams
• Ability to create strategy decks & solution proposals
• Strong stakeholder management
If you like wild growth and working with happy, enthusiastic over-achievers, you'll enjoy your career with us!