About this role
Salary: £60,000 - 110,000 per year
Requirements: Experience with Unix toolsStrong analytical skills, attention to detail, and effective communication abilitiesExperience in a fast paced, dynamic organizationExperience prioritizing and handling multiple assignments at any given time while maintaining commitment to deadlinesMasters Degree in Applied Linguistics, Computational Linguistics, Natural Language Processing (NLP), or related fieldExperience with Large Language Models, NLP, or Machine LearningExperience with Python libraries for data analysis such as pandas and scikit-learnFamiliarity with AI coding assistantsPhD in Applied Linguistics, Computational Linguistics, Natural Language Processing (NLP), or related technical fieldExperience with SQL and GitExperience building RAG or agentic systemsExperience conducting quantitative analysisExperience building data pipelinesExperience with AWS services (Bedrock, S3, EC2, etc.)Knowledge of user experience concepts and methodsFamiliarity with online retail (e-commerce) Responsibilities: Develop LLM-as-a-judge systems to support human-in-the-loop evaluationsAutomate operations and perform data analysis using scripting languages such as PythonAuthor, optimize, and manage system prompts for customer-facing LLM systemsIntegrate API calls into Retrieval Augmented Generation (RAG) systemsEvaluate model performance and annotation quality to produce reports for stakeholdersProduce, process, and manipulate different types of language dataContribute to defining platform requirements for internal tooling by developing prototypesRaise the quality bar on editorial workflows and SOPs through standardization, documentation, and periodic audits and investigationsSupport processes and mechanisms to onboard and upskill Editors and AI Tutors on an ongoing basisDesign, implement, and refine control mechanisms, metrics, and methodologies to ensure editorial and annotation qualityCollaborate with editors, applied scientists, engineers, and product managers to deliver an optimal customer experience by defining metrics, guidelines, and workflowsDeliver across parallel workstreams, balancing timelines, impact, and stakeholder requirements Technologies: AIAPIAWSEC2GitSupportLLMMachine LearningPythonRAGSQLUnixpandasCloudMobileSecurityUX UI DesignWeb More:
We are the Conversational Shopping team, a high-performing org behind Alexa for Shopping, and we are focused on making it easy for customers to find and discover the best products for their unique needs through recommendations, comparisons, product Q&A, and more. This role sits within the AI Shopping org and the CMX-Lang-Tech team, a technical sub-team of the AI Shopping Content team, responsible for AI response quality through evaluation and prompting of our AI models. We value a collaborative, cross-functional environment and work closely with global product, design, science, engineering, editorial, and research partners to deliver quality, speed, and consistency. Amazon is an equal opportunities employer with an inclusive culture and a commitment to privacy and workplace accommodations.
last updated 36 week of 2026