Functional Leader - Quality Assurance

Quality Assurance (QA) Engineer – Conversational AI & Agentic AI Job Overview We are looking for a Quality Assurance (QA) Engineer with 3-7 years of experience in testing, QA, and reporting, specifically within the field of Conversational AI and Agentic AI systems. The ideal candidate will have a strong understanding of standard testing practices and be passionate about ensuring the highest quality standards in AI-driven applications. This role involves validating conversational experiences, AI-powered workflows, autonomous agent behaviors, integrations with external tools and APIs, and ensuring the reliability, safety, and performance of AI systems in production environments. Key Responsibilities • Develop and execute test cases, scripts, and procedures for Conversational AI and Agentic AI applications. • Collaborate with product managers, developers, data scientists, prompt engineers, and other stakeholders to ensure quality standards are met throughout the development lifecycle. • Perform functional, regression, integration, performance, and end-to-end testing across conversational AI platforms and agent-based AI systems. • Validate AI agent decision-making, task execution, multi-step workflows, reasoning paths, and interactions with external tools, APIs, and knowledge sources. • Test LLM-powered applications for accuracy, relevance, consistency, hallucinations, guardrail compliance, and response quality. • Evaluate agent orchestration capabilities, including multi-agent collaboration, task decomposition, context management, memory retention, and tool utilization. • Identify, document, and track defects using JIRA and provide detailed reports on testing results. • Analyze requirements and design comprehensive test strategies covering conversational flows, autonomous agent behaviors, integrations, and edge-case scenarios. • Conduct prompt, response, and workflow validation to ensure optimal performance and user experience. • Perform security, privacy, bias, and responsible AI testing to validate compliance with organizational and regulatory standards. • Generate test reports, dashboards, and quality metrics for stakeholders, ensuring clear and actionable insights. • Contribute to the continuous improvement of AI testing processes, frameworks, methodologies, and automation capabilities. Required Qualifications • 3-7 years of experience in software testing and quality assurance, with exposure to Conversational AI, Generative AI, Agentic AI, or similar AI technologies. • Strong knowledge of QA methodologies, SDLC, STLC, defect management, and validation techniques. • Experience testing conversational AI platforms, chatbots, voicebots, virtual assistants, or AI-driven applications. • Understanding of Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), prompt engineering concepts, and AI agent architectures. • Experience with JIRA for defect tracking and project management. • Strong proficiency in Excel for data analysis, reporting, and test metrics. • Familiarity with API testing tools such as Postman, Swagger, or equivalent. • Experience in functional, integration, performance, security, and user acceptance testing. • Understanding of AI evaluation metrics, model behavior analysis, and conversational testing frameworks. • Ability to work effectively in cross-functional teams and adapt to fast-paced environments. • Strong analytical, troubleshooting, and problem-solving skills. Desired Qualifications • Experience with Agentic AI frameworks such as LangGraph, CrewAI, AutoGen, Semantic Kernel, LangChain, or similar platforms. • Experience with automated testing tools and scripting languages (Python, JavaScript, etc.). • Exposure to NLP, NLU, LLM evaluation, prompt testing, and AI model validation. • Knowledge of cloud AI platforms such as Google CCAI/Dialogflow, Azure AI, AWS Bedrock, Amazon Lex, or OpenAI-based solutions. • Familiarity with version control systems (Git, GitHub, Azure DevOps). • Experience developing AI test automation frameworks and evaluation pipelines. • Understanding of Responsible AI principles, AI governance, model monitoring, and risk assessment. Personal Attributes • Attention to detail and a commitment to delivering high-quality work. • Strong communication and stakeholder management skills, both written and verbal. • Self-motivated with a passion for AI innovation, quality engineering, and continuous improvement. • Ability to manage multiple priorities in a dynamic, fast-paced environment. • Curiosity to explore emerging AI technologies and evaluate complex AI agent behaviors. • Strong critical thinking skills with the ability to identify risks, edge cases, and quality gaps in AI systems. Preferred Experience Areas • Conversational AI Testing (Chatbots, Voicebots, Virtual Assistants) • Generative AI & LLM Testing • Agentic AI Testing • RAG Validation • Prompt Testing and Optimization • AI Safety & Responsible AI Testing • API and Integration Testing • Test Automation • Cloud-based AI Solutions Quality Assurance (QA) Engineer – Conversational AI & Agentic AI Job Overview We are looking for a Quality Assurance (QA) Engineer with 3-7 years of experience in testing, QA, and reporting, specifically within the field of Conversational AI and Agentic AI systems. The ideal candidate will have a strong understanding of standard testing practices and be passionate about ensuring the highest quality standards in AI-driven applications. This role involves validating conversational experiences, AI-powered workflows, autonomous agent behaviors, integrations with external tools and APIs, and ensuring the reliability, safety, and performance of AI systems in production environments. Key Responsibilities • Develop and execute test cases, scripts, and procedures for Conversational AI and Agentic AI applications. • Collaborate with product managers, developers, data scientists, prompt engineers, and other stakeholders to ensure quality standards are met throughout the development lifecycle. • Perform functional, regression, integration, performance, and end-to-end testing across conversational AI platforms and agent-based AI systems. • Validate AI agent decision-making, task execution, multi-step workflows, reasoning paths, and interactions with external tools, APIs, and knowledge sources. • Test LLM-powered applications for accuracy, relevance, consistency, hallucinations, guardrail compliance, and response quality. • Evaluate agent orchestration capabilities, including multi-agent collaboration, task decomposition, context management, memory retention, and tool utilization. • Identify, document, and track defects using JIRA and provide detailed reports on testing results. • Analyze requirements and design comprehensive test strategies covering conversational flows, autonomous agent behaviors, integrations, and edge-case scenarios. • Conduct prompt, response, and workflow validation to ensure optimal performance and user experience. • Perform security, privacy, bias, and responsible AI testing to validate compliance with organizational and regulatory standards. • Generate test reports, dashboards, and quality metrics for stakeholders, ensuring clear and actionable insights. • Contribute to the continuous improvement of AI testing processes, frameworks, methodologies, and automation capabilities. Required Qualifications • 3-7 years of experience in software testing and quality assurance, with exposure to Conversational AI, Generative AI, Agentic AI, or similar AI technologies. • Strong knowledge of QA methodologies, SDLC, STLC, defect management, and validation techniques. • Experience testing conversational AI platforms, chatbots, voicebots, virtual assistants, or AI-driven applications. • Understanding of Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), prompt engineering concepts, and AI agent architectures. • Experience with JIRA for defect tracking and project management. • Strong proficiency in Excel for data analysis, reporting, and test metrics. • Familiarity with API testing tools such as Postman, Swagger, or equivalent. • Experience in functional, integration, performance, security, and user acceptance testing. • Understanding of AI evaluation metrics, model behavior analysis, and conversational testing frameworks. • Ability to work effectively in cross-functional teams and adapt to fast-paced environments. • Strong analytical, troubleshooting, and problem-solving skills. Desired Qualifications • Experience with Agentic AI frameworks such as LangGraph, CrewAI, AutoGen, Semantic Kernel, LangChain, or similar platforms. • Experience with automated testing tools and scripting languages (Python, JavaScript, etc.). • Exposure to NLP, NLU, LLM evaluation, prompt testing, and AI model validation. • Knowledge of cloud AI platforms such as Google CCAI/Dialogflow, Azure AI, AWS Bedrock, Amazon Lex, or OpenAI-based solutions. • Familiarity with version control systems (Git, GitHub, Azure DevOps). • Experience developing AI test automation frameworks and evaluation pipelines. • Understanding of Responsible AI principles, AI governance, model monitoring, and risk assessment. Personal Attributes • Attention to detail and a commitment to delivering high-quality work. • Strong communication and stakeholder management skills, both written and verbal. • Self-motivated with a passion for AI innovation, quality engineering, and continuous improvement. • Ability to manage multiple priorities in a dynamic, fast-paced environment. • Curiosity to explore emerging AI technologies and evaluate complex AI agent behaviors. • Strong critical thinking skills with the ability to identify risks, edge cases, and quality gaps in AI systems. Preferred Experience Areas • Conversational AI Testing (Chatbots, Voicebots, Virtual Assistants) • Generative AI & LLM Testing • Agentic AI Testing • RAG Validation • Prompt Testing and Optimization • AI Safety & Responsible AI Testing • API and Integration Testing • Test Automation • Cloud-based AI Solutions BS/BE 6-8 years of relevant experience