Key responsibilities
Chatbot Enablement:
• Oversee the deployment and optimization of chatbot systems.
• Refine chatbot outputs by improving the underlying knowledge management system and ensuring accuracy and relevance.
Collaboration:
• Work with platform teams to provide feedback on model improvements.
• Collaborate with developers and data scientists to refine algorithms
• Partner with business teams to understand their needs and tailor chatbot solutions accordingly.
Knowledge management:
• Collaborate with knowledge management specialists to maintain and update the chatbot's knowledge base.
• Ensure that the chatbot understands context and provides meaningful responses.
Data quality assurance:
• Preprocess test data and verify the integrity of training data.
• Identify issues related to data bias or insufficiency
Developing testing strategies:
• Design and implement robust testing strategies
• Create automated test suites for AI/ML pipelines and APIs
Testing Generative AI Systems:
• Validate outputs for accuracy, consistency, and fairness.
• Conduct functionality, performance, load, and stress testing
• Ensure compliance with ethical standards by identifying biases
Documentation and reporting:
• Document test cases, procedures, and results.
• Analyse trends in validation data to identify areas for improvement
Training and feedback:
• Conduct training sessions for business users to maximize chatbot adoption.
• Gather feedback from users and stakeholders to inform ongoing improvements.
Monitoring and reporting:
• Monitor chatbot performance metrics (e.g., response accuracy, user satisfaction).
• Provide regular reports to stakeholders on system effectiveness and areas for enhancement.
Skills required
Soft Skills:
• Strong communication skills for cross-team collaboration.
• Critical thinking to anticipate potential risks in AI models and knowledge management systems.
• Analytical mindset to refine outputs based on feedback and performance data.
• Ability to interpret complex data and identify trends.
Technical Skills:
• Proficiency in AI, programming languages such as Python or Java.
• Experience with testing frameworks like PyTest.
• Familiarity with ML frameworks (e.g., TensorFlow, PyTorch) and LLMs (e.g., GPT 35).
• Understanding of CI/CD pipelines and tools Azure DevOps
Qualifications
• Bachelor’s degree in computer science, software engineering, or a related field.
• 2–3+ years of experience in QA engineering or validation roles focused on AI/ML systems.
• Knowledge of ethical AI principles and regulatory compliance standards.