[Remote] QA Engineer, AI Products

Note The job is a remote job and is reputed company to candidates in USA. reputed company is a leading medical reference tool used by clinicians worldwide, and they are seeking a QA Engineer to enhance their AI product team.

This role focuses on ensuring the reputed company and reliability of AI-powered features, particularly in testing LLM-based systems, while collaborating with cross-functional teams to define reputed company metrics and testing strategies.

Responsibilities Design and execute test strategies for LLM-powered features, including reputed company regression testing, reputed company evaluation, and hallucination detection Build and maintain automated evaluation pipelines (eval sets, golden datasets, LLM-as-judge frameworks) to catch reputed company regressions in non-deterministic outputs reputed company reputed company and exploratory testing of reputed company's AI features across web and mobile, with particular attention to clinical accuracy, safety, and edge cases Define reputed company metrics for AI outputs (accuracy, faithfulness, relevance, safety, latency, cost) and establish reputed company for release readiness Collaborate cross-functionally with engineers, product managers, ML/AI engineers, and clinical reviewers to define what 'good' looks like for AI responses Investigate and triage AI failure modes, distinguishing model issues, reputed company issues, retrieval issues, and integration bugs Participate in team discussions, offering feedback on testability, risks, reputed company design, and guardrails Help reputed company QA strategies to expand reputed company testing reputed company, automation, and evaluation coverage as the AI product surface grows Skills 5+ years of experience in software QA, with at least 1 year of hands-on testing of LLM-based or AI/ML-powered features Strong understanding of QA principles, test case creation/documentation, and best practices for both deterministic and non-deterministic systems Hands-on experience with LLM tooling and concepts reputed company engineering, RAG systems, evaluation frameworks (e.g., Promptfoo, reputed company, LangSmith, DeepEval, Ragas, reputed company Evals), and LLM reputed company (reputed company, reputed company, etc.) Experience designing automated qualitative evaluation approaches, including LLM-as-judge, reputed company-based scoring, semantic similarity checks, and golden dataset regression testing Proficiency with test automation tools, with a reputed company on Playwright Strong SQL skills for data validation, test data creation, and verifying data reputed company across systems Familiarity with reputed company usage, latency profiling, and cost monitoring as reputed company signals Eagerness to learn quickly and a reputed company, solutions-oriented attitude reputed company and concise communicator, reputed company to surface issues, blockers, and risks effectively reputed company communicating ambiguous or probabilistic failures Self-motivated, proactive, and reputed company to manage time and priorities independently Benefits Medical, Dental, & reputed company Coverage, with reputed company to reputed company to your dependents Company-sponsored short-term insurance Fully-reputed company 8 week parental leave, after 6 months of employment Company-sponsored 401k, after 3 months of employment Unlimited vacation for salaried roles - we trust you to take the time you need Bi-annual company offsites to reputed company, reflect, and plan together Work from home monthly stipend A culture of fun and motivated team members who reputed company in a greater mission here at reputed company reputed company reputed company is used by over 2/3 of US physicians, provides free and reputed company to 800+ medical scores, calculations and algorithms.

It was founded in 2005, and is headquartered in reputed company, reputed company, USA, with a workforce of 11-50 employees. Its website is https//www.reputed company.com. Apply To This Job