AI-Powered Quality Assurance System: Boys Town

The Arizona State University Artificial Intelligence Cloud Innovation Center, powered by Amazon Web Services (AWS), collaborated with Boys Town to develop an innovative AI-powered Quality Assurance system that revolutionizes how the organization evaluates hotline call performance. 

This initiative focused on transforming Boys Town's manual, time-intensive QA process into an automated system that can assess hotline calls at scale using artificial intelligence. The proof-of-concept successfully integrates AWS cloud services and generative AI to transcribe, analyze, and score hotline calls against Boys Town's established 18-point QA rubric with high accuracy and consistency, enabling comprehensive evaluation of counselor performance while reducing manual review time and surfacing targeted feedback for improvement.

Boys Town operates a vital 24/7 National Hotline providing crisis intervention support to individuals in need. While maintaining consistent, high-quality counselor performance is central to its mission, the existing quality assurance process presented significant challenges. 

The entirely manual QA review system was time-consuming and resource-intensive, allowing only a small fraction of total calls to be evaluated by staff. This limitation prevented Boys Town from maintaining uniform service standards across all interactions, providing timely feedback to counselors, and scaling its QA efforts to match its call volume growth. The organization needed an automated solution that could assess hotline calls comprehensively while maintaining the confidentiality and sensitivity required for crisis intervention work.

The ASU CIC team developed an AI-driven evaluation pipeline that automates the assessment of hotline call recordings against Boys Town's established 18-point QA rubric. The system integrates AWS Transcribe Call Analytics to convert audio recordings into speaker-separated transcripts, then uses Amazon Bedrock Nova Pro to analyze calls and generate structured feedback and scores across four key categories. A user-friendly React.js web interface hosted via AWS Amplify allows QA reviewers to upload recordings, monitor processing status, and access detailed evaluation results with counselor profiles and historical performance tracking.

The solution was powered by several key AWS services:

• Amazon Transcribe Call Analytics - Converts audio recordings to text with speaker separation and detailed metadata extraction
• Amazon Bedrock (Nova Pro) - Executes AI-based call evaluation aligned with the QA rubric, generating structured feedback and scores
• AWS Step Functions - Coordinates and sequences all backend processing functions in a stateful workflow
• AWS Lambda - Handles event-driven processing for orchestration, transcription, formatting, scoring, and database updates
• Amazon DynamoDB - Stores counselor profiles and historical evaluation data for real-time retrieval and dashboard visualization
• Amazon S3 - Stores audio files, transcriptions, formatted transcripts, and analysis results securely
• AWS Amplify - Hosts the React.js web application with integrated deployment pipeline
• Amazon API Gateway - Provides secure REST API endpoints for frontend-backend communication

The automated workflow processes uploaded recordings through transcription, analysis, scoring aggregation, and database storage, providing QA reviewers with comprehensive evaluation results including timestamped evidence and performance insights for targeted counselor development.

This AI-powered QA system transforms how crisis intervention organizations can maintain and improve service quality at scale. By automating the evaluation process, Boys Town can now assess a significantly larger percentage of hotline calls, ensuring more comprehensive quality oversight and consistent application of their established standards. The system provides objective, rubric-aligned evaluations that eliminate subjective variations in manual reviews while generating detailed, actionable feedback for counselor development.
 

Working the AWS and ASU teams was incredibly easy and collaborative.  These teams worked diligently to build an AI QA bot that will revolutionize our ability to assess the quality of counseling staff.  The ASU students were incredibly professional and polished in their engagements and product delivery.  Boys Town is incredibly thankful for our partnership with AWS and ASU, the impact of this work will be immense.

Oscar Gonzalez, Director National Hotline, Boys Town.

The automated QA framework developed for Boys Town offers significant potential for adaptation across crisis intervention services, mental health organizations, and customer service operations. The system's architecture supports different evaluation rubrics, diverse content domains, and varying organizational quality standards, making it applicable to suicide prevention hotlines, domestic violence support services, healthcare call centers, and other organizations requiring consistent quality assessment of sensitive conversations.

Government agencies, nonprofit organizations, and healthcare providers managing large volumes of critical communications can implement similar AI-powered evaluation systems to improve service delivery, ensure compliance with quality standards, and provide data-driven insights for staff development and operational improvement.