Ensuring quality in Out-of-School Time (OST) requires an understanding of the following areas (click to jump to the section): Quality Improvement Approaches, Systems-Building Efforts, Ways to Measure Program Quality, Sample Program Assessments and Standards, the Cost of Quality, Program Case Studies, Hiring and Developing Staff, and other topics.
quality improvement approaches
The Beyond The Bell Toolkit (2015)
This 4th Edition Beyond The Bell Toolkit contains practical, easily transferable content that can help program leaders build or enhance an out-of-school program. Its focus is on assisting program leaders in the process of designing, implementing, and evaluating their programs through time, and it gives examples of what actual programs are doing in the field. It has 96 ready-to-use tools covering everything from program management and partnerships to evaluation to continuous program improvement measures. It is only accessible through purchase; however, there is a free “Tool of the Month." Before you purchase the toolkit, you can watch a presentation to determine its practicality for your purposes.
Expanded Learning, Expanded Opportunity (2015)
This report from America’s Promise Alliance profiles four communities in different stages of strengthening and effectively using Out-of-School Time networks to benefit high school youth. It looks at expanded learning opportunities in Grand Rapids, Michigan; Louisville, Kentucky; Memphis, Tennessee; and Rochester, New York; all of which have convened GradNation summits. It highlights successes, challenges, and innovation these communities have faced, and it also draws conclusions that have implications for OST programs of all types across the country. One of the most significant findings it shares is the proven importance of OST programs' engagement in cross-sector collaboration throughout the community in order to establish a network and community of support for OST programs.
Why Afterschool Quality Matters (2015)
In this focus brief, the National AfterSchool Association (NAA) describes its role in the afterschool field, clarifies what is and is not an afterschool program, explains who funds and provides afterschool programs, mentions who uses afterschool programs, and highlights the need for quality measures.
Early Warning Indicators—An Afterschool Guide (2013)
This guide, created by the National Institute on Out-of-School Time (NIOST) for the United Way of Greater Philadelphia and Southern New Jersey, examines the role of OST providers in increasing graduation rates. Using the Early Warning Indicators (EWI), this guide explains how OST programs can partner with schools to support youth with EWIs. It provides tips for building partnerships and tools for targeting and designing effective OST activities and experiences for youth.
Think Outside the Clock - Planners Link After-School Programs to Classroom Curriculum (2011)
This brief article, sponsored by the Wallace Foundation, discusses the importance of the issue of Out-of-School Time for educators. It asserts that schools can play a critical role to ensure alignment, coordination, and quality, particularly in the case of school-based expanded learning programs.
After-School Toolkit: Tips, Techniques and Templates for Improving Program Quality (2008)
This toolkit, commissioned by the James Irvine Foundation and developed by Public/Private Ventures, offers program managers a practical, hands-on guide for implementing quality programming in the afterschool hours. The kit includes the tools and techniques that increased the quality of literacy-focused programming and helped improve student reading gains in the Foundation's Communities Organizing Resources to Advance Learning (CORAL) initiative, an eight-year, $58 million afterschool endeavor to improve education achievement in low-performing schools in five California cities.
Putting it All Together: Guiding Principles for Quality After-School Programs Serving Preteens (2008)
This report examines the specific characteristics of quality programs that have produced positive outcomes for preteens. The following six characteristics were developed based on studies of existing afterschool programs: a focused and intentional strategy - programs have a clear set of goals, target specific skills, and use a youth development framework; exposure (depth, intensity and breadth) - programs are available for a significant number of hours per week and include a variety of activities; supportive relationships - programs emphasize positive adult-youth relationships regardless of content; family engagement - programs strive to include families through various strategies; cultural competence - programs have diverse staff, are inclusive of a variety of populations, and help participants to understand and value diverse cultures; and continuous program improvement - programs strengthen quality through ongoing staff training, coaching and monitoring, and data collection and analysis.
Supporting Success - Why and How to Improve Quality in After-School Settings (2008)
The CORAL Initiative (Communities Organizing Resources to Advance Learning) was launched by the James Irvine Foundation in 1999 to strengthen the academic achievement (particularly in reading) of students attending low-performing schools in California. This study distills lessons learned for improving Out-of-School Time program quality based on efforts undertaken by the CORAL initiative. Key recommendations for strengthening program quality and improving student academic achievement include: instituting a process for continuous improvement that includes ongoing monitoring and opportunities for professional development; making the financial investments necessary to implement the continuous improvement process; and securing the commitment and participation of staff at all levels.
Characteristics of Effective Summer Learning Programs in Practice (2005)
Based on observation of various summer programs, this research report explains that high quality summer programs that effectively support student academic growth and youth development share nine characteristics that focus on a program's approach to learning and infrastructure, including: intentional focus on supporting learning, firm commitment to youth development, proactive approach to summer learning, strong leadership, collaborative planning, opportunities for staff development, strategic partnerships, commitment to rigorous evaluation and improvement, and a focus on sustainability.
NAA Quality Conversations
This series of podcasts from the National AfterSchool Association examines what afterschool quality entails, how it can be achieved, and why it is so important. Over the course of five episodes, Executive Gina Warner speaks with experts, researchers, practitioners, and funders about the various aspects of afterschool program quality.
SYSTEMS-BUILDING EFFORTS TO ENHANCE QUALITY
Building Citywide Systems for Quality: A Guide and Case Studies for Afterschool Leaders (2012)
This guide from The Wallace Foundation and the Forum for Youth Investment explores how cities and intermediaries can collaborate with afterschool providers across communities of varying sizes to build a high-quality system. It explains how to build or strengthen a Quality Information System (QIS)—an intentional, organized effort to improve afterschool program quality.
Collecting and Using Information to Strengthen Citywide Out-of-School Time Systems (2011)
The guide presents examples of how mayors, council members, and other municipal leaders across the country have implemented various strategies to improve the Out-of-School Time opportunities in their cities and towns. Their efforts highlight the powerful role city leaders can play in supporting data collection at the program and city levels.
Investments in Building Citywide Out-of-School Time Systems: A Six-City Study (2009)
This report serves as a companion to a previous resource developed by Private/Public Ventures and The Finance Project: The Cost of Quality Out-of-School Time Programs (included at the bottom of this page), which shares detailed information regarding both the average full cost and the average out-of-pocket expenditures of a broad range of OST programs. It builds on these resources by establishing a systems-building framework for OST and explicitly discussing the system-level investments and strategies the case study cities used. The authors use their research to support their suggestions that cities invest in a strong OST infrastructure by: providing community leadership and vision; improving program quality; expanding access to and participation in quality programs; and financing and sustaining quality programs.
Building Quality Improvement Systems: Lessons from Three Emerging Efforts in the Youth-Serving Sector Executive Summary (2007)
This executive summary looks at quality improvement processes in three Out-of-School Time networks: Girls Incorporated Quality Improvement Process; YouthNet of Greater Kansas City Organizational Assessment and Improvement Project; and the Michigan Department of Education After-School Quality System Demonstration. It synthesizes the quality improvement efforts of these respective networks including: key partners involved, core components of the improvement process, evidence of early impact/sustainability, challenges, and lessons learned.
research on measuring program quality
Out-of-School Time Program Research & Evaluation Database (2015)
This database from the Harvard Family Research Project compiles research on Out-of-School Time initiatives. The content it provides access to ranges from research and evaluation work on small OST programs to large-scale OST initiatives. It is focused on information that gives insight into high-quality evaluation techniques and the impact that has on OST programs success.
2014 Atlanta Quality Matters Youth Program Quality Intervention Report: Executive Summary (2014)
This summary shares highlights from a quality improvement effort in certain OST sites throughout Atlanta in 2014. Led by the Atlanta Ready by 21 Southeast Challenge City, in partnership with the David P. Weikart Center for Youth Program Quality Assessment, this effort examines quality indicators, programmatic successes, and analyses from multi-year staff.
Continuous Quality Improvement in Afterschool Settings: Impact Findings from the Youth Program Quality Intervention Study (2012)
This report conveys the findings from a study by the Weikart Center for Youth Program Quality on methods of quality improvement. It found that the Youth Program Quality Intervention, an evidence-based continuous improvement model, improves quality among many types of afterschool systems. It also explains how a cycle of assessment improves the services that youth receive in afterschool programs. The executive summary can be found here.
Afterschool Evaluation 101: How to Evaluate an Expanded Learning Program (2011)
This guide from the Harvard Family Research Project is targeted toward program staff who do not currently use an evaluation system for their program. It offers a step-by-step process for programs to follow in order to effectively approach evaluation and the program’s capacity for it, to focus the evaluation and choose a design, and finally to analyze and present the evaluation data.
Measurement Tools for Evaluating Out-of-School Time Programs: An Evaluation Resource (2011)
This Harvard Family Research Project tool can be used to guide program evaluations on a wide array of scales. It explains instruments and tools that programs can get for on-the-ground evaluation of their programs to assess the aspects and outcomes of an Out-of-School Time program.
Self-Assessment of High-Quality Academic Enrichment Practices (2011)
This resource from NIOST Afterschool Matters proposes that more 21st Century Community Learning Center programs use self-assessment as an alternative to external assessment to assess program quality. It provides reviews of program elements that have shown to impact academic growth, and it describes the extent to which program practices are actually measured by the self-assessment tools that are available. The self-assessment tactics could be applied to non-21st CCLC programs as well.
Measuring Youth Program Quality - A Guide to Assessment Tools, Second Edition (2009)
This guide, updated from an earlier version, compares assessment tools used in Out-of-School Time programs. The tools included in the guide are all research-based, designed to be deployed in a variety of settings and to utilize program observation. Tools are compared in terms of purpose/history, structure/methodology, content, reliability and validity. Some of the assessments reviewed in the guide include: Assessing Program Practices Tool (APT); Out-of-School Time Observation Tool (OST); the School Age Care Environmental Rating Scale (SACERs); New York State Afterschool Network Quality Self-Assessment Tool (QSA); and the Youth Program Quality Assessment.
Improving After-School Program Quality (2007)
This research report builds on the work of two previous research studies (Durlak & Weisberg, 2007; Yohalem & Wilson-Ahlstrom, 2007) to answer key policy questions related to Out-of-School Time programming: what are the characteristics of programs that produce good outcomes? Can programs simultaneously produce results in multiple outcome areas (e,g., safety, academic improvement, youth development)? How can programs best be held accountable for outcomes? How can stakeholders improve program effectiveness? It shares the major findings: research on programs supporting youth development outcomes can also help to improve academic performance; programs with SAFE features (Sequenced activities, Active learning techniques, Focus on social or personal development, and Explicit objectives) are correlated with positive outcomes for participants; the increased availability of programming in many communities makes it more politically feasible to shift the focus to program quality; the specific content of a program is less important than the focus on implementing it well; and program evaluations should examine staff and program practices related to implementation as well as youth outcomes.
Quality Time Afterschool (2007)
This research brief summarizes research conducted by Public/Private ventures of the Beacons Centers managed by Philadelphia Safe and Sound. The Beacon centers provide a range of out-of-school programming focused on enrichment, leadership, heath, arts, and recreation. The brief shares the key finding that effective group management and support from adults were two of the most important factors in promoting youth engagement, participation, and learning, and it also explains that the greater role that youth had in shaping program activities, the greater their engagement and enjoyment of the activity.
Sample Program assessments & Standards
Youth Program Quality Assessment (PQA) and School-Age Program Quality Assessment
This resource from the Weikart Center for Youth Program Quality provides assessment tools to help OST programs determine the quality level of learning environments and identify staff training needs. The tools it provides can be accessed through PDF downloads in the link provided. In addition to sharing a Youth Program Quality Assessment Handbook, it includes the Youth PQA (in both English and Spanish), Supplements to the PQA, a STEM PQA, Camp PQA, Health and Wellness PQA, and an Arts PQA.
A Program Assessment System (APAS) Training
This resource from the National Institute on Out-of-School Time (NIOST) helps OST programs connect quality and youth outcomes in a flexible, comprehensive manner. It includes three measurement tools, shown below, that allow OST programs to focus on realistic outcomes for youth and improve program quality. Each of these tools is no-cost but requires a training.
- Survey of Academic and Youth Outcomes (SAYO) S and T (completed by staff and teachers)
- Survey of Academic and Youth Outcomes (SAYO) Y (completed by youth)
- Assessment of Program Practices Tool (APT)
New York State Afterschool Network (NYSAN)'s Quality Self-Assessment (QSA) Tool
NYSAN created this Tool to emphasize the importance of performing self-assessments regularly and in a quality manner. It intends for this Tool to help programs identify their own strengths and weaknesses and be able to respond accordingly, instead of relying on external evaluation and monitoring. It should be used in conjunction with other formal and informal methods of evaluation, such as focus groups, youth, parent and staff meetings, and external evaluation and monitoring for maximum impact.
FOUNDATIONS' Quality Assurance System (QAS)
This online tool from FOUNDATIONS, Inc.combines program improvement and afterschool program evaluation. It takes you step-by-step through an examination of your program, then guides you through a structured analysis before helping you create detailed improvement plans. This tool is helpful for programs assessing more than one site, because it gives a Cumulative Report and lets you view progress across sites. It also uses graphing to give an immediate, comprehensive look at teh progress your program has made and the areas that still need improvement.
School-Age Care Environment Rating Scale (SACERS)
From the Frank Porter Graham Child Development Institute at The University of North Carolina Chapel Hill, this rating system is designed to assess group-care programs for students age five to 12. The scale it uses includes 49 items broken down into seven subscales: Space and Furnishings, Healthy and Safety, Activities, Interactions, Program Structure, Staff Development, and Special Needs.
Rhode Island Program Quality Assessment (RIPQA)
Based on the David P. Weikart Center for Youth Program Quality’s school-age and youth PQA tools, the RIPQA measures programs in relation to four main areas: safe environment, supportive environment, interactions, and engagement. It was created primarily by the Providence After School Alliance with help from other youth-serving organizations. Also in the tool is an administrative checklist which tracks the ability of organizations to support staff development and professional development, family and civic engagement, and administrative functions.
Washington State Quality Standards for Afterschool & Youth Development Programs
Based on national research, these quality standards define what quality looks like in an afterschool and youth development program setting. The standards apply to all programs for youth between the age of five and young adult across Washington state.
North Carolina Center for Afterschool Programs (NCCAP) Program Self-Assessment
This self-assessment and planning tool from NCCAP helps programs chart their progress in the following nine areas housed within three categories: 1) Goals for Program Management and Delivery: Safe, Healthy, and Orderly Environment; Qualified and Diverse Staff; Opportunities to Learn in Diverse Environments, 2) Goals for Program Connections: Positive Participant and Staff Interactions; Active Family and Community Partnerships; Consistent Participant Attendance, and 3) Goals for Program Participants: Greater Personal Responsibility; Improved Academics Achievement; Greater Creativity and Well-Being.
cost of quality
Cutting Cost - Keeping Quality (2010)
This research brief from The Finance Project highlights strategies that youth serving organizations are using to maintain quality services despite financial difficulties caused by a challenging economic environment.
The Cost of Quality After-School Programs (2009)
Based on a sample of 111 programs across six cities, this Wallace Foundation research report discusses the costs of implementing high-quality Out-of-School Time programs. The study finds that the costs of quality vary significantly based on program goals, content (singularly focused v. multiple activities), ages served, and times of operation. The discussion of costs includes non-monetary (or in-kind) contributions like physical space and volunteers.
OST Cost Calculator
Check out the Wallace Foundation's Cost Calculator to determine the impact of making quality-focused improvements on the rest of your organization's work!
case studIES
Believe It. Build It.: Minnesota's Guide to Effective Afterschool Practices
This Believe It. Build It. online guide from Minnesota's Afterschool Network connects readers to tips and resources in a variety of areas, from intentional program design to responsiveness to culture & identity to continuous program improvement. A Toolkit accompanies the guide for those who want to share it with partners and/or stakeholders.
Promising Practices - Showcase of Award Winning Programs (2011)
New Jersey After 3pm is a private/public partnership dedicated to maximizing resources to support afterschool programs throughout the state. This brief publication highlights Out-of-School Time programs honored as part of New Jersey After 3's Promising Practices and Competition Showcase. Promising practices highlighted in the brief focus on systems, processes, or activities designed to strengthen family, school and community partnerships and/or program content.
Afterschool Grows Up - How Four Large American Cities Approach Size and Scale in Afterschool Programs (2008)
This Robert Wood Johnson Foundation report provides in-depth profiles of 4 emerging city-systems for Out-of-School Time: The Afterschool Corporation (New York City); LA's Best (Los Angeles); Afterschool Matters (Chicago); and San Diego 6 to 6 (San Diego). It presents their work as part of growing efforts to provide Out-of-School Time opportunities at scale, rather than the piecemeal, fragmented approaches that still dominate the OST landscape. The report examines each city-system's approach to Out-of-School Time funding, staffing, evaluation, champions/leadership, and authority/control.
HIRING and developing staff
Body and Soul: Reflections on Two Professional Development Credential Pilots in Massachusetts (2010)
This report provides an overview of two credentialing models developed in Massachusetts: the School Age Youth Development (SAYD) Credential and Professional Youth Worker Credential (PWYC). The report includes staff reflections on the impact of earning a credential on their commitment to the field and program practices. It concludes that credentialing systems work best when they include: a training system (including method of approving trainers), career lattices, incentives (including wage increases), and a quality rating system.
Out-of-School Time: Leveraging High Education for Quality (2010)
This paper discusses how partnerships between institutions of higher education and the Out-of-School Time field can strengthen the quality of learning opportunities beyond the regular school day. Increasingly higher education institutions are recognizing the OST field as a unique space requiring unique approaches to curriculum and content delivery. In response, they are developing courses, certificate programs, and practicums in OST. This paper helps advance a shared understanding of how higher education can play an important role in professionalizing the OST field, developing a skilled workforce, and providing potential pathways for individuals seeking to pursue a career in Out-of-School Time.
Credentialing for 21st CCLC Staff An Overview of the Benefits and Impacts (2009)
As the field of Out-of-School Time matures, more professional development models are emerging— credentialing systems are one such approach to professionalizing the field, improving staff practices, and enhancing program quality. This report summarizes two long-term models of credentialing and provides an overview of the benefits and the potential barriers to credentialing. The report also discusses essential elements of a strong credentialing system, including detailed analysis of core competency frameworks. The report also describes the current state of credentialing systems in afterschool and youth development and includes a case study of the Missouri Youth Development Credential.
additional quality resources
Tough Times, Tough Choices in After-school Funding: Pathways to Protecting Quality (2012)
This white paper from The Wallace Foundation argues for the importance of prioritizing quality in afterschool decisions, instead of always opting to maximize the number of children served. It pulls from interviews with agency leaders from three major cities to examine the approach these leaders take to make afterschool decisions, the amount they prioritize quality, and the factors they consider in the decision-making process.
Administrative Management Capacity in Out-of-School Time Organizations: An Exploratory Study (2008)
This report summarizes the management concerns and challenges of organizations that provide Out-of-School Time programming. The findings are based on a study of 16 high-quality out-of-school providers. It highlights how a key challenge for programs is a lack of ability to invest in administrative staff and infrastructure (e.g. IT, space), which forces program staff to spend time on these tasks and diverts the time that could be spent on ensuring program quality and strategic planning. The findings are further broken down into these categories: financial management, human resources, information technology, and facilities.