In 2020, BellXcel engaged McClanahan Associates, Inc. (MAI) to conduct a literature review to learn more about quality assessment (QA) in Out-of-School Time (OST) programs and understand the research support for BellXcel’s quality standards and indicators. SCRI talked with Wendy McClanahan, president and founder of MAI, to distill the findings and share her unique perspective on QA in OST.

Q: Describe quality assessment in OST. What does it look like? What’s the purpose?

Wendy: Quality assessment (QA) in OST is part of a larger cycle of continuous quality improvement. It allows programs to zoom the lens out a little to see the bigger picture. Ideally, programs assess their quality by collecting data through self-assessment or external observation. They use the data they collect to see where they are meeting their goals and where they are falling short, and they make changes to improve program implementation. With each new program cycle, they begin the QA cycle again. As they strengthen programs, they should see more positive benefits for their constituents.

Q: What are programs measuring, and how?

Wendy: Researchers and program operators use many different tools to assess quality, but all of them are measuring things that are aligned with youth development theory. Observation is the main source of data, and interviews and document reviews are also common, particularly for measuring the quality of program administration.

There are several substantive components common to most QA in OST. Nicole Yohalem (2010) reviewed nine quality assessment tools and found six domains of quality measured across all of them. Other substantive measures look at youth leadership; youth voice and choice; program management; community linkages; and aspects of staffing. BellXcel’s quality indicators are organized within five quality standards and align to these areas.

No matter what tool is being used, the real idea behind QA in OST is continuous learning and improvement.

Q: How did you conduct this literature review?

Wendy: BellXcel asked us to look at the research literature to see whether, and if so, how, their quality indicators were supported. We used a very specific set of eligibility requirements: the article had to be published since 2000; focused on the measurement of quality using observation techniques; reporting about the use of QA in academic enrichment interventions taking place outside of school hours; and targeting students in grades K-8. We searched academic journals, grey literature (research reports), and resources intended for practitioners.

Our search yielded 48 studies that met the criteria, and we reviewed 30 in greater detail.

Q: What’s unique about BellXcel’s quality standards?

Wendy: Taken together, BellXcel’s quality standards cover the same six common elements that I described previously, so there’s overlap with other well-known tools in the field. Where BellXcel’s quality standards differ a little is the focus on family engagement and evaluation and assessment. These sections aren’t as common in other QA tools, especially family engagement. You’re more likely to see family engagement nested in a set of standards related to community partnerships than in a standalone domain. Another slight difference is that there’s a really deep emphasis on relationships in the field, but it’s less pronounced in BellXcel’s indicators.

Q: Generally, what did you find? What surprised you?

Wendy: Our main finding is there is a literature base for all of BellXcel’s quality standards and indicators. The indicators have been explored by other researchers and are aligned with other quality tools. Of BellXcel’s standards, there is the most empirical support for the curriculum and instruction and culture and climate indicators.

We found that researchers are using many different quality assessments—some developed on their own, and others used well-known assessments. Some of the most commonly used included the YPQA, School-Aged PQA, Program Promising Practices Rating Scale, Comprehensive Assessment of Summer Programs, and Summer Learning PQA.

What surprised us is how little literature there was that explored relationships between quality standards and youth outcomes. In general, there are just very few studies connecting quality indicators to outcomes. About half of the studies were descriptive and didn’t explicitly test the link to outcomes. Several of the studies did show positive correlations between quality and outcomes, and some showed null or negative findings. Other studies assessed quality in programs and assessed outcomes, but did not make a link between the two.

So, we can’t say if there are strong youth and adult relationships, that means positive academic outcomes. There is a lot of “face validity” around these measures. It’s just surprising to us that there isn’t more research that makes statistical linkages.

Q: Is quality assessment a widely used practice? What are the challenges to QA?

Wendy: There is a lot of emphasis on quality in OST, but there’s still no consensus on specific standards or tools; many states and localities have developed their own quality standards.

A challenge is that QA can be resource-intensive depending on the approach and tools. Some of the tools suggest that you’re trained first, use multiple observers, and have long observation windows. I remember one study of the Youth Program Quality Assessment (YPQA) showed that program managers dedicated 50-plus hours over 18 months to QA, and three additional staff spent about 70 hours combined, on average. That might not sound like a lot of time, but I think that it can be for programs that don’t have a lot of capacity, and I think that doing QA routinely along with continuous improvement could take even more time. From my perspective, OST programs typically need more funding to not only do QA, but to be even more impactful programs.

Q: Do you have any examples of programs who used the data they gathered to change their practice? Specifically, what did they learn through QA and what practice(s) did they change as a result?

Wendy: Yes, there are many organizations that have benefitted from quality assessment and used it to strengthen their program models. One OST organization I worked with had a very broad QA process that included external program observations (using a tool that was adapted from the YPQA) and participant and staff feedback. One of the key lessons learned was that there was variation in the extent to which youth felt emotionally safe in the OST center. Youth experienced high levels of emotional safety in certain program areas and not others. The organization—which already had emotional safety protocols in place such as anti-bullying policies—was surprised, but also wanted to ensure that participants felt emotionally safe in every program across the center. Participant feedback helped them understand reasons why youth were not feeling safe all the time. One theme that emerged was that some participants felt overwhelmed in busier programs where there was less structure. They also learned that while staff supported and implemented the center’s anti-bullying policies, some staff felt they could benefit from additional training in “classroom management” so that they could be more proactive and less reactive when emotional safety issues arose. In response, this organization started a safety campaign, provided staff with additional training, and developed quiet spaces for participants who preferred less activity.

Q: What are some first steps that a program can take to address quality assessment, especially those with time and resource constraints? If you can’t do a whole Cadillac QA, where can you start?

Wendy: I recommend that OST programs focus on two things: 1) developing (or refining) a programmatic logic model (LM) or organizational theory of change (ToC) and 2) building or enhancing their learning culture. With all the changes in the landscape, now is the perfect time to take a step back and redesign for quality.

First, LMs and ToCs clarify the program model, create a common language and alignment across staff, and stimulate innovation and learning.  These tools form the foundation of continuous improvement efforts by making the program design, including outputs and outcomes, explicit. They explain what you do, who your constituents are, how you do your work, and what you expect to achieve.

A learning culture is also important because it supports an organization’s ability to acquire, interpret, and apply knowledge. A strong learning culture encourages collaboration, risk taking and experimentation, and participatory and data-informed decision making. It also includes formal systems and practices that reward and reinforce learning and enable access to and availability of data and information. QA is designed to identify strengths and weaknesses — a strong learning culture can help organizations make meaning of the assessment findings and innovate to better meet the needs of the youth they serve.

There are several quality assessment options that can provide great information and require fewer resources. Self-assessments are a good option to get started. They are less expensive than external assessments, can be based on perceptions rather than observations (which are more resource intensive) and can be more successful at securing buy-in from staff than external assessments.

Here are two free tools for nonprofits planning to develop a Theory of Change of Logic Model:

• Theory of Change: Theory of Change Overview Guide with Template
• Logic Model: W.K. Kellogg Foundation’s Logic Model Development Guide

Here is an article that discusses strategies for growing a Learning Culture:

• Strategies for Cultivating an Organizational Learning Culture

My last piece of advice is to not let perfect be the enemy of the good. This work takes time. They are ongoing, iterative processes designed to build a stronger foundation and program. Think about how you can achieve small wins — whether it be spending two hours as a team self-assessing quality in one area, drafting program outputs and outcomes, or discussing what you’re learning about your program.