A few weeks ago, Rise had the opportunity to co-present on developmental evaluation at the Minnesota Council of Nonprofits' Annual Conference. With the Twin Cities Medical Society (TCMS), we discussed how Rise used developmental evaluation to help TCMS build an innovative advocacy training program for med students. It was a great chance to talk about what developmental evaluation is (and isn't) and how nonprofits can use it as a tool to develop new approaches to complex problems. Check out some of the most commons Q and A's about this approach below.

Q: What IS developmental evaluation anyway?
A: Developmental evaluation is a type of evaluation that helps social innovators develop solutions to complex and dynamic problems. It applies the thinking and methods of evaluation to problems that have multiple and overlapping causes (like poverty), where the environment is constantly shifting (like the start of the pandemic), or where there are many different stakeholders with conflicting views (like pretty much any social system).

Q: How does developmental evaluation differ from other types of evaluation?
A: Traditional forms of evaluation help us understand programs that are already well-defined, meaning that activities and  outcomes are known in advance. Traditional evaluations typically happen after a program has been implemented.  Developmental evaluation is different because it happens as an initiative is developing. Activities and outcomes aren't necessarily known in advance. While the methods are similar (surveys, interviews, quantitative data analysis), developmental evaluation relies on rapid-cycle data collection and analysis to help innovators learn about their initiatives - and adapt them - in real time. It also prioritizes systems-level thinking over program-level thinking, which means thinking about relationships between and across settings rather than discrete program components.

Q: Is developmental evaluation approach for my program, initiative, or organization?
A: Well, it depends. Developmental evaluation works best in situations that involve complex or changing environments, where you want to build or develop a new approach (i.e. innovate). Highly structured programs with strict accountability measures are not a great fit. For this approach to be successful, program staff and organizational leadership need to be comfortable with ambiguity and willing to adapt program components and methods in response to new information.

Q: How does developmental evaluation differ from our continuous improvement efforts?
A: Like continuous improvement, developmental evaluation involves ongoing data collection and analysis to provide information on a program or initiative. With continuous improvement, the focus is on the program itself. Data inform how the program is working - is it reaching the intended audience, achieving expected outcomes, etc. With developmental evaluation, the focus is on how the system is changing as an initiative develops. Data are used to inform key elements of a  initiative and may lead to large changes in the activities and expected outcomes.

Q. How do I convince leadership to use developmental evaluation?
A. Supportive leaders (and funders!) are critical to the success of developmental evaluation. At Rise, we've found that it helps to identify the limits of traditional types of evaluation - which are great for understanding programs that are well-developed but less great at changing systems to enable big change.

Want to learn more about whether developmental evaluation is right for your organization? Let's connect!

Across Minnesota, counties collect data in the normal course of administering human service programs. When someone calls the county in need of food assistance, for example, a  county worker will record information about their family, job, income, and assets. This information is used to determine eligibility for programs like MFIP (cash welfare) and SNAP (food support), monitor program participation, and provide referrals to community organizations.
 
Few MN counties use this data for program improvement.

I'll admit, I was shocked when I learned this. And not just because I love data! Not using existing data is a lost opportunity – even simple analyses can help programs better understand the families that they serve, their needs, and whether these needs are being met.
 
So, why aren’t counties using the data they already collect for program improvement? There are actually a few reasons:

1. Counties in MN lack easy access to aggregate client data for many social programs. Although workers enter data locally, most data actually "live" in state data systems (eg. MAXIS or SSIS). Only a few counties have access to these state systems. When counties without access want to know something about program participants as a whole, they have to make a formal request to state research staff and wait for it to be filled. And because state research staff are busy, there is often a delay.
2. Counties - especially small ones - lack the capacity to use aggregate data for program improvement. Resources are thin and many localities simply do not have the resources/staff to devote to analyzing aggregate participant data. This is less a critique of local governments and more a plea for the state and other funders to help counties build this capacity.
3. Data systems in human service programs are outdated and do not ‘talk' easily to one another. Many systems currently in use were developed decades ago to monitor basic program participation. They weren’t designed to help counties (or the state, for that matter) understand whether or how a program is working. Because the systems developed apart from one another, the data are hard to link. This means that families that interact with multiple programs have to give their personal information to different workers at different times for different systems. This is invasive and inefficient, to say the least. And it makes it hard for those at the local and state level to understand the holistic needs of those they serve.
4. Related to this point, data systems are difficult to learn. True story: A few years ago I tried to learn about a dataset for a major public program, for a project aimed at helping one MN county use that data for program improvement. In my investigation, I learned no codebook (ie. a map to the data) even existed! All information about that dataset literally lived in the heads of various state workers. This makes no sense!

 
Admittedly, some of these problems are easier than others to solve. And to be clear, there are still major limitations of existing datasets - namely, that they are embedded in systems of structural racism and can perpetuate disparate treatment and outcomes by race and ethnicity.
 
But even recognizing these limitations, there is still much that can be accomplished if counties can better understand program participants, activities, and outcomes. Using existing data - with necessary caveats - can enhance efficiencies and minimize the additional burden placed on program participants. 

So what can we do?

• First, the state should make it easier for county human service programs to access aggregate data about their clients.
• Second, funders can invest in technical assistance to build local capacity to use data for program improvement.
• Third, we can strengthen the connection between county governments and researchers/evaluators to complement local capacity to understand and improve human service programs.

Want to know more about using data for program improvement? We'd love to chat! Contact us to learn more.

You can’t get very far these days without hearing about the need for systems change. The term is used to describe everything from transforming public assistance to reforming democracy. But what exactly does it mean?

In a nutshell, systems change is an approach to solving complex social problems.
 
At Rise, many of our partners are working to solve problems that don't have an easy solution. These so-called ‘wicked problems’ are distinguished by their complexity: they are hard to define, are intertwined with other problems, and are subject to varying interpretations by different stakeholders. Poverty, for example, is the product of many interconnected problems including the lack of educational opportunity, declining real wages, rising housing costs, and systemic racism.
 
Complex problems are tricky to solve at a program level. A program that provides housing assistance doesn't address educational inequities, low wages, or institutional racism. Complex problems need solutions that change systems, in addition to changing programs.
 
And that's what systems change initiatives do. They aim to change the underlying structures and practices that make a system or set of systems function in a certain way.
 
That sounds like a bunch of jargon, I know! Let me give you a real-life example:
 
For several years the Rise  team has been working to reduce ethnic and racial disparities in Minnesota’s juvenile justice system. This is a complex problem with multiple, overlapping causes. Programs certainly matter for addressing disparities. Implicit bias trainings, for instance, are important for changing the behavior of front-line staff. But systems-level interventions are also critical.

That’s why Rise is working to change how the system itself perpetuates ethnic and racial disparities, by:

• Bringing together diverse stakeholders to better understand how the system impacts different groups of youth
• Using data and evidence to understand where disparities are most pronounced across the system and across MN
• Helping to improve community access to data/evidence for use in advocating for policy change
  
• Identifying policies that disproportionately impact youth of color and supporting efforts to change them.

 
Working to change systems requires the ability to work across programs, agencies, and stakeholders and to bring different groups of people together in solving complex problems – skills we actively work to cultivate at Rise. If you want to connect with us to talk about how we can help your organization work at a systems level to tackle complex problems, email me at [email protected].