Episode Notes

Learn more:

• Stream the full webinar, which includes specific tactics that you can leverage to operationalize the principles discussed: Best Questions to Ask for Measuring Impact

• Read about Choosing a Grant Management Software to Support Trust-Based Philanthropy

Interested in more quality content focused on reducing grantee burden and improving equity? Here are a few Submittable resources:

• Redistributing Power in Philanthropy: A Conversation with Shaady Salehi (Webinar)

• How to Center Equity in Your Grantmaking (Blog)

• The Next Generation of Grants Management Software (Guide) 

• The Review (Bi-monthly newsletter by Sam Caplan)

Full Transcript

Welcome to Impact Audio. I’m Rachel Mindell. This special episode is focused on the topic of impact measurement, with insights and actionable strategies from Natalya DeRobertis-Theye, Submittable’s Senior Product Marketing Manager. So, without further ado... [drumroll]

RACHEL: Hi, Natalya. Welcome to Impact Audio. 

NATALYA: Thank you Rachel, it's a pleasure to be here. 

RACHEL: Yeah, how's your Friday going? 

NATALYA: It's a pretty good Friday. This is the third project in a row I'm working on with our producer Jordan Marvin. And I'm wearing a tie dye shirt, so I would say so far so good. 

RACHEL: It's an amazing shirt. I was kind of sad when you put that sweater on. Because that is an amazing tie dye shirt. And since this is audio, people should know that it's rainbow-colored. Yeah? Or-- 

NATALYA: Yup, I made it myself. 

 RACHEL: Awesome, awesome. Well, so this is maybe a little more formal than our usual video chat and Slack, but I'm really looking forward to digging into the topic of impact measurement with you today, and thinking about the right questions to ask. Especially because I know you've done a ton of research on this topic, and you've identified seven strategies for trust-based impact measurement. How did you land on those? 

NATALYA: So as you're aware Rachel, because you do some of this work as well. Our team were constantly deep in this world of what we kind of call like, the next generation of grant management, or evolved grant management. And Sam Caplan, our VP of Social Impact here at Submittable, has kind of broken it down into three different pieces, all with the overarching principle of being grantee-focused, and reducing the burden on grantees. And those three pieces are trust-based, streamlined, and equitable grantmaking. 

So kind of using that perspective, I dug into the ideas, and the leadership, and the research out there. And I came up with seven principles.

RACHEL: Well, I am excited to dig into these seven principles with you. So for the first one, I see that you have prioritized the work. What does that mean? 

NATALYA: Sure. So prioritize the work is the way that I wanted to communicate, an organization funds their grantees, so that they can do the work of administering their program and making a positive difference in their communities. The funding is not, they don't want their grantees to spend an inordinate amount of time filling out questionnaires or jumping through hoops to come up with specific data points at your behest. 

So the idea behind prioritize the work, is that if you keep the mission as your primary focus, that will help you determine whether in your impact measurement, you're asking useful essential questions that are serving that mission. The way Shaady Salehi of the Trust-Based Philanthropy Project put it in conversation with you, Rachel, is to take your grantee's point of view. And so I put this as a first principle because I think it's always important to begin here. 

RACHEL: Fantastic. Yeah, I have learned so much from Shaady and her colleagues at the Trust-Based Philanthropy Project. If you're not familiar with them, please check them out. And yeah, I think we're also seeing a lot of this with other funders, and even Submittable clients, that are really putting these ideas into practice. So I love that first principle. 

So for number two, you have do your part. 

NATALYA: Yeah, do your part really means taking on what reporting burden yourself, that is possible and reasonable for you. So here I'll reference Fix the Form, which is a movement focused on reducing the burden on grantseekers, spearheaded by Laura Solomons and Kari Aanestad. 

RACHEL: And I should jump in here to just recommend that people check out the webinar that you helped facilitate Natalya, with Kari Aanestad and our VP of Social Impact, Sam Kaplan, on the topic of Fix the Form with live form assessment. It's really amazing. So continue. 

NATALYA: Absolutely. It was very fun. So Fix the Form's whole idea was to identify pain points for grantseekers, and helping funders fix those pain points to make a better experience for grantseekers worldwide. And a major pain point that Fix the Form identified, is requiring applicants to provide publicly-available information. Which if you think about it, is completely ridiculous. If information's publicly available, you shouldn't have to write it into a form again and again, every time you apply for a grant. 

So doing your part here means doing the research and taking on the task of finding anything that's available online yourself. And then to take it a step further, you can think about how you can participate in the measurement work itself. So for example, this is a big trend that we're seeing, is verbal check-ins. So if you're doing a verbal check-in, can you, as the funder, record the call and upload it to your grant management software yourself, or upload your notes. 

You may also be in a position to apply your resources to some of this work. For example, the Robert Sterling Clark foundation in New York, hires external evaluators to review recorded conversations and assess progress toward trust-building goals. So do your part, do your research, take on what tasks you can yourself, and use your resources to help you do so. 

RACHEL: I love that. I just think about, for example, for like a really busy community organizer at a non-profit, for them to be able to just jump on a call and do a check-in. Rather than filling out a 500-word statement, and editing, and crafting, and all of that. It just makes so much sense. Great. So tell us about the third principle. 

NATALYA: Sure principle number 3 is embrace the complexity. As I've learned, again and again, and I'm sure as anyone who's tried to do impact measurement work knows as well, it's just so complex. You're dealing with short-term outputs and long-term outcomes, and some consequences are intended and some are incidental. And some can't be foreseen at all. So embracing the complexity here, I'm just meaning to remind everyone that, as you're collecting impact data, to resist the temptation to look for a really neat linear story, or strictly quantitative data points. Or just in general, reducing a larger story to a single stat, like impact per dollar. 

It also means that sometimes what might look like a failure is really building for the future, that's your phrase Rachel. So let's consider an example, imagine that you had funded a program that was trying to get a political initiative passed. And that initiative ultimately failed at the ballot box. So from one point of view, it failed. But from another point of view, the organization may have educated a ton of people about the issue, they may have contributed to moving the Overton window on what is possible for the future. And just really laying the groundwork for future success the next time around. And they may also have learned strategies in communication and field work, what was effective for getting their message out, or for persuading people. 

So I think that example really illustrates that a failure can contain so much success. And I think that that's true for the vast majority of significant social change work. One more point about embracing the complexity, I think that it also really encompasses the concept of timing. I love to refer to a quote I heard from Neil Buddy Shah at the Yale Philanthropy Conference some time ago, which is "You risk good ideas failing when you run an impact evaluation that is too rigorous, too early." 

RACHEL: That's great. Yeah, I really enjoyed that panel too, I remember from the conference. And I remember my takeaway being that, basically exactly what you're saying, that impact measurement is very complex and also very unique to the organization. Like if you, what is the phrase, if you've done impact measurement at one organization, you've done impact measurement at one organization. It's always, very unique. 

It reminds me actually, a quick aside, when I interviewed Alnoor Ebrahim whose work I know you spent some time with. He wrote a book called "Measuring Social Change, Performance and Accountability in a Complex World." And I interviewed him about social impact strategy, and I loved this analogy, which I just want to share. Which is about really how important it is to think about the unique circumstance of every organization. 

RACHEL: So he gives a really great example that I love about an ambulance service. And the idea is basically that we know that the faster an ambulance service arrives on the scene, the greater chance that someone will survive. So when we measure the impact that the ambulance service is having, we're going to think about how quickly they arrive on the scene, and the quality of care they're providing. But we're not actually going to measure the ultimate health outcome, because they're only involved in one small piece of that, right. When they get the person to the hospital, what happens beyond that is not relevant to the impact that we're measuring for the ambulance service. 

NATALYA: That's a really great point. And it actually kind of leads into the next principle that I've got here as well. Certainly some overlap, but the next one is to think holistically.

RACHEL: Yeah, do them out of order, why not. 

NATALYA: OK. And the idea behind that one is that, just as an example you just described, programs do not exist in a vacuum. And a holistic mindset can apply both to the issue you're looking to tackle, in that case, health outcomes, or it could be childhood hunger. But it also applies to your grantmaking program. 

And there are themes to be gleaned, as well as lessons to be learned, across the spectrum that can help and inform related work, as well as your other grantees. So I wrote a blog on the topic of measuring social impact a while ago. And I was thinking about, when you're thinking holistically, you really need to take into account attribution. So there's your organization's activities, there could be organizations of people that you partner with, there could be activities by others totally independent of your efforts, but maybe with overlapping goals or maybe with contrary goals. And then there's just forces of nature macroeconomic, political issues, all of these things can influence outcomes. 

And so as I was reminded again by Sam Caplan, both the short-term and the long-term results are important to consider here. And there's a difference between how is a project going and are we achieving our mission. And what's more, they can really be totally independent of each other. 

RACHEL: Absolutely. So I see next that you have centering people as a principle. Will you tell me more about that? 

NATALYA: This is such an important one. The ultimate beneficiaries of social impact work are always people. Individuals with families, and friends, and hopes, and fears, and their own stories to tell. And it's so important not to reduce a dynamic human experience to just a data point. So centering people is meant to remind you to keep the human element of this work always front and center in your impact measurement. So for example, we're often seeing Submittable clients asking their grantees to share the stories of the community members who are benefiting from the work. To share testimonials, pictures, video, audio, that just makes it all the better. 

RACHEL: That really just ties back to trust-based philanthropy doesn't it. Where one of the principles is support beyond the check, right. So organizations can take these testimonials potentially and help some of these organizations with their own marketing, and help to boost them. 

That was something that when I talked with Marcella Tillet, from the Brooklyn Community Foundation, she said one of the things they do to support their grantees is to collect photos, collect materials, that they can then use to help celebrate the work of their grantees. 

NATALYA: Absolutely. One of the tactics that I tied to an earlier principle that we discussed, is to make sure that you're taking action. So from a trust-based approach, you want to be sure to ask like what you can do, and how you can help. Just like Marcella was describing they're doing at the Brooklyn Community Foundation. And then being sure to take action on that feedback, and provide that support beyond the check. 

RACHEL: And it seems to me that the idea of taking action and really listening with an open heart to feedback really relates into the next principle that you talk about, around humility. 

NATALYA: Absolutely. So number six here is to be humble. And part of this work, I believe, is accepting that you might not have all the right answers. Even if you've been in the industry or doing this work for a long time. For example, have you checked that what you determine to be the most important evidence of success matches with what your grantees consider them to be? 

A great example I found that [INAUDIBLE] gave in a piece I read recently about the concept of effectiveness is, let's say you're evaluating a mentorship program based on high school graduation rates. But what the community being served actually cares about most is that the kids are happier. So being open to the idea that your definition might not be the only definition, or even the best one, of success. 

And similarly, not all of the feedback that you get is going to be positive. And you just kind of need to make it your practice to accept that learning, and again, building for the future is an important positive outcome for the long term. Satonya Fair, put in conversation with Sam Caplan on this program, “You can be the President of the United States, you better still be learning.”

RACHEL: Absolutely. I mean I've heard you say many times, it makes so much sense for so much of this work around evolving or grant making processes. It's not a one-and-done. It's a long-term, ongoing, continual learning process. And then talk to us about principle seven. 

NATALYA: Definitely. So our last one here is apply an equity lens. And again, it's related to the humility point, there's so much overlap in all of these principles. But when you think about equity, it's important to consider that traditional methods of measurement are often the result of a white-dominant institution or evaluators. And may not reflect what is valued by those who are doing or benefiting from the work. 

And then furthermore, these traditional measures and rigid requirements, especially, will typically favor white-led nonprofits. And then with those white-led led nonprofits, best able to demonstrate those traditional measures, they're going to get most of the funding. And then the cycle perpetuates itself. So when doing equity work, I like to say, remember that it's work. 

So one pernicious aspect of white privilege is its invisibility, oftentimes to white people. So to make progress towards equity, we need to choose to see it and to dismantle white privilege. As I heard Erik Clemons, again at the Yale Philanthropy Conference say, to really achieve equity you need to share power. And I know Shaady uses that term as well, I believe. So it kind of, it comes to sharing power. It comes down to sharing power. 

RACHEL: Thank you so much, Natalya. This was really fantastic. I really have been really excited about your work, and I love that in your role at Submittable, because it's tied to the product, you can be thinking about how these principles that you're identifying, relate on the tech end. And vice versa, you know, like how we can improve Submittable's tech to better meet these principles. So thank you so much for talking with me today. 

NATALYA: Yeah, definitely. Thank you, Rachel. It's been a real pleasure. And I would just say for anyone who's curious to dive into this topic a little bit deeper, there are tactics that we identified that correspond with each of these principles. And if you want to watch the full webinar, you can dive into those as well. 

RACHEL: I highly recommend it. It was one of our most well-attended and engaged webinars that we've had in 2021. And it's a great learning experience. So check it out. 

NATALYA: Thanks so much, Rachel. 

RACHEL: Thank you. 

For more on impact measurement, be sure to visit the resources included in our episode notes. 

Thank you for joining us today. Impact Audio is edited and produced by Jordan Marvin and our crew at Submittable. Submittable is a cloud-based social impact platform designed to help your team make better decisions and have a bigger impact. We’d love to partner with you to maximize social good and create lasting change through smarter technology—find out more at Submittable.com. And until next time, take good care.