CHANGE Lessons learned the first year of the CHANGE Grants program at the Hawai‘i Community Foundation (HCF): A formative evaluation, commissioned by HCF, to inform and improve future grant-making

CHANGE Applicants

The Hawai‘i Community Foundation received 509 applications for CHANGE Grants funding.

HCF staff spent more than 340 hours with over 450 nonprofits across the state to listen and assess where the foundation’s resources can make the most impact.

The median annual operating budget for all applicants was $480,369 (mean = $3,318,076), with a minimum of $0 and a maximum of $178,000,000.

The median organizational age (in years as of January 2021) was 20 (mean = 28.2) with a minimum of 0 and a maximum of 175.

The median total staff size (full-time and part-time) was 7 (mean = 40.9) with a minimum of 0 and a maximum of 2000.

According to interviewees, several dimensions factored into stakeholders’ perceptions of the application experience:

Organizational bandwidth

Many interviewees spoke favorably about the streamlined application with clear questions, simple budget, and lack of attachments; they hoped for more grant applications like the CHANGE Grant application. An interviewee with an organization that was overwhelmed as they tried to adjust to COVID-19 said, “This was the most strenuous and difficult process that we’ve been through...Honestly, it was the worst experience.”

Organizations that decided not to apply or who were unhappy with the process felt overburdened by the application process. Several organizations spoke specifically about the struggle to apply as they were dealing with repercussions from the COVID-19 pandemic. “I am doing the work of three positions,” said an interviewee in the health sector. “For just anybody, COVID had an impact. But for people in health care, we were working 14-hour days.” Another person shared that their organization had struggled a great deal “to stay alive during COVID.” She felt that trying to understand the complexity of the CHANGE Framework and where their organization fit into it was too challenging during such a tenuous time. “When I have to read through a [CHANGE Framework] document,” she said, “I am taking time away from putting out fires in all other parts of the organization” during a pandemic.

Interactions with HCF staff

Interviewees’ direct experiences with HCF staff members affected their perception of the CHANGE Grant process. Most interviewees spoke about good experiences with staff. One stakeholder gushed, “[They are] the sweetest, most helpful, most on-it, most professional, most everything! We feel we have so much support. We have a person we can call to help. Can’t say enough about them!” At the opposite end of the spectrum, a stakeholder who had a negative application experience said she experienced trouble from the “customer service angle.” For other interviewees, they would have liked to continue working with program officers with whom they already had a relationship. One person provided this example: “We used to work with the same person, and they knew our projects so well that they may have been better able to help us fit our metrics into the CHANGE Grant metrics. But the person I talked to didn’t know us at all, and so they really couldn’t help us.”

Applicants felt that the CHANGE interviews served as an important measure of whether a grant would be received. If an organization thought the interview went well but then did not receive a grant, they were caught off-guard. If an organization thought the interview was lackluster and then did not receive a grant, they blamed the interview for the unsuccessful result. Still, almost all the people we interviewed appreciated the CHANGE Grant interviews. One person whose statement is representative of what we heard said, “The interview was really wonderful—the highlight of the CHANGE grant…The genuine interest in listening to what [organizations] do in a way they can share instead of a rigid box with character counts…It felt authentic and genuine.” Many people expressed a desire for more of this type of communication with HCF staff. One person said the interviews had the benefit of “humanizing the organization and the program officers.”

Clarity about criteria

The statement representative of the sentiment we heard about the criteria was, “It seems like they were more interested—I’m just guessing—in nonprofits that were already established. The big names with big groups of people involved. Groups that had a lot of funding in the past…I can understand that. But it would have been good to express that…We can’t compete with the folks who get paid hundreds of thousands just to write grants.”

Other groups were frustrated by the criteria to be current with all HCF grant reports. One person specifically recounted that some HCF grants did not require reporting during COVID, and she interpreted that as an HCF-wide rule. She was surprised when her organization’s grant proposal was rejected for noncompliance when she thought they were compliant.

The CHANGE Grants program was created to galvanize measurable improvement across six sectors. For each sector, HCF has curated data points, or metrics, to assess improvement. The CHANGE Framework metrics were a source of frustration for potential applicants. Most of the people we interviewed said the metrics were “irrelevant” to their work. Several nonprofit leaders wondered if any Hawai‘i nonprofits had been involved in developing the metrics because they seemed so disconnected to the work on the ground. The metrics requirement kept some groups from applying. For example, one group that decided not to apply because of the metrics said, “It’s a $50,000 grant, but getting the data is a $100,000 headache for us.” Another person, when asked why they decided not to apply to CHANGE, said: “The metrics. We’ve been using certain metrics for 10-plus years that have worked for other funders, but they didn’t seem to fit in with the CHANGE Grant metrics. We didn’t want to spend the time reinventing our metrics for just $10,000.”

Several stakeholders mentioned that in follow-up conversations with HCF staff after being denied funding, they learned that their data-gathering was not good enough. For example, one person recalled, “The feedback we got was that we weren’t advanced enough with data-tracking. And we put a lot of effort into our data-tracking, so I don’t understand it still…What are they looking for?” This person, along with others who heard similar feedback, said they thought the application materials should be clear about what criteria HCF is using to assess groups’ data-tracking capabilities.

Did the CHANGE Grants process enable and encourage smaller, lower-resourced groups to apply for funding? To receive funding?

HCF’s process enabled lower-resourced organizations to apply for CHANGE Grants funding at approximately the same rates of higher-resourced organizations.

Organizations with budgets in the highest quartile were more likely to receive an award, while organizations with budgets in the lowest quartile were less likely to receive an award. (This result was statistically significant using Pearson’s chi-squared test, X2 (3, N = 509) = 41.74, p = <.001). The age of organizations and staff size did not make a difference in whether they received an award.

Organizations with headquarters on O‘ahu were more likely to receive an award (X2 (6, N = 509) = 15.24, p = <.018). However, we also calculated the strength of relationship between service location (with the categories of Kaua‘i, O‘ahu, Maui County, Hawai‘i Island, statewide, or multiple), which did not return a statistically significant result. Some organizations with O‘ahu headquarters are serving communities and people on neighbor islands. This analysis did not consider where grant dollars actually would be used because that data was not available to HCF or Groundswell Services.

What did HCF learn about Hawai‘i nonprofits?

As HCF staff reviewed applications and conducted interviews with more than 500 Hawai‘i nonprofits, they gained invaluable knowledge about the CHANGE sectors. They learned about facilitators and barriers to success for these organizations, which we summarize below.

Facilitators

Nonprofit organizations often were engaged in a variety of strategies and activities that crossed multiple sectors. How can the CHANGE Framework encourage innovation and intersectionality?

Networks were important holders of space, often comprising the highest capacity members in sectors.

Nonprofit organizations communicated that technical assistance has supported their work, and they mentioned different types of assistance that could be helpful to them. The technical assistance they asked for was as follows:

• Support for advocacy, such as information about allowable and unallowable advocacy activities as nonprofit organizations; tools and training for virtual testimony; and tools and resources to advocate effectively at different political levels, including advocating for better agreements with government agencies.
• Network-building support, including support for developing collective goals and shared measures, in addition to strategies to foster systems change.
• Assistance and support for technology as it continues to evolve—for example, teleheath, recording-keeping, and outreach.
• Tools, training, and resources for monitoring and evaluation, including sufficient financial resources for this work; different technological tools for automated data-tracking; and how to leverage existing databases or data sources.
• Training and information about fundraising and organizational sustainability, including raising earned revenue through service contracts. This was especially relevant to small and medium-size organizations.
• Accessing and effectively managing federal funding, especially for larger organizations.
• Support for strategic planning, board training, and other nonprofit management needs.
• Support for staff development and training, including support to participate in broader national or international networks.
• Resources and support for outreach and communications.
• Support for workforce development to train and build a quality nonprofit workforce.

Barriers

Organizations of all sizes and in all sectors said they needed more resources to achieve their desired goals. Most organizations knew they were addressing symptoms rather than root causes, but they said they did not have the capacity to address root causes because the symptoms-related needs were so great. These needs have been exacerbated by the COVID-19 pandemic.

Multiple organizations in various sectors spoke about the difficulty of securing a qualified workforce. This was especially true for neighbor island organizations.

Organizations also described government as a barrier or problem. They discussed inequitable contracts, slow contracting processes, unreasonable data sharing policies, a lack of adequate investment, and policies that are openly hostile or harmful to their goals (such as policies that restrict arts in the schools, for example).

Organizations are challenged to collect data, adequately monitor their outputs and outcomes, and understand and communicate impact. When groups were gathering data, they were more often doing so for grant compliance rather than to assess effectiveness or desired change. Also, groups that were engaged in monitoring and evaluation typically were not using indicators conducive to or aligned with the CHANGE Framework. Finally, most organizations were not assessing equity or even able to describe how they addressed equity.