Developing an Evaluation Capacity Building Network in the Field of Early Childhood Development

This reflective essay traces the development of an evaluation capacity building network within the early childhood development field. First, we describe the context for building the network using a community-based participatory approach and provide rationale for our specific focus on early childhood development. Second, we provide an explanation of the purpose and processes involved in three areas of significant engagement: partner, stakeholder, and student. We reflect on the methods of engagement used across these three areas and their impact on the outcomes that we achieved. Finally, we conclude the paper with some final considerations for guiding engaged scholars and with the next steps in our own work.

2015; Pei, Feltham, Ford, & Schwartz, 2015;Viswanathan et al., 2004).Fewer examples exist of sustainable community-university partnerships and networks that encompass all aspects of CES (e.g., research, teaching and learning, student engagement, knowledge mobilization) with core resources (human and financial) from both university and community partners.Although less common, examples of such partnerships include the Community-University Partnership Program at the University of Brighton, Community-based Research Canada, and Participatory Research in Asia.In the province of Alberta, the Community-University Partnership for the Study of Children, Youth, and Families (CUP) in the Faculty of Extension at the University of Alberta represents another example.CUP was launched in 2000 through shared community and university leadership to improve the development of children, youth, and families by promoting interactions among researchers and community members (e.g., practitioners, policymakers, families) in the areas of research, knowledge sharing, and lifelong learning (Chapman, 2015;McCaffrey, 2007).CUP has the mandate to nurture environments where evidence is used effectively to develop practices, programs, and policies that support the healthy development of children, youth, families and communities across four priority areas: policy, poverty, early childhood development, and evaluation.
Over its 17 years of operation, university and community partners have shared joint responsibility for guiding and sustaining CUP, and are represented by CUP's Steering Committee.Reflecting the partnership, an academic and community member co-chair the CUP steering committee, and its membership is currently comprised of 27 members that represent foundations, municipal and provincial government, academics, research and policy centres, and community-based agencies.This governance structure has long provided the impetus for collaborative opportunities in CUP's key focus on evaluation and early childhood development from its many community requests for research, measurement, and evaluation support (Bisanz, Edwards, & Shaw, 2013).These requests have resulted in sharing resources, brokering relationships with other faculty and graduate students on campus, and developing participatory research and/or evaluation projects.However, with ever-increasing requests from community-based agencies that are not always accompanied by resources, it became necessary for CUP to determine a more systematic and effective way to respond to these needs.
This reflective essay provides an in-depth account of how community and university members of CUP spearheaded the development of a network to advance evaluation capacity in the early childhood development field, using a CBPR approach.First, we describe the context for the development of the Evaluation Capacity Network (ECN) 1 and the reason for focusing on early childhood development.This is followed by an explanation of the purpose and processes involved in three areas of significant engagement: partners, stakeholders, and students.We reflect on the methods of engagement used across these three areas and the impact of these methods on the outcomes that we achieved.Finally, we conclude the paper with some final considerations for guiding engaged scholars and with the next steps in our own work.

Addressing Community Evaluation Needs Through Partnership
CUP has received a steadily increasing number of requests from community agencies for assistance with research and evaluation (Bisanz et al., 2013).This reflects an increasing demand across Canada for evaluation.Funders, program planners, and policymakers are seeking rigorous and reliable evidence to inform resource allocation and improve essential services (McShane, Usher, Tandon, & Steel, 2015).However, the demand for evaluation currently surpasses the required resources and supply of evaluation knowledge and expertise available to many community agencies (Gauthier et al., 2010).This has placed community agencies in an untenable situation.With limited funding, human resources, and evaluation expertise to collect and use evaluation evidence, community agencies often struggle to justify continued support through evaluation (Bakken, Núñez, & Couture, 2014;Cousins, Goh, Elliott, & Bourgeois, 2014;Janzen et al., 2017).
To gain a deeper understanding of widespread evaluation capacity issues with the intent of developing an effective response, CUP hosted two focus groups in 2012.The 14 participants included leaders (e.g., CEOs, executive directors, and managers) from nonprofits, foundations, and provincial government, as well as professional evaluators and university academics familiar with evaluation related issues.What emerged from the focus groups was a locally-relevant understanding that community agencies find the process of evaluation challenging in common ways.These findings resonated with what others had already reported in the literature: funders often request specific evaluation methods and outcomes to meet their needs for accountability that do not realistically reflect organizational strategic learning goals, time, and resources (Carman & Milleson, 2005;Leviton, 2014); agencies find the process of evaluation challenging due to insufficient funding, and lack of human resource capacity (Bakken et al., 2014;Cousins, et al., 2014;Janzen et al., 2017); experts find it difficult to provide all the required resources, knowledge, and capacity to community agencies; and evaluation outcomes often prove uninformative for program development and practice.Common across the focus group participants was the urgent need for further dialogue among intersectoral stakeholders who support evaluation of programs.They also validated the need to create a central point where stakeholders could access coordinated evaluation capacity building resources, and ensure high quality training, practice, and research in evaluation.

Forming the Partnership
Focus group findings were presented to the CUP steering committee, and a working group was established to realize the evaluation initiative.The working group consisted of a foundation CEO, two executive directors of large nonprofit agencies, and an academic and research associate affiliated with CUP.The key task of the working group was to identify and bring together leaders (funders, nonprofits, academics, and government) from the social sector to form a partnership.The intent of the partnership was to develop and operationalize a robust, coordinated plan for increasing the availability of quality evaluation knowledge, resources, expertise, and tailored capacity building opportunities.In September 2013, a group of 18 leaders was brought together for a full-day meeting to determine (1) what the focus of the evaluation initative should be, (2) how to fund the initative, (3) what individuals and organizations needed to be involved, and ( 4) what steps the partnership should take moving forward.The group decided that the focus of the initative would be on evaluation in the early childhood development (ECD) field.Much of the group's discussion focused on the need to foster and support "evaluative thinking" in the ECD sector.Evaluative thinking has been defined as Critical thinking applied in the context of evaluation, motivated by an attitude of inquisitiveness and a belief in the value of evidence, that involves identifying assumptions, posing thoughtful questions, pursuing deeper understanding through reflection and perspective taking, and informing decisions in preparation for action.(Buckley, Archibald, Hargraves, & Trochim, 2015, p. 378).
To stimulate and support the intiative, several partners offered in-kind and cash contributions to support an application for a Social Sciences and Humanities Research Council Partnership Development Grant (SSHRC PDG), which we were successful in obtaining in 2014.As determined during the September planning meeting, the primary objectives of the SSHRC PDG were to (1) conduct an intersectoral needs assessment using community forums with the aim of identifying common evaluation knowledge (i.e., language, metrics, methods, theories and issues) and capacity gaps; (2) develop and deliver educational resources and training opportunities that address these gaps and to subsequently evaluate and refine the resources and training; and (3) nurture and sustain an Evaluation Capacity Network that supports ongoing dialogue of evaluation experts, government, funders, and community agencies at a national level, and knowledge mobilization of community-engaged evaluative practices across the range of sectors that impact ECD.The partnership now had a clear focus on building a network to advance evaluation practice in the ECD field, and the necessary funding and committed leaders to move things forward.

Rationale for Focusing on Early Childhood Development
The rationale for the partnership's focus on evaluation in ECD was based on several considerations.First, robust scientific evidence in the areas of child and family health and wellbeing demonstrates that experiences and environments in the early years profoundly impact children's development (Akbari & McCuaig, 2017;Hertzman & Boyce, 2010;Shonkoff & Levitt, 2010;Shonkoff, 2017).Incorporating multiple sectors and systems, the field of ECD is interdisciplinary and complex yet provides critical opportunities for innovations in social policy and practices.When policies and programs that target the early years are responsive to the complexity of the field, they can reduce expensive interventions in later years (Akbari & McCuaig, 2017;Duncan et al., 2007;Heckman, 2008;OECD, 2012;Shonkoff, 2017).Despite significant investments in early years programming, approximately one in four Canadian children lack the social, emotional, and cognitive capacities to benefit from the public education system (CIHI, 2014).This rate doubles for Indigenous children and English-language learners of immigrant and refugee backgrounds, who experience social vulnerabilities as a result of economic, cultural, and language differences (Cabrera, 2013;Georgis, Gokiert, & Kirova, 2018;Gokiert et al., 2014;Government of Canada, 2011).The prevalence of mental health disorders among children in Canada is about 13% (Waddell, Shepherd, Schwartz & Barican, 2014), and roughly a quarter of Canadian children are living in low-income households (Statistics Canada, 2011).Furthermore, in 2016 UNICEF ranked Canada 26th of 35 high-income countries for child well-being.Such statistics indicate there is a pressing need for improvement in existing Canadian child-and family-focused policies and practices.
Second, policy frameworks have been emerging for over a decade that encourage a common understanding of development in the early years, promote shared language and outcomes, create continuity across jurisdictions and settings, and serve as a resource to support program and policy development (CMEC, 2014;Government of Alberta, 2013;Government of Manitoba, 2013;Government of NWT, 2013;Munro, 2006).Unfortunately, these policy frameworks tend to create unintended complications for the early childhood system.Funders adopt these policy frameworks, and do not always provide clear expectations on how to use them or contribute to them in meaningful ways.This leaves the public sector collecting considerable amounts of data that result in a significant "data burden," as nonprofits invest resources beyond their means to produce performance data that is of little use to both the organizations producing the data and the funders requesting it (e.g.Snibbe, 2006;Carman, 2010;Leviton, 2014;Liket, Rey-Garcia, & Maas, 2014).
Finally, there is no common approach to quality evaluation knowledge, resources, expertise, and capacity-building opportunities tailored to the needs of the intersectoral and interdisciplinary nature of the ECD field.This void compromises high-quality research, training, and practice in evaluation, and ultimately impacts the programs and practices in ECD.A coordinated approach was the solution, and so we extensively engaged partners, stakeholders, and students to understand the needs and assets of the ECD field.In the next sections, we will explore our process of developing the Evaluation Capacity Network (ECN) through three distinct but related themes of engagement: (1) partner engagement, (2) stakeholder engagement, and (3) student engagement.

Partner Engagement: Developing a CBPR Partnership for Evaluation
As the ECN is grounded in a CBPR approach (Israel et al., 1998), it integrates research, action, reflection, and communication.As such, a partnership was the first step towards building the ECN because it provides principles and methods to guide the work of the network, can stimulate intersectoral and interdisciplinary dialogue, and can ensure that community needs and values are at the foundation of the network.Partners were carefully chosen for the important expertise they had, and the role they could play in working together to recognize and address the complexity of intersectoral evaluative thinking.Partners represented stakeholder groups whose definition(s) of evaluation effectiveness, practices, and outcomes are influenced by the sector within which they worked and the role they played in supporting ECD programs, practices, or policies.

ECN Partnership Governance
The partnership that was developed and now sustains the ECN comprised three main governing bodies and involves the authors in various roles: a steering committee, core research team, and project management team (see Figure 1).The 19-member Steering Committee had representation from government, community agencies, funders, evaluation consultants and academia.The Steering Committee was the decision-making body for the ECN and provided high-level guidance, support, and direction for partner engagement, research, data interpretation, knowledge mobilization, and funding initiatives.The Steering Committee comprised 19 members, including academics (Gokiert, Kingsley), funders, government representatives, evaluation consultants, and nonprofit representatives.The Core Research Team met monthly and was responsible for the design, development, and implementation of the research component of the ECN.The Core Research Team comprised nine members, inclusive of an executive director from a national nonprofit (Hopkins), a government representative, five academics (Cor, Gokiert, Poth, Springett), and one postdoctoral fellow (Kingsley).The Project Management Team met weekly and was responsible for the development and monitoring of the plans, schedules, budgets, and deliverables of the ECN within the established time frames and quality guidelines approved by the Steering Committee.The Project Management Team comprised the principal investigator (Gokiert), a postdoctoral fellow (Kingsley), graduate research assistants (El Hassar, Tink, Tremblay), practicum students, and student volunteers.

Engaging Partners through Focus Groups
To develop guiding goals, common principles, indicators of success for the partnership, and a framework for building a provincial evaluation agenda, we hosted two focus groups with 16 individuals across the levels of governance.The questions posed were: What was the original reason you agreed to participate in the ECN and what do you hope to gain from your involvement?What are the essential elements of the community-university partnership that will contribute to developing a successful ECN?What would success look like for the ECN?The focus groups were audio-recorded, and three graduate research assistants took extensive notes.Members of the Project Management Team completed a thematic analysis of this data and organized the information into core partnership principles, project outcomes and particular actions that partners described wanting from the ECN.
The principles that arose from the focus groups were highly reflective of the guiding principles of a CBPR approach (Israel et al., 1998).They included the need for trust, mutual benefit, equity, co-creation, accessibility, collaboration, strong communication, a commitment to action, and engagement at all levels.The focus groups provided an important opportunity not only to identify key principles to guide the partnership, but to also ascribe meaning to these principles in the context of the ECN.
The partners identified several immediate and intermediate outcomes that were important to guide the ECN towards the ultimate outcome articulated through focus groups: children are provided the best possible start through evaluation-informed policies and practices.Immediate outcomes included the need to increase awareness and understanding of the purpose of the ECN, increase the perceived importance of evaluation, and increase evaluation capacity through elevated evaluative knowledge and practice.Intermediate outcomes included the need for a culture that values evaluation, risk-taking and transparency, an increase time and financial investments in evaluation, evaluations in ECD that are relevant and meaningful, and the integration of evaluation into organizational processes through a utilization focused approach.
Finally, a number of actions were identified that partners felt would contribute to the success of the ECN.Actions included the need to engage and connect current and future evaluators, users of evaluation, and evaluation capacity builders; provide evaluation mentorship and expertise; develop education tools and resources; develop common evaluation language through effective communication mechanisms; provide professional development and learning opportunities; and share best practices in evaluation.For these actions to occur, it was determined that there would be a need to mobilize and create synergy between the levels of governance and the broader membership of the Network.

Challenges and Opportunities in Partner Engagement
The focus group process was beneficial as a means of engaging partners in a dialogue about shared principles, outcomes, and actions that they wanted for the ECN.The focus groups also generated ideas, enthusiasm, and a shared commitment for the ECN that otherwise might not have happened.Despite the positive impacts of the focus groups, maintaining a concerted level of engagement over time and adhering to the principles that were identified by the partners has been challenging for a number of reasons.For one, the breadth of the project -engaging partners from across so many contexts (universities, funding agencies, nonprofits, government, policy and research centres and consulting firms) -has, to some extent, affected the depth of engagement that we were able to achieve.Having 19 people on the steering committee alone limited the extent to which face-to-face meetings could be held and rich discussion could be facilitated.In addition, many of the steering committee members were high-level decision makers and leaders in the field.As a result, finding meeting times when everyone was available and expecting significant engagement in the ECN was difficult beyond meeting once per year and communicating project milestones through email.
Although we initially intended the Steering Committee to be the primary decision making body for the ECN, the Core Research Team took on many of the responsibilities of the Steering Committee and provided significant expertise in measurement, evaluation, and engagement.Although not initially intended, the Core Research Team members suggested meeting monthly to gain momentum on the project.The Steering Committee still functioned, however, with a more distal role than we originally intended, with strategic thinking, funding opportunities, and provincial connection being the main areas of focus.To compensate for this limited engagement, we have instead engaged with members of the Steering Committee on an individual basis depending on the task at hand or the stage of the project, reflecting a need to connect on a more personal level.

Stakeholder Engagement: Understanding the Evaluation Needs and Capacities of the Field
Building an agenda for advancing evaluation practices in the field of ECD required determining evaluation needs and assets, which in turn required extensive stakeholder engagement.Stakeholder, in this sense, refers to individuals who might be affected by decisions about an evaluation agenda (Freeman, 2010) -namely funders, evaluators, nonprofit and government employees, and academics.Consistent with a CBPR approach, we wanted to develop an agenda that had mutual benefit across the ECD field (Israel et al., 1998).It was therefore essential to understand the needs and assets of the field from the perspectives of stakeholders working across a range of contexts.A subsequent goal of stakeholder engagement was to gain a wider sense of ownership and momentum across the ECD field towards working collectively to address the evaluation needs that exist.
To accomplish the needs and assets assessment, we used three primary methods of stakeholder engagement across the province of Alberta: surveying stakeholders, priming stakeholders and consulting stakeholders.The methods were complementary, in that an evaluation capacity building survey and stimulus paper were sent to the participants in advance of attending the forums, and the survey results and stimulus paper were used to spark discussion throughout the forums.We will describe each method of engagement in more detail with its contribution to understanding the needs and assets of the ECD field.

Evaluation Capacity Building Survey
It is important to recognize that evaluation is not simply about developing in people and organizations the capacity to do evaluation but also about developing the capacity to use evaluation (Cousins et al., 2014).The construct of evaluation capacity has been conceptualized and defined in the literature and operationalized in the form of assessment tools (e.g., Labin, 2014;Preskill & Boyle, 2008;Taylor-Ritzler, Suarez-Balcazar, Garcia-Iriarte, Henry, & Balcazar, 2013).However, many of these frameworks and tools are not context specific, making them less reflective of the complex field of ECD and thus were not appropriate for understanding the individual and organizational capacity of our stakeholders.As a result, it was necessary to draw on these frameworks and tools, through an extensive review of the literature and to develop a survey that better reflected our context.A doctoral student (El Hassar) took a leadership role in developing the survey as part of her doctoral thesis.She engaged the Core Research Team at several points in the development process such as determining the most contextually relevant survey components and items, measurement scales, and survey format (online).The final survey consisted of items across three main areas of evaluation capacity: individual capacity (e.g., attitudes, motivation, knowledge, and skills), organizational capacity (e.g., leadership, organizational processes, and available resources), and training and professional development (e.g., training experiences, desires for capacity building).Stakeholders that were invited to the evaluation capacity building forums (described below) were purposefully selected based on experience, expertise, or leadership in evaluation and/or early childhood development in the province of Alberta.The survey was sent out to these stakeholders prior to their attendance at the provincial forums.Approximately 164 surveys were sent to invitees, and a total of 101 surveys were returned.As mentioned, the resulting data were presented back to participants at the forums.

Evaluation Capacity Building Forums
In total, the ECN hosted four forums across the province of Alberta in Winter/Spring 2016 as a way to further understand the collective evaluation needs and capacities of the ECD field.The specific purpose of the forums was to engage influential stakeholders in conversation about the evaluation barriers, facilitators, and needs experienced in their work and to generate innovative and collective solutions for addressing them.A total of 164 leaders from 78 different organizations (funding, government, nonprofit, university, and consulting) were invited, and a total of 122 attended.A large facilitation team guided and stimulated discussion during each forum and comprised a mixed group that mirrored the diversity of the stakeholders.The team included current and retired leaders in the ECD field from school boards, funding agencies, and nonprofits; academics (Gokiert & Kingsley) and graduate students from CUP (El Hassar, Tink, & Tremblay); and a highly-respected consultant in the social sector.For each forum, there was a primary moderator who led the agenda and several co-facilitators sat at each discussion table to guide conversation, take notes, and report back to the larger group.The agenda for the forums was co-created by the Core Research Team, the Project Management Team, and the same external consultant who aided the facilitation.
To provide contextual grounding, shared language and definitions for the forum, and to spark some initial ideas, we developed and sent a stimulus paper to all the invitees (Evaluation Capacity Network, 2015).The stimulus paper comprised five main sections that situated evaluation capacity building in the field of early childhood and asked participants to think about the possibilities for advancing meaningful evaluation in the field.Discussion questions were integrated throughout and provided in an accompanying worksheet to generate ideas in preparation for the forum (e.g., What ECD evaluation framework, if any, have you found most useful and why?What characteristics does your organization have that support an evaluation learning culture?).The stimulus paper was one of several engagement techniques implemented with the aim of stimulating innovative, change-focused ideas.We also used a combination of small and larger table discussions, group plenaries, a world café (www.theworldcafe.com), and a design thinking challenge (Stanford University, 2017).The facilitation team made modifications to the agenda through a reflective debrief after each forum.
The information shared during each forum was captured on participant worksheets that were given to stakeholders at the start of the day, in addition to sticky notes used during some of the group activities.This information was electronically inputted and a basic thematic analysis was conducted to organize the ideas.These ideas were presented as three action areas for the ECN to focus on to advance evaluation in the field of early childhood and were reported back to stakeholders in a "What We Heard" summary document (Tink, Kingsley, & Gokiert, 2016).In addition, feedback from stakeholders during the forums indicated that they wanted us to send the survey out more broadly in order to reach all levels of an organization, from frontline staff through to the leadership.With this advice, we modified the survey to be reflective of a more diverse audience and redistributed it.We sent the survey to all participants and asked that they forward it through their organization to staff using a snowball technique of sampling.The second administration generated over 329 responses.The survey data was reported back to stakeholders in a report that is posted on the Evaluation Capacity Network website (Tink, Gokiert, Kingsley, & El Hassar, 2017).
Authentic stakeholder engagement is not an easy or straightforward process, even with substantial experience.For this reason, we relied heavily on pre-existing, well-established, and trusting relationships with pivotal stakeholders and organizations across the province (developed over the past decade of CUP's history).One of the ECN partners, the Muttart Foundation, had carried out a series of forums across Alberta and Saskatchewan with the ECD sector to determine a system of early learning and care (Muttart Foundation, 2013).We reached out to the Muttart Foundation for advice about their engagement process, costs, ideal locations and venues, skilled facilitators, and the leaders they had previously invited.It was under their guidance that we recruited facilitators in each region as they had the relationships necessary to increase the likelihood that stakeholders would participate.The Foundation used stimulus papers for each of their forums, and so we adopted this idea and created a stimulus paper to contextually ground our forums.They also provided insight into the professional cultures of each location so we could adapt our style of engagement.For example, one of the larger cities in which we hosted a forum had more of a 'corporate' culture, which was quite different from the 'relaxed' culture of one of the smaller rural locations.We made subsequent decisions about the locations based on these differences, with the larger urban forums held at a conference centre and hotel in the downtown core, compared to the rural forums which were held at a public library and a school.
We wanted the forums to be change-oriented rather than deficit-focused.With this in mind, we purposefully invited leaders who we believed were driven to find unique solutions to complex problems, and possessed the influence to effect such change.Despite these intentions, the extent to which this made a difference to the quality of the discussion is difficult to gauge.In focusing on inviting change-oriented leaders, we also restricted the number and breadth of perspectives shaping the direction of the ECN, particularly those of frontline staff and service providers.
Positioning the forums around change created expectations from our stakeholders for action to result from the discussions.This was exacerbated by the fact that a number of other consultations had been recently conducted in the field.For this reason, it was important to provide time-sensitive reports to summarize learning, and to find a balance between community need and academic definitions of rigour (Kingsley & Chapman, 2013).In this process, our ability to respond to demands for action has been impacted by our own limited resource, time and expertise, which must grow significantly if we are to adequately respond to the needs presented during the forums.This is further tested by the interest and awareness generated through the engagement process.Raising the profile of the ECN through public dialogue has also resulted in an increased number of requests for evaluation support and resources.In an attempt to meet this need, we have drawn upon graduate students as a source of capacity for the ECN, the details of which are described in the next section.

Student Engagement: Training and Mentorship
Graduate student engagement has been fundamental to the success of the ECN, as they have provided significant capacity.Some students have completed their masters' and doctoral research with the ECN, while others contributed through independent studies, research assistantships, practicums and course placements, and volunteering.For example, two students joined the project management team to fulfill a 150-hour practicum as part of an embedded graduate certificate in community-based research and evaluation.Two graduate students from the Master of Arts in Communication and Technology program in the Faculty of Extension developed a communications plan and website for the ECN as part of an independent study course.One of these students became further involved in the ECN and completed her final capstone project with the ECN.Two doctoral candidates are pursuing their dissertation research focused on different research questions relating to the ECN (El Hassar, Poth, Gokiert, Kingsley, & Krishnan, 2016;Gokiert et al., 2017;Tremblay et al., 2016).In the three-year span of the ECN thus far, eleven graduate students and one postdoctoral fellow have been mentored through, and provided support to, the ECN.Mentorship is generally provided by academics that are community-engaged scholars themselves and/or who specialize in evaluation.

Emerging Community-Engaged Scholars
The intent of engaging students in the ECN is to support their development as communityengaged scholars and evaluators.The focus on such enhanced learning opportunities is based on a growing need for professionals who are equipped to navigate a social sector with complex challenges (Armitage & Levac, 2015;Cantor, DeLauer, Martin, & Rogan, 2015).Students seem to be recognizing this need to expand their learning and build new skills, which has resulted in an increasing number seeking community-based research and evaluation opportunities.These emerging scholars are acquiring skills, knowledge and experiences that would not be available through conventional academic training.For example, students are exposed to several research methods in an interdisciplinary context, and developed skills in knowledge mobilization and engagement with academic and non-academic audiences.More specifically, they have facilitated meetings and forums, produced grey literature and scholarly manuscripts, and developed and maintained the ECN's online presence through email campaigns and blog posts.

Community Service Learning
In addition to preparing students as community-engaged scholars, the ECN also supports students to develop their skills as evaluators.As mentioned, there is a substantial need for evaluation support in the social sector, which was reiterated by the information gathered during the forums.To meet this need, while providing experiential learning in evaluation for students, service learning placements were established.Service learning is a method to enhance students' learning and development through organized service experiences in the community, and is integrated into academic curriculum (Taylor et al., 2015).Service learning, particularly in community-based participatory projects, is widely considered to expand the student learning, providing opportunities to develop critical thinking and a sense of civic responsibility (Taylor et al., 2015).Service learning also provides additional capacity to the organizations in which the placements occur.Learning opportunities are therefore intended to be of equal benefit to both the student and the recipient of the service (Furco, 1996).
The service learning placements in this project are embedded within three graduate-level evaluation courses at the University of Alberta: Health Promotion Planning and Evaluation in the School of Public Health, Program Evaluation in the Department of Educational Psychology, and Program Planning and Evaluation in the Department of Human Ecology.To fulfill their course requirements, graduate students are asked to work with community organizations to build contextually relevant evaluation plans.To build a comprehensive plan, they meet with organizations several times to gain a contextual understanding of their evaluation needs, develop a logic model and gain feedback from staff.The course instructors mentor the students to enhance learning and serve as a form of quality assurance for the community and government agencies.This approach has been generally positive; however, there is a need to further extend this service learning to assist organizations with evaluation beyond the creation of an evaluation plan.To this end, advanced courses in evaluation are required that involve more extensive and concerted student engagement over time.This need to build the evaluation capacity of university students in Canada is well supported in the literature (e.g.McDavid & Devine, 2009;McShane et al., 2015), and will become a focus for the ECN moving forward.

Funding Graduate Students
Due to limited funds to engage graduate students in the ECN, we have sought other funding opportunities to supplement student involvement.Several funders have supported the ECN's student engagement and created various learning opportunities.The Women and Children's Health Research Institute (WCHRI) based at the University of Alberta has several student funding streams including the Patient and Community Engagement Training program and conference travel.Through this program, graduate students and postdoctoral fellows conducting engaged research are funded for 8 months to support their work.In each funding year, a community of practice is formed to discuss issues relating to engagement.One doctoral student (El Hassar) and one postdoctoral fellow (Kingsley) have participated in the training program and received travel grants to present their research at conferences.Mitacs, a Canadian funding agency that builds partnerships between academia and nonprofit or industry partners, has also provided substantial graduate student funding support.Mitacs funded a postdoctoral fellow (Kingsley) for three years, two doctoral (Tremblay) and one masters' student up to one year, to assist community agencies with evaluation and evaluation capacity building.
Graduate students have added significant capacity to the ECN; however, we have also run into some challenges with a project reliant on such extensive student involvement.Finding funding for students has been difficult at times and as mentioned, it was necessary to seek alternative forms of funding.Applying for this funding consumes available resources such as time.In addition, students -no matter their level of experience -require mentorship, or at the very least, management and supervision.Allocating university mentors or supervisors who have sufficient time to work with students, above their regular academic responsibilities, is also challenging.For this reason, much of this workload falls on the project management team, and is the reason that having a postdoctoral fellow involved in the project has been essential for sharing the supervision load of students.
An additional issue relates to student retention, as most students are only available on a short-term basis, either because they finish their degrees, accept alternative opportunities, or because our ability to fund them diminishes.Although we have had some students volunteer for us in the past, we would prefer to be able to pay individuals for their time.Finally, as with all CBPR projects, there can often be a dissonance between community and academic timelines.Graduate students have milestones they need to achieve before they can progress with their research (e.g., a candidacy exam), yet the project needs to progress regardless.This can put additional pressure on the student during what is already a stressful process and has the potential to prevent the project from moving ahead as planned.Similarly, academic standards and expectations that inform student processes and products are sometimes at odds with the flexible, responsive approaches needed when partnering with community.For example, one of the students (El Hassar) took the lead on the development of the evaluation capacity survey as part of her dissertation and engaged partners during this process.In addition to reflecting conventional standards of validity, this engagement (and the tool itself) also reflects an alternative form of rigour that more closely reflected a community-based research approach (Kingsley & Chapman, 2013).To allow for a broader conceptualization of rigour, it is important to form graduate committees with faculty who have some understanding of community-based and engaged research to avoid potential friction between academic and community needs and expectations.

Concluding Remarks
The value of co-created understandings is highlighted by the evaluation-focused efforts of McShane and colleagues (2015) who state, "community engagement is often touted as a goal for universities and community collaboration is increasingly viewed as favourable in research" (p.149).Evaluation experts have long recognized the importance of the interests, views, involvement, needs, and roles of all stakeholders in evaluation practice and theory (e.g., Alkin, 2004;Cockerill, Myers, & Allman, 2000;Cousins & Earl, 1992).Using a CBPR approach and through multiple forms of engagement, the ECN was developed to bring together many voices from across academic, government, nonprofit and consulting contexts around the issue of evaluation in the field of ECD.
The purpose of this reflective essay was to provide an in-depth account of the development of the network and reflect on our engagement processes.Illustrating the context for the development of the ECN, we provided a detailed description of the purpose and process of engagement with partners, stakeholders, and students in the development of a provincial agenda for the ECN, reflecting on each method and its impact on the outcomes we were able to achieve.In these concluding remarks, we offer some final considerations for communityengaged scholars relating to these three forms of engagement, provide details of our intended next steps for the ECN, and invite individuals and organizations to contact us for more detailed information.
In a summary of learning from a conference co-hosted by Community-Campus Partnerships for Health (CCPH) and the University of Guelph, Wenger, Hawkins, andSeifer (2011, 2012) articulated the need for flexibility in community-engaged scholarship.Through our engagement with ECN partners, we also learned early on that we needed to be flexible in our approach.Despite a comprehensive plan for governance -three committees with specific roles, responsibilities, and expectations -it did not play out in practice as we originally intended.This did not appear to negatively impact the quality of the process, but did require us to adjust our engagement expectations and modify our collaboration processes to align with these.To be respectful of partners' time, we only convened the entire group for specific and necessary purposes and instead met one-on-one to access specific expertise on an ad hoc basis.It is difficult to know the effects of this shift, and it may have led to limited ownership and learning amongst those less engaged.To access this information, it would be beneficial to ask our partners if this reflected their preferences and how it may have impacted their involvement.
Relating to stakeholder engagement, the role of CUP was pivotal in providing a stable foundation for reaching out to community and government agencies and receiving such a positive and enthusiastic response.With CUP's 17-year history as an effective relationshipbuilder, in developing the ECN we drew heavily on CUP's existing relationships as a trusted organization within the province's social sector to determine scope and pool resources.The benefits of this are unsurprising since trust is identified as a fundamental pillar of authentic community-based research partnerships (e.g., Cargo & Mercer, 2008).Key partners, such as the Muttart Foundation, which we relied on for guidance in our provincial forums, were critical in avoiding missteps, gaining momentum quickly, and increasing our reach substantially.Through our stakeholder engagement process, we also identified a number of assets that have led to new evaluation-related opportunities in the community.Finally, engaging students was a mutually beneficial process for the ECN and for the students themselves.Students, through research assistantships, practicums, course-based learning, and volunteering, provided much needed capacity.In return, students themselves were mentored as community-engaged, interdisciplinary scholars who were better equipped to respond to "wicked problems" (Cantor et al., 2015, p. 407).Engaging students to the extent that we did helped to highlight the pressing need to build better infrastructure to support student engagement.Although there are a number of courses through which students are connected to the ECN, there is currently a significant lack of evaluation courses in Canadian universities generally (McDavid & Devine, 2009), and the University of Alberta specifically (Bisanz et al., 2013).This makes both the training and recruitment of students who have evaluation experience difficult and limits the supply of students available to the ECN.In addition, hiring students or facilitating practicum placements requires mentorship, supervision, and management.This is currently an area of lack for the ECN, with most of the supervision falling on the principle investigator and a postdoctoral fellow on the project.This will not be sustainable over time and is an area that requires attention.
To our knowledge, the ECN is the first initiative of its kind to use a systems approach to build evaluation capacity that extends beyond the scope of a particular organization.To effect change, we believe a broad systems approach is essential to mobilize influential players from across the entire early childhood field (Sanderson, 2000;Suárez-Herrera, Springett, & Kagan, 2009;Waldrop, 1992).Doing so will not only foster a collective and coordinated effort to build capacity, it will increase the likelihood that the questions asked of evaluation are valuable to the field as a whole and will be more effectively used to support improvements to early childhood policies and practices.In addition, while this broad reach is necessary, we must also develop resources and opportunities that are tailored to different users and therefore contextually relevant.As such, the ECN aims to be, simultaneously, a broad yet localized approach.This has required us to draw on the various forms of engagement described in this paper.However, fostering deep relationships with partners remains a challenge with a project of this scale and an area requiring further attention.

Next Steps
The Evaluation Capacity Network (ECN) was developed to support dialogue among intersectoral stakeholders and create a central point through which stakeholders could access coordinated evaluation capacity building resources tailored to the field of ECD, and ensure high quality training, practice, and research in evaluation.The ECN has provided, and continues to provide a mechanism for fostering dialogue among academics, funders, government representatives, evaluation consultants, and nonprofit representatives.It also provides a way to more easily share and develop capacity building resources, expertise and opportunities.Through our engagement with partners, stakeholders, and students, we have developed a provincial agenda for the ECN that we intend to implement over the next several years, while continuing to expand the network and establish new partnerships in Canada and internationally.It is also through engagement that our learning and those we engaged with remain in motion and challenge us to adapt in new ways.In light of this, it is important to acknowledge that engagement is not an inherently necessary or equally advantageous process.Although the various forms of engagement through the ECN have helped to generate interest and develop a mutual agenda, it will be essential as we move forward to continually assess whether the ways we are engaging are meaningful and appropriate.To evaluate our partnership and our engagement processes systematically, we will use multiple methods and tools (e.g. the PARTNERtool; Varda, Chandra, Stern, & Lurie, 2008), which will allow us to respond to unanticipated challenges throughout the life of the partnership.
With an increasing number of institutions attempting to distance themselves from an ivory tower status (Furco, 2001), the concept of engagement appears to have gained significant momentum across the academy.There is a subsequent need for caution when making decisions about the forms and extent of engagement appropriate in each research project to avoid tokenism (at best) and tyranny (at worst) (Cooke & Kothari, 2001).We commit ourselves to this ongoing deliberation and invite others to join us in a collective effort to engage carefully.

About the Authors
Rebecca Gokiert, PhD, R.Psych, (corresponding author) is an Associate Professor and Associate Director, Community-University Partnership (CUP), University of Alberta.Her teaching and research is focused in community engagement and participatory research in cross-cultural early childhood contexts.She is a registered psychologist in the province of Alberta and works in partnership with many school divisions.Email: rgokiert@ualberta.caJane Springett, PhD, is a Professor in the School of Public Health at the University of Alberta.She has expertise in participatory practice, network and system development, and realist approaches to synthesis and evaluation.She brings over 25 years of international evaluation experience and graduate supervision to the ECN.

Figure 1 :
Figure 1: Evaluation Capacity Network Governance

Bethan Kingsley ,
PhD, is a postdoctoral fellow with the Community-University Partnership (CUP) at the University of Alberta.Bethan's current focus is to build and support the Evaluation Capacity Network, with a keen interest in community-based research and evaluation.Her doctoral dissertation explored marginalizing practices in recreation for young people living with lower incomes.Cheryl Poth, PhD, CE, is an Associate Professor in the Centre for Research and Applied Measurement and Evaluation (CRAME), University of Alberta.Cheryl has expertise conducting research and teaching in the areas of program evaluation, mixed methods and qualitative research, classroom assessment and health sciences education.Karen Edwards, Med, is the Director of the Community-University Partnership (CUP) at the University of Alberta.In her role, she fosters ongoing network and partnership development, facilitates the implementation of the organizational strategic plan, brokers requests from network partners, and pursues funding and resource development.Btissam El Hassar, MPP, is a doctoral candidate in Measurement and Evaluation through the Centre for Research and Applied Measurement and Evaluation (CRAME), University of Alberta.Her dissertation is on the development and validation of an evaluation capacity instrument that is contextually relevant to the early childhood development field in Alberta.Lisa Tink, MA, is a doctoral student in the School of Public Health at the University of Alberta.Prior to pursing a doctorate degree, Lisa held numerous positions in the non-profit sector, provincial government, and the academy.Lisa's work focuses on how historical social and political arrangements create the conditions for policy development.Melissa Tremblay, MSc., is a doctoral candidate in the School and Clinical Child Psychology Program at the University of Alberta.Melissa has been a graduate research assistant with the ECN project management team.Through her doctoral research, Melissa is examining the impact of a supportive housing program for teen parents.