The Utilization of Process Evaluations in Childhood Obesity Intervention Research : A Review of Reviews

Process evaluations are an essential component to evaluating health promotion programs, however they are consistently under-utilized and oftentimes not reported upon in the literature. This study reports the use of process evaluations in childhood obesity prevention interventions implemented over the past three decades. Seven metaanalyses and systematic reviews were located for this review or reviews, and from these, 119 unique references were identified. Each article was retrieved and read for appropriateness, and 20 were excluded for a variety of reasons (ex. not published in English language), resulting in 99 articles included for this study. Overall, process evaluations were not well reported upon. Only 38 studies reported the fidelity of program implementation, 25 studies tracked participant attendance, 29 studied evaluated participant satisfaction, and 49 studies reported how staff members were trained. Additionally, one-third of the studies did not report using a single type of process evaluation, and only 5 studies reported using all four types. Results from this study suggest that the use of process evaluations has been low in this area of research, which may explain why many obesity prevention studies have reported mixed or modest results. Suggestions for implementing simple, yet effective process evaluations in future studies will be presented.


INTRODUCTION
Obesity is a major public health concern, which threatens the health of our current generation and those to follow.Medical consequences of obesity have been well documented, and include premature mortality and greater risk of type-2 diabetes, cardiovascular disease, sleep apnea, asthma, non-alcoholic fatty liver disease, and many forms of cancers [1].Psychological consequences have also been found and include bias, discrimination, social marginalization, low self-esteem, and depression [2,3].In response to this epidemic, a growing body of research has amassed pertaining to issues surrounding causative factors related to obesity including those at the genetic, behavioral and environmental levels, effective treatments of obesity, including behavior change therapy, drug therapy, and weight loss surgeries such as gastric banding or bypass, and the prevention of obesity, including health promotion and health education interventions.
With regards to the prevention of obesity, many literature reviews have been written in previous years, including systematic reviews and meta-analyses, and suggest that school-based and community based health promotion interventions have experienced varying degrees of success.For example, in one metaanalysis, researchers analyzed 19 school-based interventions published from 1995 to 2007 and found that programs were overall significantly effective in reducing the prevalence of obesity, but treatment interventions were not effective when compared to a comparison or control intervention [4].Additionally, one meta-analysis on obesity prevention programs delivered in the school environment found that when compared to control groups, treatment interventions had a significant, but small overall effect on BMI (r=0.05;p<0.001) [5], however in another metaanalysis, researchers reported that school-based obesity interventions had no impact on BMI [6].
While much research has focused on the efficacy and effectiveness of obesity prevention interventions, and there ability to modify important impact and outcome measures, such as BMI-percentile, obesogenic behaviors, behavioral antecedents, and overall quality of life, less has focused on process evaluations.It is commonly understood that process evaluations are important in health promotion research, as it appears in many, if not all, major community planning models.For example, conducting a process evaluation is Step 6 of the PRECEDE-PROCEED model [7], part of Step 6 of the Intervention Mapping model [8], and is included in the Multilevel Approach to Community Health Model (MATCH) and the Planned Approach to Community Health (PATCH) [9,10].The term process evaluation refers to measuring the extent to which an intervention is delivered according to how it was planned or conceptualized.As such, the goal of process evaluation is to carefully document how much and how well an intervention was delivered to its intended audience.While there is no fully agreed upon set of criteria that defines a complete or proper process evaluation, many have proposed essential targets that are crucial for process evaluations, including program fidelity, attendance, and audience satisfaction.Given the lack of attention process evaluations have received in popular literature reviews, and potential that process evaluations have in interpreting the results of intervention studies, the purpose of this study was to serve as a 'review or reviews' in order to determine the extent to which process evaluations have been reported upon in the previous 30 years, and discuss the implications underreporting process evaluations may have in our understanding of how effective obesity prevention programs have been.

METHODS
Given the large amount of intervention studies that have been published in area of child obesity prevention in past decades, a number of popular meta-analyses and systematic reviews were located using the databases Academic Search Complete, CINAHL, ERIC, PUBMED, and SPORTDiscus.Overall, seven review articles were located, of which included a total of 119 unique citations [4][5][6][11][12][13][14]. Each article was located and reviewed for appropriateness.Inclusion criteria included: the primary purpose of the article was program evaluation, articles were full text and not abstracts published as conference proceedings, must be published in the English language, must be peer reviewed, and must include elements of health promotion.Upon further examination 20 studies were eliminated for a variety of reasons (Figure 1).Seven studies were not intervention studies, or had some other purpose than program evaluation [15][16][17][18][19][20][21], four studies were abstracts or commentaries on other articles [22][23][24][25], three studies were written descriptions of the program, but did not present any results [26][27][28], three studies were published in languages other than English [29][30][31], one was an exercise training study [32], one was not a peer-reviewed article [33], and one did not include a health education component [34].The process for our article selection can be found on Figure 1.
The remaining 99 articles were independently reviewed by both investigators of this article, in an effort to find evidence that process evaluations were conducted and/or reported upon.To accomplish this review, both investigators thoroughly read each article and independently evaluated the presence or absence of each type of process evaluation.In order to assist both investigators in the review, key types of process evaluations were operationalized and coded categorically to indicate the presence or absence of each type of process evaluation.Implementation Fidelity was operationalized as any evaluation that took place in order to monitor program activities to assure the program was implemented as planned.Research methods could include self-report, direct observation, video surveillance, or any mix of the three types.Attendance was operationalized as an evaluation to track attendance of program participants.Attendance could occur once during the program, every session of the program, or on an infrequent basis.Training was operationalized as any description of how program facilitators were trained to adequately implement the intervention.Training could occur once before the start of the program, periodically during the implementation of the program, or on an infrequent basis.Satisfaction was operationalized as any evaluation that collected information about the satisfaction of the primary or secondary audience of the intervention.For example, the audience could have been the children receiving the program (primary audience), or the parents or teachers of the children in the program (secondary audience).Both qualitative (ex.focus groups) and quantitative (ex.surveys) designs were considered appropriate for this criteria, however if the author made an anecdotal statement in the article, or stated that participants enjoyed the program without describing how they collected data to support the claim, then it was not counted.Finally, Context was operationalized as any description of how environmental or outside factors could influence the impact or outcome evaluation implemented in the study.Upon reading and rating each article, both investigators compared findings, and settled any discrepancies between evaluations by re-reading the article together.
In addition to reporting the presence or absence of each type of process evaluation it was hypothesized that the average date of publication and the current impact factor of the journal the article was published, may differ between the studies reporting each type of process evaluation.Therefore, an independent t-test was used to compare both variables between groups.

RESULTS
Overall, 99 studies were reviewed for this study.Table 1 shows each study with what type of process evaluation was reported.According to observations, 38 studies (38%) reported program fidelity, 25 studies (25%) reported taking attendance, 49 studies (49.5%) reported some details about training program facilitators, and 29 studies (29%) studies reported the satisfaction of the primary or secondary audience (Table 2).Additionally, 33 studies (33.3%) reported using no type of process evaluation, 25 studies (25.3%) reported using one type process evaluation, 21 studies (21.4%) reported using at least two types of process evaluations, 15 studies (15.2%) reported using at least three types of process evaluations, and 5 studies (5.1%) reported using all four types of process evaluations.Not included on Table 2 were results on context, as only 6 studies reported using this type of process evaluation [35][36][37][38][39][40].The years of publication for the studies in this review ranged from 1983 to 2008.An independent samples t-test was performed, comparing studies that reported each type of process evaluation (excluding context), and there was no significant difference found for any variable.The impact factor for the journals of studies in this review ranged from 0.0 to 53.0.An independent samples t-test was performed, comparing studies that reported each type of process evaluation (excluding context), and there was no significant difference found for any variable.Means and standard deviations for each variable and group can be found on Table 3.

DISCUSSION
Process evaluations are an essential component to evaluating health promotion programs, such as those directed towards the prevention of childhood obesity, however results from this study indicate that they are oftentimes underutilized.Take into consideration a basic logic model of how an obesity prevention intervention is planned.Typically resources are gathered in order to plan, implement and evaluate the intervention, the intervention is subsequently implemented, and data is collected regarding whether the objectives of the intervention were met, such as changes health-related behaviors, and behavioral antecedents.It should be apparent that process evaluations are essential for interpreting impact and outcome evaluations.Said another way, unless a proper process evaluation is conducted, a practitioner or researcher will not know if the results from an impact or outcome evaluation are trustworthy.Another advantage to conducting a proper process evaluation is that the information collected from them can be used discern effective intervention activities versus ineffective intervention activities.For example, through an evaluation of satisfaction, participants may report minor enjoyment for some aspects of an intervention, and great enjoyment other aspects.By tracking attendance, it may also become apparent that at some point during the intervention participant interest waivers by evidence of lower participation rates.Finally, a proper process evaluation can help practitioners and researchers standardize program activities so they are implemented identical across various settings.For example, it is likely the case that programs will be implemented by a number of practitioners in a variety of settings.By identifying key intervention elements in an evaluation of program fidelity, all program staff members can be trained to implement the program uniformly.
As Steckler and Linnan [131] reported, the practice of performing process evaluations is not necessarily a new concept in the field of public health and health promotion.They note that the idea of process evaluations has been around since the 1960's, however it was not until the 1980's that process evaluations became well known.Surprisingly however, in our study we did not see a trend that indicated process evaluations are becoming more popular with time.This finding did bring about the following conundrum however: "Are process evaluations not conducted and therefore, are not reported?"Or, "Are process evaluations conducted but not reported upon in the literature?"In the first case, if process evaluations are not done, then a logical follow-up question is "Why are process evaluations not collected?"There are many possible explanations for why process evaluations may not be collected.Improper training or a low level of awareness of researchers and health practitioners may be two possible explanations.In the systematic reviews and meta-analyses for which the studies were retrieved for this article, rarely did authors mention process evaluations as a limitation to program evaluation.Another explanation may be a lack of resources, such as time, money or trained personnel to collect process evaluations.It may also be that researchers and practitioners place more value on impact and outcome evaluations.If this is indeed the case that process evaluations are not being collected, then it would be recommended that more emphasis should be given to them in academic training and other venues such as grant requests for proposals (RFP's).
In the second case, if process evaluations are done and not reported upon, then a logical follow-up question is "Why are process evaluations not reported upon?"This was observed in a few of the studies reviewed in this article.For example in Taylor et al. [122], a brief statement mentioning process evaluations was present, however this statement referenced unpublished data, making it impossible to find more information about the actual procedures of the process evaluation.Another example comes from Bush et al. [47], a study reporting the effectiveness of the popular 'Know Your Body' intervention.While there were no mention of process evaluations in this report, and no citation to any studies documenting the implementation of the program, another report by Taggart et al. [135], documented the implementation and process evaluation of the program, and referenced the Bush et al. [47] study.Another reason process evaluations may not be reported upon in greater detail, is that researchers may consider this type of evaluation similar to testing statistical assumptions, which have also been found to be poorly documented in many cases [134,135].
There were a few notable limitations to this review.First, our search strategy was limited to articles that have been previously reviewed, and therefore we could have excluded some studies.Although, it should be noted that systematic reviews and meta-analyses typically have higher standards when retrieving articles to review, therefore even if some articles were omitted, it is likely that they were of lesser quality or level of rigor.As mentioned previously, it may also be the case that process evaluations are being implemented, but not reported upon, which limits our ability to generalize about the current state of research.However, a common saying in the medical field among practitioners is "If it wasn't documented, it never happened" and this statement should hold true for public health and health promotion and education.
In conclusion, process evaluations are not a new phenomenon that would be expected to be absent in our current literature.Health promotion and education is a science, and according to the scientific method, scientists should carefully document their methodologies so that others may replicate their research if desired.Process evaluations can also be easily incorporated into program design, at very little cost.Attendance is an important process evaluation that ensures that the program dose reaches the target audience.Researchers and practitioners can keep basic attendance, and set minimum acceptable attendance rates for each program session.While it may be unrealistic to expect 100% attendance for an intervention, acceptable or realistic attendance rates, such as 70% or 80% can be considered.Participant satisfaction can be evaluated using simple qualitative or quantitative measures.Qualitative measures can be obtained with focus groups, individual or group interviews, and any other method that enables the users to respond to open ended questions.Quantitative measures can be obtained through surveys.Program fidelity is important as it ensures that every stage of a program is being implemented entirely as designed.Program fidelity can be collected in a number of ways, including using outside observers or self-report.Fidelity can be evaluated by creating a set of objectives the intervention is intended to address, and keeping track of whether these objectives were met.Finally, program training is an important process evaluation that can have profound effect on the implementation of the program.When writing final reports, researchers should mention if the participating staff of the program was trained, and provide a basic description of how the staff members were trained.
In order to move forward in this area of public health and health promotion researchers and practitioners should be mindful of what gaps exist in research and practice.Determining what components of an intervention are successful, for whom, and under what conditions, are all questions that can be answered by appropriate process evaluations.It is apparent from this review that in this area of childhood obesity prevention research, process evaluations are highly underutilized.This is not to say that there have not been improvements in this area however.Take into consideration the CATCH program.Multiple reports have been published that solely focus on process evaluations for the physical activity component [135], classroom component [136], family component [137], and food service component [138].More attention should be given to process evaluations in future studies.

DISCLOSURE
There are no financial disclosures or conflicts of interests related to this study.

Figure 1 :
Figure 1: Process of Program Article Selection.