7/27/2019 0000258

1/9

Journ al ofEvaluation and Progmm Planni ng, Vol. 1, pp. 41-49 11978 . Pergamon Press. Printed in USA

THE GOAL-ORIENTED APPROACH TO EVALUATI ON:CRITIQUE AND CASE STUDY FROM DRUG ABUSE TREATMENT

J EANNE?C. MARSHInstitute for Social ResearchUniversity of Michigan

ABSTRACTTh e specifi cation and measur ement of program goals remai ns cent ra l to most eval uat ionresearch strategies, y t pro edures for implementi ng thi s approach are not well -art icul ated.I t is the pur pose of thi s di scussion to descri be a stepwi se pr ocedu re for pr ogramm ati c goalsett i ng and monitor i ng used in a demonstrat i on dru g treatment program for women. T hr eeimp l ement ati on steps ar e descri bed: (a) goal sett i ng, (b) checki ng for consi stency, (c)moni tori ng and feedback. Th e advan tages and limi tati ons of thi s appr oach ar e di scussedand useful compl ementar y measur ement strat egies are suggested.

INTRODUCTIONThe specification and measurement of program goals is central to most evaluation research strategies described in theli terature (Riecken, Boruch, Campbell, Caplan, Glennan, Pratt, Rees, Williams, 1974; Suchman, 1967; Tripodi, Fellinand Epstein, 1971; Weiss, 1972). The nature of an evaluative study is such that it must document what a program doesand then determine what effect i t has. The failure to specify the focus and level of an intervention can lead to themeasurement of irrelevant variables and the generation of misleading results (Charters and J ones, 1973; Rossi, 1972).While the documentation of the focus and impact of program implementation can be accomplished in a variety of ways,the specification and measurement of program goals provides perhaps the most frequently articulated approach. Despitegeneral agreement on the role of goal setting and measurement in evaluation research, the task is complex and itsdimensions are not well-defined.

The inadequate understanding of most social problems as well as the minimal knowledge of effective ameliorativestrategies makes the task of defining and operationalizing goals difficult for even the most competent program staff andresearchers. For some social programs the nature of the problem and the intervention are such that goal setting provesless problematic. Programs such as population control programs have a well-defined intervention, the provision of birthcontrol education and devices, and an objective outcome measure, population growth. Many social problems, however,are concerned with less clear-cut problems, e.g., treatment and prevention of mental illness, delinquency control andprevention, or substance abuse prevention and control. Each of these phenomena are open to numerous definitions. Andthe targets for intervention for these problems are likely to be equally diverse.

The goal setting task is further complicated by the fact that as programs mature, needs for evaluative informationchange. For new programs, when major attention must be given to establishing new modes of service delivery, a focus onprocess objectives is needed to determine if services are being provided as intended. Later, when service strategies areestablished, studies of whether and under what conditions services are having the desired impact on clients becomesimportant. Eventually, an evaluation of the impact of the program on other organizations and on the community isrelevant. Tripodi, Fellin and Epstein (1971) emphasize the importance oftailoring program goals to the various stages ofprogram development and call this approach dif ferent ial evaluat ion. They suggest the analysis of the effects of theprogram intervention are premature and unproductive in the early stages of program development. The achievement ofgoals related to the ultimate outcome of the program is often dependent on the success with which problems of programimplementation have been managed.

Preparationf this article for publication was aided by support from National Institute for Drug Abuse Grant H -81 DA-01496-02.Request for reprints should be sent to J eanne C. Marsh, 5027 Institute for Social Research, The UniversityofMichigan, Ann Arbor, Michigan 48106.41

7/27/2019 0000258

2/9

42 J EANNti C. MARSH

SPECIFICATION AND MEASUREMENTThe specification and measurement of program goals iscentral to evaluation research strategies which focus onthe assessment of social programs (Suchman, 1967;Weiss, 1972; Riecken, Boruch, Campbell, Caplan, Glen-nan, Pratt, Rees, Williams, 1974). However, goal settingand monitoring has been found to be useful in other effortsas well. F or some time, social psychologists havedocumented the strong relationship ofmotivation to goalsand of the impact of individual goals on outcomes for boththe individual and the group (Atkinson, 1968; Zander,1971). Mental health professionals have capitalized onthe efftciencyof goal-directed behavior and have incorpo-rated goal setting and monitoring into treatment ap-proaches to enhance therapeutic gain and to aid theevaluation of treatment outcome (Bandura, 1969;Kiresuk and Sherman, 1968; Houts and Scott, 1972; Reidand Epstein, 1972). Kiresuk and Shermans Goal Attain-ment Scaling is perhaps the most widely disseminated ofthe goal-oriented approaches to the evaluation of indi-vidual treatment effectiveness which have been de-veloped in recent years (Mager, 1972; Davis, 1973; Miller1973; Bolin and Kivens, 1974). The identification of goalsand values is also an important component of the multi-attribute utility analysis developed by Guttentag, Ed-wards and Snapper (1975) to aid decision-making of fed-eral level policy makers. In business, goal setting andmonitoring is central to at least one approach to contem-porary management practice. The MB0 (Management byObjectives) system represents a widely distributed exam-ple of this approach (Raia, 1974).

While goal-oriented approaches to treatment, evalua-

tion, decision-making and management practice haveproliferated, few descriptions of these approaches containspecific information to guide the evaluation researcherseeking to establish and measure program goals. It is thepurpose of this discussion to present an explicit procedurefor programmatic goal setting and monitoring which wasused in a demonstration drug treatment program for wo-men. A discussion of the advantages and limitations of theprocedure specifically as well as the goal-oriented evalua-tion in general will follow.

The implementation of goal-oriented evaluation in adrug treatment program provides a particularly informa-tive test of this method. Effective strategies for the treat-ment of heroin addiction have not been determined. Ap-propriate treatment methods for heroin addicted women,in particular, remain unidentified (Doyle and Levy, 1975;Rosenthal, Spillane and Greene, 1976). Thus, the discus-sion will describe the use of goal-oriented evaluation in aprogram with minimal information as to appropriate di-rection and focus. Furthermore, the paper wil l describethe use of a goal-oriented method in the early stages ofprogram development when goals focus on process, i.e., onthe tasks required for organizational development: secur-ing resources, developing treatment technology, facilitat-ing staff skill development, coordinating activites withinthe program. Except for changes in the content of goals,the goal setting and monitoring procedure remains essen-tially constant throughout the life of the program. Thepresent discussion will focus on and present data from theimplementation period when goal-oriented evaluationmethods were introduced to the program staff.

PROGRAM DESCRIPTIONThe WOMAN (Women Organized to Move Against Nar-cotics) Center is a community-based methadone mainte-nance program for heroin-addicted women in Detroit. Agrass-roots effort, the program was conceptualized anddesigned by a group of women in the Cass Corridor whofelt women addicts in their neighborhood were not receiv-ing adequate treatment in traditional methadonemaintenance programs. So the program was broadly de-fined to meet the unique needs of women and included

an all-female staff and a child care center. The originatorsof the program took positions on the Board of Directorsand they hired women, the majority of whom were para-professionals from the neighborhood, as program staff.The characteristics of the program that are particularlyrelevant to this discussion are (1) that it was a newly-developed program and (2) that the staff were relativelyinexperienced in program development and evaluation.

PROCEDURE FOR GOAL SETTING AND MONITORINGThere are three aspects of goal setting and monitoring asimplemented at the WOMAN Center. The first involvesproviding program staff with a rationale and instructionsdescribing the purpose as well as techniques for settingobjective, measureable program goals. The second stepinvolves assessing goals the program identifies for consis-tency with objectives of the program as set forth by othergroups or individuals with investment in the program,e.g., program founders, a Board of Directors, or a fundingsource. The final step involves goal monitoring and feed-back for purposes of program planning and development.These steps are summarized in the diagram in Figure 1and described below.

Step 1. Goal settingStaff members who have not previously been involved inan evaluation effort are likely to be unfamil iar with therelevance of clearly specified program goals for programplanning and development as well as for a meaningfulevaluation. For example, the majority of the WOMANCenter staff had no previous experience in the develop-ment and evaluation of a treatment program, and theyhad not considered the usefulness of clearly defined goalsfor the program or for the evaluation. Initially, the evalu-ation staff may have to educate program staff regardingthe importance of translating treatment procedures andintended effects into researchable terms and provide

7/27/2019 0000258

3/9

The Goal-oriented Approach to Evaluation: Critique and Case Study from Drug Abuse Treatment 43

GOAL SETTING CHECKING FOR CONSISTENCY > GOAL MONITORING

IFEEDBACK

For program planningand development For programevaluation

Figure 1 Goal setting and monitoring steps

them with techniques for doing so. Frequently, the beststrategy requires that staff become involved in settingprogram goals immediately and acquire insight into theimportance of goal setting for the evaluation as a functionof this activity. Evaluation staff can provide one or tworules to program staff to guide them in their goal settingefforts and then work with them to identify actual pro-gram goals and to translate them into researchable terms.

In order for all aspects of the program to be representedin the goals, all program staff (or representatives of pro-gram staff) including administrators, treatment and sup-port staff should be involved in the goal setting process.Within a given program, administrators and treatmentstaff are likely to be concerned with different aspects ofthe program (Etzioni, 1960). Intervention staff are, ofcourse, concerned with the delivery of service while ad-ministrative staff must be concerned with the mainte-nance ofthe program, recruitment of additional resourcesand the relationship of the program to outside forces -funding sources, other agencies and community groups.Thus, depending on position within the organization,program staff are likely to be concerned with differenttypes of goals. By involving staff from all aspects of theprogram, goals are likely to be more representative of thetotal program. Indeed, at the WOMAN Center, each com-ponent of the program developed individual componentgoals which were then incorporated into the total pro-gram goals.

A number of factors contribute to the difficulty of thistask. First, particularly in a new program, there is likelyto exist a lack of consensus regarding the most importantindependent and dependent variables. Additionally, thedivergent values and orientations of treatment andevaluation staff are likely to become obvious as a result ofthis process. Treatment staff conceptualize the treatmentprogram in terms of multiple independent and dependentvariables. Evaluation researchers, on the other hand,seek less complex relationships and would like the pro-gram staff to sort out the most critical and relevant vari-ables for measurement. Treatment staff are primarilyconcerned with the efficient delivery of best availableservice. Evaluation researchers seek to determine what isthe best, or most effective service. These conflicting orien-

tations are inherent in the evaluation task, and to theextent that they can be acknowledged and anticipated,their impact can be minimized.

The acceptability of the goal setting procedure to pro-gram staff will be enhanced to the extent that it is per-ceived as an aid to program planning and development aswell as an evaluation tool. Goal setting can be describedas a useful means for delineating the focus and directionof a new program. For example, it was pointed out toWOMAN Center staff that as part of the initial goalsetting process, various staff members conceptualiza-tions of the purpose and function of the program could benegotiated until a consensus emerged in the form of pro-gram goals. Additionally at this time, the frequency andformat of goal achievement feedback was discussed sothat evidence of regular and tangible outcome of staffefforts was apparent.Step 2. Checking for consistencyOnce program goals have been written, it is necessary todetermine their consistency with expectations of othergroups or individuals having interest or investment in theprogram. These groups can include initiators of programproposals, a funding source, a Board of Directors, a licens-ing agency. This phase is useful for a number of reasons.In cases where federal or regional directives exist, thiscomparison provides a clear picture of whether or notthese directives are being met. In other cases where pro-gram expectations are prescribed by groups such as anadvisory board, a licensing or funding source, this consis-tency check determines the degree to which the programgoals parallel or diverge from the expectations of thesegroups. The consistency check provides an opportunity foridentifying discrepancies and obtaining consensus whenthis is appropriate.

In many cases, the expectations of outside groups havenot been committed to paper. I n such cases verbal ap-proval of program goals by outside groups may be theappropriate strategy. In cases where expectations arewritten in some form (i.e., the funding proposal, licensingagency directives, a Board of Directors philosophy state-ment), it is useful to formally match program goals withdirectives or statements of philosophy. A more formal

7/27/2019 0000258

4/9

44 J EANN e C. MARSH

strategy is particularly useful when written expectationsare available from more than one unit. For example, ex-pectations from all units can be placed on one axis of amatrix. Program goals can then be placed on the otheraxis so that goals and expectations can be systematicallymatched. This procedure makes it possible to determinegraphically that at least one goal statement addressesevery set of expectations. This procedure was used at theWOMAN Center to relate program goals to thephilosophy statements set forth by the Board of Directors.A subset of goals and philosophy statements appear inFigure 2.

As is apparent from Figure 2, expectations from dif-ferent units can be overlapping, but they should not becontradictory. When it is impossible for the program tologically fulfill contradictory expectations from twosources, the program must negotiate with one or the otherunit to resolve this constraint. This step was particularlyvaluable at the WOMAN Center where program foundersremained involved on the Board of Directors. These indi-viduals had a strong investment in the program and deli -nite expectations about appropriate program goals thatwere not totally consistent with the expectations of theprogram staff. These inconsistencies were clarified and

FI GURE 2.

CONSISTENCY MATRIX

GOALS

1.

2

3.

4.

5.

6.

The Community Resource Coordinatorwill maintain a current and up-to-datelog of community organizations, agencies

institutions which may be useful toWOMAN Center. This log will:a. be systematized by 2115176,b. be updated monthly,c. include name, services, location, con-

tact person notes regarding contactswith WOMAN program.

The Community Resource Coordinatorwill meet weekly with community aidesto share resources, problems to com-municate recruitment efforts.Team Coordinators will develop an auditsheet for WOMAN members which willreflect the members progress throughthe program dates on which each stepin the process is completed including:a. completion of intake process,b. completion of orientation process,c. assignment of counselor,d. development of treatment plan,e. methadone dosage when appropriate.The Treatment Coordinator will reviewall treatment plans on a quarterly basis.Individual counseling sessions will beheld with children in the program on aweekly basis.The Program Director will assign eachtreatment staff person an area of advo-cacy.

Develop qual i ty programwi th funded volun ta rystaff to assist w omen wit hmental , ph ysical or en-vironmental problems.

Provide a mechani sm for col-lectin g sharin g inform a-ti on, resources skil ls tohelp women with survivalproblems.

X

X

X

Develop relationship wit hother org. groups that dealwit h issues affectin g UI O-mens lives.

X

7/27/2019 0000258

5/9

The Goal-oriented Approach to Evaluation: Critique and Case Study from Drug Abuse Treatment 45

reduced early in the program development as a result of levels of goal achievement), this can consist of an evalua-this step in the goal setting process. tion staff member who uses methods of naturalistic obser-Step 3. Monitoring and feedback vation to document goal achievement. Some evaluatorsOnce program goals have been developed, a procedure have attempted to make this a more quantitative proce-must be established for monitoring achievement of goals. dure by assigning numbers to levels of goal achievementThen this information must be provided to the program in and weightings to various goals so that a goal achieve-a form that can be used readily for program planning and ment score can be calculated. In some cases, interval leveldevelopment as well as for evaluation purposes. The statistics have been applied to test for post-treatmentevaluation staff must derive a data collection system that differences (Kiresuk and Sherman, 1968). Otherallows for accurate, efficient summary of these data both evaluators who have rated levels of goal achievement,for purposes of the program and the evaluation. Since simply placed levels of achievement on graphs and up-these data are basically qualitative (i.e., descriptions of dated the graphs monthly (Austin, 1974). Selection of a

FI GURE 3.FEE DBACK FORMAT FOR SELECTED SUBSAMPLE OF WOMAN CENTER GOALS

PHILOSOPHY STATEMENTS

GOAL

1.

2.

3.

4.

5.

6.

The Community Resource Coordinatorwill maintain a current and up-to-datelog of community organizations, agenciesand institutions for use of WOMANCenter. This log will:a. be systematized by 2115176,b. be updated monthly,c. include name, services, location, con-

tact person notes regarding contactswith WOMAN program

The Community Resource Coordinatorwill meet weekly with community aidesto share resources, problems to com-municate recruitment efforts.Team Coordinators will develop an auditsheet for WOMAN members which willreflect the members progress throughthe program dates on which each stepin the process is completed including:a. completion of intake process,b. completion of orientation process,c. assignment of counselor,d. development of treatment plan,e. methadone dosage when appropriate.The Treatment Coordinator will reviewall treatment plans on a quarterly basis.

Individual counseling sessions will bcheld with children in the program on 5weekly basis.The Program Director will assign eachtreatment staff person an area of advo-cacy.

?EVIEW DUEDATE

2115176and monthly

2129176

212176

Quarterly Partiallymet

211176 Not met

212176 Met

GOALSTATUS

Met

Met

Met

COMMENTS

This log has been developed, incorporatesthe stated information and is up-dated asnew information becomes available.

The individual counselors attempt to meetthis goal but have not been totally success-ful.Reduction in the number of Child Care staffrendered this goal impossible.

The advocacy areas include: education,employment, food resources, legal, dental,medical, housing, mental health socialservice assistance. Some advocates handlemore than one area of advocacy.

7/27/2019 0000258

6/9

46 J EANNE C. MARSH

monitoring and feedback system depends on how the in-formation will be used. If the information is collected onlyto derive a goal attainment or change score, then quan-titative procedures are appropriate. If, however, thesemethods are designed to aid in program planning anddecision-making, more qualitative information is neces-sary.

The WOMAN Center used naturalistic observation tomonitor goal achievement and then provided feedback inthe form of tables which included a rating of goalachievement. These ratings were on a simple three pointscale: met, partially met, not met. The ratings were madeby a member of the evaluation staff working on-site at theWOMAN Center who observed program functioning withspecial attention to goal-related activities. In addition todirect observation, other data sources, such as memos,minutes from meetings, client folders, were used to deriveinformation upon which ratings were based. Feedback tothe WOMAN Center also included subjective commentson possible implications of achievement or non-achievement. (See Figure 3 for a sample feedback for-mat. 1

Feedback on all goals was provided at six month inter-vals during the first two years of program functioning.The WOMAN Center staff used the feedback as a correc-tive device in program planning and development. Theinformation appeared to be particularly useful for staff

and resource allocation. Upon receipt of the feedback,program staff began modifying old goals and setting newones. The simple ratings provided a clear indication of (1)areas in which goals had been met allowing the programto progress to the next logical set of goals; (2) areas wheregoals were not met and required additional resources (or amore realistic assessment of the feasibility of goals givenpresent resources); or (3) areas where goals were onlypartially met and would require continued attention.Comments from the evaluator were most frequently pro-vided for goals which were not met or only partially met.The comments typically described the evaluators as-sessment of the reasons goals were not achieved. Thisinformation was designed to aid the program in theirdecision to pursue a goal, perhaps with more resourcesand more time, or to drop an unrealistic or inappropriategoal. For evaluation purposes, the biannual feedback wasuseful for gauging program development and determin-ing the adequacy and appropriateness of outcome mea-sures.

The goal setting and monitoring procedure utilized atthe WOMAN Center represents the specification of agoal-oriented evaluation approach. I nherent in this gen-eral approach are certain strengths and limitationsevaluation researchers must be aware of when utilizingthe approach.

ADVANTAGES OF THE GOAL ORIENTED APPROACHGoals may be tailored to stages of program developmentAlthough the determination of program goals is not aneasy task, the consistency between program goals andprogram outcome has been established as a basic measureof effectiveness in program evaluation. Furthermore, thegoal-oriented approach permits differential evaluation,i.e., definitions of effectiveness to change as the programor organization progresses through various stages of de-velopment. As a program develops, it becomes more im-portant to measure goals related to the impact of theservice innovation on the clients. The evaluation of pro-cess goals is designed to provide information for improv-ing the program. A good process evaluation allows theprogram to reorganize and improve so that it will make abetter showing on the evaluation of outcome.Program determined goals reduce influence ofevaluator biasGoal-oriented evaluation represents an objective and re-liable approach because it uses goals established by theprogram as criteria for judgment. The beliefs and valuesof the evaluator are less likely to influence the study sincethe effectiveness of the program is determined by itscapacity to achieve those objectives identified as impor-tant by the program staff. And because this approachidentifies effectiveness measures that are idiosyncratic tothe program, it affords flexibility that makes it useful in avariety of settings.

Goal setting has beneficial by products for the programIn addition to its usefulness for evaluation, there areby-products of the goal-oriented approach that are benefi-cial to the program. The process of selecting certain expec-tations as program goals accomplishes two things. F irst,selecting a subset of goals out of all those possible servesto clarify program purpose and function. Particularly innew and innovative programs, there may be disagree-ments or misconceptions regarding the purpose of theprogram. The process of selecting certain goals and com-mitting them to paper requires some consensus beachieved among relevant groups. Second, by establishingcertain expectations for itself the program can assessprogram activities in terms of these expectations andmake changes and modifications in order to meet goals. Inthis way the goal-oriented approach capitalizes on theself-fulfilling prophesy phenomenon or what has beendescribed in the context of evaluation research as targettropism (Davis, 1974). I t is obvious that the carefulclarification of a goal is a necessary, and at times suffi-cient, step for achieving the goal.Goal setting may be incorporated into programfunctioningAn additional advantage of the goal-oriented approach isthat straight-forward procedures can be developed andincorporated into ongoing program functioning. It is clearfrom the use of programmatic goal setting at the WOMAN

7/27/2019 0000258

7/9

The Goal-oriented Approach to Evaluation: Critique and Case Study rom Drug Abuse Treatment 41

Center that while goal setting and monitoring requires veloped readily requiring ohly instruction and practice.staff time and effort, and insome cases, negotiation in the And once program staff have acquired the skill , it can beselection of goals, the ability to select goals and to write usefully applied to program planning and development asthem in researchable terms is a skill that can be de- well as to evaluation tasks.

LIMITATIONS OF THE APPROACHIn addition to the positive aspects of this evaluative ap-proach, there are some limitations that must be con-sidered in its application. For example, it has beensuggested that a goal-oriented approach promotes abiased evaluation in the sense that certain outcomes areidentified for measurement while others are ignored. Un-derstanding that all methods introduce bias, it is impor-tant to recognize the particular sources of bias accom-panying the use of goal-oriented evaluation.Stated goals may not be real goalsFirst, despite the fact that a goal-oriented model utilizesprogram-generated criteria for the assessment of effec-tiveness, the goals it claims to pursue may not be those itactually follows. If the formal goals are very differentfrom the informal goals, information derived from theirevaluation can be misleading. Some evaluation research-ers claim formal goals as real goals of the organization;others choose them because they are easier to determine(Etzioni, 1960). It remains, however, that if program ac-tivities are extremely different from what program staffsay they are trying to do, the information derived from agoal-oriented evaluation will be of little value.Unintended outcomes may be ignoredA second source of bias in the goal-oriented procedureresults from the possibility that other valid but unin-tended program outcomes will be ignored. I dentificationof program goals creates expectations. This increases thepossibil ity that change will be seen in the identified areasand decreases the possibil ity that it will be found other-wise. A clear example of this occurred in the WOMANCenter. Program goals were specified and agreed upon byall components of the program including administrators,treatment staff and support staff. Al l of these goals re-lated to the treatment and prevention of heroin addictionamong women. However, it became increasingly appar-ent to evaluators that-most likely because the facility iscalled the WOMAN Center (a name which in no wayindicates that it is a facility designed to meet only theneeds of drug-addicted women) - the receptionist re-ceived a number of calls for information related to needs ofwomen other than drug-addicted women. She spent someportion of her time and resources each day providinginformation and making referrals to, among other things,feminist financial institutions, rape crisis centers andabortion clinics. This activity was not related to any ofthestated goals. The fact that it was accepted as legitimate byall program staff suggests that the organization was pur-suing goals in addition to those described for purposes ofevaluation of the drug treatment program. The descrip-tion of program goals limited to the treatment of heroin-addicted women led the evaluation to describe programoutcomes for heroin-addicted women and to miss out-comes which indicated this facil ity was meeting needs of a

broader population of women in the community.1 In thisparticular case where the treatment facility intends toseek additional funding from non-drug-related agencies,some documentation of non-drug-related efforts couldhave been useful. An alternative evaluation strategydesigned to avoid this particular source of bias is goal-freeevaluation explicated initially by Michael Striven (Sala-sin, 1974). In a goal-free evaluation, knowledge of pro-gram goals is avoided in the determination of programeffectiveness. The goal-free approach is based on the no-tion that there is no need to know goals in order to assessprogram outcomes. According to Striven, a goal-freeevaluation is designed to evaluate what a program actu-ally does rather than what is trying to do The procedurefollowed by a goal-free evaluator involves avoiding directcontact with the program staff as well as avoiding discus-sions about program history or program goals (i.e., infor-mation that would influence his determination of pro-gram effectiveness). Instead, the evaluator attempts toobserve those program inputs and outcomes that appearsignificant. The goal-free evaluator identifies the goals ofthe program to the extent that they are expressed inobservable treatment efforts and outcome.

Although the WOMAN Center evaluation has not in-corporated the methods prescribed by Striven, the evalu-ation has attempted to include a number of different out-come measures relevant for drug treatment of women inaddition to those suggested by the program goals.Possibility of cross-program comparison may belimitedMany evaluation efforts are designed to compare the ef-fectiveness of one program with another. In some cases itmay be possible to standardize at least some programgoals. To the extent that standardized goals are relevantto the programs to be compared, it may be possible todetermine program effectiveness in terms of the achieve-ment of these goals. However, to the extent that differentprograms with ostensibly the same purpose identifyslightly different or idiosyncratic outcome goals, a goal-oriented approach is limited in its usefulness for cross-program comparison.Effects of organization may not be measuredSocial programs are multifunctional organizations. Inaddition to the stated or real goals of the organization,some time and resources must be devoted to non-goalfunctions such as recruiting personnel and resources toperform goal activities, maintaining resources andfacilities, facil itating social integration of staff. All suchactivities - as well as goal-oriented activities - arefunctional and increase organizational effectiveness. In-deed, an organization that devotes all its efforts to fulfil-ling one functional requirement, even if it is that of per-forming goal activities, wil l undermine even this activity,

7/27/2019 0000258

8/9

48 J EANNB C. MARSH

because activities such as recruitment of means, mainte-nance offacil ities and the social integration of staffwil l beneglected. Implications for a goal-oriented evaluation arethat this model fails to measure large segments of organi-zational activity that are not identified in terms of pro-gram goals. To the extent that program goals are deter-mined by staff of diverse program components, and aneffort is made to include process and outcome programgoals, a broader representation of goals can be achieved.Nevertheless, without additional measurement, manyprogram dynamics contributing to goal achievement arelost.

In response to this limitation of the goal-orientedmodel, a systems model of organizational analysis hasbeen suggested (Etzioni, 1960; Schulberg and Baker,1968). Such a model assesses the allocation of organiza-tional resources rather than goals to determine the effec-tiveness ofprogram functioning. Al though this model has

been applied in a number of settings, procedures for itsuse are not yet well-developed. However, the literaturedescribing strategies for organizational analysis andchange is developing rapidly. And the assessment of thestructure and function of the organization can provideuseful information for program evaluation - if only forthe interpretation of goal achievement or lack ofachievement.To address the inattention to organizational considera-tions of a goal-oriented approach, the WOMAN Centerevaluation includes an organizational analysis which at-tempts to describe the formal and informal structure ofthe organization, the sources of power, the decision-making process, staff satisfaction and dissatisfaction(Hluchyj, 1976). These data provide a rich source of in-formation for interpreting and understanding programprogress and impact.

SUMMARYA discussion of a goal-oriented approach to programevaluation based on the use of this method in a drugtreatment program for women indicates this to be a valu-able approach for several reasons. The goal-orientedmethod limits influence of evaluator bias by using pro-gram identified goals as criteria for measurement.Further, the goal-oriented method has great flexibilitythat allows for its use in conjunction with human serviceprograms progressing through various stages of develop-ment. I ts use has important programmatic benefits in thesense that it forces programs to specify goals which, ofcourse, increases the possibil ity they will achieve thesegoals. Whether a goal-oriented strategy is used on thelevel of individual treatment evaluations or as a programevaluation strategy, the specification of goals is recog-nized as a necessary, and at times sufficient, step for goal

achievement. Finally, goal-oriented evaluation can beimplemented with staff who have minimal experience orsophistication with respect to evaluation.

The limitations or sources of bias that accompany thismodel derive from the fact that the specification of pro-gram goals (1) is useful only to the extent that the mostimportant or representative goals are selected for meas-urement, and (2) increases the possibil ity that evaluatorswill miss or ignore unspecified or unintended programeffects. Further, when programs with the same purposespecify slightly different outcome goals, the goal-orientedapproach is restricted in its usefulness for cross-programcomparisons. One last limitation of the goal-oriented ap-proach rests in its failure to account for the organizationalstructure and function that ultimately contributes to theachievement of program goals.

CONCLUSIONSThere are several implications for the evaluation en-deavor that emerge from this critique of the goal-orientedapproach. F irst, information critical for the assessment ofprogram function and outcome, i.e., what a program istrying to do and how it is trying to do it, can be obtainedthrough program goal setting and monitoring. Further,information that is obtained is useful for program plan-ning and development as well as for program evaluation.And the procedure for obtaining the information isstraightforward and can be readily incorporated intoprogram activities. However, the evaluation researcherwho elects to use a goal-oriented evaluation strategy must

be aware of the limitations or biases introduced by thismethod. Specifically, this approach may lead the re-searcher to neglect informal organizational goals, unin-tended outcomes, or organizational dynamics that influ-ence the achievement of program goals. With knowledgeof these biases an informed decision can be made to sup-plement a goal-oriented approach with appropriate addi-tional measurement strategies. Depending on the pur-pose of the evaluation effort, goal-free strategies as wellas organizational analytic strategies have been identifiedas particularly relevant for use in conjunction with agoal-oriented evaluation.

7/27/2019 0000258

9/9

The Goal-oriented Approach to Evaluation: Critique and Case Study from Drug Abuse Treatment 49

REFEREN ESATKINSON, OHN W. Motives i n Fan tasy, Action and Society. N ewYork: Van Nostrand Co.. 1958.AUSTI N, NANCY. The use of goal attainment scaling in commun-ity mental health: Training and implementation. In GeoffreyGarwick and J oan Brintnall (Eds.), Pr oceedi ngs of th e SecondGoal Attai nm ent Scalin g Conference. Minneapolis: ProgramEvaluation Resource Center, 1974, 111-124.BANDURA, ALBERT. Pri ncipl es of Behavior Modi f icat ion. N ewYork: Holt, Rinehart and Winston, 1969.BOLIN,DAVID C. and K IVE NS, LAWRENCE .Evaluating a commun-ity mental health center: Huntsvill e, Alabama. Eualuat ion, 1,3,1973, 17-18.CHARTERS,W. W. and J ONES, OHN E. On the risk of appraisingnon-events in program evaluation. Educational Researcher,1973, November, 5-7.DAVIS, HOWARD. Four ways to goal attainment. Evaluation, 1,2,1973, 43-48.DAVI S, HOWARD. Change technology and goal-oriented evalua-tion. In Geoffrey Garwick and J oan Brintnall (Eds.),Proceedingsof the Second Goal Attai nm ent Scalin g Conference. Minneapolis:Program Evaluation Resource Center, 1974, l-24.DOYLE , KATHLEEN M. and L EVY, STEPHEN . Th e female client;H ow tr eated i n dr ug abuse program s. Paper presented at theAnnual Meetings of the American Psychological Association,1975.EDWARDS, WARD, GU~ENTAG, MARCI A, and SNAPPER, KURT. Adecision-theoretic approach to evaluation research. I n E. L.Struening and M. Guttentag (Eds.1, Han dbook of Evaluat i onResearch. Beverly Hills: Sage, 1975, 139-182.ETZION I , AMITAI. Two approaches to organizational analysis: Acritique and a suggestion. Adm in istrat ive Sciences Quar terly,1960, 5, 257-278.HLUCHJY,TE RRY. Human service or social movement: Organiza-tional considerations in a drug treatment program for women.WOMAN Center Evalu ation. Year 2: A reoort to N ID A. Unnub-lished report, The University of Michigan, 1976.HOUTS,PETER S. and Scorr, ROBERTA. Goal Plann ing in MentalHeal th Rehabi l i tat ion. Department of Behavioral Science, ThePennsylvania State University, 1972.KIRESUK, THOMAS . Basic goal attainment scaling procedures.Chapter One, Program Evaluation Project Report, 1969-1973.Minneapolis: Program Evaluation Resource Center, 1974.

KI RESUK , THOMAS . and SHERM AN,ROBERTE. Goal attainmentscaling: A general method for evaluating comprehensive com-munity mental health programs. Communi ty M ental H eal thJourna l , 1968, 4, 443-453.MAGER, ROBERT.Goal Analysis. Belmont, Ca.: Fearon, 1972.MI LLER, GARY . A variant of goal attainment scaling: ProGAST.Eualuation, 1,3, 1973, 17-18.RAIA, ANTHONY P. M anagi ng by Objectiu es. Glenview, Illinois:Scott Foresman, 1974.REI D, WILLI AM . and EPSTEI N,LAURA.Task-Centered Casework.New York: Columbia University Press, 1972.RIECKEN,HENRY W., BORUCH,ROBERTF., CAMPBELL,DONALDT.,CAPLAN, NATHAN, GLENNAN,THOMASK., PRAIT, J OHN W., REES,ALBERT, WILL IAMS, WALTER.Social Experi mentati on: A M ethodfO Planni ng and Evaluat i ng Social Intervent ion. New York:Academic Press. 1974.ROSENTH AL, BARRY J ., SPILL ANE, WILL IAM H ., and GREENE ,BRADFORD . Treatment outcome: Is sez a factor. Paper presentedat the National Drug Abuse Conference, 1976.ROSSI,PETERH. Testing for success and failure in social action. I nP. Rossi and W. Williams (Eds.), Eual uat in g Social Programs.New York: Seminar, 1972.SAL ASI N, S. Exploring goal-free evaluation: An interview withMichael Striven. Eualuat ion, 1974, 2, 9-16.SCHULBERG, ERBERTC. and BAKER, FRANK. Program evaluationmodels and implementation of research findings. AmericanJourn al of Publ ic H eal th, 1968, 58, 1248-1255.SUCHMAN,EDWARDA. Evaluation Research: Principles andPrac-t ice in Publi c Servi ce and Social A ction Pr ograms. N ew York:Russell Sage Foundation, 1967.TRIPODI,TONY, FELL IN , PHI LL IP and E PSTEIN, RWIN. Social PFO-gram Evaluat i on: Guid el in es for H eal th, Educat ion and Welfar eAdminis t rators . I thasca, Il linois: Peacock, 1971.WEISS, CAROL H. Evaluation Research. Englewood Cliffs:Prentice-Hall, 1972.ZANDER, ALVIN. Motives and Goals in Groups. N ew York:Academic Press, 1971.

FootnotesEtzioni has described one of the functions of formal or publicgoals to be that of recruiting support (inputs) to the organization

from groups that would not support the private goals. The goalsdetermined by the WOMAN Center were those most relevant tothe source of funding, the National Institute of Drug Abuse.