Anthony N. Maluccio, Cinzia Canali and Tiziano Vecchiato (Eds.) Assessing Outcomes in Child and Family Services: Comparative Design and Policy Issues. Hawthorne, NY: 2003. $49.95 hardcover, $24.95 papercover.
The passage of the amendments to the Social Security Act (1994) gave the United State federal government a mandate to examine the nation’s child welfare service delivery system focusing on safety, permanency, and child well being outcomes. Since 2000, states have been participating in Child and Family Services Reviews conducted by the federal Children’s Bureau. While these reviews rely largely on administrative data from established management information systems, they also include examinations of a randomly selected small sample of complete case records and interviews of professional and community partners involved in child welfare services. Public policy resulting from these evaluations will affect what and how children’s and family services are funded for many years.
That is why this collection of papers on conducting outcome research in child and family services is timely for evaluation researchers and practitioners to read. Originally presented at the “Outcome-based Evaluation: A Cross-national Comparison” seminar in Volterra Italy in 2001, this volume is one of two. The companion volume is entitled, Evaluation in Child and Family Services: Comparative Client and Program Perspectives.
The focus of this volume is on sharing perspectives on challenges raised by the intersections of evaluation design and public policy. The reader can expect to examine program evaluation using examples from the United Kingdom, United States, Australia, South Africa, and Italy. Of particular interest is the notion of how complex these evaluations can be and how similar the problems are from country to country.
Ward offers an examination of the Looking After Children study, a national evaluation of fifty performance indicators for children in foster care in the United Kingdom. Two themes raised here and repeated by other contributors are that management information systems are not sensitive enough to drill down to the level of information needed to evaluate outcomes and that social work practitioners do not understand the importance of accurate thorough individual case documentation in evaluation.
Fein, reports on the evaluation of two intensive family preservation sites first emphasizing the importance of comparing what the social workers actually did with the clients rather than measuring time spent. Secondly she articulates the necessity for the definition of a “good” outcome, that is what is considered a good evaluation depends on how “good outcomes” are defined in the beginning. Next, Chaskin applies the theory of change model of evaluation that involves convening and evoking various stakeholders’ theories of change in community building efforts and how complex that can be.
In chapter four, Vecchiato notes the challenges to applying national indicators to detect regional differences, and presents discussion about “the limits of national planning.” In contrast, Goerge presents the case that paying close attention to selecting intervention measurements and control groups will overcome many challenges of non-experimental designs, using national welfare reform as the example.
Chapters six through eight address specific design issues. Pilati and associates offer an example of a public health intervention evaluation for smoking and compares it to other cross-national studies. Landsverk and Davis note that system improvements in child mental health systems do not demonstrate individual clinical level improvements. Pompei promotes the importance of informed documentation and how it enables embedded program evaluation using an example of residential care.
Berry and Cash’s intriguing study of risk assessment resulted in the conclusion that differential response family assessment and service provision are disconnected from the risk assessment process. Next, Wright and Paget’s paper specifically discusses the genesis of the United States’ federal reviews from a learning organization perspective, applying the logic model of evaluation. In chapter 11 Ainsworth discusses the cultural issues involved in the research agenda setting process and across professional disciplines, using an institutional review process example from Australia.
Finally Lightburn raises concerns about applying experimental evaluation designs before programs are mature enough to implement the services as intended, using the Family Resource Centers evaluation as an example. This reflects the condition of many new programs which target the most needy: those parents with multiple needs who live in dangerous or isolated communities, and who take longer to engage in services offered.
This book truly does not offer a cross-national comparison. However, the similarity of design and policy perspectives among nations should encourage more information sharing forums. Several contributors caution about the unintended consequences of focusing evaluations on indicators and systems, in view of lack of evidence that changes in systems influence outcomes for particular children. This book raises the caveat that, to paraphrase Amitai Etzioni in Modern Organizations, organizations under scrutiny of evaluation tend to neglect doing those things that are less easily measured because some things lend themselves to measurement better than others. As Vecchiato has discussed, evaluation efforts should be directed at building better theories to support effective children and family services.