|
|
Project:
90CT0072/01
Training for Managers and Supervisors to Enhance Their Capability to Understand and Implement ASFA Evaluation Report for the Project Period October 1, 2000 to September 30, 2001 Introduction This report presents the evaluation findings for the first full year of this project's operations. It is intended to be a companion document to the second Annual Progress (Performance) Report submitted by the Project Director to the Children's Bureau, detailing activities and accomplishments for that six-month period. However, this evaluation report looks retrospectively at the entire twelve months recently completed, and offers a preview of evaluation methods and events planned for the second project year. Evaluation Scope This evaluation has two components: (1) an evaluation of process, or the extent to which the work of the project is proceeding or has proceeded as expressed in the work plan; and (2) an evaluation of outcomes, or the extent to which anticipated milestone events and results and products have emerged as the consequence of the work of project staff and their leaders and various collaborators. Evaluation Findings It should be noted here
that data sources for much if not nearly all of what follows have been
created and administered by project staff for their own internal uses
in tracking the progress of their work and assuring that critical tasks
are carried out according to schedule. These tools routinely inform
and guide decisions and actions in every respect. 1. Process Findings The major year one tasks are presented in detail within the matrices for project objectives one through seven, attached. These matrices were created at the outset and have guided discussions at the meetings of the project team. The matrices attached to this report are updated periodically to reflect current status and planned activities for the following period. No significant variations from the work plan developed at the start have been observed. The only variation to the timeline originally proposed was that of establishing an evaluation approach. However, this should not be viewed as significant since the evaluation approach could not be completed until the draft curriculum was complete. The delay (which resulted from the retirement of the first evaluator) simply put both the evaluation and the curriculum on the same timeline. Therefore, the project has completed all its critical tasks on time, under budget, and in good order. The progress attained in each substantive area of work has been described in some detail within each of the Project Director's first year annual progress reports. Significant steps have been taken to develop the new ASFA curriculum and generate interest in the field, including participantion by the Project Director and team members at the CWLA 4th National Child Welfare Data Conference and NCANDS Meeting in April 2001 and the AHA's ASFA Conference in August 2001. A website (http://www.muskie.usm.maine.edu/asfa) was established to publicize the project, to solicit pilot state participants, and to share results on an ongoing basis. In addition, the project team provided a hardcopy of the report Building the Child Welfare Team Promising Practices 2001 Phone Poll in May 2001 to the Regional ACF Staff. The development of the curriculum has been guided by the active involvement of the Commonwealth of Kentucky's Department of Community Based Services representatives. Early in the development process, the Project Director worked with the Department to assemble a team of managers from key areas to advise the project team, review curriculum materials, and provide critical feedback. Project team members have held two on-site meetings in Kentucky to get their input on the competencies and curriculum, and later feedback on the draft curriculum and evaluation approach and forms. This support has been critical in ensuring that the curriculum is grounded in practice and meets the "real world" needs of child welfare managers and supervisors. Among the innovative approaches to this curriculum is the briefing for senior child welfare managers that precedes the training. The briefing highlights the main concepts being presented to managers and supervisors, provides an opportunity for senior managers to have input into and shape the curriculum, and to commit to reinforcing the main training concepts and the evaluation approach. This design element is critical--personal and organizational commitment from senior managers is necessary for ensuring that the training has the desired impact on practice. Year two is dedicated to field-testing the curriculum. Kentucky will be the first pilot field test, which will be conducted by the end of this calendar year. The training participants will be managers and supervisors from at least two Regions. During the second half of year two, the curriculum will be piloted in two additional states. The evaluation activities in Kentucky will inform any needed changes to the pilot curriculum prior to being delivered in the other states. 2. Outcomes Findings All outcomes and products scheduled for year one have been completed in good order. One of the most noteworthy aspects of the curriculum is the degree to which it incorporates ASFA guidance and technical assistance documents produced by other sources and experts and then builds on this expertise. The curriculum development recognized that ASFA is not new to child welfare practitioners who likely have received numerous briefings, trainings, memos, policy statements and the like on the facts of ASFA. The curriculum builds on these to create a unique combination of a somewhat traditional briefing on ASFA and a series of activities that guide managers and supervisors in assessing their own competencies and practice and the changes required as a result of ASFA. This approach, combined with the module structure, is critical to making the curriculum a valuable tool central in an ASFA implementation strategy. Other products have emerged as a function of the curriculum design process, and they include the following:
As with the first year, Year Two evaluation of process and outcomes will employ methods of competency, consistency, utility, and implementation analysis. The focus of the study will include implementation issues having to do with the impact of training on practice and barriers to full-scale implementation and their training implications. The project will also carefully track the various ways states modify the curriculum to fit their own requirements. Indeed, the module-based approach embodied in the curriculum invites such adaptations, and applicant states have been asked to anticipate which modules they expect to use in their own training activities. Participating states will be required to complete pilot training activities by the end of May 2002, to enable full-scale evaluation and set the stage for national dissemination The Project Director is committed to tackling the challenge of evaluating the impact of the training on practice. The evaluation activities used in the Integrated Information Management in Child Welfare Supervision training have informed the evaluation designed for this training. Year two pilot training evaluation will consist of the following activities:
1. Year One Project Task
Completion Timetable (Objectives 1.1-1.7)
Attachment 1 Major Activities and
Accomplishments During for the Project Period
Attachment 2 Activities Planned
for the for the Project Period
Attachment 3 PROPOSED PROJECT TIME
FRAMES
|
 |
|||||
