by Robert Kornfeld , Kalman Rupp
Office of Research, Evaluation, and Statistics,
and Policy, Social Security Administration

    WASHINGTON D.C. -- The Social Security Administration (SSA) initiated Project NetWork in 1991 to test case management as a means of promoting employment among persons with disabilities. The demonstration, which targeted Social Security Disability Insurance (DI) beneficiaries and Supplemental Security Income (SSI) applicants and recipients, offered intensive outreach, work-incentive waivers, and case management/referral services. Participation in Project NetWork was voluntary.
     Volunteers were randomly assigned to the "treatment" group or the "control" group. Those assigned to the treatment group met individually with a case or referral manager who arranged for rehabilitation and employment services, helped clients develop an individual employment plan, and provided direct employment counseling services. Volunteers assigned to the control group could not receive services from Project NetWork but remained eligible for any employment assistance already available in their communities.
     For both treatment and control groups, the demonstration waived specific DI and SSI program rules considered to be work disincentives. The experimental impact study thus measures the incremental effects of case and referral management services.
    The eight demonstration sites were successful in implementing the experimental design roughly as planned. Project NetWork staff were able to recruit large numbers of participants and to provide rehabilitation and employment services on a substantial scale.
    Most of the sites easily reached their enrollment targets and were able to attract volunteers with demographic characteristics similar to those of the entire SSI and DI caseload and a broad range of moderate and severe disabilities. However, by many measures, volunteers were generally more "work-ready" than project eligibles in the demonstration areas who did not volunteer to receive NetWork services.    
      Project NetWork case management increased average annual earnings by $220 per year over the first 2 years following random assignment. This statistically significant impact, an approximate 1 l-percent increase in earnings, is based on administrative data on earnings. For about 70 percent of sample members, a third year of followup data was available. For this limited sample, the estimated effect of Project NetWork on annual earnings declined to roughly zero in the third followup year. The findings suggest that the increase in earnings may have been short-lived and may have disappeared by the time Project NetWork services ended.
     Project NetWork did not reduce reliance on SSI or DI benefits by statistically significant amounts over the 30-42 month followup period. The services provided by Project NetWork thus did not reduce overall SSI and DI caseloads or benefits by substantial amounts, especially given that only about 5 percent of the eligible caseload volunteered to participate in Project NetWork.
     Project NetWork produced modest net benefits to persons with disabilities and net costs to taxpayers. Persons with disabilities gained mainly because the increases in their earnings easily outweighed the small (if any) reduction in average SSI and DI benefits. For SSA and the federal government as a whole, the costs of Project NetWork were not sufficiently offset by increases in tax receipts resulting from increased earnings or reductions in average SSI and DI benefits. The modest net benefits of Project NetWork to persons with disabilities are encouraging. How such benefits of an experimental intervention should be weighed against costs to taxpayers depends on value judgments of policymakers.
     Because different case management projects involve different kinds of services, these results cannot be directly generalized to other case management interventions. They are nevertheless instructive for planning new initiatives. Combining case and referral management services with various other interventions, such as longer term financial support for work or altered provider incentives, could produce different results. The effects of case and referral management services could also be sensitive to numerous site-specific factors, such as the implementation process, labor market conditions, and the availability of local services for persons with disabilities.
     This is one of a series of papers, summarizes the key outcomes of Project NetWork, a return-to-work program for persons with severe disabilities,(1) The Social Security Administration (SSA) initiated Project NetWork in 1991 to test the feasibility and effects of outreach and case and referral management services for beneficiaries of Social Security Disability Insurance (DI) and applicants for and recipients of Supplemental Security Income (SSI). From 1992 to 1994, members of the target population in eight demonstration sites were recruited to volunteer to receive Project NetWork services.
     A comprehensive evaluation component was included in the demonstration design, including the random assignment of 8,248 volunteers to a "treatment" group receiving case management services or to a "control" group of persons who did not receive case management services but who could obtain services on their own. To increase the incentive to work, volunteers in both the treatment and control groups were also offered waivers of SSI and DI program rules considered to act as work disincentives. The evaluation of Project NetWork is thus a rigorous study of the effects of case management services to encourage persons with disabilities to obtain work, a high-priority public policy issue.
     The comprehensive evaluation design included several major study components:
     * Process study of implementation of the demonstration at all eight sites;
     * Participation analysis focusing on targeting and self-selection among eligibles, including comparisons of the characteristics of volunteers who participated in the demonstration and the characteristics of eligible persons who did not volunteer;
     * An experimental study based on the random assignment of volunteering participants to treatment and control status to measure the net incremental effect of case management on the receipt of DI and SSI disability benefits, earnings, and other outcomes during the post-randomization followup period;
     * Supplemental statistical analysis to measure possible waiver effects on both treatment and control cases; and
     * An analysis of the overall costs and benefits of Project NetWork from the perspective of disabled study participants, taxpayers, and various levels of government.
     This article focuses on the "net outcomes" (or net impacts) of the Project NetWork demonstration. The phrase net outcomes connotes ultimate results, such as the longer term effects on earnings, benefit receipt, and other outcomes. However, from a broader perspective there are other outcomes of interest that also relate to process or intermediate variables. For example, the implementation outcomes are of interest in assessing the feasibility of implementing a complex demonstration design in different organizational and institutional settings. Likewise, the selection of participants among project eligibles is an intermediate outcome of great interest in its own right, and the intermediate outcome of service receipt is relevant for the interpretation of net outcomes, and directly enters the calculation of the overall benefits and costs of the demonstration.
     The evaluation is based in part on an extensive database obtained from both administrative records and personal surveys. Basic demographic data on both the 8,248 volunteers and the 138,613 eligible nonparticipants who lived in the demonstration areas but who did not volunteer for the demonstration were compiled from SSA administrative records based on a simulation of program eligibility rules.(2) These data serve as a basis for analyzing the decision to volunteer for Project NetWork. SSI and DI benefit information for the months immediately before and during the followup period were obtained from administrative records. Annual data on earnings covered by Social Security were compiled from the Master Earnings File (MEF) data system. In addition, baseline survey interviews of 3,439 randomly selected eligible nonparticipants and treatment and control group members took place near the start of the demonstration. Followup survey interviews of 1,521 treatment and control group members took place 2-3 years after random assignment. The surveys obtained information on issues such as health and well-being and attitudes toward Project NetWork.
     The article is organized as follows. The key features of the Project NetWork demonstration and implementation are summarized in section II. Section III provides information on the selection of participants who volunteered for the demonstration. This sets the stage for the analysis of net impacts since the impact results are conditional on the characteristics of the participants who volunteered for the demonstration.
     The next section presents a discussion of the data sources and methodology for the net impact analysis. Section V summarizes the estimated experimental net impacts of case management on earnings, the receipt of DI and SSI benefits, and other outcomes. Section VI provides the results of subgroup analyses of net case management impacts by demographic and programmatic variables.
     A summary of the results of the analysis of costs and benefits from various perspectives is presented in section VII. Section VIII details the key lessons learned from the Project NetWork experiment for the design and implementation of future demonstration evaluations. Lessons learned from the demonstration and a discussion of implications for new initiatives are presented in section IX.
    II. Demonstration Design - The Project NetWork demonstration was designed to test the efficacy of case management services in facilitating employment among severely disabled DI beneficiaries and SSI recipients and applicants who responded to the program's intensive outreach. This outreach to all members of this large target group was rooted in the belief that the traditional vocational rehabilitation (VR) system did not have the resources to serve many severely disabled beneficiaries, and that the program should instead reach out to the full range of disability beneficiaries.     It was believed that many persons with severe disabilities are good candidates for employment intervention if the appropriate mix of services is provided. Case management was seen as a tool for facilitating employment-oriented interventions customized for each individual, given that persons with disabilities face a range of barriers to work. It was hoped that these services would lead to increased earnings and reduced receipt of benefits. The demonstration tested whether these services could be implemented on a large scale.
     Voluntary participation was another important feature of this demonstration and in many similar employment and training experiments. On a smaller scale, a similar voluntary model has been used for the previous Transitional Employment Training Demonstration at SSA , which tested these services for SSI recipients who have mental retardation. The voluntary model is thought to facilitate the success of the intervention by providing services for only a self-selected group of motivated persons with disabilities. The voluntary model also responded to ethical and operational considerations that are important in setting up fair and workable demonstrations of alternatives to the status quo. Volunteers who participated were offered work-incentive waivers to facilitate the demonstration. Both the voluntary nature of the demonstration and the work-incentive provisions must be considered in interpreting the net outcome results.
     For the demonstration volunteers, case managers provided a variety of services, such as intake, face-to-face contact with clients, and the direct provision of job search assistance. They ordered and evaluated vocational assessments, referred volunteers to other service providers for job search assistance, classroom training, psychological counseling, physical therapy, business skills training, and other services. Most important, they were supposed to work with beneficiaries one-on-one. A substantial portion of their work was based on informal contacts, sometimes generating job leads. Often, case managers helped their clients to deal with several complex personal problems as well.
     The demonstration used four alternate delivery systems (models) to provide these case and referral management services. The four models had the same overall features, including identical outreach procedures and waiver provisions, but they differed somewhat in the implementation of the case management intervention. Each of the four models of case management was implemented in two of the eight demonstration sites. The first three models differed only in the nature of the organizational role and experiences of the case manager. In the SSA Case Manager Model , case management was provided by SSA staff. In the Private Contractor Model , case management was provided by private rehabilitation organizations. In the VR Outstationing Model , case managers came from state VR agencies and were "out-stationed" in local SSA offices. The fourth model, the "SSA Referral Manager Model" (, was designed to be less intensive and lower in cost: the focus was on referrals to other providers as opposed to direct services to clients.
     Project NetWork thus provided three principal bundles of services: outreach, waivers, and case management. It is important to note that these three types of services targeted different sets of DI beneficiaries and SSI applicants and recipients. The outreach component of the demonstration targeted the broadest of the three groups, while waivers were applied to a narrower subset, and case management to an even smaller, randomly selected subset. Understanding the relationship between these three groups is important for properly identifying the different evaluation questions that apply to the three groups and for interpreting the evaluation results.
     For intensive outreach purposes, approximately 150,000 DI beneficiaries and SSI recipients and applicants living in the demonstration areas ("project eligibles") were invited to participate without regard to the nature of disabilities. Outreach mailings targeted beneficiaries who were on the rolls during the demonstration without regard to their potential employability or interest in volunteering. Essentially this included all DI beneficiaries and SSI recipients aged 16 to 65 who were on the rolls. Similarly, all SSI applicants aged 16 to 65 who applied during the demonstration period were targeted for outreach.
     Work-incentive waivers were offered to 8,248 participants who volunteered for the demonstration. The 8,248 participants included two randomly assigned subgroups: treatment and control cases. The waivers were designed to facilitate work activity by project participants. Waivers (a) allowed referrals to public and private rehabilitation services in addition to state VR agencies only as required by current law; (b) provided that work performed for up to 12 months while in the project for purposes of determining a trial work period or substantial gainful activity not be counted; (c) provided that continuing disability reviews (CDRs) not be counted when an SSI participant moves into 1619 status. The most important waiver was to stop the TWP clock for 12 months among DI beneficiaries. The waivers were less significant for the SSI group.
     Finally, case management services were offered to 4,160 persons randomly assigned to the treatment group. As described earlier, case managers performed a variety of employment-related services directly or through outside vendors.
     The Project NetWork recruitment and intake process is shown in chart 1. The two boxes on the top of the chart represent the two separate streams of persons subject to outreach: the applicant stream refers to SSI applicants who were informed of the opportunity to volunteer for Project NetWork by case managers; the beneficiary stream represents persons already on the DI or SSI disability rolls who were informed through an outreach mailing effort.
     The flowchart shows that identifying participants was a multi-step process starting out with the individual expressing interest (pre-application) and ending up with an informed decision jointly made with the case manager to sign up after receiving more detailed information about the project. The box on the bottom represents the Project NetWork participants who volunteered for the demonstration. These participants were randomly assigned to treatment and control status with a 50-percent chance of assignment to each group. This randomization took place immediately after signing up for participation.
    The process study showed that all of the demonstration models were able to recruit large numbers of participants and to provide rehabilitation and employment on a substantial scale. The massive outreach targeting about 150,000 persons living in the demonstration areas through mailings to beneficiaries and field office referrals of new SSI applicants was successfully conducted, as was the implementation of all four case management models. The enrollment targets for voluntary participation were met: a total of 8,284 persons--98.6 percent of SSA's goal of 8,400--volunteered for the demonstration. Most participants completed assessment and employment planning and received some employment-related services in all models. The demonstration convincingly showed that broad-based return-to-work services can be implemented on a large scale in a variety of institutional arrangements.
     Most treatment group members who responded to the followup survey recall having met with their case/referral managers and had positive opinions of the helpfulness of Project NetWork. The intensity of interactions with case managers varied somewhat, although the vast majority reported that they met with a case/referral manager at least once. It is notable that about 3 of 4 respondents reported positive experiences with the case managers, but only about 1 of 3 reported that Project NetWork helped them to get a job.     According to the management information system that kept track of services purchased for clients by case/referral managers, about 45 percent of treatment group clients received purchased rehabilitation services. This figure is roughly similar to the 49 percent of treatment group members who reported receiving Project NetWork services in the followup survey.
     IV. Data Sources and Methodology -  SSA administrative records are the most reliable source of information on several key outcomes of interest for all persons in the demonstration. These data provide at least 30 months of post-random assignment data on SSI benefits, at least 42 months of post-random assignment data on DI benefits, and a complete benefit history during the pre-demonstration period.(3) These administrative records were also used to obtain the universe of eligible individuals solicited for the demonstration(4) and to collect basic demographic information, such as gender, race, age, and primary impairment, measured at the time of random assignment.(5) The Master Earnings File (MEF) provided annual (calendar year) SSA-covered earnings reported by employers. A management information system (MIS) recorded receipt by treatment group members of specific categories of services funded by Project NetWork.
     We also used data from in-person interviews with treatment and control group members conducted at baseline and followup. To estimate impacts, we relied on administrative data as much as possible because survey data were collected for only a subset of randomly assigned volunteers and may suffer from recall bias. Nevertheless, we used survey data to estimate impacts on outcomes not recorded in administrative data files.
     Baseline interviews were conducted with a sample of treatment, control, and nonparticipant cases from March 1993 through December 1993. A total of 3,439 baseline interviews were completed, including 2,555 with treatment and control group members, and 884 with nonparticipants. From June 1996 through November 1996, a total of 1,521 followup interviews were conducted with volunteers who completed a baseline interview.
     The survey instruments contain questions about education and training, health and functional limitations, transportation limitations, employment history and earnings, personal attitudes and outlook, income and benefits, emotional and cognitive status, receipt of training and rehabilitation services from Project NetWork and other sources, respondent assessments of Project NetWork, and knowledge of rules determining benefit levels, eligibility, work incentives, and the effect the demonstration waivers had on these rules.
     Impact Estimation Methods - The Project NetWork demonstration featured the use of a classical experiment to test the impact of case and referral management services on volunteers. The evaluation randomly assigned volunteers to either a treatment group or to a control group. The post-random assignment experiences of the control group members indicate what would have happened to the treatment group members in the absence of the demonstration services.
     Because random assignment generally ensures that the pre-random assignment characteristics of treatment and control groups are similar on average, any post-random assignment differences in outcomes can be interpreted as unbiased estimates of the incremental impact of demonstration services.
     Other designs for estimating impacts are often subject to selection bias, which will occur if the treatment and comparison groups differ in ways that are correlated with the outcomes of interest. As the participation analysis showed, demonstration volunteers differ from nonparticipants in terms of motivation and other characteristics that are not recorded in administrative records, so it would have been difficult to identify a comparison group of persons similar to the volunteers without the use of random assignment. Impact estimates based on random assignment are also more reliable than estimates based on "pre/ post" comparisons of outcomes before and after service receipt, because outcomes often change over the demonstration period for reasons unrelated to demonstration services.
     The estimates of impacts of demonstration services measure only the incremental impacts of case and referral management services for the self-selected group of volunteers. The experimental evidence cannot be generalized to the broader population of SSI applicants and recipients and DI beneficiaries, most of whom did not volunteer for the demonstration. For both treatment and control groups, the demonstration waived specific DI and SSI program rules considered to act as work disincentives, so the experimental impact analysis cannot isolate the effect of these waivers. A nonexperimental analysis showed no clear evidence of waiver effects on earnings and benefits.
     Impacts were estimated by comparing the outcomes of the treatment and control groups, using standard tests of statistical significance to determine the level of confidence we can have that the estimated impact represents a real effect, rather than a difference that could be expected on the basis of chance alone. In this analysis, any estimated impact that is larger than what could be expected on the basis of chance alone 90 percent of the time is deemed evidence of a real effect. To adjust for chance differences between the treatment and control groups and thereby provide more precise impact estimates, treatment/ control differences in outcomes have been adjusted by regression analysis wherever possible to account for any chance differences in the measured baseline characteristics of the two groups.
     The results of these standard hypothesis tests should be interpreted with care. Whenever an estimated impact is not statistically significant, two explanations are possible. The first is that Project NetWork truly had no effect on the outcome. The second is that Project NetWork really had an impact, but the size of the true impact is too small to detect given the available sample size.
     Another potential problem with these tests, which are intended to identify effects that have a probability of less than 10 percent of occurring by chance alone, is that there is also a 10-percent chance that a single estimate will be statistically significant by chance alone, even when the true effect is zero. At least a few of these false positives are bound to appear whenever we examine a large number of impact estimates for many outcomes and subgroups.
     Experimental Analysis of Net Impacts - Services
     According to the followup survey responses, Project NetWork increased the rate of receipt of return-to-work services by a statistically significant amount. Fully 69 percent of control group members reported receiving employment and rehabilitation services. The most common services were psychological counseling, physical therapy, assessments of work potential, and job search assistance.
     Project NetWork increased the percentage of volunteers receiving any of these services to 75 percent, a statistically significant impact. Treatment group members received, on average, significantly more job search assistance, business skills training (training in a trade or business school), and assessment of work potential than did their counterparts in the control group. It is important to note, however, that even where statistically significant, these treatment/control service differentials are not large; even without Project NetWork, many volunteers would have obtained services.    
    

[Editor's note: For more detail on the findings of the study referencve is made to Social Security Bulletin, Vol. 63, No. 1, 2000. ]

Comment