1

Reengineering the SIPP: The Evolution of a Phoenix

Joint Statistical MeetingsJuly 30, 2007

David Johnson

2

Reengineering the SIPP

3

The Unique Value of SIPP

• To provide a nationally representative sample for evaluating:– annual and sub-annual dynamics of income

– movements into and out of government transfer programs

– family and social context of individuals and households

– interactions between these items

4

Purpose of SIPP• Recommendation 2-1: The two primary goals for SIPP should be

to provide improved information on the distribution of income and

other economic resources for people and families and on eligibility

for and participation in government assistance programs. Within

these two goals, most attention should be paid to improving the

information for people who are economically at risk. A third

important but subordinate goal is for SIPP to have a capability to

respond to current policy needs for data in topical areas that are

related to the core subjects of SIPP.

– The Future of the Survey of Income and Program Participation, NAS, 1993

5

• Department of Agriculture

– Model food stamp eligibility and measure food stamp participation

• Department of Health and Human Services

– Measure the economic effect of disabling conditions on children and

adults, and determine "triggers" that cause people to go on or to go

off programs.

• Social Security Administration

– Model SSI benefits, and the restructuring of Social Security such as

age threshold changes.

• Congressional Budget Office and Congressional Research Service

– Use micro-simulation to measure participation in major government

programs

Current Uses of SIPP

6

7

Reengineering Goals

• Cost reduction

• Improved accuracy

• Improved timeliness and accessibility

• Improved relevance

Use of annual data collection

Focused content selection

Lower attrition rates

Integrated administrative data

Improved documentation

Improved processing system

Ongoing content determination

process; use of “hooks”

Use of ACS

8

Main Components of the Reengineering Process

• Improved Processing System and Collection Instrument

• Development of Event History Calendar• Meetings with stakeholders, development of

survey content, and use of reimbursable supplements

• Use of ACS for sample• Use of administrative records data to

supplement and evaluate survey data

9

Timeline of SIPP/DEWS/Re-engineered SIPP(The Order of the Phoenix)

Stopped work on converting existing SIPP to BLAISE and reprogramming post-data collection processing to SAS

December 2005

January 2006 Preparing for budget release, sessions with key federal stakeholders were held to discuss future SIPP plans

Publicly launched the “Program of Income, Wealth, and Health Insurance Measurement” and possible end of SIPP data collection in Sept 2006

February 2006

March/May 2006

CEPR circulated “Save the SIPP” letter. Census presented progress to date, and suggested using linked CPS data and administrative records data

June 2006 Brookings Workshop: Concerns expressed by stakeholders about possible data gap and usefulness of admin data. Stakeholder matrices sent to major users in order to identify critical items. It was determined that admin records would only assist in data quality evaluation.

10

Timeline of SIPP/DEWS/Re-engineered SIPP

First open stakeholder meeting held at Census. Publicly stated the Bureau was refocusing effort to use admin data to supplement data collection (not replace). New survey plans presented – EHC and ‘hooks’ to allow opportunities for reimbursable supplements. Announced plans to continue SIPP with reduced sample from 10/06-5/07

August 2006

Dec 2006/Jan 2007

March 2006

April/May 2006 Topical content meetings held

With additional funding, announced plans to continue SIPP with reduced sample through Sept, 2007

Another open stakeholder meeting at Census where more details were presented on content, structure, and plans for the new survey. Census presented plans for at CNSTAT Panel meeting.

June/July 2006 Senate and House Appropriations Comm propose additional funding to continue SIPP collection in 2008

11

SIPP ’04 Panel Collection2/04-1/08

(reduced sample 10/06-1/08)

2004 2005 2006 2007 2008 2009 2010 2011 2012

Improved PanelData Collection2011/12 and on

Reengineering Planning2006 – 2011

SIPP ’08 PanelFull Sample w/ TMSame as ’04 Panel

Dress RehearsalTest/Evaluation

9/09

EHC Paper Test

3/08

Current SIPP Status (subject to funding)

12

Improving Processing System

• Convert CAPI instrument to BLAISE• Document processing system

– Convert from FORTRAN to SAS– Document edit and imputation procedures

• Develop processing system in conjunction with survey for reengineered SIPP

13

Event History Calendars

• An EHC interview is centered around a customized calendar that shows the reference period under investigation

• The calendar contains timelines for different domains, for example, residence history, household composition, work history, and other domains relevant to the topic of study

• Landmark events, such as holidays and birthdays can be used to aid the respondent’s memory

14

EHC Format

• Survey Instrument –– Annual administration, Jan-April

– Follow movers

– Limited feedback

• Calendar –– Improvement on other designs by

integrating more closely with Blaise, the instrument programming language.

• Simultaneous development of processing and instrument

Front

Coverage

Demographics

Calendar Universe Screeners

Residence History

Marriage/Cohab/Fertility

Labor Force (emp/unemp/leave)

Soc. Welfare and Other Progs.

Health Insurance Coverage

Commuting, Work Schedule, Child Care

Assets / Balances

Annual Programs – Taxes – Lump Sum – Social

Insurance – Child Support

Adult Well-Being, Child Well-Being,Food Sufficiency

Health (Insurance, Expenditures, Disability

Wrap-up and MissingFollow-up visit/call Info

Back

Wkrs Cmp/Disab/Unemp Cmp

Landmark EventsEHC

CALENDAR

Type 2 Roster/Info

Relationship

Enrollment

15

AdditionalValidation

with Admin.Records

Prepare SOW for

Instrument & Training

2008 EHC Paper Test2007 2008

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep

Contract with RTIto prepare paper

EHC and Training

Field S

taffT

raining

Field EHCEvaluation

Test inMD & IL

Reference period for paper EHC Evaluation

2007 2008Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep

Field EHCEvaluation

Test

Coding

AnalysisEHC and SIPP 2004

ReceiveAdmin.Records

MD & IL

Administrative Records from MD & IL to validate SIPP - EHC

Census development of procedures for administering

the paper EHC Evaluation

SIPP 2004 Waves 10 - 12

16

Event History Calendar Conference

• "The Use of Event History Calendar Methods in Panel Surveys" December 5-6, 2007 at the US Census Bureau

• Agenda

– Examining the Impact of Event History Calendar Interviewing on Data Quality from Disadvantaged Respondents - Belli

– EHC in PSID

– EHC in NLSY97

– EHC in Netherlands

– EHC in LA Family and Neighborhood Survey

– EHC in British Panel survey

– Seam Bias in SIPP

17

Stakeholders and Content Review• Responding users indicated a broad need for most of SIPP core content.• About 40 stakeholders completed matrix• Select areas were added based on lost topical module content.• Held five subject area meetings to discuss specific content

– Health– General income/Government programs– Assets and wealth– Labor force– Demographics and other items

18

Use of ACS Data in Survey Design• Sampling Frame

– American Community Survey (ACS) interviewed cases

– Existing SIPP Primary Sampling Units (PSUs)– Goal is to use 2-years worth of ACS sample

• Oversample based on similar criteria as SIPP

• Sample size similar to SIPP (budget permitting)

19

Use of Administrative Records

• National-level prototype– SIPP linked with administrative records data at the

person level

• Creation of SIPP Synthetic Beta File matched with SSA data

• State-level prototype– Many social programs administered at the state-

level– SIPP data linked to social program data from the

states of Maryland and Illinois

20

CNSTAT Panel

• Goals of the panel are to evaluate:– the costs/benefits of various strategies for data linkage,

– accessibility of relevant administrative records,

– operational feasibility of linking administrative records

and survey data,

– quality and usefulness of linked data, and

– strategies for providing public access to the linked data

while protecting the confidentiality of individual

respondents

21

URL: http://www.sipp.census.gov/sipp/dewsSource: U.S. Census Bureau, Demographics Survey Division,

Survey of Income and Program Participation branch

Census Bureau Links:   Home · Search · Subjects A-Z · FAQs · Data Tools · Catalog · Census 2000 · Quality · Privacy Policy · Contact Us

                                                                                      

Assessing Users’ Needs

22

2008 EHC Evaluation

2009 Re-engineered SIPP Dress Rehearsal

2007 2008 2009 2010 2011Jan --- May --- Sep --- Jan --- May --- Sep --- Jan --- May --- Sep --- Jan --- May --- Sep --- Jan

2009 SIPP Re-Engineering – Instrument Dev.Processing and

Evaluation

SIPP 2008 Panel – Waves 1 – 9 (planned)

Eval. Analysis

Systems Tests/Preparation

2009 Re-engineered SIPP Dress Rehearsal Reference Period

Field A

ctivities

23

Matching SIPP responses with Social Security Administrative Data on

Retirement Benefits

NO Retirement

Benefit on

SSA file

Retirement

Benefit on SSA

file

NO Retirement

Benefit reported

in SIPP

84.8% 1.3%

Retirement

Benefit reported

in SIPP

2.9% 10.9%

24

Matching SIPP responses with Social Security Administrative Data on

Retirement Benefits

NO Retirement

Benefit on

SSA file

Retirement

Benefit on SSA

file

NO Retirement

Benefit reported

in SIPP

84.8% 1.3%

Retirement

Benefit reported

in SIPP

2.9% 10.9%

25

Matching SIPP responses with Social Security Administrative Data on

Retirement Benefits

NO Retirement

Benefit on

SSA file

Retirement

Benefit on SSA

file

NO Retirement

Benefit reported

in SIPP

84.8% 1.3%

Retirement

Benefit reported

in SIPP

2.9% 10.9%