Appendix A
Panel Data-Gathering Activities
The panel conducted the bulk of its business in open forum, eliciting input from staff of the Economic Research Service (ERS) and the National Agricultural Statistics Service (NASS) as well as key data users, policy makers, and additional technical experts. The panel’s public sessions included a workshop on statistical methodology, June 8-9, 2006; a session for data users at the annual meeting of the American Agricultural Economics Association, July 24, 2006; a workshop on concepts and measurement, September 28-29, 2006; a workshop on inference, December 7-8, 2006; and an open discussion of cost-of-production issues, January 18, 2007.
At the panel’s introductory meeting in February 2006, Katherine Smith, then associate director of ERS, and Carol House, associate director of NASS, set the stage for the work of the panel. They discussed the purpose of the study from their perspective, and the charge to the panel. Their frank and open exposition on the need for a review of the survey were instrumental in focusing our attention on key issues to be addressed in the study and in enhancing our understanding of the partnership between these agencies that directs and sustains the survey. In Carol House’s presentation, the panel was challenged to consider not only the question of whether the agencies were doing the right things in terms of content, but also whether they are doing things correctly in terms of approach and methodology. Katherine Smith emphasized the critical importance of the survey as the main source of information on farm economics, production, and structure.
These presentations were supported and elaborated by a thorough discussion of the state of the survey by Robert Bass and Mary Bohman, who
addressed issues arising from the complexity of the Agricultural Resource Management Survey (ARMS): the complicated survey form, the problem of missing observations and nonresponse, the lack of comparability of measures across subsamples, and the difficulties in analysis imposed by the complex estimation and other procedures, among other topics.
At our second meeting, the panel focused on issues of statistical methodology and heard mainly from staff of NASS and ERS with responsibility for various aspects of survey design and data collection, estimation, and processing. In the discussion of questionnaire content, the panel learned from Doug Kleweno, Jim Johnson, Carol Jones, and Bill McBride about the determinants of the content of ARMS from the perspective of resource use, farm business performance, farm household well-being, and commodities, respectively. Two presentations were made on the process of turning concepts into questions, as well as the methods of testing and implementing the questionnaires by Danna Moore of the University of Washington, and Kathy Ott of NASS. Collection methodology and other statistical aspects of the survey were discussed by Bob Bass, Bill Iwig, Alix Riley, Richard Barton, and Chadd Crouse. In the final session, Phillip Kott, Bob Dubman, and Nigel Key introduced the challenging topic of methods of imputation and analysis of the ARMS data. The panel revisited the difficult issues involved in imputation with a complex survey design in our fourth meeting.
At the annual meeting of the American Agricultural Economics Association (AAEA) in August 2006, several panel members met informally with a large group of frequent ARMS data users to discuss accessibility of the data and data dissemination. The users also introduced issues regarding response rates, survey design, content, classification, editing and imputation, coding, cross-survey comparisons, and ongoing user forums. A summary of the discussion was reported to the entire panel at our third meeting.
The third meeting focused on the needs of major public- and private-sector data users and on data dissemination. Useful insights on data needs by major federal users were provided by Jim Langley, Joseph Cooper, and Joseph Glauber, representing the perspective of the Congressional Budget Office, the Council of Economic Advisers, and the Office of the Chief Economist, U.S. Department of Agriculture (USDA), respectively. Bob Young discussed the data needs of the general interest U.S. American Farm Bureau Federation, and Gary Adams laid out the requirements of the statistics program of the National Cotton Council, one of the major commodity-oriented interest groups. Discussing concepts and measures for understanding environmental analysis, commodities, and farm household well-being were Timothy Kiely and Arthur Grube, U.S. Environmental Protection Administration; Jeff Goebel, National Resources Conservation Service, USDA; and Roger Claasen, Bill McBride, Carol Jones, and Jim Johnson, ERS.
The panel conducted a lively discussion of issues involved in dissemination of survey data, particularly to users of individual record data for research purposes, supported by presentations by Mitch Morehart, ERS, and Ron Jarmin, U.S. Census Bureau.
Two substantive topics were revisited in the fourth and fifth panel meetings, which otherwise were largely focused on report preparation and discussion of potential recommendations. The panel delved more deeply into the problem of data analysis for complex surveys, with presentations by William Greene, New York University, who served as a consultant to the panel, Phillip Kott, and Jim MacDonald, ERS, who elaborated earlier presentations.
Bill McBride rejoined the panel to add detail to prior discussions of the treatment of commodities in ARMS, with a focus on the implementation of recommendations by a special AAEA panel on cost of production.
Agendas of the panel’s data-gathering meetings appear below.
MEETING AGENDAS
First Meeting
February 2-3, 2006
Goals for the first meeting:
Introduce panel members to each other and to the supporting staff.
Conduct the bias and conflict of interest discussion.
Hold an open session with representatives of the sponsoring agency and other experts to discuss the charge to the panel and to learn more about the Agricultural Resource Management Survey.
Organize the approach to the task.
Set dates for future meetings.
Thursday, February 2
12:00 – 1:00 p.m. |
Welcome and Panel Introductions Bruce Gardner, Chair |
|
Michael Feuer, Executive Director, Division of Behavioral and Social Sciences and Education |
|
Constance Citro, Director, Committee on National Statistics |
1:00 – 1:45 p.m. |
Overview of the Agenda Bruce Gardner, Chair |
|
Begin Discussion of the Statement of Work Panel Discussion |
1:45 – 2:00 p.m. |
Break |
2:00 – 3:00 p.m. |
Charge to the Panel |
|
Kitty Smith, Associate Director, Economic Research Service |
|
Carol House, Associate Director, National Agricultural Research Service |
3:00 – 4:30 p.m. |
Status of the Agricultural Resource Management Survey/Issues to Be Addressed |
|
Mary Bohman, Director, Resource Economics Division, ERS |
|
Robert Bass, Director, Census and Survey Division, NASS |
4:30 – 5:00 p.m. |
Bias and Conflict of Interest Discussion Kirsten Sampson-Snyder |
5:00 – 5:45 p.m. |
Discussion of ARMS Program Issues Bruce Gardner, Chair |
5:45 p.m. |
Adjourn for the Day |
Friday, February 3
8:30 – 10:00 a.m. |
Working Session |
|
|
10:00 – 10:15 a.m. |
Break |
10:15 a.m. – 12:00 p.m. |
Working Session Continues |
12:00 – 1:00 p.m. |
Lunch |
1:00 – 3:00 p.m. |
Working Session |
|
|
3:00 p.m. |
Adjourn |
SECOND MEETING
June 8-9, 2006
Goals for the statistical methodology workshop:
To discuss statistical methodology issues in the ARMS program.
To plan for the upcoming Concepts and Measurements workshop.
To develop an outline for the final report.
Thursday, June 8
8:30 – 9:00 a.m. |
Panel Business Bruce Gardner, Chair |
|
Bias and Conflict of Interest Discussion Kirsten Sampson-Snyder |
9:00 – 9:15 a.m. |
Welcome and Overview of the Agenda Bruce Gardner, Chair |
9:15 – 10:00 a.m. |
Interagency Interaction on Issues of Statistical Methodology |
|
Robert Bass, Director, Census and Survey Division, NASS |
|
Mary Bohman, Director, Resource Economics Division, ERS |
10:00 – 10:15 a.m. |
Break |
10:15 a.m. – 12:00 p.m. |
Survey Management and Data Collection Issues and Statistical Methodology Issues: Questionnaire Content |
|
Cynthia Clark, Moderator |
|
Information on Resource Use and |
|
Allocation at the Field Level |
|
Doug Kleweno, NASS |
|
Information on Farm Business |
|
Performance: Resources, Environment, |
|
Other Enterprise Information |
|
Jim Johnson, ERS |
|
Information on the Farm Household: |
|
Economic Well-Being |
|
Carol Jones, ERS |
|
Information on Commodities |
|
Bill McBride, ERS |
12:00 – 1:00 p.m. |
Lunch |
1:00 – 5:30 p.m. |
Survey Management and Data Collection Issues and Statistical Methodology Issues |
|
Fred Conrad, Moderator |
|
Questionnaire Design |
|
TBA |
|
Questionnaire Development |
|
Kathy Ott, NASS |
|
Collection Methodology—Current and Options (CATI, CAPI, web-based); |
|
Recruiting and Training of Interviewers |
|
Bob Bass, NASS |
5:30 p.m. |
Adjourn for the Day |
Friday, June 9
8:30 – 10:00 a.m. |
Survey Management and Data Collection Issues and Statistical Methodology Issues |
|
Arthur Kennickell, Moderator |
|
Sample Design and Frame (list and area) |
|
Bill Iwig, NASS |
|
Nonresponse (unit and item) |
|
Alix Riley, NASS |
|
Estimation and Imputation |
|
Richard Barton, NASS |
|
Nonresponse and Calibration |
|
Chadd Crouse, NASS |
10:00 – 10:15 a.m. |
Break |
10:15 a.m. – 12:00 p.m. |
Survey Management and Data Collection Issues and Statistical Methodology Issues: Methods for Analysis |
|
David Binder, Moderator |
|
Complex Design; Jackknife |
|
Phil Kott, NASS |
|
ERS Procedures for Statistical Analysis |
|
Bob Dubman, ERS |
|
Scientific Uses in Research |
|
Nigel Key, ERS |
|
Review of Statistical Packages for Improving Analysis |
|
TBA |
12:00 – 1:00 p.m. |
Lunch |
1:00 – 3:00 p.m. |
Working Session |
|
|
3:00 p.m. |
Adjourn |
THIRD MEETING
September 28-29, 2006
Goals for the concepts and measurements workshop:
To discuss conceptual and measurement issues in the ARMS program.
To explore the needs of data users.
To consider possible improvements in data dissemination, particularly for protected dissemination of sensitive individual records for research purposes.
To explore issues regarding tests of inference for regression analysis using ARMS data.
To further refine the outline for the final report and make plans for the upcoming meeting on December 7-8, 2006.
Thursday, September 28
8:30 – 9:00 a.m. |
Panel Business Bruce Gardner, Chair |
9:00 – 9:15 a.m. |
Welcome and Overview of the Agenda Bruce Gardner, Chair |
9:15 – 10:00 a.m. |
Report on the AAEA Data Users Forum Bruce Gardner, Panel Members |
10:00 – 10:15 a.m. |
Break |
10:15 a.m. – 12:00 p.m. |
Data Uses and Users |
12:00 – 1:00 p.m. |
Lunch |
1:00 – 5:30 p.m. |
Concepts and Measures |
5:30 p.m. |
Adjourn for the Day |
Friday, September 29
8:30 – 10:00 a.m. |
Tests of Inference for Regression Analysis |
10:00 – 10:15 a.m. |
Break |
10:15 a.m. – 12:00 p.m. |
Discussion of Methods of Analysis Issues David Binder, Moderator |
12:00 – 1:00 p.m. |
Lunch |
1:00 – 3:00 p.m. |
Working Session |
|
|
3:00 p.m. |
Adjourn |
FOURTH MEETING
December 7-8, 2006
Goals for the inference workshop:
To discuss issues regarding tests of inference for data analysis using ARMS data.
To further refine the outline for the final report.
To discuss preliminary findings and recommendations.
To make assignments for tasks for completing the final report.
To make plans for the upcoming meeting on January 18-19, 2007.
Thursday, December 7
8:30 – 8:45 a.m. |
Introduction to Problem of Inference and Summary of Prior Discussion on the Topic |
|
Bruce Gardner, Chair |
8:45 – 9:30 a.m. |
Issues in Data Analysis for Complex Surveys |
|
William Greene, New York University (Consultant) |
9:30 – 10:00 a.m. |
Alternatives to Current Methods of Estimation and Variance Computation |
|
David Binder, Discussion Leader |
10:00 – 10:15 a.m. |
Break |
10:15 – 10:45 a.m. |
Practical Issues in Applying Methods of Inference to Complex Data Sets |
|
Bill Greene, Discussion Leader |
10:45 – 11:00 a.m. |
Summary of Issues in Complex Data Analysis |
|
Bruce Gardner, Chair |
11:00 a.m. – 12:00 p.m. |
General Discussion of ARMS Issues |
|
Bruce Gardner, Chair |
12:00 – 1:00 p.m. |
Lunch |
1:00 – 5:30 p.m. |
Working Session |
|
Panel discussion of issues |
5:30 p.m. |
Adjourn for the Day |
Friday, December 8
8:30 – 10:00 a.m. |
Working Session |
|
Review of information gathered Refine outline for final report |
10:00 – 10:15 a.m. |
Break |
10:15 a.m. – 12:00 p.m. |
Working Session |
|
Assignments for preparation of final report |
12:00 – 1:00 p.m. |
Lunch |
1:00 – 3:00 p.m. |
Working Session |
|
|
3:00 p.m. |
Adjourn |
FIFTH MEETING
January 18-19, 2007
Goals for the meeting:
To discuss issues regarding cost of production estimates from the ARMS survey with ERS/NASS staff.
To further refine the final report.
To further develop sections written since the December meeting.
To make additional assignments for tasks for completing the final report.
To validate initial findings and recommendations.
To make plans for the upcoming meeting in March 2007.
Thursday, January 18
8:30 – 9:00 a.m. |
Discussion of Approach to this Meeting |
|
Bruce Gardner, Chair |
9:00 – 11:00 a.m. |
Discussion of AAEA Cost of Production Report and Status of Implementation of Recommendations in ARMS |
|
Bill McBride, ERS |
11:00 a.m. – 12:00 p.m. |
General Discussion of ARMS Issues with Department of Agriculture Staff |
|
Bruce Gardner, Chair |
12:00 – 1:00 p.m. |
Lunch |
1:00 – 2:30 p.m. |
Break-out Working Sessions |
2:30 – 5:30 p.m. |
Panel Discussion of Issues |
5:30 p.m. |
Adjourn for the Day |
Friday, January 19
8:30 – 10:00 a.m. |
Working Session |
|
Refine Final Report |