Meta-Analysis for Public Management and Policy

Book excerpt: Praise for Meta-Analysis for Public Management and Policy "In his usual rigorous but readable style, Evan Ringquist and co-author Mary Anderson have produced a tour-de-force on the topic of meta-analysis in public policy and management research. Meta-analysis is badly needed in the all-too-common situation when researchers have low confidence in summarizing the overall results of dozens of studies on the effectiveness of some policy. This book has a nice combination of conceptual overview, methodological details, and applications that will make it possible for researchers to conduct their own meta-analysis. It is tempting to require all graduate students to write a meta-analysis as a chapter in their dissertation, or include meta-analysis as a standard offering in the research methods curriculum of social science graduate programs. The more people that adopt Ringquist and Anderson's approach, the less resources will be wasted on conducting studies that do not contribute to cumulative scientific knowledge. " —Mark Lubell Department of Environmental Science and Policy Director, Center for Environmental Policy and Behavior University of California-Davis “Ringquist and his colleagues deliver value and add to canon of public management methods by delivering an analytical framework that makes the case for systematic research using the tools of meta-analysis. This book will be a must read for all committed to strengthening evidence-based research that improves public policy and management decision making.” —David M. Van Slyke The Maxwell School of Citizenship and Public Affairs Syracuse University “In Meta-Analysis for Public Management and Policy Evan Ringquist and his colleagues provide a lucid and practical roadmap for policy and public management scholars who use meta-analysis in their research. But this is more than a “how to” volume; it provides background on why meta-analysis is a potent means for accumulating and synthesizing empirical research findings, and shows how its use has evolved in recent decades. Specific applications of meta-analysis to long-standing policy and management debates are given, essentially providing an array of developed “templates” through which scholars and practitioners can assess how to approach different kinds of analytical problems using meta-analysis. Particularly valuable to me is the careful development and presentation of the necessary stages of meta-analysis, from conceptualization through data coding and bias assessment to advanced modeling. All of the statistical analyses can be conducted in Stata, utilizing readily available “.ado” modules. I will use this book, both in research and in the classroom. Overall it is one of the most useful methodological contributions I’ve seen in some time.” —Hank Jenkins-Smith Department of Political Science Director, Center for Applied Social Research University of Oklahoma “Meta-Analysis for Public Management and Policy conveys the considerable untapped potential of meta-analysis to strengthen and advance bodies of knowledge and evidence in public management and policy. This book takes students and researchers deep into the methods of meta-analysis and details of their empirical application, without losing sight of the important policy questions and the implications of choices that researchers make in their empirical work for the production of evidence for public managers and policymakers. This book will serve as an excellent practical guide for those conducting their first meta-analysis, while at the same time supporting critically-focused consumption of existing meta-analyses and discussion of where the field can gainfully take this approach to enhance our research and knowledge bases. It draws in a range of valuable and important examples of applications of meta-analysis techniques throughout the book and rounds off with four full-fledged applications of the method. Although the book reaches out to an audience of public management and policy researchers and consumers of this research, it should be of interest to a broad range of applied social science researchers and students as well.” —Carolyn Heinrich Sid Richardson Professor of Public Affairs Director, Center for Health and Social Policy LBJ School of Public Affairs University of Texas – Austin “Even for incredibly specialized techniques, public management and policy scholars have a multiplicity of methods texts from which to choose. Yet it is truly surprising that a strong guide to applied meta-analysis — a rigorous framework for the organization of empirical findings — has not been available. Ringquist and Anderson provided just that with an accessible guide to sophisticated techniques. Marrying an instructive text to a set of exemplary standalone studies, Meta-Analysis for Public Management and Policy offers unparalleled guidance for instructors and students and more than a little wisdom for seasoned scholars. It is destined to become the standard reference for our field.” —Anthony Michael Bertelli CC Crawford Chair in Management and Performance USC Price School of Public Policy USC Gould School of Law University of Southern California “This comprehensive treatment of meta-analysis is an excellent guide for scholars and students in public management and public policy. The carefully done exposition demonstrates why meta-analysis should have greater use in the profession.” —Kenneth J. Meier Charles H. Gregory Chair in Liberal Arts Department of Political Science Texas A&M University “This remarkable book reviews the history of the use of meta-analysis in the social sciences, argues forcefully for its importance, value, and relevance for public managers, and provides one-stop-shopping for those who want to learn how to do it or understand how others have done it. The detailed coverage of each step in the process allows a student to learn the technique completely while fully understanding the logic and intellectual goals of the enterprise. Most importantly, the authors review techniques from a range of disciplines, drawing most of their positive suggestions from the field of medical statistics rather than the social sciences. The examples and applications, on the other hand, stem from the world of government and public policy. Four chapters provide new syntheses of research on individual policies using the techniques and practices introduced in the earlier chapters. The result is original research, a strong argument for the value of meta-analysis in a field (political science and public administration) that uses it little, and a complete tool-kit for those who would want to apply these powerful ideas on their own. A very impressive and useful text.” —Frank R. Baumgartner Richard J. Richardson Distinguished Professor Department of Political Science University of North Carolina at Chapel Hill “Meta-analysis is a valuable tool for accumulating knowledge about how management matters from across a range of policy areas and disciplines. It is also an underused tool, in large part because of the lack of a comprehensive and useable guide on the topic. Ringquist remedies this problem by offering clear instruction on how to apply the technique wisely, as well as highly useful empirical demonstrations. The field of public management needs this excellent book.” —Donald Moynihan Professor of Public Affairs University of Wisconsin-Madison “Professors and students frequently face decisions about how deeply to invest in a statistical procedure, a new technology, a new theory, or some other development in their discipline. The authors of Meta-Analysis for Public Management and Policy support such a decision about meta-analysis by making a convincing case for its value and increasing utilization, including such steps as a careful consideration of criticisms of the method. Evan Ringquist then provides clearly, engagingly written chapters on the major concepts, procedures, and issues in the techniques of meta-analysis. His coauthors then provide effectively-presented examples of meta-analytic studies about such topics as school voucher effectiveness, public service motivation and performance, and public sector performance management. The accessible and reader-friendly explanations, coupled with the illustrative examples that walk the reader through how to do it, make this a distinctively effective methodological text. In so doing, it offers a distinctively valuable resource for those of us who want to learn more about this important statistical method.” —Hal Rainey Alumni Foundation Distinguished Professor Department of Public Administration and Policy University of Georgia “James Heckman’s Nobel lecture described the combined influence of micro surveys, advances in computers and software, and the development and dissemination of multivariate statistical methods on applied economic research. His comments apply equally well to empirical research throughout the social sciences. These forces have created a “flood of numbers” and advances in technology since he wrote about them have assured that the process is accelerating. We need to transform the ways we learn from empirical analyses and create a science for the analysis of the secondary data from applied statistical and econometric models. This science would include methods for summarizing what has been learned from estimates and tests. It would provide methods for diagnostic screening of results to gauge the importance of modeling assumptions and the types of primary data for the findings being reported. Finally, it may well lead to the development of meta-models—integrating findings intended to describe a single system but viewed thru distinctive empirical lenses. Meta-analysis is a method that takes an important step in developing this science. It is a collection of methods that is a product of the transformation in applied research in the past half century. Initially much of this research was the domain of social scientists working on the evaluation of educational interventions. In these applications the primary data from different studies were routinely available, but the outcome and control variables differed across studies. As a result, the focus for these meta-analyses was on data combination with multiple, distinctive measures for asset of latent variables associated with the hypothesized underlying process. The texts describing meta-analysis focused on these situations. As applications of meta-analysis expanded to economics, political science, and sociology, the data structures changed. The new data came from empirical models –as estimated parameters or summaries of test results. The challenges posed in developing these types of data and understanding what they reveal were distinctly different. A text developed by scholars who appreciate how these types of summaries are different was missing until Ringquist and Anderson’s Meta Analysis for Public Management and Policy. Explaining a process that blends the best of qualitative and quantitative research is a challenge. This book has met this challenge and delivered researchers a great platform for teaching these methods to their students and for updating their own skills. At least four features distinguish this book: 1. The authors display a clear understanding of the strengths and the weaknesses of meta- analysis. Their treatment describes how care in data construction, variable coding, relevant statistical methods and, especially, careful attention to interpreting the findings from a meta-analysis can reinforce the strengths and mitigate the weaknesses. 2. There are real examples presented throughout the book along with a genuine understanding of the importance of the details in developing meta-analyses. 3. The coverage of relevant statistical methods is comprehensive and clear. And 4. The Appendices offer the detail researchers need to see in order to genuinely learn how to use meta analytic methods. It should be in the library of every serious teacher or practitioner”—V. Kerry Smith Regents Professor and W.P. Carey Professor Department of Economics Arizona State University “There are several texts for meta-analysis available, most notably “The Handbook of Research Synthesis and Meta-Analysis” by Cooper, Hedges and Valentine, but none specifically directed to public administration and policy scholars. In fact the points of emphasis and examples make the existing texts both difficult and poorly suited for the applied social sciences. Ringquist’s book is a spectacular success in filling this lacuna. Ringquist provides a clearer encapsulation of “the basics” in its opening section, and the “basics” are tailored to “problem-oriented” policy sciences (noting for instance, that meta-analyses in public management and policy will almost always use random-effects over fixed-effects). The empirical examples woven throughout as well as the actual analyses on PSM and school vouchers are exceptionally useful in identifying the stages of the process. At the same time, the book doesn’t spare the gritty details of confronting commonly required procedures, like bootstrapping and dealing with clustered robust SE, hierarchical modeling, etc. For readers with no exposure to meta-analysis, the text eases the transition by offering a refresher on how statistical techniques are used in original research, then how they differ when used in meta-analysis. Ringquist offers guidelines for syntheses, formulating problems, data evaluation, turning studies into data, techniques in meta-analysis, “the language of meta-analysis”, coding strategies and publication bias. The author also notes that the context and even techniques of meta-analysis are different for public management and public policy compared with medicine and psychology, and education. Public administration and policy analysis provide great opportunities for meta-analysis, but these fields also present considerable challenge. Great care is needed in synthesizing differently designed studies, which are observational and quasi-experimental or correlational designs, because the statistics of meta-analysis were originally developed to synthesize results from experiment design. Measurement issues are tricky because authentic scales are used less frequently than in psychology or medical research. In addition PA and policy as fields of scholarship are diverse and eclectic in research design which makes comparison of parameter estimates exceedingly difficult. Ringquist adroitly compiles an approach to meta-analysis adapted to reflect this context. While Section 1 consists of seven chapters, which discusses techniques of meta-analysis, Section 2 including Chapters 8, 9, 10 and 11 illustrates actual studies using meta-analysis conducted in public management and policy research: evaluating the effectiveness of educational vouchers, performance management in public sector, the effects of federal poverty deconcentration efforts on economic self-sufficiency and problematic behaviors, and the relationship between public service motivation and performance. The book is an easier read than other texts in it guides from project inception through lit review and analysis in a manner tailored to policy and management, and it actually provides a much more accessible and thorough coverage of many of the basic building blocks, random effects, r-based effect sizes, and bootstrapping, making it far more indispensable for any PA meta-analysis. The check-lists for coding articles are especially useful. Provision of Stata commands and practical data management suggestions (creating a command file for data set transformations, for instance) is a great advantage for this text. Adding an addendum with R programming options, in the next edition might be helpful too. The conclusion both compelling and concise but I would like to have seen some of the arguments presented here at the beginning of the book, reserving the conclusion for a fuller encapsulation of what the overall strategy of the book accomplishes in stages – rebutting criticisms that meta-analysis in social science is a waste of time because study estimates are non-comparable and effect sizes non-independent with careful examination of research design and models. This book is essential reading for any scholar in public administration and policy considering undertaking meta-analysis. I expect it will gain many readers in other social science disciplines as well. For serious users of meta-analysis Ringquist’s book will not be the only one on the shelf, but it is a valuable addition.” —Richard Feiock Augustus B. Turnbull Professor Askew School of Public Administration and Policy Florida State University