Hi, I'm Nick Hart, an expert on evidence-based policymaking and an advocate for using data to make better decisions in government. Ask Me Anything!

Nick how do you avoid the use of fact shaping in evidence based politics where, for example, two candidates can use the same data set from the same study and hand pick which facts matter and which facts don’t so they reach different conclusions for different agendas?

Thanks for the question. It turns out this is really tricky. The point of evidence-based policymaking is to have high-quality information that can inform decisions about policies. Rarely will the evidence direct a policymaker what the should do, which takes into account value systems and resource availability, as well as the body of evidence. Is an unemployment rate of 5% high, therefore justifying action, or low and justifying a focus on other priorities. How we determine whether a problem exists and how to define it can be shaped by evidence, but also relies on other factors.

In practice, it can be difficult and sometimes challenging to interpret what the evidence says or even means. For one reason, policymakers often aren't trained in reading or understanding scientific information. For another reason, sometimes there are conflicting studies that use slightly different methodological approaches and arrive at different conclusions.

So what do we do? Well, for starters there's a role for science communication, trusted intermediaries, and knowledge brokers to help responsibly convey findings. We can also engage in better efforts to make research transparent and openly available to encourage thoughtful dialogue about meaning, limitations, and implications for practice. We can also promote activities like having systematic reviews, which consider an entire body of research and information on a given topic to generate summary conclusions.

Hi,

I've worked on privacy and data policy, but one issue that comes up every time government use of data appears is privacy.

I'm wondering how you think governments can better balance the need for transparency, good use of data, and personal privacy? Especially as the goals of transparency and privacy can often conflict with each other.

Thanks for the great question. We know that when data about people or organizations are used to study complex problems, the privacy of data subjects (the unit of analysis) is important. This is especially the case when government is collecting data and has pledged confidentiality!

The U.S. Commission on Evidence-Based Policymaking was specifically tapped by Congress and President Obama in 2016 to study this question. They issued a report in 2017 with a unanimous set of recommendations that recognized the essential role of accessibility of data in parallel with strong privacy safeguards.

In short, these goals can be achieved simultaneously -- and there are great many examples where this is done well today! But as the uses of data become more complex, the American public rightfully should also expect their government to clearly articulate how government is using data and what the benefits of using those data really are.

Hi, Nick! Thanks for doing the AMA. Question : with all of the fire and fury over data and corruption from outside sources/countries.... what is being done to secure the data being used going forward for policy-making?

Great question! The U.S. government has a great infrastructure for protecting data -- though there can always be improvements. A commission on cybersecurity back in 2016 made some really targeted recommendations for government to step up its game. And in 2017 the Commission on Evidence-Based Policymaking made recommendations about ensuring senior leaders are in place to protect privacy as part of data lifecycle management.

Here's one new thing that is happening today -- government agencies are currently establishing chief data officers. This is a requirement that came about as part of a new law called the Foundations for Evidence-based Policymaking Act (P.L. 115-435). The goal is that this senior leader will have as a daily charge to work in partnership with systems experts (CIOs), privacy officers, and other agency leaders to ensure that data are protected appropriately but also beneficially used to gain insights about programs and policies.

It may seem like a small step, but today most agencies don't have someone filling this role. Moving forward, this alone will likely lead to significant changes in how we implement certain data practices in the federal government.

Hello Nick. For starters, thanks for doing the AMA. There are topics that are very partisan, with each side believing that they are being "factual". Which topics do you see getting a little more resolution (actually change in bipartisan opinion based on facts), and what resolution seems likely? Ex: gun control, abortion, immigration, socialized health care, etc

Thanks for the question. I agree. There are definitely lots of issues that incite personal and political values in our societal discourse. But that doesn't mean evidence can't or won't play a role. The comments below reference climate change, and I'd just say that some issues unfortunately take a longer time to get on the policy agenda than we would like. Major air pollution legislation didn't emerge in the United States until the 1970s, even though we knew much longer before about the potential for ill effects. Personally, I hope our elected leaders will give the existing and compelling body of research about climate change serious consideration. However, even that information will not dictate the course of action we should take as a country, or globally.

To your specific question, beneath the headlines and cameras there is a great deal of bipartisan collaboration and cooperation on a great many issues. In 2018, for example, Congress came together to pass major criminal justice reforms and major improvements to the child welfare system, among other actions. Both the CJ and child welfare reforms, also included targeted provisions to support evaluation and future study of the policies to ensure we actually achieve the goals that were intended. I find that to be an important component of good program design!

Moving forward, there are some bipartisan solutions that have already emerged in targeted areas related to immigration, education, health, and infrastructure that offer promising opportunities for bipartisanship in the future. And if it's not clear...I'm an optimist.

The data tells us that the skills and attributes required to be a successful politician - arrogance, dogmatism, self righteousness, lust for power - combined with the fact that almost none of them have a STEM background or an entrepreneurial background (many might have a business background, but usually one earned only by inheritance and nepotism and thus not based on their skills) mean you have an audience almost entirely uninterested in a pragmatic, evidence based approach to how they live their lives. How do you change minds among such a group who have huge incentives to stick to rigid points of view in the face of overwhelming evidence?

This is a fantastic question. Believe it or not, there is an entire field of research devoted to adult learning! But realistically in politics it can be difficult to convince individuals with long held ideological beliefs that there may be some erroneous nature to the view. At the same time, value systems are an important part of how we make decision in democratic society. Evidence-based policymaking is about ensuring that information is available and has a seat at the table for informing decisions, it's rarely about dictating what the decision should be.

To the extent our evidence is increasingly open, accessible, and available to both policymakers and the American public, it will force productive conversations and dialogue about what results mean and how to interpret uncertainty. I'm a believer in both the American electorate and the power of evidence.

1) Oftentimes evidence-based decision making relies on quantitative data. What are your thoughts on using qualitative data and mixed methods research and program evaluations?

2) Data requires interpretation and context to make decisions. How can open data and the new law provide that so that raw quant data isn’t pulled out of context?

3) Also, to what extent the nudge unit will have a say in how the policies are created?

Oftentimes evidence-based decision making relies on quantitative data. What are your thoughts on using qualitative data and mixed methods research and program evaluations?

Great question -- there's no doubt in my mind that we need both! Our questions drive the methods. I've seen excellent experimental studies that failed to incorporate implementation research that would identify what happened inside the black box for understanding how or why certain impacts occurred. If that knowledge is needed, then we will almost always need mixed methods designs with some qualitative data.

Data requires interpretation and context to make decisions. How can open data and the new law provide that so that raw quant data isn’t pulled out of context?

We can unfortunately never have a guarantee that someone won't take information out of context. How many times have you seen news media take averages of polling results, with blatant disregard for margins of error? It happens all the time. But open data and the requirements of the Foundations for Evidence-Based Policymaking Act will increase the likelihood that the American public will have access to certain core data to validate claims and assertions.

Have you read Bryan Caplans "The Myth of the Rational Voter?", and if so, what do you think about it?

Caplan looks at voter behavior through the lens of public choice economics, and argues that from the perspective of the average voter, their seemingly irrational behavior is not that irrational.

The value of a vote is negligible. To the extent that a vote has value, the cost of bad policy is socialized, and the benefits privatized.

At the same time, getting informed is costly. It takes time, and ridding yourself of whatever beliefs you hold that are proven false through study is painful. On the other side, using your beliefs as a shortcut to an identity, or as a way to fit in in your social circle has actual benefits, and it's not costly at all.

Basically, there are good reasons for why people treat politics as a tribal sport, and these reasons are endemic to the incentives found in a democracy.

My question is: How would you overcome these incentives such that evidence based policy has a chance at even coming into play?

I have not read it -- but will look into it!

I have read the vast literature on rational choice decision-making. That literature base suffers a similar challenge in satisfying the core assumption that actors are inherently rationale and capable of both accessing and processing an infinite amount of knowledge. It sounds like a similar premise.

In general I am an advocate of a concept called "knowledge brokering." My friend Karol Olejniczak has written extensively about the idea and even created a game/simulation to help policymakers understand how to use the concept in practice. The basic idea is a recognition that policymakers who want or need access to information often need some assistance because of the variety of demands placed on their time. That is, we can't assume they have an infinite amount of time to conduct new research or even read journals themselves. At the same time, we shouldn't assume the burden is on academics to figure out how to be useful to policymakers. Knowledge brokers serve as intermediaries who can help connect the dots, and offer technical knowledge about policy substance or how to devise a strategy for a dialogue on developing a strategy.

The Foundations for Evidence-Based Policymaking Act sets the stage for a type of brokering function -- through the establishment of what's called a "learning agenda."The basic idea is that government agencies and policymakers would be better at sending signals about what they need help addressing through the establishment of strategic plans for evaluation and research that offer targeted questions. Researchers then have access to information about policy-relevant questions they can answer that would b e timely to spend limited energy and resources addressing. In turn, more timely evidence can support questions that need to be answered.

Even then, brokering still requires folks who know what they're doing to translate and share knowledge. This is why the Evidence Commission recommended chief evaluation officers -- and the concept is embedded in the Evidence Act.

But yes -- much work to be done!

Do you agree with the NIJ recommendations for fixing gun violence versus the political suggestions popular today?

Thanks for the question. Note that the NIJ recommendations you link to appear to be for a somewhat narrower configuration of gun violence issues than the question and discussion might suggest.

There is no doubt in my mind that gun violence in this country constitutes a problem for which there are reasonable and meaningful policy solutions. That said, I do think our existing evidence is limited on what solutions will be most effective in different contexts (though that alone is not likely a sufficient reason for inaction).

One initiative that is worth highlighting here is the National Collaborative on Gun Violence Research, sponsored by Arnold Ventures. Laura and John Arnold deserve a tremendous amount of credit for taking the initiative to help out country fill in the gap in knowledge to understand how to make the country -- and schools -- safer with effective gun policy. They specifically acknowledge a belief that government under-invested in gun violence research for more than 20 years. I agree. While the National Center for Health Statistics has collected some data as part of a program they run for years, it stops short of the kind of policy research that is needed for real policy solutions to be debated and advanced. Personally I hope that this research becomes available and acted upon before my own son enters primary school in a few years.

"...Late last year Congress passed the Foundations for Evidence-Based Policymaking Act with overwhelming bipartisan support..."

This statement seems really fantastic on the surface; what is your perspective on how lawmakers will truly take this once they realize that evidence-based approaches do not allow for the reactionary or lobbied approaches that they are currently accustomed to?

I absolutely love your effort and it saddens me that more emphasis isn't put on this across our Western society. I believe that this is probably the most important issue facing the West right now. Thank you for your efforts.

Great question. To be honest, Members of Congress have a tough job. They are constantly bombarded with information from voters, constituents, media, researchers, lobbyists, friends, etc. We expect them to magically sift through the array of information to always make good and informed decisions. In practice, lawmakers often have an incredible staff operation that supports their work, helps them prioritize information, and enables translation or filtering of research. In my experience, I've seen Republican and Democratic committee staff work together on common solutions to addressing the opioid epidemic, resolving criminal justice disparities, strengthening the disability insurance program, and even building systems to encourage evidence-based policymaking. These staff used available evidence and expertise to tackle tough problems. The staff were then instrumental in working with respective Members of Congress to educate, consider their views, and ultimately address complex policy issues.

So if we're interested in helping lawmakers have the expertise to address complex challenges faced by our society, we should also identify ways to set up institutions to have the capacity to support defining problems and identifying solutions based on evidence. I've written fairly extensively about the issues facing the U.S. Congress in the past and lets just say there's definitely a lot of room for improvement, but also a lot of success stories!

​

What are some great examples from around the world in evidence based policy making?

The Commission on Evidence-Based Policymaking offered a few great examples in its final report. They referenced the work of Raj Chetty on moving to opportunity and understanding economic disparities. One of my favorite projects Chetty undertook was re-analyzing the data from the HUD Moving to Opportunity Study. When initially completed, HUD had only short-term outcomes. But Chetty extended the analysis with more years of information to generate new findings, which have revolutionized how we think about the potential benefits of housing vouchers. Just one example of many!

Hi Nick. To which Federal agencies does the Foundations for Evidence-Based Policymaking Act apply? And do all provisions apply to the same agencies?

Great question. And the short answer is, it depends! The Foundations for Evidence-Based Policymaking Act (P.L. 115-435) has three key titles:

• Title 1 of the law focuses on evidence capacity. It applies to the 24 departments listed under the Chief Financial Officers Act (e.g., DOD, EPA, GSA)
• Title 2 of the law is the OPEN Government Data Act and provides some directives around open data and establishing chief data officers. These provisions apply broadly to executive branch agencies.
• Title 3 of the law is something called the Confidential Information Protection and Statistical Efficiency Act. While virtually every executive agency is eligible to use these authorities, the White House's Office of Management and Budget has to approve each unit to ensure they can comply with the strong privacy safeguards built into the law.

Hi Nick. To what degree are we talking about using Data here? Are we talking about an app that pools information for specific topics for politicians? Using a crowd-sourcing platform to evaluate potential policy decision? Large scale models designed to project knock-on effects of certain decisions? What technologies are available today?

You ask a complex but important question. There are a great many ways we can use data to be productive in society, including for government. The proliferation and encouragement of open data will likely result in more APIs that offer targeted benefits through services, entertainment, etc. Open government data also puts the American public in the position of being able to support efforts to understand and evaluate whether programs and policies satisfy their intended purposes.

Title 2 of the Foundations for Evidence-Based Policymaking Act includes the OPEN Government Data Act, which is a directive that government agencies make more data open and publicly accessible when possible. We can't always do so, but when we can the benefits are almost endless.

Users, please be wary of proof. You are welcome to ask for more proof if you find it insufficient.

OP, if you need any help, please message the mods here.

Thank you!

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

This is a great post to see -- in a discussion about evidence-based policymaking!

I study quantitative analysis in school and focused my research on human trafficking. How do you approach policy questions where there is a major issue in the data collection?

Great question! You are essentially flagging that data quality is an important element of producing good evidence that is useful for policymakers. There are some circumstances and situations that likely face challenges in collecting high-quality information. I know agencies like the U.S. State Department have spent a great deal of effort in their evaluation operations thinking about data collection in conflict zones, bearing in mind the need to have quality data for their analyses. At the end of the day, if data quality is low or their are threats to either measurement validity or measurement reliability, we do need to be cautious in interpreting findings or results.

​

How does this square with President Trump's new panel filled with science deniers?

The Trump Administration deserves credit where credit is due. While not all decisions have strictly fallen in a favorable light for evidence-based policymaking, the administration's development of a Federal Data Strategy is an effort that we can applaud. In fact, I've said so publicly.

Hi Nick!

I also work with big data (finance/insurance) and when I heard about this evidence-based policymaking I was super interested. Thank you for doing the AMA!

Can you provide some examples or scenarios in which data analysis could help with policymaking?

I imagine there are tons of ways but I was hoping you can highlight some key ideas you and your team might have.

Thanks so much!

Thanks for the question. Here's one example -- I met with a federal agency today that was discussing how to improve data quality for a system that will help them understand how certain items are ordered and where cost discrepancies are. The existing infrastructure they have limits their ability to analyze anomalies, which means there may inadvertently be high costs where they shouldn't exist. This kind of basic cost analysis is essential, it is necessary, but may not always happen. Government has a lot of antiquated systems that need to be updated. And in some cases one system can't "talk" to another, limiting our ability to understand real impacts or key indicators that are essential knowledge for effective management.

More broadly, we face the same kind of issues in trying to think about long-term outcomes that are intended from many of our major government investments. We have a great many income support programs that aim to improve self-sufficiency for families by providing sustainable jobs or some threshold for income and earnings. Yet, many of these programs can't routinely measure the outcome of interest -- income and earnings -- because of limits on how government-collected data can be used. The Commission on Evidence-Based Policymaking described in its final report that income and earnings data are essential for understanding the impacts of some government programs -- and they included a targeted recommendation to say government needs to do better to make these data accessible for research and evaluation.

One potential option, for example, is to expand access to a resource called the National Directory of New Hires. It already exists and contains data about quarterly earnings, unemployment insurance, and when individuals in the US begin new jobs. But those data are restricted use by federal law, and can only be used for an incredibly narrow set of purposes. We could imagine the creation of a privacy protective environment (such as that created under Title 3 of the Evidence Act) where those data could be made slightly more accessible, but without reducing privacy protections. This would, in turn, mean that some of our key programs could have aggregate statistics that would help program managers and policymakers better understand program impacts. I mention this example because it is a low-cost, high-impact solution that exists that could vastly improve our ability to understand whether major federal investments are achieving their intended goals. This knowledge would be invaluable for framing future political debates about how or whether to modify those programs.

Hi Nick, Healthcare is one of the biggest topics in US politics. Most people want major healthcare reform, but they don't really know the best way to do it. The United States is in a weird position where our healthcare is not a free market, nor is it a single payer controlled system.

Have you or anyone you know collected data for evidence-based policy making on healthcare? What does the data say we should do?

BPC has a great project conducts research in the health care space. I won't pretend to know all the nuances myself. But what I do know is that health care inflation is a major factor in our society that we will need to come to terms with in coming years, especially as health care crowds out other priorities in budgets. One way to better handle this would be to study cost-effectiveness rather than just effectiveness in the health care space. it's surprising to me that we often put all our attention on what works best, but fail to also take into account whether the most efficacious approaches can also be reasonably implemented. More attention to cost-effectiveness research would go a long ways!

Do you claim not to be influenced by your own values and biases in the gathering, analysis and interpretation of data—that is, that you are conducting value-free science?

Not at all. I have values just like you and every single decision maker. It's important for researchers to be aware of their potential biases and value systems which affect everything from what we research to the types of data we collect. The American Evaluation Association even addresses this in the organization's principles and practices.

Nick do you find it ignorant that you titled it as 'in government' when most of the world is not in fact part of the U.S. goverment? This is nitpicking but honestly I think somebody so well educated would laugh at this.

Thanks for the comment. My work at BPC is focused on government. It's not to say that much of the same concepts and principles would not apply to businesses or in private life -- indeed they do!

Nick, Can you explain the pros and cons of Medicare for all? Most of what I read on the topic seems to be slanted left or right by the author.

Acknowledging the limits of my own knowledge -- I will admit that unfortunately I cannot. But I will ask my colleagues from the Bipartisan Policy Center's Health team to weigh in!

Nick, will the data you want collected be publicly accessible or only available to government officials and/or corporations?

Great question!

I spend a lot of time talking about government-collected data that can be made open and publicly accessible as well as confidential information that needs strong protections.

Take weather data, for example. Weather data pose not immediate privacy risk to an individual or company but offer tremendous potential for helping us understand conditions to plan for travel, recreational activities, etc. Weather data are also one of the great examples of what happens when government makes information publicly accessible -- we can realize the benefit of the data and use them as a strategic asset.

There are other types of information collected by government through the course of running programs where personal or business identifiers might be attached. In these cases, this sensitive information in complete form likely can't or shouldn't be released publicly (though there are exceptions). Think about records at the Census Bureau collected as part of the American Community Survey -- these data are protected because they contain important confidential information that is also useful for analyzing the data. That said, the Census Bureau is able to develop tabulations and statistics that are made publicly accessible to realize the benefit of these data -- creating a form of open government information that also leads to benefits for the American public.

​

Hey Nick,

Can you talk about how the federal government is largely failing to link its data sets, preventing important research and evaluation?

Thanks for the question -- this is a really important issue! The Commission on Evidence-Based Policymaking was created by Paul Ryan and Senator Murray in part to address this exact issue.

There are several barriers in place today. On the commission talked about explicitly was legal constraints, sometimes they are intentional but oftentimes they are not. The Evidence Commission recommended including legal clarifications to the extent possible (see CEP Rec. 2-3), and one step toward the legal clarification is included in Title 3 of P.L. 115-435 that President Trump enacted earlier this year. Hopefully it helps; but the language won't solve all problems. That's because we also know the capacity to engage in data linkages in a privacy protective and transparent manner is somewhat limited today. This is why the Evidence Commission recommended the creation of a National Secure Data Service (see CEP Recs. 2-1, 2-2, 4-1, 4-2, 4-3, 4-4, 4-5), as a way to establish a shared service center that could conduct linkages in a privacy protective environment, while also having public accountability and radical transparency in place. The Evidence Act (PL 115-435) takes a small step to begin implementation of the Data Service, by establishing a new advisory committee to plan for its implementation.

Much work would still be needed, including public discourse and congressional deliberation, before the Data Service can come to fruition.

Thank you all for tuning in for the AMA. I'm thrilled to see the enthusiasm for the topic and appreciate your excellent, thoughtful questions. At this point I am heading over to a federal agency to help them plan for implementing the new law, but we were able to respond to more than 40 questions during the AMA. Sorry we could not get to all! Here are some resources to help us continue the discussion:

• Background on the Foundations for Evidence-Based Policymaking Act
• A Primer on Evidence-Based Policymaking
• Our Research on the Evidence Use in Congress
• Learn about the Evidence Project at BPC

You can also stay up to date with what's happening in the field by following me on twitter at @NickRHart.

Hi Nick,

What can the average citizen do to be more direct about confronting individuals in power or position who are willingly spreading false or misleading information?

Thanks for the question. First thing, do your research. Take a look at credible sources with reliable and valid information when you perceive something might be off. In the words of Daniel Moynihan:

"Everyone is entitled to his own opinion, but not to his own facts"

Then, you should call it out and identify misinformation when it exists.

In our society, we can have healthy debates about how to solve complex, wicked problems. But we want these debates to be framed by facts.

Hey Nick, whats your favourite drink?

Boulevard Wheat. Hands down.

Proof: it's currently in my refrigerator.

So do you have a special pillow reserved for screaming into ? I would with your job

I do have a comfortable pillow, though my toddler handles the screaming in our house (coupled with lots of giggling...).

Obviously focusing on evidenced based policy, you have lots of free time through 2020. How do you spend all these hours?

Today I've spent over 3 hours on Reddit. Is that good? But yes -- much to do!!

Hi Nick,

What are your thoughts on how the quality of the data should impact how we use the data to make policy decision. Would the methodology of how the data be created an important consideration if it should be used for policy decisions?

Thanks for the question. Certainly, data quality is really important. We must always monitor measurement validity and reliability in determining how and even whether to go about using (or how to use) data in making decisions. We also know that the more data are used, the more likely we are to see improvements in quality over time. For example, when the Foundations for Evidence-Based Policymaking Act is fully implemented, researchers may still identify issues in existing administrative datasets. But once those issues are identified, we will be able to develop possible strategies for improving the data quality moving forward.

Does your work actually make a difference? Because last I checked government is based on politics and in politics fact and evidence don't matter

Thanks for the comment. Yes -- it does make a great difference. Decisions about policies are made by quite literally thousands of tremendous public servants and program administrators in hundreds of government agencies. For a great many of these decision-makers, they want their programs to be as effective and efficient as possible. Evidence-based policymaking is a strategy that helps them achieve this goal!

Hello Nick, what decision, based on the evidence and the numbers, has not been made that should have been made ages ago?

I like to describe the example in the United States of listing Saccharin as a hazardous waste. For years, we regulated this sweetener under one of the most onerous regulatory infrastructures (and important!) that exists in our country. Our initial research on the topic suggested the chemical might be a carcinogen -- but we later learned that the mode of action from that existing research wasn't something that was transferable from mice to humans. Thus, not likely a carcinogen. Even with this knowledge, it took our regulatory infrastructure over a decade to change the policy because there was not a routine process in place to evaluate these decisions. Fortunately, the best available science was eventually brought to light thanks to a petition process, and the regulations were revised.

​