False Positives in Scienti�c Research

Jonathan Libgober

Department of Economics, Harvard University

September 3, 2015

Abstract. �is paper develops a model of costly information acquisition, focus-

ing on an application to scienti�c research. When research protocols are not fully

transparent, scientists are incentivized to make their experiments more susceptible

to false positives, despite their preference for be�er information. On the other hand,

non-transparency can induce a scientist to undertake a costlier but more informative

experiment if it also enables her to commit to acting scrupulously. Our analysis sug-

gests, counterintuitively, that policies establishing greater transparency in scienti�c

methodology might therefore ultimately lead to some scientists undertaking research

that is worse for those interested in the results.

Keywords. False positives, sender-receiver games, information acquisition, experi-

mentation, transparency.

JEL Codes. D82, D83.

Any analysis that relies upon statistical inference inevitably risks arriving at an incorrectconclusion. Nevertheless, as argued in Ioannidis (2005), there are compelling reasons to believethat mistakes in published research cannot be explained by statistical error alone, and in fact arisedue to bias in experimental design and conduct. �is observation raises questions as to whatmight incentivize scientists to introduce bias in their experiments, and how policymakers shouldevaluate the e�ects of policies which seek to remove this bias.

Contact. jlibgober@fas.harvard.edu. I am deeply indebted to Drew Fudenberg, Ben Golub, Sco�Kominers, Eric Maskin and Tomasz Strzalecki for their encouragement and advice, and to Johannes Horner forextensive guidance during early stages of this project. I am also grateful to Philippe Aghion, Isaiah Andrews, VivekBha�acharya, Kirill Borusyak, Gonzalo Cisternas, Mira Frick, Jerry Green, Oliver Hart, Ryota Iijima, Yuhta Ishii,Harry Di Pei, Neil �akral, Juuso Toikka, Carl Veller, Alex Volfovsky and Heidi Williams for helpful conversations,and seminar participants at Harvard for excellent feedback. Any remaining errors are my own.

Jonathan Libgober 2

In response to the perceived high degree of wastefulness in biomedical research, partially dueto the prevalence of false positives, the prestigious academic medical research journal �e Lancetpublished a series of articles in January 2014 asking how to improve the e�ciency of fundingdecisions.1 While the series highlighted many potential problems in the publication and fundingprocess for biomedical research, it also discussed several concrete policy recommendations. Oneissue highlighted by the series is the lack of documentation requirements regarding researchprotocols, which some authors argued contributes to the incidence of false positives. In particular,one of these papers, Ioannidis et al. (2014), advocated for more widespread use of pre-registration,whereby scientists describe all of their planned research steps prior to experimentation. �eirexplicit policy recommendation is to “make publicly available the full protocols, analysis plans orsequence of analytical choices, and raw data for all designed and undertaken biomedical research.”While transparency requirements di�er across disciplines, researchers in a variety of �elds havepushed for similar policies. Indeed, some have even considered the e�ects of such policies ineconomics and social science (Miguel et al. (2014), Co�man and Niederle (2015), Olken (2015)). Anatural question is whether such policies will lead to be�er research being pursued.

To help answer this question, this paper formally analyzes the connection between trans-parency requirements and the incidence of false positives, or type I errors. In our model, a scientist(she) chooses an experiment with an observable outcome (success or failure) that is seen by adeveloper2 (he). �e experiment imposes a cost on the scientist, but provides information onwhether the developer might be able to successfully develop a drug (which would yield a bene�tto both players) by exerting costly e�ort, or whether such e�ort will be futile.

We assume that the scientist receives a positive bene�t if the developer is successful. Wealso allow for the scientist’s payo� to depend on (1) whether her experiment produced a positiveresult, and (2) the belief that her hypothesis is true. We demonstrate that with this speci�cation,the scientist has an incentive to acquire be�er information, provided the desire for a positiveresult in itself is not too strong. In fact, it will turn out that, without a preference for positiveresults or experiment costs, both developer and scientist would share the same preferences overexperiments.

�roughout the paper, we suppose that the scientist chooses a research proposal, which wethink of as a high-level description of the strategy used to address a problem, and a protocol,which we think of as all of the steps taken during the course of the experiment. Protocols di�er intheir propensity for false positives, and we will refer to protocols which are more susceptible to

1�e biologist Ed Wilson wrote a le�er in response that, while praising the series in general, speci�cally lamented thelack of involvement from economists.

2We use “developer” here in order to make our story more concrete. However, for interpretation purposes, we canthink of this person as someone intrinsically interested in the results of the scientist’s experiment, for whateverreason.

2

False Positives 3

such errors as “biased.” Fixing the choice of proposal, we assume more biased protocols yield lessinformative experiments. In practice, a proposal could be the basic description of the experiment,say, toward the beginning of a paper, whereas the protocol could correspond to all of the judgmentcalls to be made in the lab. �ough our analysis is general, to be concrete, one could imagine thatthe proposal re�ects the amount of data the researcher plans on collecting, whereas the protocolre�ects the number of speci�cations the researcher plans on testing. We are mainly interested incomparing the case where the research protocol is observable to the case where it is not, re�ectingthe transparency counterfactual.

Our analysis seeks to answer two questions. First, we address whether false positives can bea�ributed to the lack of transparency requirements. We show that non-transparency incentivizesthe scientist to follow biased protocols even when they are no less costly, despite the aforemen-tioned preference for be�er information. Bias is encouraged due to the fact that the choice ofprotocol cannot in�uence the perceived informativeness under non-transparency, and yet highersignal realizations still yield higher payo�s ex-post. Second, we address whether transparency isoptimal for the developer. Since non-transparency induces false positives, one might be temptedto conclude that transparency is always optimal, but we are able to show otherwise. We obtainthis result by noting that the research proposal (which is observed by the developer) chosen bythe scientist may di�er between regimes. While one proposal may be more costly than another,this extra cost may be worth it if the other experiment is devalued due to non-transparency. Ifthe more costly proposal is more informative and harder to bias, then non-transparency canincentivize the scientist to choose it, making the developer be�er o�. However, this conclusionis only true if the scientist’s preference for developer success is su�ciently important, sinceotherwise non-transparency will not have this bene�cial e�ect.

From a theoretical perspective, our paper studies the interaction between the incentives todistort an informative signal and the incentives for information acquisition. In our model, the la�eris represented by the convexity of the scientist’s expected payo�s as a function of the developer’sbeliefs. In contrast, the marginal bene�t from distorting can be most clearly seen by studyingthe slope of the expected payo� conditional on the state (i.e. the truth of the hypothesis). �eseparation between the incentives for distortion and information acquisition is the primary pointof departure from other models where agents can in�uence beliefs through signal distortion,most notably Holmstrom (1999) and Dewatripont, Jewi� and Tirole (1999). We will discussthe connection between our model and theirs throughout the paper, but we highlight that ingeneral this distinction is important. Indeed, our main result states that it can be be�er to havetransparency over the informative action, and non-transparency over the distortive action. Byexplicitly providing a distinction between these incentives, we hope our paper will provide adeeper understanding of these other models as well.

3

Jonathan Libgober 4

Our main model is a sender-receiver game where the sender’s action consists of a choice ofexperiment, and is therefore reminiscent of Kamenica and Gentzkow (2011). Our benchmarkmodel under transparency di�ers from theirs due to our restrictions on the set of experimentsthat can be chosen, and the presence of costs for experiments3. Both departures seem appropriategiven our main application, and are necessary for our main results—in our main model, a scientistthat could choose a fully informative experiment costlessly would always do so, provided thepreference for positive results is not too large. However, our counterfactual of non-transparencycorresponds to the case where the sender can commit to part but not all of the experiment, andthis feature does not nest within their model. We view our way of modelling partial commitmentas a feature that distinguishes our model from others o�ered in the literature.

We contribute to the aforementioned policy debate by clarifying the connection betweenthe underlying preferences of scientists and the merits of transparency requirements. �ere aretwo key features of the model which drive our conclusions: First, scientists care about followon research, and second, di�culty or costs associated with experiments in�uence experimentchoice. Taken together, these assumptions imply that when one dimension of the experiment isunobservable, the incentives driving the choice on the other (observable) dimension will change.It turns out that the change can be so signi�cant that the research ends up being less useful forthe developer under full transparency. Our formal model clari�es precisely why this is the caseand the conditions under which this will occur.

We proceed as follows. We highlight the main features of the model with an example in Section1, and proceed to introduce the model and its assumptions in Section 2. In Section 3, we considerthe scientist’s equilibrium behavior in the case where her research methodology is fully observedand the case where it is not. �is section shows when false positives are more prevalent due to thelack of observability on research protocols. In Section 4, we use this analysis to give interpretableconditions under which drug developers are be�er o� when scientists do not face transparencyrequirements. We proceed to consider a number of alternate speci�cations for the analysis inSection 5, review the literature in Section 6, and conclude in Section 7.

1. A SIMPLE EXAMPLE

Our main insights can be illustrated by the following simple example. �e scientist is endowedwith a hypothesis, the validity of which is given by a state θ ∈ {T, F}. �e scientist’s prior that

3Gentzkow and Kamenica (2014) study a version of this model with experiment costs. However, our se�ing willtypically violate their assumption that cost is proportional to informativeness of the signal. We think this violationis sensible for our se�ing, since it may be as costly for an agent to choose a biased experiment as it is to choose anunbiased one, and the two may be indistinguishable just from reading a research article. �eir paper would requireus to view these two as being distinguishable and with di�erent costs, both of which we believe to be out of linewith our particular application.

4

False Positives 5

θ = T is p0 = 1/4, and the scientist chooses an experiment, with some cost, which induces a(state-dependent) distribution over a signal y ∈ {0, 1}. We interpret y = 1 as the event that thescientist’s experiment produces a signi�cant result, and y = 0 as the event that no such resultwas obtained. Assume that the scientist �rst decides whether to perform an experiment of type aor type b, and then chooses which protocols to follow. Call a protocol d, with d ∈ {0, 1/6}, andsuppose the probability distribution on results is as follows:

Pa(d)[y = 1 | T ] = 2/5 + (3/5)d, Pa(d)[y = 1 | F ] = d,

Pb(d)[y = 1 | T ] = 1, Pb(d)[y = 1 | F ] = 0,

where Pe denotes the probability measure when experiment e is chosen. One can check that as dincreases, the experiment a(d) becomes less informative in the sense of Blackwell (note that ddoes not a�ect the informativeness of an experiment with proposal b, which perfectly reveals thestate θ).

An interpretation of the experiment type {a, b}, for example, might be the number of obser-vations collected, with a being a small sample or b being a large sample. In contrast, d could bethe aggressiveness of applying a treatment, or the number of speci�cations the experimentermight plan on testing. While such speci�cations for dmight not precisely yield experiments of theabove form, the key point is that �rst, these actions might make an experiment susceptible to falsepositives, while on the other hand, a positive result is already very likely when the hypothesis istrue, the dataset is large, and the experiment is unbiased. For simplicity, we suppose a large dataset guarantees a success if the hypothesis is true.

�e scientist cares about the beliefs of an interested observer, namely a drug developer wholearns about the viability of a drug from the experiment. �e developer sees y and updates hisbelief that θ = T from p0 to p(y). Suppose that the scientist receives a payo� of 1 with probabilityp(y) if θ = T and 0 if θ = F .4 Hence the scientist receives a higher payo� if the developer is moreoptimistic when the hypothesis is true, and receives no payo� if the hypothesis is false. Finally,suppose an experiment of type a costs 0, but an experiment of type b is costly.

Suppose the developer sees the complete experiment chosen by the scientist—that is, whethera or b was chosen, as well as the distortion parameter d. If an experiment of type b is chosen, theposterior jumps to 1 if θ = T , and 0 if θ = F , and hence the expected payo� (gross of the cost)from this experiment is just the prior, in this case 1/4. Le�ing πS(a(d)) be the expected payo�from choosing experiment a(d), one can calculate that πS(a(0)) = 1/8 and πS(a(1/6)) = 1/12.Since b is more informative than a(0), which in turn is more informative than a(1/6), we verify

4�is is a reduced form of a more complete set-up in which the developer chooses higher e�ort–thereby increasingthe probability of a success in the drug development stage–when his beliefs regarding the state are more optimistic.

5

Jonathan Libgober 6

that the scientist strictly prefers more informative experiments. Still, due to costs, she may notchoose b.

Now suppose that the developer cannot distinguish between the scientist’s choice of a(0) ora(1/6), but can observe whether or not b was chosen. In this case, if the scientist picks a(d), itwill never be an equilibrium to set d = 0. To see this, simply note that, as long as a(d) is chosen,the scientist’s expected payo� is always higher following a signal of y = 1 than a signal of y = 0.On the other hand, the developer cannot distinguish a(0) and a(1/6), and so in equilibrium hisbelief will not change with the choice of d. Since a(1/6) generates a higher probability of y = 1

when θ = T , it cannot be that a(0) is chosen in equilibrium.

Does this mean that the experiment the scientist chooses is less informative when d is unob-served? Not necessarily. To see this, suppose the cost of a type b experiment is cb = 3/20. �enthe payo� from choosing this experiment is πS(b(d))− cb = 1/4− 3/20 = 1/10. So, the payo�from b is less than πS(a(0)) = 1/8, and larger than πS(a(1/6)) = 1/12. So if the developer seesd, the scientist will choose a(0), as this gives her the highest payo�. But the previous paragraphargued that if d is unobserved, d = 0 cannot be chosen in equilibrium. Hence when d is observed,a(0) is chosen, but when d is unobserved, b is chosen. So even though making the distortionsunobserved prevents the scientist from choosing a(0), she ultimately chooses a more informativeexperiment in equilibrium.

�e example highlights three main messages which are useful for understanding our mainresults. First, the scientist exhibits a strict preference for be�er information.5 Speci�cally, theabove calculations veri�ed that the scientist would always prefer a more informative experimentif the extra information were free. We explain why this is the case formally in Section 3, andindeed such a formal description will be necessary in order to reconcile this observation withthe aforementioned incentives for distortion. Notice, however, that the scientist does not gainanything if the developer’s beliefs are unduly optimistic when θ = F , and in fact strictly prefersthat the developer have correct beliefs when θ = T . Furthermore, the more informative theexperiment, the higher beliefs are (in expectation) when θ = T . We will explain below how thestate dependent nature of the scientist’s payo� as a function of beliefs translates into an incentivefor information acquisition, which will allow us to distinguish these incentives from the incentivesfor distortion.

Second, despite this preference for be�er information, when d is unobserved the scientist losescredibility for scrupulousness. Since beliefs do not respond to the choice of distortion, and since theposterior rises if and only if y = 1, increasing d increases the probability that p is higher when

5�is does not follow immediately from Blackwell’s �eorem, since the scientist is not the decision maker who usesthe information. See Kim (1995) for a discussion of the use of Blackwell informativeness in a principal-agent modelwith moral hazard.

6

False Positives 7

θ = T . Hence even though the scientist would like to set d = 0, she cannot commit to doing sowhen d is unobserved, since the developer will realize that it is pro�table for her to deviate tod = 1/6. We also study this idea formally in Section 3 and highlight some of the subtleties in theextensions.

�ird, despite the loss of credibility for scrupulousness under non-observability, the scientistcompensates by exerting costly e�ort which is ultimately bene�cial for the developer. Since thescientist has a preference for be�er information, the only reason a scientist would choose a lessinformative experiment over a more informative one would be on account of costs. So by makingit impossible to commit to a(0), the scientist is induced to take a costly action which provesher scrupulousness, which in this case takes the form of choosing an even more informativeexperiment. In other words, the scientist needs to exert more costly e�ort in order to prove thatthe experiment is actually informative. �is illustrates an argument against pre-registration—bynot having pre-registration requirements, the incentives of researchers are shi�ed toward choosingexperiments for which bias is less likely to be problematic. �ese experiments, in turn, may bemore bene�cial to those who are interested in the outcomes of scienti�c research.

In light of the �rst two points, a key feature of the model is that the scientist is able to takecertain actions which increase the informativeness, and other actions which increase the bias.Of course, these two are related, since increasing the bias also decreases the informativeness,but in order to distinguish these incentives and study the possibility of partial non-transparency,we insist on viewing the action of the scientist as two-dimensional. In the description of themodel, we viewed the scientist as �rst choosing a or b, and then choosing d; since informationonly arrives a�er the full experiment is chosen, whether one views these decisions as being madesimultaneously or sequentially will not change any of the results.

In our model, we implicitly assume that the scientist cannot credibly make d observable on herown. �e conditions under which this assumption holds are beyond the scope of this paper. Still,there are many reasons why this could be the case—for example, without a formal mechanismfor doing so, credibly demonstrating that the protocols followed were scrupulous might imposeprohibitive costs on the scientist. If there is no way to verify or certify the experiment a�er theexperiment concludes, then this would be a reasonable assumption. In our model, we abstract awayfrom this, assuming that only an external agency would be able to impose such requirements, andtaking the incentives of this agency as either irrelevant or aligned completely with the developer.Still, the extent to which these kinds of actions could be credible is an intriguing question forfuture empirical work.

Our model in Section 2 di�ers in three respects. We �rst allow for richer preferences of thescientist, to illustrate what these e�ects rely upon. Additionally, we substantially generalize theset of information structures that the scientist chooses her experiment from, thereby clarifying

7

Jonathan Libgober 8

the features of the information structure which are necessary for the results. Finally, we introduceexplicit preferences for the developer in order to provide a microfoundation under which thescientist’s preferences will take the form described above. Doing so will allow us to relate thisanalysis to the application more directly, describe welfare in a meaningful way, and directly relateour work to other papers in the literature.

2. MODEL

A scientist is endowed with a hypothesis whose validity is given by θ ∈ {T, F}, drawn by nature.�e scientist is able to conduct an experiment, the results of which are of interest to a drugdeveloper. It is helpful to think of the scientist as being a sender, or an agent, and the developer asbeing a receiver, or a principal. All players share a common prior on θ, with P[θ = T ] = p0. Wewill always take p0 to be interior, and we will think of T as being the “good” state, and F as the“bad” state.

An experiment consists of two components—a research proposal, denoted by α ∈ A, and aprotocol, denoted by d ∈ [0, d]. We think of α as part of the experiment that the researcher wouldneed to describe to a funding agency, whereas the methodology consists of other details whichare determined in the lab. We allow each experiment (α, d) to have an associated cost, which wewill denote cα(d). We outline the assumptions on the cost function below.

For greatest simplicity we focus on the case where there are two possible research proposals;that is, A = {α1, α2}. We allow for the scientist to have private information regarding whichexperiments are available—let qα denote the developer’s belief that proposal α is available, andagain for simplicity we will take qα1 = 1.6 �is allows us to capture the scientist’s privateinformation regarding the capabilities of her lab, such as whether there are enough lab assistantsavailable for a potentially more intensive experiment. Assume that the outside option of thedeveloper is 0, and for the scientist it is su�ciently low so that she will always perform anexperiment if possible.7

�e experiment produces an outcome y ∈ {0, 1} according to a distribution that depends on θand (α, d). �is outcome y ∈ {0, 1} is observable to the developer, and we will refer to the eventy = 1 as a “positive result,” and y = 0 as a “negative result.” A false positive occurs when y = 1

and θ = F .A�er observing y, the developer updates his prior from p0 to p(y). We will assume for most

of the paper that the developer observes α, but we are interested in comparing the case where6As we explain in Section 4, this form of private information is necessary in order to rule out a solution where thefunding agency dictates that the scientist must choose α2. With the type of private information we assume, givingthe scientist discretion will be strictly optimal, as α2 will never be chosen with probability 1. See also Armstrongand Vickers (2010) for another model where the agent’s private information is the set of feasible projects.

7We will return to this point in Section 5.2.

8

False Positives 9

he also observes d to the case where he does not. Since the developer’s beliefs will depend onequilibrium behavior, we will typically suppress the dependence of the posterior p(y) on theexperiment choice of the scientist, but occasionally we will make this dependence explicit whenthe choice is known by denoting the posterior p(α,d)(y).

�e developer chooses a level of e�ort e ∈ [0, 1] at cost cR(e) = k2e2. �is determines the

realization of a random variable x ∈ {0, 1}, the distribution of which is given by:

P[x = 1 | θ = T, e] = e, P[x = 1 | θ = F ] = 0.

We think of x = 1 as the event that a drug is developed, and x = 0 as the event that it is not. Ifthe drug is developed, the developer obtains a payo� of b > 0, and the scientist obtains a payo� ofλx. We also suppose the scientist receives two additional payo�s; �rst, a bene�t of λp · p whenthe public belief is p at the end of the game, and second, a bene�t of λy · y. We assume thatλx, λp, λy ≥ 0. So the scientist receives an additional payo� when the result is positive (namelyλy), and may also exogenously prefer for people to believe that her hypothesis is more likely to betrue. Hence �nal payo�s for the scientist are

λx · x+ λy · y + λp · p− cα(d),

whereas for the developer, they are

b · x− k

2e2.

To focus the analysis, we impose some additional assumptions on the model, which we willdefend as appropriate at the end of the paper (speci�cally, Section 7.1). Much of the analysis willdepend on the experiment when the proposal is α = α1, which will be partially informative.

Assumption 1. Under α1, positive results are not guaranteed in either state, but are more likely whenthe hypothesis is true. �at is, P[y = 1 | θ, α1, d] = hθ(d) < 1, with hT (d) > hF (d). Additionally,hθ(d) is increasing and continuous in d, for all d and θ.

Notice in particular that since hT (d) > hF (d), we also have p(1) > p(0) whenever α1 ischosen. �is assumption holds for the information structure in the example, where d is a garblingparameter. However, it will turn out that the key feature of the example is not garbling per se, butrather the fact that the distortive actions increase the probability of a signi�cant result both whenthe hypothesis is true and when it is false. For example, testing a hypothesis with a larger numberof speci�cations will not only make a false positive more likely, it will also make a true positivemore likely since true positives are not guaranteed. Continuity is assumed for technical reasons,to ensure the existence of a pure strategy equilibrium when d is unobserved.

9

Jonathan Libgober 10

We impose the following condition on α2, which will be the experiment that the developerprefers but is more costly:

Assumption 2. �e experiment α = α2 does not change with d. Additionally, positive results aremore likely when the hypothesis is true—that is, P(α2,d)[y = 1 | T ] > P(α2,d)[y = 1 | F ].

�is assumption ensures that the advantage of α2 is its lower susceptibility to bias, and henceallows the scientist to commit to a more scrupulous protocol.8 Notice that since positive resultsare more likely when the hypothesis is true, beliefs are more optimistic following a positive result,as is the case when the proposal is α1.

Next, we will make the following assumption on the scientist’s costs:

Assumption 3. �e research proposal α2 is more costly for the scientist than proposal α1, no ma�erwhich value of d is chosen—that is, mind cα2(d) > maxd cα1(d).

So, for example, if α2 involves collecting a larger amount of data, it makes sense that thiswould be more costly for the scientist, if only due to the greater di�culty.9 It is this assumptionthat is the main source of con�ict between scientist and developer. Since the developer does notincur the costs of the experiment, he would prefer that the scientist simply choose the one thatis most informative. But if there are experiment costs, then she may not necessarily choose thedeveloper’s favorite option, even if she prefers be�er information (as we will show to be the case).

We also make the following assumption to ensure that developer e�ort is interior:

Assumption 4 (Interiority). Whenever the belief of the developer is between 0 and 1, so is his e�ortchoice. In other words, b < k.

�is assumption ensures that the scientist is always strictly incentivized to induce higherbeliefs for the developer, even when λp = 0. Without this assumption, it could be the case that thedeveloper’s e�ort choice is equal to 1 for all signal realizations, in which case there is clearly noincentive for the scientist to alter the experiment if λp = 0. On the other hand, as long as e(p) isstrictly increasing, the key features of the scientist’s incentives (preference for more informationand distortions under non-transparency) will hold.

2.1. Measuring Uncertainty

We introduce the following assumption to ensure that the developer prefers lower d:8We highlight that this assumption can also be relaxed, but we impose this assumption to highlight this proposal’sadvantage. See Corollary 5 or the discussion in Section 7.1.

9Notice that Kamenica and Gentzkow (2014) make an assumption on costs which is signi�cantly more restrictivethan ours (though they also allow for any information structure to be chosen in principle, which we disallow).

10

False Positives 11

Assumption 5. For proposal α1, increasing d decreases the variance of the developer’s posterior,E(α1,d)[p(y)2]− p20.

�ere is a formal sense in which this assumption is equivalent to saying that d increasesthe developer’s uncertainty, which relates our paper to others which have discussed measuringuncertainty—e.g. Blackwell (1953), Ely, Frankel and Kamenica (2015), Gentzkow and Kamenica(2014). Notice that Assumption 1 allows for a large degree of generality over information structures;so much generality that, without Assumption 5, it need not even be the case that the developerwould prefer a lower incidence of false positives. Since the case of interest for our application isone where false positives are socially harmful, an additional assumption is clearly necessary. Tosee what this would require, de�ne πR(p(y)) to be the developer’s utility when her belief is p(y),and observe that:

πR(p(y)) = maxebRp(y)e− cR(e),

One can check that p(y)πR(1)− πR(p(y)) is a measure of uncertainty as discussed in Ely, Frankeland Kamenica (2015) and Gentzkow and Kamenica (2014).10 In these papers, this de�nition was inturn motivated by Blackwell (1953), since a more informative experiment in the sense of Blackwellalways reduces uncertainty. Notice that, for a �xed experiment (α, d), since E(α,d)[p(y)] = p0, wehave the expected uncertainty is p0πR(1)− E(α,d)[πR(p(y))].

So, if we call p0πR(1) − E(α,d)[πR(p(y))] the developer’s measure of uncertainty, then thisassumption states that increasing d increases the developer’s uncertainty. Since cR(e) = k

2e2,

stating the the developer prefers experiments with higher posterior variance is formally equivalentto stating that a higher incidence of false positives increases his uncertainty.

3. SCIENTIST’S EQUILIBRIUM BEHAVIOR

We perform the analysis in two parts. In this section, we focus on how transparency in�uencesthe scientist’s choice of d, whereas in the next section we study conditions under which thedeveloper is be�er o� under non-transparency. Along the way, we will provide more economicand geometric intuition behind the forces that drive the scientist’s experiment choice. �e �rstresult shows that the incentives to acquire information arise due to the di�erence in the payo� ofthe scientist as a function of the developer’s beliefs across states—that is, because the returns todeveloper e�ort di�er when θ = T from when θ = F . Second, our �eorem 2 gives conditionsunder which bias is introduced under non-transparency.

�e scientist takes the e�ort choice of the developer as given, and the expected payo� froman experiment is a function of the posterior belief of the developer following y, denoted by p(y).10Indeed, this expression is 0 at p(y) = 0 and positive at p(y) = 1. Whenever cR(e) is strictly convex in e withc′R(0) = 0, we have that e(p) is strictly increasing and πR(p(y)) is strictly convex as well.

11

Jonathan Libgober 12

Payo�s are therefore:

p0E[λxe(p(y)) | θ = T ] + λpE[p(x, y)] + λyP(α,d)[y = 1]− cα(d) (1)

Whereas the scientist only cares about the developer’s belief conditional on θ = T , thedeveloper chooses e�ort a�er taking an expectation over the state. His payo� is proportional tothe expected variance of the posterior, and this does not involve taking a conditional expectation.However, we can write the payo�s of the scientist without conditioning on the state:

Lemma 1. In any pure strategy equilibrium where the experiment is correctly inferred as (α, d), thescientist’s payo�s can be wri�en as:

λpp0 + E(α,d)[λxp(α,d)(y)e(p(y)) + λyy]− cα(d) (2)

Proof. See appendix.

We explain some intuition for this lemma. Notice that the λx term in equation (1) �rst takesan expectation over signal realizations conditional on the state, and then takes an expectationover the state. In contrast, equation (2) �ips this order, �rst taking an expectation over the state,conditional on the signal realization, and then takes an expectation over signal realizations. Hencewhereas p0 multiplies λxe(p(y)) in (1), p(y) multiplies this term in (2). �ese two perspectives areequivalent because the scientist and the developer share the same information following eachsignal realization.

Most importantly, equation (2) clari�es that λx generates preference for information acquisition.It is well understood that convexity of payo�s in beliefs can generate incentives for informationacquisition, but it is hard to see how to apply this intuition directly from looking at equation (1).But if we do not condition on the state, as in (2), the payo� from the developer’s action is λxpe(p),and pe(p) is convex whenever e is increasing and not too concave, as is the case here.

To understand this further, notice that the scientist’s ex-post payo�s are state dependentwhenever λx > 0, because it is impossible for the developer to be successful if θ = F . Ifλx = λy = 0 and λp > 0, then the speci�cation of preferences would be a simpli�ed version ofcareer concerns, such as Holmstrom (1999) or Dewatripont, Jewi� and Tirole (1999). In thesemodels, the agent is compensated simply according to the belief in the state. �e novelty in ourmodel relative to these, therefore, is that λx induces state-dependence in the utility function.Furthermore, λp does not in�uence the incentives for information acquisition at all, since thedeveloper’s beliefs are a martingale and hence a constant. �ese ideas underlying the scientist’spreferences over information when λy = 0 are illustrated graphically in Figure 1. �is intuitionalso applies when λy > 0, though the graphical intuition is less clean due to the fact that payo�s

12

False Positives 13

no longer depend solely on the beliefs following each signal realization.11 Also note that theextent to which λy increases incentives for information acquisition depends on the prior, sinceP(α1,d)[y = 1] = p0hT (d) + (1− p0)hF (d).

In fact, not only are there incentives for information acquisition, the following immediatecorollary tells us the sense in which the preferences of the scientist and the developer are aligned,at least when λy = 0:12

Corollary 1. Suppose λy = 0. In any pure strategy equilibrium, the scientist’s expected bene�t froman experiment is linear in the developer’s expected revenue, bp(y)e(p(y)).

Corollary 2. Suppose λy = 0. �en if d is observable to the developer and cα(d) is weakly increasing,the scientist sets d = 0—that is, distortions are minimized.

Proof. Increasing d decreases the developer’s payo�s, by Assumption 5. With cR(e) = k2e2,

however, the developer’s payo�s are proportional to his revenue, which is proportional to thescientist’s payo�s. Whenever cα(d) is weakly increasing, d = 0 will hold if d is observed.

Remark 1. We brie�y discuss the role of Assumption 5 for this result, and how the analysis wouldchange if it were replaced. �is proof relies upon the fact that for the cost function we have chosen forthe developer, the developer’s cost, bene�t and pro�t are all proportional to one another. In fact, thisalignment of preferences would hold so long as cR(e) = ken for some n > 1, and the analysis wouldbe unchanged had we assumed this functional form for the developer (though as mentioned earlier,Assumption 5 would be di�erent). �e analysis would be similar for more general functional formsas well, though one would need an additional assumption in order for both the developer’s payo�and the scientist’s payo� to be decreasing in d, and we do not do so to avoid placing assumptions onendogenous objects.

We can contrast this result to what happens when y is all that ma�ers:

Corollary 3. Suppose λp = λx = 0. �en for λy su�ciently large and p0 su�ciently low, d > 0 inany equilibrium.

We now turn to the case where d is unobserved, and give conditions under which d 6= 0

in equilibrium. Even though the payo� in equilibrium can be wri�en as an expectation oversignal realizations, the gain to having higher values for d is indeed state dependent—even thoughincreasing d increases the probability of y = 1 in the state θ = F , the developer’s belief is moreimportant for the scientist if θ = T .11In particular, the line describing the ex-ante payo�s in Figure 1 would no longer be convex when λy > 0—instead,since p(1) > p(0), this line would have a jump of size λy at p0.

12Our interest in this case will become clear from Corollary 4.

13

Jonathan Libgober 14

�e following theorem gives conditions under which distortions occur in the non-transparencyregime. Fixing the choice of α, the condition on equilibrium behavior we provide in the �rst partof the theorem essentially states that when choosing d, the scientist equates marginal bene�ts andmarginal costs. While the former simply depends on cα(d), the marginal bene�t is the sum of threeterms, where each term depends on one of the three variables in�uencing the scientist’s payo�(namely, y, x, and p). While such a condition is intuitive, we highlight that the marginal bene�tdepends on beliefs, which are endogenous. Additionally, this intuition requires the existence ofa pure strategy equilibrium, which our theorem provides. If the scientist were mixing, then shewould have private information on the informativeness of the experiment (knowing the realizedvalue of d), and hence the developer and scientist would not share the same beliefs following eachsignal realization.13 On the contrary, our theorem guarantees that the scientist and developershare the same distribution over posterior beliefs. �e second part of the theorem provides acondition for this to be the unique equilibrium.

�eorem 2. Suppose that cα(d) is weakly convex in d and hT (d), hF (d) are weakly concave in d.

(1) A Perfect Bayesian equilibrium in pure strategies exists when d is unobserved. In this equilibrium,if d is interior we have:

c′α(d) = λy(p0h′T (d) + (1− p0)h′F (d)) (3)

+λx(e(p(1))− e(p(0)))p0h′T (d) (4)

+λp(p(1)− p(0))(p0h′T (d) + (1− p0)h′F (d)) (5)

whereas if the le� hand side is always smaller we have d = 0 and if the right hand side isalways smaller we have d = d.

(2) If either convexity of cα(d) or concavity of hT (d) is strict, there is no equilibrium in mixedstrategies.

Proof. See appendix. 14

A key feature of this theorem is that hF (d) does not enter into (4), the bene�t the scientistreceives when the developer is successful—this only ma�ers insofar as hT (d) and hF (d) both areincreasing in d. Even when the scientist only cares about the developer’s sucess, as in Section 1,an incentive to increase d still arises because doing so increases the probability that a true positiveoccurs. On the other hand, informativeness decreases because it also increases the probability of a13We will return to this point in Section 5.1.14In the appendix, we prove this theorem for more general cost function of the developer. As stated in the previousremark, we do not adopt more general cost functions in order to avoid placing assumptions on endogenous objects.

14

False Positives 15

false positive, and hence this is costly for the scientist. When the state is not fully revealed whenθ = T , there is an incentive to increase d, because higher signal realizations lead to higher payo�s.

We observe the following simple corollary by comparing �eorem 2 and Lemma 1:

Corollary 4. Suppose λx = λp = 0 and λy > 0. �en the scientist’s choice of (α, d) is the sameunder observability or non-observability.

�e intuition for this result is straightforward—if the scientist is purely interested in producinga success, then how the developer updates his belief is completely irrelevant. Since transparencyonly in�uences how the developer updates his belief, we have that the behavior is exactly thesame in the two regimes. With this corollary in mind, we will henceforth suppose λy = 0, as itshould be clear that a pure preference for positive results is not relevant to whether there shouldbe transparency over d.15

We have already seen that higher λp does not increase the incentive for information acquisition,whereas higher λx does. On the other hand, if hT (0) = 1—that is, if the experiment when θ = T

and d = 0 fully reveals that the state is good—then the incentives to distort only arise fromline (5), which is the incentive to push the beliefs higher. �is observation is suggestive thatexperiments that are more informative may also be less susceptible to distortion, since the changein the true positive rate will be much smaller relative to the change in the false positive rate, dueto the true positive rate being already closer to 1. However, as this would depend on the speci�cparameterization, we leave the question regarding the extent to which this is true open to futureempirical research. Hence we see that the incentives from λp and λx have di�erent e�ects on theincentives for distortion as well. �e following corollary highlights this:

Corollary 5. Suppose λy = 0, h′T (d) = 0, h′F (d) > 0 and c′α(0) > 0. �en d = 0 for all λx if λp issu�ciently small. In contrast, d > 0 if λp is su�ciently large.

Proof. By �eorem 2, under the hypotheses of the corollary, the equilibrium condition for interiord is

c′α(d) = λp(p(1)− p(0))(1− p0)h′F (d).

So d = 0 if λp ≤ c′α(0)(p(1)−p(0))(1−p0)h′F (0)

, and d > 0 otherwise.

�is corollary allows for a clear contrast with the career concerns literature. If λp = 0—that is,if the scientist were purely motivated by the developer’s success—then distortion only occurs if it15�is may be a surprising observation at �rst, since a common justi�cation for transparency is that it makes it easierto know how to interpret theoretical results. As also recognized by Glaeser (2006), however, in any pure strategyequilibrium, protocols should be inferred correctly, and therefore such an explanation is only compelling of there isuncertainty over which protocols are available. We do consider this extension in Section 5.1, but as we will see, thisspeci�cation is much less straightforward to analyze.

15

Jonathan Libgober 16

increases the probability that y = 1 when θ = T . In contrast, if λp > 0 and the scientist caresabout the belief that the hypothesis is true, then the prospect of increasing the probability thaty = 1 when θ = F drives the incentives for distortion.

We conclude with the following immediate corollary which is helpful for the following section.

Corollary 6. If c′α1(d) ≤ 0, then when d is unobserved and α = α1 is chosen, d = d in equilibrium.

To summarize: the incentives for information acquisition arise due to the state dependence ofthe scientist’s payo� as a function of the developer’s posterior, since the convexity of this line iswhat generates incentives for information acquisition. But the loss of credibility occurs due to thepositive slope of the scientist’s payo� conditional on θ—that is, because the payo� is still higherwhen the developer’s belief is higher. By devloping this model and comparing the in�uence of λpand λx, we have also shown why the forces highlighted are distinct from others that have beenproposed, most notably in the career concerns literature.

4. TRANSPARENCY REQUIREMENTS

�is section considers the e�ect of transparency requirements. Whereas the previous analysisholds α �xed and considers the scientist’s choice of d, here we consider how transparency a�ectsthe choice of α. We recall that transparency may not be optimal for the developer due to thethird feature of the example—namely, since the lack of credibility over scrupulousness leads tocompensation by choosing the more costly proposal. Our theorem clari�es the forces which thisconclusions relies upon.

It is easy to see that the scientist is always strictly worse o� without transparency requirements,since it essentially prevents her from commi�ing to a particular information structure. �roughoutthis section, we will assume cα(d) does not depend on d, which by Corollary 6 implies that whenα = α1 is chosen and d is unobserved, d = d. For greatest clarity, we will take cα1 = 0 withcα2 > 0, and we will also assume that α2 is fully informative, though the underlying messages arenot sensitive to these assumptions.

Recall that our interpretation in this exercise is a funding agency who can approve an ex-periment, and can require that d be made observable to the developer, but cannot dictate theactual choice of d. If qα2 = 1, then the funding agency can simply reject proposals of α = α1, andthis will maximize the payo� of the developer. However, when qα2 < 1, the scientist has privateinformation regarding the set of feasible experiments, and hence such rejections may force thescientist to take her outside option.16 But under non-transparency, the funding agency sacri�ces16�is concern is also present in Armstrong and Vickers (2010), who consider a general model of delegated projectchoice.

16

False Positives 17

p(y)

0.2

0.4

0.6

0.8

1.0Scientist Payoff

Unconditional Payoffs vs. Developer Posterior (λp=0)

p (0) p0 Ep (y) T p (1)

πs (α,d)

p(y)

0.2

0.4

0.6

0.8

1.0Scientist Payoff

Payoff Conditional on θ=T vs. Developer Posterior (λp=0)

p (0) p0 Ep (y) T p (1)

p(y)

0.2

0.4

0.6

0.8

1.0Scientist Payoff

Unconditional Payoffs vs. Developer Posterior (λp>0)

p (0) p0 Ep (y) T p (1)

πs (α,d)

p(y)

0.2

0.4

0.6

0.8

1.0Scientist Payoff

Payoff Conditional on θ vs. Developer Posterior (λp>0)

p0 Ep (y) T Ep (y) F

θ=F

θ=T

Figure 1: Graphical illustration of the model when λy = 0 (in particular, Lemma 1). All graphsexpress the scientist’s payo� as a function of the developer’s belief. �e top row considers themodel where λp = 0, and the bo�om row considers the model when λp > 0, where were normalizethe ex ante payo� when p = 1 to 1 for both cases. �e le� column displays payo� of the scientistfrom the ex-ante perspective, taking an expectation over the state. Notice that the line is moreconvex—and hence there are more incentives for information acquisition–when λx is relativelylarger than λp. In contrast, the right column writes the scientist’s payo� conditional on each state.Lemma 1 relates the le� column and the right column. Notice also that is clari�ed from thispicture, since the payo� line conditional on θ = F is �at (i.e. 0) in the upper right picture, butupward sloping in the bo�om right picture.

17

Jonathan Libgober 18

some informativeness of α = α1 in a way that could align the incentives of the scientist and thedeveloper.

Recall that πR(p(y)) = maxe bRp(y)e− cR(e) is the developer’s utility with belief p(y). Wehave the following theorem:

�eorem 3. Let λx = βλ and λp = (1 − β)λ, se�ing λy = 0. Take cR(e) = 12e2, and assume

experiment costs are constant in d and α2 is fully informative. Non-transparency is strictly optimal forthe developer if cα2 is signi�cant but not prohibitive (that is, in a range [c, c] with 0 < c < c <∞),qα2 is less than 1 but not too small, and drug development is important for the scientist (i.e. β > 0).

Proof. See Appendix.

Hence, one feature of this theorem is that given the other parameters of the model, thereare always values for β, cα2 and qα2 such that non-transparency is optimal. Notice that in orderfor non-transparency to be strictly preferred, the preference for information acquisition must besu�ciently strong. If this is not the case, then the scientist may not �nd it worth it to take theaction that commits to acting scrupulously (i.e. se�ing α = α2). While the assumptions on thecost function are meant to direct the analysis at this tradeo� explicitly, we remark that one shouldbe able to obtain a similar theorem with more general cost functions and information structures,as long as there is a channel through which our compensation e�ect can arise. A preference forinformation acquisition is the crucial feature which allows this force to arise.

4.1. What about complete non-transparency?

Our main counterfactual regarding the observability of d is primarily due to the relevance for ourapplication, but in general we should emphasize that the type of non-transparency ma�ers forthe conclusion of the above theorem. We illustrate this idea heuristically, showing that partialnon-transparency can outperform full non-transparency as well. Imagine that the developercould observe neither the research proposal α, nor the protocol d, but would need to infer bothin equilibrium. Suppose momentarily that qα2 = 1 and β = λ = 1. Again taking α2 to be fullyinformative, notice that in order for this to potentially be optimal, this must induce the scientistto undertake the fully informative experiment, and for this to be an equilibrium we would need:

p0 − cα2 ≥ p0hT (d). (6)

Compare the right hand side of this equation to the rightmost side of (9), which we can write as:

p0(hT (d)e(p(1)) + (1− hT (d))e(p(0))

).

18

False Positives 19

We consider conditions under which (6) implies (9), which will be true whenever

hT (d) > hT (d)e(p(1)) + (1− hT (d))e(p(0)).

To help interpret this expression, suppose (α1, d) is barely informative. �en e(p(1)) ≈ e(p(0)) ≈p0, and we have

hT (d) & p0 (7)

Now we are able to provide a sense in which partial non-transparency will typically be be�erthan complete non-transparency. If α1 is very uninformative when fully biased, and either hT (d)

is high or the prior that θ = T is low, then partial non-transparency will be no worse than fullnon-transparency and strictly worse for some cα2 . We therefore maintain our insistence thatpartial non-transparency is an important benchmark.

5. EXTENSIONS

�is section discusses the robustness of the aforementioned results. While we have sought torigorously clarify the mechanisms underlying the three features of the main example, it is stillnatural to ask how the conclusions might change were we to use alternate speci�cations. We doso in this section. We focus on the case where λy = 0 in order to highlight the issues raised byeach extension most clearly.

5.1. Private Information on Distortability

In the version of the model presented in Section 2, whether or not distortions are feasible iscommon knowledge. �is is a sensible assumption if, for example, the research proposal fullycharacterized the set of methodologies which the scientists would have access to. On the otherhand, if the scientist has specialized knowledge about the experiment in the �rst place, then theymay also have private information information regarding the set of possible d. �is extensionshows how private information changes the geometric intuition from the previous sections.

We augment the model by giving the scientist some additional private information. Speci�cally,we suppose that with probability t, the scientist can choose any d ∈ [0, d], and with probability1− t, they are forced to choose d = 0. �e model presented in the Section 2 is a special case ofthis more general model, when t = 1.

Figure 2 illustrates this situation, focusing again on the case with costless choice of d. �ereare two features which distinguish this version from the previous analysis. First, under non-transparency, the developer’s beliefs need not form a martingale from the perspective of thescientist, when the experiment α1 is chosen—that is, for the scientist, E[p(y)] 6= p0. To see this,

19

Jonathan Libgober 20

p(y)

Scientist PayoffPrivate Information on Distortability

p0

A

B

C

D

E

Figure 2: Illustration of the extension in Section 5.1. �e dashed line corresponds to experimentα2, with scientist payo�s given by point A. �e do�ed lines correspond to the experiment α1: �ehigher do�ed line corresponds to the case where the developer assumes the scientist will chooseα2 whenever it is available, and the lower do�ed line corresponds to the case where he assumesshe will choose (α1, d) if possible, even if α2 is available. �e expected belief of the developerwill depend on the scientist’s private information, and will therefore no longer be a martingalefrom her perspective, in contrast to the baseline model. �e points C and E represent expectedpayo�s if distoritons are feasible (hence to the right of the prior), and points B and D representexpected payo�s if distortions are infeasible (hence to the le� of the prior). If the cost of α2 movesthe payo� from A to a point lower than C but higher than E, then the equilibrium behavior willinvolve mixed stratgies.

20

False Positives 21

note that with probability t, the scientist chooses d > 0 when choosing α1, and with probability1− t the scientist is forced to set d = 0. Hence the probability that y = 1 is larger for the scientistwho is allowed to distort than it is for the scientist who cannot. But the developer cannot observewhether the scientist is able to distort, and his beliefs are a martingale from his perspective. �eresult is that if the scientist can pick d > 0, then E[p(y)] > p0, but if the scientist cannot, thenE[p(y)] < p0, emphasizing again that p(y) is the probability the developer, and not the scientist,assigns to the event that θ = T .

To see the second di�erence, notice that when considering the informativeness of an experi-ment, we were able to treat the experiment choice as given and then ask which level of distortionswould be picked. In this case, however, the informativeness of the signal α1 will depend onwhether a player who is able to distort would prefer to choose α2 or α1. If the scientist choosesthe proposal α = α1 when distortions are available, then the experiment is less informative thanit would be if she were to choose α = α2 when distortions are available. When t = 1, these twocases are the same, but in general they need not be. While it is still true that the scientist distortswhenever possible when α1 is chosen, the point is that in this extension, whether α1 is chosenbecause α2 is unavailable or because distortions are possible depends on equilibrium behavior.

�emain di�culty in this extension arises because there need not be a pure strategy equilibrium,and the scientist who can choose α2 or distort might mix between (α2, 0) and (α1, d). Indeed, itmay be the case that, without mixing, if the developer thought the scientist would always chooseα2 if available, the scientist would prefer to choose α1 and set d = d, whereas if the developerthought the scientist would only choose α2 if distortions were impossible, then the scientist wouldalways prefer to choose α1. Since these issues seem orthogonal to the main questions of thispaper, we do not develop this case in detail. However, we brie�y remark that one could stillprove an analog of theorem 3 for this case, and that non-transparency would still be optimal,with similar intuition.17 We remark that several other papers consider principal agent problemswhere the agent follows a mixed strategy due to a lack of commitment at the time of contracting;see, for example, Fudenberg and Tirole (1990). Still, to the best of our knowledge, we believe themechanism isolated here for mixed strategies in this se�ing is new.

5.2. Outside Options

�roughout the analysis, we assumed that the outside options of the scientist was su�cientlylow. �is assumption simpli�ed the analysis greatly, because it allowed us to focus on the casewhere the scientist always chose one of the two experiments in equilibrium, which allows for the17In fact, it would be optimal even if the funding agency had even more authority than we allowed in our main model,namely if they were able to limit the set of feasible d available to the scientist. �is is due to the fact that restrictingthe set of feasible experiments may prevent the scientist from undertaking any experiment at all, which would becostly to the developer.

21

Jonathan Libgober 22

simplest possible comparison. In practice, however, this is not without loss, so we discuss howthe problem changes with signi�cant outside options. �e analysis is straightforward when thedeveloper’s outside option is positive, so we focus on the scientist.

Suppose that the outside option of the scientist is uS , and that this greater than her payo�from choosing experiment (α1, d), but less than her payo� from choosing experiment (α2, 0). Inthis case, if the payo� under non-transparency is greater than uS , non-transparency may have theundesirable e�ect of having the scientist take the outside option, instead of the desired e�ect ofchoosing α = α2. Indeed, this should be expected since non-transparency unambiguously leavesthe scientist worse o�, which is more likely to cause a problem if her reservation utility is higher.However, we remark that when the outside option is signi�cant, whether transparency is optimalwill depend on whether the outside option can be exercised before the scientist sees the feasibleexperiments or a�er. If the scientist can only do so before, then as long as the expected payo� tothe scientist from non-transparency is higher than the outside option, the outside option will notbe chosen. However, If the scientist retains the right to exercise the outside option throughout thegame, then for these values of the outside option (that is, when uS is between her payo� from(α1, d) and her payo� from (α2, 0)), an extra ine�ciency is introduced from non-transparencydue to the fact that there is a chance that the scientist might choose her outside option insteadof choosing experiment (α1, d). In other words, if the outside option can be exercised a�er thescientist observes her private information, then it may be be�er for her to do so if she cannotchoose α2. In any case, it may be the case that transparency becomes optimal for a certain valueof the outside option when it was not optimal for lower values.

5.3. Transfers

�ere are many possible interpretations of the bene�ts to the scientist, and in this paper we preferan interpretation where these bene�ts are non-monetary. For example, if drug development issuccessful, other researchers may view this work as a valuable contribution, and follow-on byworking on similar problems. However, since there are many mechanisms by which monetarytransfers can be introduced, we think it is worth considering the case where the developer is ableto pay the scientist in order to undertake the more informative experiment.

�is scenario is illustrated in Figure 3, which plots the expected payo�s of the developer againstthe payo�s of the scientist. Each experiment corresponds to a particular developer-scientist payo�pair. When there are both transfers and transparency, we suppose the developer induces theexperiment (α2, 0) by paying the scientist the di�erence between her utility from α = α2 andα = α1 with transparency. In this case, the scientist prefers to choose α = α2 when it is available,and chooses α = α1 otherwise, while still not incurring a loss of informativeness by having d beobservable. �e expected ex-ante payo�s are a convex combination of the payo�s from each of

22

False Positives 23

0.00 0.05 0.10 0.15 0.20Scientist Payoffs0.00

0.05

0.10

0.15

0.20Developer Payoffs

Payoffs with Transfers

Transparency

Non-Transparency

α1,d

(α2,0)

(α2,0) with Transfer

Transparency with Transfer

Figure 3: An illustration of the optimality of non-transparency when there can be transfers.Transfers induce the agent to choose (α2, 0) when it is available under transparency. In thiscase, the payo� pro�le under each regime is a convex combination of payo� pro�les under twoexperiments, withweight on (α2, 0) of qα2 . �is �gure illustrates that as qα2 → 1, non-transparencyremains optimal. However, there will be a range of values for qα2 for which transparency is optimalwith transfers but suboptimal without transfers.

the experiments, with weight qα on the α = α2 experiment.

On the one hand, it may be the case that allowing for transfers makes the developer be�er o�under transparency. But ultimately, the conclusion of�eorem 3 remains valid, even with transfers(though there are cases under which transfers with transparency outperforms non-transparency).Indeed, when qα2 is large, the cost of devaluingα = α1 is quite small, since under non-transparencyit is unlikely that this experiment is chosen. Still, the size of the payment necessary in order toensure the scientist prefers α = α2 does not vanish as qα2 approaches 1. Hence transfers withtransparency will still be worse than non-transparency for qα2 su�ciently large.

6. RELATED LITERATURE

We �rst relate our main result on the potential optimality of non-transparency to other paperswhich arrive at similar conclusions. A few papers derive conditions under which a principal isbe�er o� by not observing certain actions undertaken by an agent, most notably Cremer (1994),Prat (2005) and Bergemann and Hege (2005). In these papers, the intuition behind the optimalityof non-transparency is that it gives the principal additional commitment power which would

23

Jonathan Libgober 24

not be credible under full observability. 18 In contrast to these, the present paper obtains non-transparency as a way of burning surplus in a way that aligns the incentives of the principal andthe agent19. �e presence of another action that is incentivized by this “money burning” is crucialfor our result, and does not have a direct counterpart in these papers. Importantly, this featurerelies upon the high degree of alignment between preferences of scientist and developer. In thissense, our results are closer to Angelucci (2014) or Szalay (2005), whereby the principal alignsincentives by taking actions which seem to hurt both principal and the agent. On the other hand,in these papers, the distortions take other forms. Ambrus and Egorov (2014) consider cases wheremoney burning is part of the optimal contract in general principal-agent se�ings, though ourcounterfactual of non-transparency does not nest in their framework.

Next, our paper contributes to the literature on communication games. As stated in theintroduction, the key features of our communication technology are restrictions on feasibleexperiments, costs, and lack of commitment. Ho�mann, Inderst and O�aviani (2014) also areinterested in the la�er property, but view the lack of commitment as arising from a disclosureproblem. �e connection between communication and experimentation has also been studiedby Kolotilin (2015), and several papers have consider cases where distortions of information cantake particular forms, such as fraud as in Lacetera and Zirulia (2008), or selective disclosure as inHenry (2009) and Felgenhauer and Schulte (2014). Henry and O�aviani (2014) study a dynamicdecision problem where an agent decides when to stop collecting information, and where senderand receiver disagree over how long to spend doing so. �eir application is more concerned witherrors in clinical trials, and not academic publication per se, but they do consider the incentivesto distort. Aside from using a one-shot model, which we will justify as more appropriate for ourapplication in the conclusion, the model here allows for quite a bit more generality regardingexperiment choice.20 Other papers have studied information aquisition and communication,but instead opted to model the communication between sender and receiver as cheap talk; seeArgenziano, Severinov, and Squintani (2014), as well as Pei (2014).

Finally, this paper relates to the literature on academic publication, also studied by, for example,Aghion, Dewatripont and Stein (2005) and Kiri, Lacetera and Zirulia (2015). Glaeser (2006) inparticular also studies the question of the incentives behind false positives, and similarly arguesthat it may be socially bene�cial to have some false positives, though his reasons for this do notdirectly relate to our overcompensation e�ect, and he instead focuses on the choice of hypothesis.

18Other papers give conditions under which the principal is be�er o� when the agent is not able to perfectly observea state variable, for example, Jehiel (2014) and Ederer, Holden and Meyer (2014).

19As mentioned earlier, we think of the scientist as being a sender, or an agent, and the developer as being a receiver,or a principal.

20We give general conditions on the information structure that generates these results, whereas they restrict toinformation generated by Brownian motion.

24

False Positives 25

�is theoretical literature also has a notable empirical counterpart. Azoulay, Bona�i and Krieger(2015) empirically study the e�ect of a retraction on a scientist’s reputation, and document that aretraction causes a drop in citations consistent with a reputation loss. Li (2015) studies evaluationsof funding decisions at the NIH, and �nds that evaluators have be�er information on projects intheir own area and therefore make be�er funding decisions.

7. CONCLUSION

7.1. Discussion of Model Assumptions

�e assumptions of the model were introduced in order to illustrate the economics forces in thesimplest way possible. We brie�y clarify the role of some of these assumptions in the analysis.

Perhaps most importantly, we have parameterized and limited the set of experiments thescientists could choose from. It does not seem unreasonable to place some restriction on experi-ments, and the additional assumptions—that experiments are one-dimensional and continuous ind—are for simplicity and tractibility. �e economically meaningful restriction that Assumption 1imposes is that hT (d) and hF (d) are co-monotonic in d. �is assumption allows us to focus onthe case where d unambiguously biases the experiment toward positive results, in the spirit ofIoannidis (2005). In our model, however, the only way the scientist can increase informativenessis by choosing α2 over α1.

Since most experiments are subject to bias in some form, one may also view Assumption 2 astoo strong. Notice that Section 3 mostly focused on experiment α1. In fact, Corollary 1 indicatesthat as long as higher d does not increase the true positive rate, d = 0 in equilibrium, providedλy = 0. Using such a strong assumption was therefore primarily to most clearly illustrate thereason the scientist is induced to choose α2. As long as the scientist gains commitment power withthis other experiment, the conclusions of �eorem 3 would still hold. Still, as stated in the example,it is plausible to think that if enough data are collected, then it may be much less plausible forthe potential sources of bias to signi�cantly contaminate the results. One may still object thatthe potential danger is only lessened, but not eliminated completely. As long as this is the case,however, our conclusions will continue to hold, though with more notation.

We have also expressed the payo�s of the scientists as additively separable and arising fromthree sources, which we a�empt to re�ect important incentives driving the scientist experimentchoice. A richer model would potentially allow for interactions between these parameters, or mayallow preferences to depend on other factors (such as the expected d). While such interactions mayarise in practice, we have neglected these considerations, as they complicate the model withoutgenerating additional insights as far as we can tell. As this paper is meant to provide intuition forwhy non-transparency encourages false positives but may nevertheless be preferred, we leave the

25

Jonathan Libgober 26

extent to which such interactions are signi�cant to future empirical work.

7.2. Final Comments

�is paper has shown why a sender’s inability to fully commit to an information structuremay make a receiver be�er o�. While we illustrated the main forces at work through a simpleexample, our general model clari�ed which features on preferences and information structureswere necessary for these conclusions. As an application, we have considered whether transparencyrequirements should be more widespread in academic disciplines. �e key insight is that non-transparency on one dimension can induce scientists to exert more e�ort or incur more costs onanother dimension, in a way that ultimately makes those interested in the results be�er o�. Inassessing this conclusion, we have primarily been concerned with the interest of those who usethe results downstream. �is is, of course, a simpli�ed view of welfare, but it is motivated by theidea that society invests in scienti�c research so that it produces results which will be useful forothers downstream.

In the main model we assumed that the only policy lever under the designer’s control iswhether the developer observes the scientist’s choice of protocols. For example, a funding agencymay be able to implement pre-registration requirements, but might not have the authority todictate speci�c steps that scientists follow. In the extensions we considered what might happen iftransfers were also available at their disposal, and demonstrated that the main conclusion wouldnot be changed. Still, the main model highlighted the relevant tradeo�s of transparency. Onthe other hand, transparency also might have an advantage of allowing for richer punishments,something we do not consider here, though they are natural places for future work.

As highlighted in the introduction, we have used our theoretical insights to comment onan active debate on the costs and bene�ts of transparency. Our main model has highlightedthat the policy debate over transparency requirements should consider the extent to whichfollow-on research in�uences which experiments scientist choose to perform. In cases where it issigni�cant, our model shows that scientists have a natural incentive to both add informationalcontent to their experiments and bias their experiments toward false positives. In this case, non-transparency encourages scientists to choose experiments which are inherently harder to bias. Ifthese experiments are particularly di�cult as well, then scientists may not be su�cient motivatedto undertake them under transparency, but if they are more informative then presumably societyis be�er o� if they do.

Finally, we remark that the intuition for many of our results would apply to the researchprocess more generally, and not just academic publication. While we called the person who actsa�er the scientist the developer, in general this could be any individual who will use the scientist’sresearch, and whose actions the scientist would be interested in in�uencing. We have shown

26

False Positives 27

that in general, a sender might be incentivized to acquire information but not distort a signal, orconversely. We provided a framework which describes when each will happen, so long as thereis some sense in which these two types of actions can be distinguished. �e observation thatnon-transparency can be used as a mechanism for money burning will likely have applicationsbeyond the focus of this paper, as these concerns are similar to those that have been raised in awide variety of contexts in organizational economics.

8. APPENDIX

8.1. Proof of Lemma 1

�at λpE[p(x, y)] = λpp0 follows immediately from the martingale property of beliefs, and clearlyλyE[y] = λyP[y = 1], so we need only consider the last term in the expression for the scientist’sbene�t. We �nd it useful to demonstrate the following, more general lemma, and simply note thatour claim follows immediately by se�ing t(p) = λxe(p) and f(p) = 0:

Lemma 2. For functions t, f : [0, 1]→ R,

p0Ey[t(p(y)) | T ] + (1− p0)Ey[f(p(y)) | F ] = Ey[p(y)t(p(y)) + (1− p(y))f(p(y))] (8)

Proof. By writing out the de�nition of the conditional expectation, we have:

p0Ey[t(p(y)) | T ] = p0∑y∈Y

t(p(y))P[y | T ]

=∑y∈Y

t(p(y))

(p0P[y | T ]

P[y]

)P[y]

=∑y∈Y

t(p(y))p(y)P[y] = Ey[p(y)t(p(y))].

An almost identical argument can be used for the other term in the expression above.

8.2. Proof of �eorem 2

We prove this theorem for any strictly convex cost function for the developer with c′R(0) = 0

(which implies that e(p) is strictly increasing).

Proof. Proof of (1) Fix α. We show that there is some d∗ such that when the developer conjecturesthat d∗ is the action of the scientist, the scientist’s best response is to follow action d∗. Since p0 isinterior, the developer always puts non-negative probability on observing y = 0 or y = 1, and

27

Jonathan Libgober 28

hence forms a belief p(1) following signal y = 1 and a belief p(0) following a signal y = 0, forany equilibrium strategy of the scientist.Let t(p) = λxe(p) + λpp and f(p) = λpp, and de�ne the function φ as follows:

φ(d) = arg maxd′∈D

p0[t(p(α,d)(0)) + hT (d′)(λy + t(p(α,d)(1))− t(p(α,d)(0)))]

+ (1− p0)[f(p(α,d)(0)) + hF (d′)(λy + f(p(α,d)(1))− f(p(α,d)(0)))]− cα(d′)

Taking dn → d, and dn ∈ φ(dn) with dn → d, since beliefs are continuous in d and f(p), t(p) arecontinuous as well (by continuity of e(p)), we have t(p(α,dn)(1))− t(p(α,dn)(0))→ t(p(α,d)(1))−t(p(α,d)(0)), and similarly for f . If d /∈ φ(d), then there exists some value δ such that:

p0(hT (δ)− hT (d))(λy + t(p(α,d)(1))− t(p(α,d)(0)))

+ (1− p0)(hF (δ)− hF (d))(λy + f(p(α,d)(1))− f(p(α,d)(0))) > cα(δ)− cα(d)

But since dn → d and dn → d, by continuity we can �nd some n su�ciently large such that

p0(hT (δ)− hT (dn))(λy + t(p(α,dn)(1))− t(p(α,dn)(0)))

+ (1− p0)(hF (δ)− hF (dn))(λy + f(p(α,dn)(1))− f(p(α,dn)(0))) > cα(δ)− cα(dn)

contradicting that dn is a maximizer of φ(dn). Hence the map φ is upper-hemicontinuous. Fur-thermore, φ(d) is nonempty and closed because [0, d] is compact and the objective function inthe expression for φ(d) is continuous. Finally, to see that it is convex, notice that if d1 < d2 bothmaximize d, the convexity of cα(d) and the concavity of hT (d), hF (d) means that we must havep0hT (d′)(λy+t(p(α,d)(1))−t(p(α,d)(0)))+(1−p0)hF (d′)(λy+f(p(α,d)(1))−f(p(α,d)(0)))−cα(d′)

constant for d′ between d1 and d2, meaning that φ(d) is convex. Hence by Kakutani’s �xed pointtheorem, an equilibrium exists when α is observed.

Now, suppose 0 < d < d and (5) fails. If the le� hand side is smaller than the right handside, then increasing d to d+ ∆ with ∆ small yields payo� proportional to the right hand side ofthis inequality, whereas the cost is proportional to the le� hand side, and hence this deviation ispro�table. �e case when the inequality is �ipped is analogous.

To show that there is also an equilibrium without mixing over proposals, suppose the devel-oper’s beliefs correspond to an equilibrium choice of d∗ when α is chosen with probability 1. Inthis case, it is a pure strategy equilibrium for the scientist to place probability 1 on whicheverproposal yields higher payo�s, completing the proof of existence.

Proof of (2) Suppose to the contrary that there is a mixed strategy equilibrium. �en the �rstorder condition in equation (5) must hold for two values of d, say d1 < d2. On the other hand, the

28

False Positives 29

developer’s beliefs do not depend on the choice of d. Keeping the notation from the previous wehave:

c′α(d1) = p0(λy + t(p(1))− t(p(0)))h′T (d1) + (1− p0)(λy + f(p(1))− f(p(0)))h′F (d1)

c′α(d2) = p0(λy + t(p(1))− t(p(0)))h′T (d2) + (1− p0)(λy + f(p(1))− f(p(0)))h′F (d2)

and subtracting the �rst equation from the second, since p(0) and p(1) are the same in bothequations,

c′α(d2)− c′α(d1) = p0(λy + t(p(1))− t(p(0))))(h′T (d2)− h′T (d1))

+ (1− p0)(λy + f(p(1))− f(p(0)))(h′F (d2)− h′F (d1))

By the mean value theorem, applied to hT , hF and cα, we have, for some dc, dt and df :

c′′α(dc)(d2 − d1) = p0(λy + t(p(1))− t(p(0)))h′′T (dt)(d2 − d1)

+ (1− p0)(λy + f(p(1))− f(p(0)))h′′F (df )(d2 − d1).

But since either hT or hF is strictly concave or cα(d) is strictly convex, either the le� hand side isstrictly positive or the right hand side is strictly negative, with both being at least weakly so, acontradiction. Hence in equilibrium, there can only be pure strategies.

8.3. Proof of �eorem 3

Proof. First, note that if β is 0, then the scientist does not have any incentive for informationacquisition, but does have an incentive for distoriton, as per Section 3. Hence if β = 0, non-transparency only induces a more biased experiment, but does not induce the scientist to undertakethe costlier experiment, and hence is worse for the developer. �erefore, for the rest of the proofwe will assume β > 0.

By corollary 6, when d is unobserved and α = α1, d = d, whereas d = 0 when d is observedor α = α2. Suppose there are no restrictions on the choice of experiment. Under transparency,the scientist chooses α = α1. Under non-transparency, by Lemma 6, the scientist chooses d = d

whenever choosing α = α1, and no distortions occur when α = α2.

29

Jonathan Libgober 30

We can show the precise conditions under which non-transparency is optimal are:

E(α1,0)[p(y)e(p(y))] ≥ p0 −cα2

λβ≥ E(α1,d)

[p(y)e(p(y))] (9)

qα2 ≥E(α1,0)[πR(p(y))]− E(α1,d)

[πR(p(y))]

p0πR(1)− E(α1,d)[πR(p(y))]

(10)

Suppose (9) fails. �en one of two cases must hold: either the scientist still prefers α1 undernon-transparency, or the scientist prefers α2 under transparency. In the la�er case, the scientistonly chooses α1 when α2 is unavailable, and hence non-transparency is suboptimal since itdevalues the experiment α1. In the former case, non-transparency does not induce the scientist tochoose α2, and hence this does not induce a change in the proposal, and only makes the developerworse o� if α1 is chosen. Hence without (9), transparency is optimal.

Now suppose (9) holds. Notice that whenever qα2 < 1, forcing the scientist to choose α2 isstrictly worse than non-transparency, since the scientist chooses α2 whenever it is available undernon-transparency, and if it is not, the developer still receives payo� E(α1,d)

[πR(p(y))], which islarger than if the scientist takes the outside option. Hence there are never any restrictions on theset of permissible experiments in any optimal regime.

By the analysis from Section 3, it follows that the developer’s payo� under transparency withno restrictions is E(α1,0)[p(y)2]. By the analysis from the same section, since α2 is fully informative,the expected payo� from this experiment is p0πR(1). Hence payo� under non-transparency is

qα2p0πR(1) + (1− qα2)E(α1,d)[πR(p(y))]

which is greater than or equal toE(α1,0)[p(y)2] if and only if beliefs are su�ciently high, completingthe proof.

30

False Positives 31

References

Philippe Aghion, Mathias Dewatripont, and Jeremy Stein. Academic freedom, private-sector focus,and the process of innovation. RAND journal of Economics, 39(3):617–635, Autumn 2008.

Philippe Aghion and Jean Tirole. Formal and real authority in organizations. Journal of PoliticalEconomy, 105(1):1–29, February 1997.

A�ila Ambrus and Georgy Egorov. Delegation and nonmonetary incentives. Working paper,Northwestern University, Kellogg, November 2014.

Charles Angelucci. Motivating information acquisition through risky decision-making. Workingpaper, Columbia Business School, July 2014.

Rossella Argenziano, Sergei Severinov, and Francesco Squintani. Strategic information acquisitionand transmission. Working paper, June 2014.

Mark Armstrong and John Vickers. A model of delegated project choice. Econometrica, 78(1):213–244, January 2010.

Pierre Azoulay, Alessandro Bona�i, and Joshua Krieger. �e career e�ects of scandal: Evidencefrom scienti�c retractions. Discussion paper, MIT, May 2015.

Dirk Bergemann and Ulrich Hege. �e �nancing of innovation: Learning and stopping. RANDjournal of Economics, 36(4):719–752, Winter 2005.

David Blackwell. Equivalent comparisons of experiments. Annals of Mathematical Statistics,24(2):265–272, June 1953.

Patrick Bolton and Christopher Harris. Strategic experimentation. Econometrica, 67(2):349–374,March 1999.

Lucas C. Co�man and Muriel Niederle. Pre-analysis plans have limited upside, especially wherereplications are feasible. Journal of Economic Perspectives, 29(3):61–80, Summer 2015.

Jacques Cremer. Arm’s length relationships. �e�arterly Journal of Economics, 110(2):275–295,May 1995.

Mathias Dewatripont, Ian Jewi�, and Jean Tirole. �e economics of career concerns, part i:Comparing information structures. Review of Economic Studies, 66(1):183–198, January 1999.

Florian Ederer, Richard Holden, and Margaret Meyer. Gaming and strategic opacity in incentiveprovision. Discussion paper, Cowles Foundation, January 2014.

31

Jonathan Libgober 32

Mike Felgenhauer and Elisabeth Schulte. Strategic private experimentation. American EconomicJournal: Microeconomics, 6(4):74–105, November 2014.

Drew Fudenberg and Jean Tirole. Moral hazard and renegotiation in agency contracts. Econometrica,58(6):1279–1319, November 1990.

Ma�hew Gentzkow and Emir Kamenica. Costly persuasion. American Economic Review: Papersand Proceedings, 104(5):457–462, May 2014.

Edward Glaeser. Researcher incentives and empirical methods. In Andrew Caplin and AndrewScho�er, editors, �e Foundations of Positives and Normative Economics, pages 300–319. OxfordUniversity Press, Oxford, 2008.

Emeric Henry. Strategic disclosure of research results: �e cost of proving your honesty. �eEconomic Journal, (119):1036–1064, July 2009.

Emeric Henry and Marco O�aviani. Research and the approval process. Working paper, November2014.

Florian Ho�mann, Roman Inderst, and Marco O�aviani. Persuasion through selective disclosure:Implications for marketing, campaigning and privacy regulation. Working paper, BocconiUniversity, November 2014.

Bengt Holmstrom. Managerial incentive problems: A dynamic perspective. Review of EconomicStudies, 66(1):169–182, January 1999.

John P. A. Ioannidis et al. Increasing value and reducing waste in research design, conduct andanalysis. �e Lancet, 383(9912):166–175, January 2014.

Alex Frankel Je� Ely and Emir Kamenica. Suspense and surprise. Journal of Political Economy,123(1):215–260, February 2014.

Philippe Jehiel. On transparency in organizations. Review of Economic Studies, (Forthcoming),2015.

Emir Kamenica and Ma�hew Gentzkow. Bayesian persuasion. �e American Economic Review,101(6):2590–2615, October 2011.

Godfrey Keller, Sven Rady, and Martin Cripps. Strategic experimentation with exponential bandits.Econometrica, 73(1):39–68, January 2005.

Bralind Kiri, Nicola Lacetera, and Lorenzo Zirulia. Above a swamp: A theory of high-qualityscienti�c production. Working paper, NBER, May 2015.

32

False Positives 33

Nicolas Klein. �e importance of being honest. Working paper, University of Montreal, July 2014.

Anton Kolotilin. Experimental design to persuade. Games and Economic Behavior, 90:215–226,March 2015.

Nicola Lacetera and Lorenzo Zirulia. �e economics of scienti�c misconduct. Journal of Law,Economics and Organization, 27(3):215–260, February 2014.

Danielle Li. Expert vs. bias in evaluation: Evidence from the nih. Working paper, Harvard BusinessSchool, 2014.

Edward Miguel et al. Promoting transparency in social science research. Science, 343(6166):30–31,January 2014.

Benjamin A. Olken. Promises and perils of pre-analysis plans. Journal of Economic Perspectives,29(3):81–98, Summer 2015.

Di Pei. Communication with endogenous information acquisition. Working paper, Massachuse�sInstitute of Technology, January 2015.

Andrea Prat. �e wrong kind of transparency. �e American Economic Review, 95(3):862–877, June2005.

Armin Rick. �e bene�ts of miscommunication. Working paper, University of Chicago, November2013.

Dezso Szalay. �e economics of clear advice and extreme options. Review of Economic Studies,72(4):1173–1198, October 2005.

Edward Wilson. Research: increasing value, reducing waste. �e Lancet, 383(9923):1123–1124,March 2014.

33