Eliciting Probabilities, Means, Medians, Variances and Covariances without Assuming Risk Neutrality

We are interested in incentivizing experimental subjects to report their beliefs truthfully, without imposing assumptions on their risk preferences. We prove that if subjects are not risk neutral, it is not possible to elicit subjective probabilities or the mean of a subjective probability distribution truthfully using deterministic payments schemes, which are predominant in the literature. We present a simple randomization trick that transforms deterministic rewards into randomized rewards, such that agents with arbitrary risk preferences report as if they were risk neutral. Using this trick, we show how to elicit probabilities, means, medians, variances and covariances of the underlying distribution without assuming risk neutrality.

• Belief Elicitation; Proper Scoring Rules; Lottery Tickets; Risk Preferences

1. Introduction

The economic literature on the elicitation of an expert’s subjective beliefs has focused on so-called proper scoring rules. These mechanisms, which are used in many economic experiments, reward the expert on the basis of post-elicitation events such that it is in the expert’s interest to report her true beliefs if she is risk neutral. The quadratic scoring rule (QSR) [1] is the most popular rule, used to elicit the probability of an event or the mean of a random variable. In the absence of risk neutrality, there is an incentive to report conservative beliefs in order to avoid large losses [2]. This is a problem, since risk neutrality is shown to be widely violated in experimental studies [e.g. 3]. Indeed, Armantier and Treich [4] show experimentally that consistent with risk aversion, elicitation with the quadratic scoring rule leads to conservative bias in reported beliefs.

There are different ways to get around this problem. Offerman et al. [5] propose a way to correct for deviations of risk neutrality and expected utility, by quantifying the size of deviations for each individual. Alternatively, an earlier literature starting with Smith [6]¹ shows how one can induce risk neutrality by rewarding subjects using binary lottery tickets. This idea has been used to show how to elicit the subjective probability of an event [e.g. 10,11] in a way similar to the elicitation of a reservation price [12].

We extend this literature in several ways. First, we prove that deterministic schemes are not adequate if one does not know the risk preferences of the expert. Second, we combine the literature on scoring rules for risk neutral preferences with the literature on incentivizing with lottery tickets to show that one can elicit a median or any quantile without making assumptions on risk preferences. We also present an alternative way to elicit a probability or mean based on the randomized quadratic scoring rule. Third, we present a new deterministic rule, and its randomized counterpart, to elicit variances and covariances when two independent observations are available.

2. Preliminaries

We consider two people, an expert and an elicitor. The expert has subjective beliefs about the distribution of a bounded random variable that yields outcomes belonging to with. The expert maximizes expected utility for some utility function on such that for some. The elicitor only knows that yields outcomes belonging to, and would like to learn some parameter of the distribution. We consider the use of a reward system or scoring rule which rewards the expert on the basis of her report and a single random realization of. Here, is a distribution over the rewards which includes a deterministic reward as a special case. In the literature, is called strictly proper for if

for all. We say that a rule elicits if for all and all with.

3. Limitations of Deterministic Rewards

Consider an elicitor who wishes either to learn about the mean of some random variable with support in, or about probability of some event. We obtain the following result.

Proposition 1. A scoring rule with a deterministic reward cannot elicit the probability or the mean.

The proof is in the Appendix. The intuition is simple. The elicitor has only one parameter, the realization, to incentivize the expert to tell the truth. On the other hand, there are two dimensions of uncertainty as the elicitor does not know and

4. Probabilistic Elicitation

We now consider elicitation using probabilistic or randomized reward functions. The idea, first elaborated by Smith [6], is that one pays the experts in lottery tickets rather than money. The size of the prize is given by the probability of winning the lottery. Hence where is now the payoff distribution awarded conditional on. Using this idea, we show how to elicit probabilities, means, different quantiles, and variances and covariances.

4.1. Randomization Trick

We use the following “randomization trick” to transform deterministic into probabilistic payoffs. First, given a deterministic reward function, determine and

such that and

Second, draw a realization from a uniform distribution on and then pay if and pay if

Formally, we replace the deterministic reward by the randomized reward

where is a lottery that pays with probality and with probability Consequently,

The expected utility of the expert equals an affine transformation of. Thus, a report that maximizes her expected utility is a report that maximizes the utility of a risk neutral expert and vice versa. In particular, elicits iff is strictly proper for.²

4.2. Eliciting Probabilities

Randomized rewards for the elicitation of probabilities have received quite some attention. Grether [10] (see also Holt [15, ch. 30] and Karni [16]) presents a simple reward function where a prize is rewarded with some probability that depends on the draw of two uniformly distributed random variables. Allen [11] presents an alternative rule that relies on a draw of a random variable that has a more complex probability distribution. Mclvey and Page [17] uses a randomized version of the quadratic scoring rule in an experimental application, which is similar to the rule we present below.

The QSR (for the event) is given by

and is strictly proper for [1]. The randomized quadratic scoring rule (for the event), short rQSR, is defined by

The following result obtains:

Proposition 2. The randomized quadratic scoring rule elicits.

Note that the expected payoffs under rQSR are identical to those under the rules of Allen [11] and McKelvey and Page [17] when.

4.3. Eliciting the Mean

To elicit the mean, we combine the randomization trick with the fact that the QSR is a strictly proper scoring rule for the mean (for risk neutral experts). Given and we obtain the randomized quadratic scoring rule as defined by

Proposition 3. The randomized quadratic scoring rule elicits.

4.4. Median and Quantiles

The quantile scoring rule, due to Cervera and Munoz [18] is a strictly proper scoring rule for the quantile of the distribution of for any given. Its reward function is given by The randomized quantile scoring rule is hence given by where

Proposition 4. The randomized quantile scoring rule elicits the quantile.

In particular, Proposition 4 shows how to elicit the median by setting.

4.5. Variance and Covariance

In order to elicit the variance of we assume the elicitor can condition on two independent realizations and of when rewarding the expert. So we conder a reward function We first construct a strictly proper scoring rule. Following Walsh (1962),

where and are indendent copies of We combine this with the quadratic scoring rule to obtain that the variance scoring rule that is strictly proper for Given

we obtain the randomized variance scoring rule by

Proposition 5. The randomized variance scoring rule elicits the variance of

Similarly we can elicit the covariance given two ranm variables and We assume that

for Here we condition on a realization drawn from Again following Walsh (1962)we use the fact that

and then use the QSR to define the covariance scoring rule that is strictly proper for Given and

we obtain the randomized covariance scoring rule by

Proposition 6. The randomized covariance scoring rule elicits the covariance of and

5. Conclusions

We have rigorously shown the limits of deterministic scoring rules for belief elicitation. To overcome those limitations, we applied the idea of paying in lottery tickets to transform known deterministic scoring rules for belief elicitation, such as the well-known QSR, into randomized rules. These rules provide agents with incentives to truthfully report parameters of a subjective probability distribution for all risk preferences, and can be used in experimental applications.

This paper has considered the theoretical side. On the empirical side, it is an open question whether these rules have the desired properties in actual applications, and how they are best presented to subjects. Selten et al. [19, see also review therein] raises doubt whether subjects rewarded using lotteries behave as if risk neutral in experiments. More recently, Harrison et al. [14,20], and Hossain and Okui [13] provide evidence that the produre can induce subjects to behave more in line with risk neutrality.

Appendix

Proof of Proposition 1. If one can elicit the mean of a random variable for all distributions in then one can also elicit the probability of an event as if is the Bernoulli random variable such that if and only if Hence it is enough to show that one cannot elicit to prove that one cannot elicit

We first show that and are differentiable almost everywhere. Once this is established the first order conditions reveal the impossibility.

Consider where So

Let

Assume that elicits for all concave. Then we have for all

For and we have

so

so

Hence we have shown that is strictly increasing in

Similarly, for we have

and since

it follows that So strictly decreasing in and hence is strictly increasing.

From the above two strict monotonicity statements we obtain that and are differentiable almost everywhere. Let be the set where they are differentiable.

For and differentiable we can calculate

and infer that

It is easy to argue with generalized version of the intermediate value theorem that there is such that Consider that is differentiable with. Then rewrite (1) as:

Since is strictly increasing in there is some such that

So when the left hand side of (2) depends on. Therefore, (2) cannot hold for all.

References

1. G. W. Brier, “Verification of Forecasts Expressed in Terms of Probability,” Monthly Weather Review, Vol. 78, No. 1, 1950, pp. 1-3. doi:10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2

2. R. Winkler and A. Murphy, “Nonlinear Utility and the Probability Score,” Journal of Applied Meteorology, Vol. 9, 1970, pp. 143-148. doi:10.1175/1520-0450(1970)009<0143:NUATPS>2.0.CO;2

3. C. A. Holt and S. Laury, “Risk Aversion and Incentive Effects,” The American Economic Review, Vol. 92, No. 5, 2002, p. 1644. doi:10.1257/000282802762024700

4. O. Armantier and N. Treich, “Eliciting Beliefs: Proper Scoring Rules, Incentives, Stakes and Hedging,” IDEI Working Paper, Vol. 643, 2010.

5. T. Offerman, J. Sonnemans, G. Van de Kuilen and P. P. Wakker, “A Truth Serum for Non-Bayesians,” Review of Economic Studies, Vol. 76, No. 4, 2009, pp. 1461-1489. doi:10.1111/j.1467-937X.2009.00557.x

6. C. Smith, “Consistency in Statistical Inference and Decision,” Journal of the Royal Statistical Society. Series B, Vol. 23, No. 1, 1961, pp. 1-37.

7. L. J. Savage, “The Foundation of Statistics,” Wiley, New York, 1954.

8. L. Savage, “Elicitation of Personal Probabilities and Expectations,” Journal of the American Statistical Association, Vol. 66, No. 336, 1971, pp. 783-801. doi:10.1080/01621459.1971.10482346

9. A. Roth and M. Malouf, “Game-Theoretic Models and the Role of Information in Bargaining,” Psychological review, Vol. 86, No. 6, 1979, pp. 574-594. doi:10.1037/0033-295X.86.6.574

10. D. Grether, “Financial Incentive Effects and Individual Decision-Making,” Working Paper 401, California Institute of Technology, 1981.

11. F. Allen, “Discovering Personal Probabilities When Utility Functions Are Unknown,” Management Science, Vol. 33, No. 4, 1987, pp. 542-544. doi:10.1287/mnsc.33.4.542

12. G. Becker, M. DeGroot and J. Marschak, “Measuring Utility by a Single-Response Sequential Method,” Behavioral Science, Vol. 9, No. 3, 1964, p. 226. doi:10.1002/bs.3830090304

13. T. Hossain and R. Okui, “The Binarized Scoring Rule,” Manuscript, 2009.

14. G. W. Harrison, J. Martffnez-Correa and J. Swarthout, “Eliciting Subjective Probabilities with Binary Lotteries,” Manuscript, Georgia State University, Atlanta, 2012.

15. C. Holt, “Markets, Games and Strategic Behavior,” Pearson/Addison-Wesley, Boston, 2006.

16. E. Karni, “A Mechanism for Eliciting Probabilities,” Econometrica, Vol. 77, No. 2, 2009, pp. 603-606. doi:10.3982/ECTA7833

17. R. McKelvey and T. Page, “Public and Private Information: An Experimental Study of Information Pooling,” Econometrica: Journal of the Econometric Society, Vol. 58, No. 6, 1990, pp. 1321-1339. doi:10.2307/2938318

18. J. L. Cervera and J. Munoz, “Proper Scoring Rules for Fractiles,” In: J. O. Berger, A. P. Dawid and A. F. M. Smith, Eds., Bayesian Statistics 5, Oxford University Press, Oxford, 1996, pp. 513-519.

19. R. Selten, A. Sadrieh and K. Abbink, “Money Does Not Induce Risk Neutral Behavior, but Binary Lotteries Do even Worse,” Theory and Decision, Vol. 46, 1999, pp. 211-249. doi:10.1023/A:1005038628305

20. G. W. Harrison, J. Martinez-Correa and T. Swarthout, “Inducing Risk Neutral Preferences with Binary Lotteries: A Reconsideration,” Journal of Economic Behavior and Organization, 2012, in Press. doi:10.1016/j.jebo.2012.09.008