Even ignoring risk premiums the efficient market hypotheses doesn't predict a 50% chance that one asset will outperform another. It predicts that any two assets will have the same *expected* returns. But if one of the two assets has an asymmetric risk profile (for example a large probability of doing worse balanced out by a tiny probability of doing *much* better), the chance of doing better might not be 50%.
Re question 11: Hdroxychloroquine is not a particularly effective treatment for covid. It does, however, seem to be remarkably effective as a prophylaxis. This is likely how India beat covid against all odds. India accounts for three-quarters of the malaria in Asia and large swaths of its population take hydroxychloroquine regularly because it is less expensive than more recently developed antimalarials.
I don't think it's necessary to conclude that you and Bucky were "worse" than guessing 50-50 on everything. For example, such guesses would imply a distribution with 0% of probability mass between 100k and 3M US COVID deaths. The predictions look pretty reasonable to me; I know that a lot of people (myself included) were underconfident about the possibility of a 2nd wave.
Pretty much every nation has lied about their death counts in an obvious way, since almost every nation shows much higher excess mortality than official covid deaths. There's almost certainly a lot of undercounting going on (though it may not be deliberate lying). The UK, with their unusually loose standard for a death within 28 days of a positive test being a Covid death, is one of the only nations where COVID deaths and excess mortality are similar - in fact the UK has slightly more COVID deaths than excess mortality.
That still isn't enough to knock the US off the top afaiw though. But of course, China's figures are worth as much as a bet on Trump being president, but I suppose it doesn't qualify for being obvious and no one will ever know the true death toll.
"Will the Bay Area stay locked down until Election Day (11/3/2020)?" was worded in your original post as "Bay Area lockdown (eg restaurants closed) will be extended… until Election Day". Both wordings to me are unambiguous: they imply a single continuous lockdown. If they meant multiple lockdowns, the former would read "Will the Bay Area be locked down on Election Day" and the latter would read "Bay Area lockdowns will continue on and off".
Zvi also explicitly clarified on his post (https://thezvi.wordpress.com/2020/05/01/slatestarcodex-2020-predictions-buy-sell-hold/): "If anything, that seems high, assuming it means continuous lockdown until then rather than being locked down on election day." I think omitting that result from his score would be better than including it, because he was explicitly predicting a different thing from you.
I'm getting different totals for the scoring. Could you/someone else check to see what's going on? Just to make sure we're doing the same calculation: Let's call p the probability that a forecaster assigned to the correct answer, then LogOdds = log(p/(1-p)).
Some examples: scott's score on the first question was log(0.6/0.4) ~= 0.405. His score on the third question was log(0.9/0.1) ~= 2.197 (balanced out exactly by his score on the second question).
Doing this computation for all questions gives total scores for Scott, Zvi, and Bucky at 7.2034, 13.2863, and 6.5006 respectively, all better than chance!
Also, I recommend switching to Brier score (squared error) for things like these. It's a proper scoring rule, which means that you cannot game the system; in expectation you'll get the best possible score by reporting your true beliefs. In contrast, something like LogOdds can be gamed since increasing the extremity of your forecast just ups the ante, but the odds of the bet don't change [i.e. log(p/(1-p) = -log((1-p)/p)]. For instance, if you think something is 90% to occur, then by reporting your true beliefs you are expecting to get 1.7578 "LogOdds points" (0.9 * log(0.9/0.1) + 0.1*log(0.1/0.9) ). However, you could expect to score more points by reporting a more extreme belief, like 95%. In that case, you would expect to score 2.3556 points (0.9 * log(0.95/0.05) + 0.1*log(0.05/0.95)). Doing the same calculation with brier score for belief p and *reported* belief x has you minimizing the function p*(1-x)^2 + (1-p)*x^2 for values of x, which occurs only when p=x (note that lower Brier scores are better).
Here's some Matlab code to run these computations:
Forgive my ignorance for question 12, "Will I personally will get coronavirus": isn't a 30% probability of this being true equivalent to 70% probability of it being false... i.e. Scott correctly predicted he wouldn't get covid?
Next time perhaps include a competent astrologer who also uses ML. Maybe a female for stronger intuition. It would be v interesting to see the results.
I still think this way of giving and judging predictions, using confidence levels, is fundamentally flawed.
Here, let me make some predictions with confidence levels:
1) The sun will rise in the east on February 24th (95%)
2) The sun will rise in the east on February 25th (95%)
3) The sun will rise in the east on February 26th (95%)
4) The sun will rise in the west on February 27th (95%)
5) The sun will rise in the east on February 28th (95%)
6) The sun will rise in the east on March 1st (95%)
...
20) The sun will rise in the east on March 15th (95%)
If we evaluate my predictions a month from now, I think you will find that out of 20 predictions I made with 95% confidence, I got 19 correct. Perfectly calibrated! Clearly I'm the greatest predictor on earth!
I think the only good way to score predictions is to compare them to other people. If a hundred people predict A, you predict B, and you're right, that's great. If 80 people predict X, 20 predict Y and you also predict Y, that's pretty good. If 5 people predict N and you and 95 others predict M, then that's not impressive at all, even if you're right.
> Also, will Bitcoin outperform the US stock market over the next five years, at 51%. I started out thinking - of course it's 50-50! By the efficient market hypothesis, if any asset was obviously going to do better than another, people would change the price until it wasn't. But on second thought that's wrong - stocks have a higher than 50% chance of beating treasuries over the same period because of a risk premium. Maybe there's no intuitive way to think about this, you have to have opinions on the underlying fundamentals, and it's only 51% by coincidence?
Yeah, this is wrong, and I was also tripped up by this a while ago. The efficient market hypothesis only tells you that E[X] = X. But, e.g., if Bitcoin could only either go up 3x or crash to 0, then the efficient market hypothesis would tell you that p * 3 * Initial capital + (1-p)*0 = Initial capital => p = 1/3 = 33.3% of going up and 66.6% of going down.
It looks like you're still collecting play-money prediction markets but haven't mentioned The Foresight Exchange, one of the oldest. (I tried to post this last week.)
Perot did not win 5+% of the vote as a third-party candidate in 1992 and 1996. He won 8% of the vote as a third-party candidate in 1996, after having won 18% of the popular vote as an *independent* candidate in 1992.
It's an important distinction, because running for President as an independent candidate implies that you're trying to pull the Gabriel-over-the-White-House trick, riding into Washington on a white horse and sweeping away the corruption with the awesome power of the presidency. If you envision *not* being a dictator and actually sharing power with other people, then the first step is almost certainly going to be finding a bunch of other people (particularly wannabe-Congressman type people) with similar goals and getting organized.
Metaculus is scoring the prediction as positive if either a third-party or independent candidate reaches 5%, so at least they're aware of the distinction. But they don't seem to think it is an important distinction, and I think it's important to push back against that.
In grad school I took a course on probabilistic risk where our tests were graded using log odds scoring. Instead of picking A, B, C or D, you wrote in your probabilistic confidence in each choice. For example, if you’re somewhat confident in A, write in 60%, and then distribute the remaining 40% of weight across the other three choices.
Because of the log scoring rule, being very confident and wrong would result in a massive penalty. There was one star student in the class who blew the curve for all of us by putting down 90%+ confidence in his correct answers because he knew what he was doing and, furthermore, knew that he knew what he was doing.
This method was really hard and I got a pretty mediocre grade in the course, but I’ve always thought it was a good idea. You really learn about your own understanding of the material this way. It’s much easier to notice gaps in your own understanding when, instead of being given partial credit, you’re actively penalized for undue confidence.
A 3rd party POTUS candidate reaching 5% is not a particularly high bar nor, during my lifetime at least, a highly unusual event. It happened in 1996, 1992, 1980, and 1968. Three of the last 11 presidential elections and four of the last 14. So even before considering anything specific about 2024, I'd take the "yes" side on 15% odds.
On the third-party candidate: I think that the naive assumption of "Last 3 of 8" making the 15% bet worthwhile is a bit simplistic of an analysis. The first thing to consider is that, for such a small sample size, if it's possible to get more granularity, one should. And we certainly can. The political environments of those 3 third-party successes, so to speak, were not nearly as polarized as things currently are. Unless you think there will be a de-stressing of tensions (Possible) or a fracturing of parties (Very unlikely, given the two-party system's strength) I think it's closer to 1%. If there's an easing of tensions, call it 20%, very generously. Take this as me predicting against this.
Incidentally, it feels to me that there is some serious ambiguity on the "second wave in fall" question. Some would say the second wave was in the summer, and the later one was a third wave. Some would say that third wave was really in the winter, because it peaked in December and January most places. Some would say that although the summer wave seemed distinct from the first wave, it was really just a delayed first wave that hit Florida, Texas, and Southern California as soon as the initial restrictions were eased. Some would say that in the northern plains region, the wave that started building in August/September was actually just their first wave. Though I think that New Mexico and El Paso were clearly in a second local wave by October/November.
Here is an interesting paradoxical problem with this type of prediction self-scoring: suppose you are especially adept at thinking up prediction questions that are "hard" precisely because they successfully "cleave (potential future) reality in half". For example, your question 9 asks if China will have 100k deaths. In reality it had 96k deaths, so the threshold in this question is a pretty good guess!
If you were insightful enough to keenly set the thresholds for all your questions in such a way, you would frustratingly find that there is no way to earn a positive score at your own game (or even have much fun playing it), since an optimal Bayesian agent should predict 50-50 for all questions.
Re: "will Bitcoin outperform the US stock market over the next five years"
Consider my new cryptocurrency, Gamblecoin. On Jan 1, 2022 I will roll a die, and if that die comes up 6 it activates a smart contract that will buy anyone's Gamblecoins for $1 USD each, otherwise nothing happens. The odds of Gamblecoin beating the market are one in six, and this is not an efficient market violation because if it does beat the market, it beats it by a lot.
On a long enough timescale, Bitcoin should either become some kind of enormous global currency worth way more than it currently is, or become basically worthless as we find something else to serve the "anonymous secure global currency" role (doesn't have to be crypto, maybe the banks get their act together and come up with an efficient way to move USD) and Bitcoin no longer has a reason to go to the moon. The odds of Bitcoin becoming an enormous global currency are probably not 50/50, so there is some time period over which prediction markets shouldn't say it's 50/50 to beat the market.
That being said, there's a decent chance that Bitcoin is still in the speculative bubble phase and we won't know whether it becomes the massive global currency until far in the future. "Does Bitcoin beat the stock market over the next five days" should be 50/50 on the efficient market grounds you mention because the next five days are (very probably) just going to be speculative trading rather than a resolution to the question of whether Gamblecoin has value or not.
Vaguely on topic: somewhere in a thread on SSC or a related site, I predicted 1 million or more excess deaths in the US during covid, explicitly including deaths due to untreated non-covid conditions, despair due to unemployment, etc.
A month or two after that, US covid deaths were still below 1/4 million, and the rate was dropping, and I felt like an idiot - but I'd made the prediction, as my first attempt at measuring my prediction results.
When I checked today, the number of explicitly covid deaths reported in the US is over 1/2 million (501,117), and my confidence in my prediction is much higher.
Excess deaths - i.e. number of deaths in each 12 month period compared to the last 12 months without covid - aren't tracked as prominently, or reported as swiftly, so I can't yet check how close we've come so far.
I'm reposting mostly to continue to hold myself accountable.
I see a number of people modeling bitcoin outcomes as binary. That may be reasonable, but just about every other financial asset is very well modeled as a random walk with drift (expected return). If we do that, stocks have about 15% annual volatility over long horizons, and bitcoin has... I actually haven't measured it but it's probably 5-10x that. Expected returns on stocks is probably in the ballpark of 4%, looking at international markets (US has done better, but using its returns to predict its future returns is probably worse than using the average international estimate). Over a 5 year period, the volatility of bitcoin is so much higher than stocks that the average return on stocks barely matters. So the random variable (bitcoin 5 year returns - stock 5 year returns) has a negative mean (if we assume bitcoin doesn't have a positive risk premium), but its volatility is so high relative to that mean that the probability of it being greater than 0 is pretty close to 50%.
>16. Will there be a general consensus that summer made coronavirus significantly less dangerous?
Maybe this is true, but I have my doubts. I found this posted on Good Judgment Open:
"Given the lack of immunity to SARS-CoV-2 across the world, if there is an effect of temperature and humidity on transmission, it may not be as apparent as with other respiratory viruses for which there is at least some preexisting partial immunity. It is useful to note that pandemic influenza strains have not exhibited the typical seasonal pattern of endemic/epidemic strains. There have been 10 influenza pandemics in the past 250-plus years—two started in the northern hemisphere winter, three in the spring, two in the summer, and three in the fall. All had a peak second wave approximately 6 months after emergence of the virus in the human population, regardless of when the initial introduction occurred."
These markets seem to be modeled on, well, markets, but the function they aim to fulfill seems to be closer to that of a consulting firm. Betting/investment markets, at least in the US, come with a prohibitive degree of red tape.
So, bizarre question: would it be possible to create a prediction "market" that functioned like "the Uber of consulting firms"? Clients contract with the firm for a given amount with a set of things they want predictions on. "Independent contractors" can buy a percentage in the contract and submit their prediction(s) for the associated questions. Payout is a function of accuracy and percentage of buy-in.
1) Could that work?
2) Would it be legally less difficult than prediction markets as currently constituted?
Re Third party 5% share: Seems like the easiest path would be a Trump Patriot Party-esque splinter. This'd require Trump to rerun (or find a charismatic substitute) and for the GOP to resist hard enough to create an election-losing vote split over it, which seems iffy, but certainly raises the chance beyond the background chance in the previous elections.
Last year, I built a site that was based on predicting the future. (Don't worry, it's dead, this is not an advert)
The idea was to make it a more general, lower-entry prediction market. Instead of people betting money, they would gain or lose reputation. The site itself would function like a geopolitics focused subreddit, except that people with good reputation have much stronger upvotes. Therefore, the smart people would rise to the top much faster, and if a strong expert made a comment a few hours too late, it would not be buried. Second, any unproductive groupthink or echo chamber effects would be broken down quite fast - the people that succumbed to this sort of stuff would lose reputation, and the people that didn't would rise up, relatively speaking.
And of course, the reputation of every poster would be visible next to their comment for bragging rights.
In the end, I couldn't figure out how to reach a critical mass of users and took it down; here is a mock-up I used that shows how it worked, if anyone is interested.
Gladly. Please check the following two publications demonstrating that planetary positions correlate with phenomena with high r2 values written by Renay Oshop my project partner.
"Astronomy of the day is more effective than seasonal decomposition in modeling and predicting the rates of Amazon review misspellings," International Journal of Scientific Research and Management, Feb 2019, Vol. 7, 2, DOI: 10.18535/ijsrm/v7i2.aa01, https://ijsrm.in/index.php/ijsrm/article/view/2040/1719
On third party vote share: Gary Johnson got 3.27% of the popular vote in 2016. I think if you re-ran the tape a few times he might have crossed 5%. Indeed, FiveThirtyEight's final forecast (https://projects.fivethirtyeight.com/2016-election-forecast/) had him at 5.0%.
Talking to some people on Twitter suddenly reminds me how, as a Gen-X er, I'm still bitter about how much the government in the form of Surgeon General Dr. Koop scare mongered straight people about AIDS. They wanted you to believe circa 1990 that all oral sex was going to give you AIDS if you didn't wear a dental dam.
I'm not saying that mattered for most people, but for many of us on the margin I think it was a big deal. Sex is awkward when you are young and the fear of AIDS made it much more awkward. Not much is written about that these days, but when I think back it was a huge deal at the time.
How do we know if we over- or under-reacted? It seems like a lot of people are locked in to defending their views on what the right level of reaction was.
By the way, here's a question for helping come up with an objective assessment of how bad covid was. The Spanish flu of 1918 hit men between 20 and 40 the hardest, thus leaving a lot of widows and orphans. With 500,000 Americans now having died from covid, how many orphans (under 18 years old) did this toll create?
Just wanna say it's so strange that neither you nor your friends got covid-19. I am russian and it highlights again, how different is our risk tolerance, comparing to the rest of the world. I remember one link you posted earlier, where some guys try to calculate the risk of some some interactions, going from the assumption that the reasonable risk of getting covid per year should be 1%. For me it's ridiculous. I would say 70% chance of getting covid per year is ok, maybe 40% if you are extra-cautious.
I got covid (asymptomatically), my wife got covid, my sister and her husband got covid, my wife's cousin and all her family got covid, my best friend got covid (asymptomatically) and his mother got covids, some other friends also got covid. (No deaths, one hospitalization).
And Russia has so few restrictions. After spending five months last year in Chile in a lockdown I enjoy the freedom to leave my house tremendously.
All response are a matter of conjecture. Posit if you will. Nobody I have seen in the public format is a prophet; it's a good thing to give and take in an opinion which can be refined to it's essence without another opinion.
I made predictions last year, inspired by Scott. I got 7/8 95% ones, 15/16 90% ones, 10/13 80% ones, 6/12 70% ones, 8/12 60% ones, and 2/12 50% ones. It seems like my largest error was in the 50% ones, strangely enough, I think due to my tendency to throw a lot of low-probability things with affirmative phrasing ("this WILL happen") there. As a fun fact, the 95% one that I got wrong was "I will still regularly be reading SSC at the end of the year: 95%"; this was incorrect for reasons entirely different than I imagined.
In retrospect, it almost seems like writing questions - in particular, anticipating all possible outcomes and then phrasing questions so there will be unambiguous resolutions given any possible outcome - is harder than assigning probabilities once you've already written the question.
I argue that 15% is on the lower end. The relevant base rate is near 20%, and four years is a long time in politics. I'd say it is too long to make strong predictions about specifics of he next presidential elections to deviate from the base rate more than 5 points.
Partisan tension may be high, but I'd argue that heightened tensions and general weirdness of recent times contribute towards general uncertainty about electoral results. In addition to established third parties, there is a non-zero chance for large scale party realignment or splits, and I can see substantial part of uncertainty splitting in that direction. Was there much reason in 1908 to suspect the great upset of 1912? There had been only one 5+% third party election in previous ~30 years, there was little indication that TR would run *against* Taft or that Debs would double his 1908/1904 level turnout. As a case of upset of smaller magnitude, Perot launched his 1992 campaign only after the primary season had started.
I would pick the general base rate as the best estimate that incorporates how often this kind of things may happen in US politics. And anyway, it still gives 80% chance for all 3rd candidates getting <5% of votes, which is the usual trend. 80% is often.
China's official case count on 12/31/20 was 95,963, so false."
Actually, depends. China publishes two different offical numbers. What you took is the number of symptomatic cases (which China, other than all other countries, enters into the official statistics websites). China also publishes the number of asymptomatic cases each day (positive test but no symptoms), but doesn't make them available in a nice and accessible format. If you count those, too, you are way above 100,000.
Long thoughtful article by Siddhartha Mukherjee. "While the virus has ravaged rich nations, reported death rates in poorer ones remain relatively low. What probing this epidemiological mystery can tell us about global health."
"I will never bet against Zvi on anything. Last year he bet against me on what restaurant I would have dinner at, without knowing anything about my situation or food preferences, and won anyway."
> will a third-party candidate win 5%+ of the popular vote in 2020?
I got very confused before clicking the link: this should be 2024
Even ignoring risk premiums the efficient market hypotheses doesn't predict a 50% chance that one asset will outperform another. It predicts that any two assets will have the same *expected* returns. But if one of the two assets has an asymmetric risk profile (for example a large probability of doing worse balanced out by a tiny probability of doing *much* better), the chance of doing better might not be 50%.
Re question 11: Hdroxychloroquine is not a particularly effective treatment for covid. It does, however, seem to be remarkably effective as a prophylaxis. This is likely how India beat covid against all odds. India accounts for three-quarters of the malaria in Asia and large swaths of its population take hydroxychloroquine regularly because it is less expensive than more recently developed antimalarials.
I don't think it's necessary to conclude that you and Bucky were "worse" than guessing 50-50 on everything. For example, such guesses would imply a distribution with 0% of probability mass between 100k and 3M US COVID deaths. The predictions look pretty reasonable to me; I know that a lot of people (myself included) were underconfident about the possibility of a 2nd wave.
Pretty much every nation has lied about their death counts in an obvious way, since almost every nation shows much higher excess mortality than official covid deaths. There's almost certainly a lot of undercounting going on (though it may not be deliberate lying). The UK, with their unusually loose standard for a death within 28 days of a positive test being a Covid death, is one of the only nations where COVID deaths and excess mortality are similar - in fact the UK has slightly more COVID deaths than excess mortality.
That still isn't enough to knock the US off the top afaiw though. But of course, China's figures are worth as much as a bet on Trump being president, but I suppose it doesn't qualify for being obvious and no one will ever know the true death toll.
"Will the Bay Area stay locked down until Election Day (11/3/2020)?" was worded in your original post as "Bay Area lockdown (eg restaurants closed) will be extended… until Election Day". Both wordings to me are unambiguous: they imply a single continuous lockdown. If they meant multiple lockdowns, the former would read "Will the Bay Area be locked down on Election Day" and the latter would read "Bay Area lockdowns will continue on and off".
Zvi also explicitly clarified on his post (https://thezvi.wordpress.com/2020/05/01/slatestarcodex-2020-predictions-buy-sell-hold/): "If anything, that seems high, assuming it means continuous lockdown until then rather than being locked down on election day." I think omitting that result from his score would be better than including it, because he was explicitly predicting a different thing from you.
I'm getting different totals for the scoring. Could you/someone else check to see what's going on? Just to make sure we're doing the same calculation: Let's call p the probability that a forecaster assigned to the correct answer, then LogOdds = log(p/(1-p)).
Some examples: scott's score on the first question was log(0.6/0.4) ~= 0.405. His score on the third question was log(0.9/0.1) ~= 2.197 (balanced out exactly by his score on the second question).
Doing this computation for all questions gives total scores for Scott, Zvi, and Bucky at 7.2034, 13.2863, and 6.5006 respectively, all better than chance!
Also, I recommend switching to Brier score (squared error) for things like these. It's a proper scoring rule, which means that you cannot game the system; in expectation you'll get the best possible score by reporting your true beliefs. In contrast, something like LogOdds can be gamed since increasing the extremity of your forecast just ups the ante, but the odds of the bet don't change [i.e. log(p/(1-p) = -log((1-p)/p)]. For instance, if you think something is 90% to occur, then by reporting your true beliefs you are expecting to get 1.7578 "LogOdds points" (0.9 * log(0.9/0.1) + 0.1*log(0.1/0.9) ). However, you could expect to score more points by reporting a more extreme belief, like 95%. In that case, you would expect to score 2.3556 points (0.9 * log(0.95/0.05) + 0.1*log(0.05/0.95)). Doing the same calculation with brier score for belief p and *reported* belief x has you minimizing the function p*(1-x)^2 + (1-p)*x^2 for values of x, which occurs only when p=x (note that lower Brier scores are better).
Here's some Matlab code to run these computations:
answers = [.6 .4 .8; .1 .1 .1; .1 .05 .05; .5 .3 .6; .9 .9 .95; .8 .9 .8; .7 .8 .6; .9 .95 .8; .7 .4 .7; .5 .4 .4; .2 .15 .6; .3 .2 .1; .6 .4 .3; .5 .3 .6; .2 .2 .1; .7 .7 .4; .3 .2 .2; .9 .8 .9; .2 .15 .3; .2 .1 .2]; %Scott Zvi Bucky
outcomes = [1 1 0 0 1 1 1 1 0 1 0 0 0 0 0 1 1 0 0 1]'; %Scott's recorded outcomes to the 20 questions
err = abs(answers-outcomes); %Compute error for all answers
accuracy = 1-err; %The probability assigned to the correct answer
brier = (1-acc).^2; %brier is squared error
LogOdds = log(accuracy ./ (1-accuracy) ); %Compute LogOdds for all responses based on accuracy
sum(LogOdds) %display total LogOdds for each forecaster
mean(accuracy) %same for average accuracy
mean(brier) %same for average brier
%% How might a well-calibrated forecaster improve their score?
p = 0.9; %well-calibrated belief
x = p; %stated probability. For now, say what you believe
(p*log(x/(1-x))) + ((1-p)*log((1-x)/x)) %compute expected LogOdds points
x = p+0.05; %say something a little more extreme
(p*log(x/(1-x))) + ((1-p)*log((1-x)/x)) %compute expected LogOdds points
%Compare with brier
x = p; %stated probability. For now, say what you believe
p*(1-x)^2 + (1-p)*x^2
x = p+0.05; %say something a little more extreme
p*(1-x)^2 + (1-p)*x^2
Forgive my ignorance for question 12, "Will I personally will get coronavirus": isn't a 30% probability of this being true equivalent to 70% probability of it being false... i.e. Scott correctly predicted he wouldn't get covid?
> I don't know how else to visually represent "mantic"
By a "mantle" (i.e., pallium). :-)
Next time perhaps include a competent astrologer who also uses ML. Maybe a female for stronger intuition. It would be v interesting to see the results.
I still think this way of giving and judging predictions, using confidence levels, is fundamentally flawed.
Here, let me make some predictions with confidence levels:
1) The sun will rise in the east on February 24th (95%)
2) The sun will rise in the east on February 25th (95%)
3) The sun will rise in the east on February 26th (95%)
4) The sun will rise in the west on February 27th (95%)
5) The sun will rise in the east on February 28th (95%)
6) The sun will rise in the east on March 1st (95%)
...
20) The sun will rise in the east on March 15th (95%)
If we evaluate my predictions a month from now, I think you will find that out of 20 predictions I made with 95% confidence, I got 19 correct. Perfectly calibrated! Clearly I'm the greatest predictor on earth!
I think the only good way to score predictions is to compare them to other people. If a hundred people predict A, you predict B, and you're right, that's great. If 80 people predict X, 20 predict Y and you also predict Y, that's pretty good. If 5 people predict N and you and 95 others predict M, then that's not impressive at all, even if you're right.
Each week I share in the Metaculus discord a list of the top comments. If people are interested, I will share that here too. (Here is last week's:)
https://metaculusextras.com/top_comments?start_date=2021-02-15
Last week's top comments:
- More discussion of Elon's prediction: https://www.metaculus.com/questions/6498/will-sn10-land/#comment-55669
- Assessment of Biden's chances of running: https://www.metaculus.com/questions/6438/will-joe-biden-run-for-reelection/#comment-55732
- Thoughts on Alcubierre drive in the wider context of theoretical physics: https://www.metaculus.com/questions/6558/working-alcubierre-drive-before-2100/#comment-55878
> Also, will Bitcoin outperform the US stock market over the next five years, at 51%. I started out thinking - of course it's 50-50! By the efficient market hypothesis, if any asset was obviously going to do better than another, people would change the price until it wasn't. But on second thought that's wrong - stocks have a higher than 50% chance of beating treasuries over the same period because of a risk premium. Maybe there's no intuitive way to think about this, you have to have opinions on the underlying fundamentals, and it's only 51% by coincidence?
Yeah, this is wrong, and I was also tripped up by this a while ago. The efficient market hypothesis only tells you that E[X] = X. But, e.g., if Bitcoin could only either go up 3x or crash to 0, then the efficient market hypothesis would tell you that p * 3 * Initial capital + (1-p)*0 = Initial capital => p = 1/3 = 33.3% of going up and 66.6% of going down.
It looks like you're still collecting play-money prediction markets but haven't mentioned The Foresight Exchange, one of the oldest. (I tried to post this last week.)
Perot did not win 5+% of the vote as a third-party candidate in 1992 and 1996. He won 8% of the vote as a third-party candidate in 1996, after having won 18% of the popular vote as an *independent* candidate in 1992.
It's an important distinction, because running for President as an independent candidate implies that you're trying to pull the Gabriel-over-the-White-House trick, riding into Washington on a white horse and sweeping away the corruption with the awesome power of the presidency. If you envision *not* being a dictator and actually sharing power with other people, then the first step is almost certainly going to be finding a bunch of other people (particularly wannabe-Congressman type people) with similar goals and getting organized.
Metaculus is scoring the prediction as positive if either a third-party or independent candidate reaches 5%, so at least they're aware of the distinction. But they don't seem to think it is an important distinction, and I think it's important to push back against that.
A pet peeve: "general consensus": If it's a consensus, it's general. "Consensus" is accurate and has less redundancy.
"<i>9. Will China’s reach 100,000 official cases?</i>
China's official case count on 12/31/20 was 95,963, so false."
That was so close. Should there be a scoring system which adjusts for predictions which are just barely true or just barely false?
In grad school I took a course on probabilistic risk where our tests were graded using log odds scoring. Instead of picking A, B, C or D, you wrote in your probabilistic confidence in each choice. For example, if you’re somewhat confident in A, write in 60%, and then distribute the remaining 40% of weight across the other three choices.
Because of the log scoring rule, being very confident and wrong would result in a massive penalty. There was one star student in the class who blew the curve for all of us by putting down 90%+ confidence in his correct answers because he knew what he was doing and, furthermore, knew that he knew what he was doing.
This method was really hard and I got a pretty mediocre grade in the course, but I’ve always thought it was a good idea. You really learn about your own understanding of the material this way. It’s much easier to notice gaps in your own understanding when, instead of being given partial credit, you’re actively penalized for undue confidence.
"But the soul is still a mantic thing; amid the market's din,
List the ominous stern whisper from the Delphic cave within,—
'They enslave their children's children who make compromise with sin.'"
You know, that really works for me.
A 3rd party POTUS candidate reaching 5% is not a particularly high bar nor, during my lifetime at least, a highly unusual event. It happened in 1996, 1992, 1980, and 1968. Three of the last 11 presidential elections and four of the last 14. So even before considering anything specific about 2024, I'd take the "yes" side on 15% odds.
On the third-party candidate: I think that the naive assumption of "Last 3 of 8" making the 15% bet worthwhile is a bit simplistic of an analysis. The first thing to consider is that, for such a small sample size, if it's possible to get more granularity, one should. And we certainly can. The political environments of those 3 third-party successes, so to speak, were not nearly as polarized as things currently are. Unless you think there will be a de-stressing of tensions (Possible) or a fracturing of parties (Very unlikely, given the two-party system's strength) I think it's closer to 1%. If there's an easing of tensions, call it 20%, very generously. Take this as me predicting against this.
Incidentally, it feels to me that there is some serious ambiguity on the "second wave in fall" question. Some would say the second wave was in the summer, and the later one was a third wave. Some would say that third wave was really in the winter, because it peaked in December and January most places. Some would say that although the summer wave seemed distinct from the first wave, it was really just a delayed first wave that hit Florida, Texas, and Southern California as soon as the initial restrictions were eased. Some would say that in the northern plains region, the wave that started building in August/September was actually just their first wave. Though I think that New Mexico and El Paso were clearly in a second local wave by October/November.
But it's really hard counting waves.
"highest death toll per capita (although all the cities that had higher death tolls per capita were small"
Which cities had higher death tolls per capita?
Here is an interesting paradoxical problem with this type of prediction self-scoring: suppose you are especially adept at thinking up prediction questions that are "hard" precisely because they successfully "cleave (potential future) reality in half". For example, your question 9 asks if China will have 100k deaths. In reality it had 96k deaths, so the threshold in this question is a pretty good guess!
If you were insightful enough to keenly set the thresholds for all your questions in such a way, you would frustratingly find that there is no way to earn a positive score at your own game (or even have much fun playing it), since an optimal Bayesian agent should predict 50-50 for all questions.
Re: "will Bitcoin outperform the US stock market over the next five years"
Consider my new cryptocurrency, Gamblecoin. On Jan 1, 2022 I will roll a die, and if that die comes up 6 it activates a smart contract that will buy anyone's Gamblecoins for $1 USD each, otherwise nothing happens. The odds of Gamblecoin beating the market are one in six, and this is not an efficient market violation because if it does beat the market, it beats it by a lot.
On a long enough timescale, Bitcoin should either become some kind of enormous global currency worth way more than it currently is, or become basically worthless as we find something else to serve the "anonymous secure global currency" role (doesn't have to be crypto, maybe the banks get their act together and come up with an efficient way to move USD) and Bitcoin no longer has a reason to go to the moon. The odds of Bitcoin becoming an enormous global currency are probably not 50/50, so there is some time period over which prediction markets shouldn't say it's 50/50 to beat the market.
That being said, there's a decent chance that Bitcoin is still in the speculative bubble phase and we won't know whether it becomes the massive global currency until far in the future. "Does Bitcoin beat the stock market over the next five days" should be 50/50 on the efficient market grounds you mention because the next five days are (very probably) just going to be speculative trading rather than a resolution to the question of whether Gamblecoin has value or not.
Vaguely on topic: somewhere in a thread on SSC or a related site, I predicted 1 million or more excess deaths in the US during covid, explicitly including deaths due to untreated non-covid conditions, despair due to unemployment, etc.
A month or two after that, US covid deaths were still below 1/4 million, and the rate was dropping, and I felt like an idiot - but I'd made the prediction, as my first attempt at measuring my prediction results.
When I checked today, the number of explicitly covid deaths reported in the US is over 1/2 million (501,117), and my confidence in my prediction is much higher.
Excess deaths - i.e. number of deaths in each 12 month period compared to the last 12 months without covid - aren't tracked as prominently, or reported as swiftly, so I can't yet check how close we've come so far.
I'm reposting mostly to continue to hold myself accountable.
Regarding bitcoin vs. stock market:
I see a number of people modeling bitcoin outcomes as binary. That may be reasonable, but just about every other financial asset is very well modeled as a random walk with drift (expected return). If we do that, stocks have about 15% annual volatility over long horizons, and bitcoin has... I actually haven't measured it but it's probably 5-10x that. Expected returns on stocks is probably in the ballpark of 4%, looking at international markets (US has done better, but using its returns to predict its future returns is probably worse than using the average international estimate). Over a 5 year period, the volatility of bitcoin is so much higher than stocks that the average return on stocks barely matters. So the random variable (bitcoin 5 year returns - stock 5 year returns) has a negative mean (if we assume bitcoin doesn't have a positive risk premium), but its volatility is so high relative to that mean that the probability of it being greater than 0 is pretty close to 50%.
Have you made the 2021 predictions? Or will you continue the cycle and make them in April?
>16. Will there be a general consensus that summer made coronavirus significantly less dangerous?
Maybe this is true, but I have my doubts. I found this posted on Good Judgment Open:
"Given the lack of immunity to SARS-CoV-2 across the world, if there is an effect of temperature and humidity on transmission, it may not be as apparent as with other respiratory viruses for which there is at least some preexisting partial immunity. It is useful to note that pandemic influenza strains have not exhibited the typical seasonal pattern of endemic/epidemic strains. There have been 10 influenza pandemics in the past 250-plus years—two started in the northern hemisphere winter, three in the spring, two in the summer, and three in the fall. All had a peak second wave approximately 6 months after emergence of the virus in the human population, regardless of when the initial introduction occurred."
https://www.nap.edu/read/25771/chapter/1
These markets seem to be modeled on, well, markets, but the function they aim to fulfill seems to be closer to that of a consulting firm. Betting/investment markets, at least in the US, come with a prohibitive degree of red tape.
So, bizarre question: would it be possible to create a prediction "market" that functioned like "the Uber of consulting firms"? Clients contract with the firm for a given amount with a set of things they want predictions on. "Independent contractors" can buy a percentage in the contract and submit their prediction(s) for the associated questions. Payout is a function of accuracy and percentage of buy-in.
1) Could that work?
2) Would it be legally less difficult than prediction markets as currently constituted?
Re Third party 5% share: Seems like the easiest path would be a Trump Patriot Party-esque splinter. This'd require Trump to rerun (or find a charismatic substitute) and for the GOP to resist hard enough to create an election-losing vote split over it, which seems iffy, but certainly raises the chance beyond the background chance in the previous elections.
Last year, I built a site that was based on predicting the future. (Don't worry, it's dead, this is not an advert)
The idea was to make it a more general, lower-entry prediction market. Instead of people betting money, they would gain or lose reputation. The site itself would function like a geopolitics focused subreddit, except that people with good reputation have much stronger upvotes. Therefore, the smart people would rise to the top much faster, and if a strong expert made a comment a few hours too late, it would not be buried. Second, any unproductive groupthink or echo chamber effects would be broken down quite fast - the people that succumbed to this sort of stuff would lose reputation, and the people that didn't would rise up, relatively speaking.
And of course, the reputation of every poster would be visible next to their comment for bragging rights.
In the end, I couldn't figure out how to reach a critical mass of users and took it down; here is a mock-up I used that shows how it worked, if anyone is interested.
https://app.moqups.com/7XPcEzrtnD/view/page/ab67a74d4?ui=0
Gladly. Please check the following two publications demonstrating that planetary positions correlate with phenomena with high r2 values written by Renay Oshop my project partner.
"Astronomy of the day is more effective than seasonal decomposition in modeling and predicting the rates of Amazon review misspellings," International Journal of Scientific Research and Management, Feb 2019, Vol. 7, 2, DOI: 10.18535/ijsrm/v7i2.aa01, https://ijsrm.in/index.php/ijsrm/article/view/2040/1719
and
"Twitter Followers Biased to Astrological Charts of Celebrities," Journal of Scientific Exploration, Vol. 29, No. 1, pp. 9–34, 2015. Data files are available for confirmation. http://www.scientificexploration.org/docs/29/jse_29_1_OshopandFoss.pdf
You may also want to look at my videoblog discussing the relationship between astrology and science. https://www.youtube.com/c/JutkasAstrologyandScience
On third party vote share: Gary Johnson got 3.27% of the popular vote in 2016. I think if you re-ran the tape a few times he might have crossed 5%. Indeed, FiveThirtyEight's final forecast (https://projects.fivethirtyeight.com/2016-election-forecast/) had him at 5.0%.
Nobody you personally know outside of patients got Covid?! Wow, either your social circle is very small or very cautious.
Talking to some people on Twitter suddenly reminds me how, as a Gen-X er, I'm still bitter about how much the government in the form of Surgeon General Dr. Koop scare mongered straight people about AIDS. They wanted you to believe circa 1990 that all oral sex was going to give you AIDS if you didn't wear a dental dam.
I'm not saying that mattered for most people, but for many of us on the margin I think it was a big deal. Sex is awkward when you are young and the fear of AIDS made it much more awkward. Not much is written about that these days, but when I think back it was a huge deal at the time.
How do we know if we over- or under-reacted? It seems like a lot of people are locked in to defending their views on what the right level of reaction was.
By the way, here's a question for helping come up with an objective assessment of how bad covid was. The Spanish flu of 1918 hit men between 20 and 40 the hardest, thus leaving a lot of widows and orphans. With 500,000 Americans now having died from covid, how many orphans (under 18 years old) did this toll create?
Just wanna say it's so strange that neither you nor your friends got covid-19. I am russian and it highlights again, how different is our risk tolerance, comparing to the rest of the world. I remember one link you posted earlier, where some guys try to calculate the risk of some some interactions, going from the assumption that the reasonable risk of getting covid per year should be 1%. For me it's ridiculous. I would say 70% chance of getting covid per year is ok, maybe 40% if you are extra-cautious.
I got covid (asymptomatically), my wife got covid, my sister and her husband got covid, my wife's cousin and all her family got covid, my best friend got covid (asymptomatically) and his mother got covids, some other friends also got covid. (No deaths, one hospitalization).
And Russia has so few restrictions. After spending five months last year in Chile in a lockdown I enjoy the freedom to leave my house tremendously.
All response are a matter of conjecture. Posit if you will. Nobody I have seen in the public format is a prophet; it's a good thing to give and take in an opinion which can be refined to it's essence without another opinion.
I made predictions last year, inspired by Scott. I got 7/8 95% ones, 15/16 90% ones, 10/13 80% ones, 6/12 70% ones, 8/12 60% ones, and 2/12 50% ones. It seems like my largest error was in the 50% ones, strangely enough, I think due to my tendency to throw a lot of low-probability things with affirmative phrasing ("this WILL happen") there. As a fun fact, the 95% one that I got wrong was "I will still regularly be reading SSC at the end of the year: 95%"; this was incorrect for reasons entirely different than I imagined.
You've inspired me to evaluate my own predictions (mostly relevant to Georgia-the-country) from last April - https://peripateticpedagogue.wordpress.com/2021/02/24/evaluating-past-predictions/
In retrospect, it almost seems like writing questions - in particular, anticipating all possible outcomes and then phrasing questions so there will be unambiguous resolutions given any possible outcome - is harder than assigning probabilities once you've already written the question.
I argue that 15% is on the lower end. The relevant base rate is near 20%, and four years is a long time in politics. I'd say it is too long to make strong predictions about specifics of he next presidential elections to deviate from the base rate more than 5 points.
Partisan tension may be high, but I'd argue that heightened tensions and general weirdness of recent times contribute towards general uncertainty about electoral results. In addition to established third parties, there is a non-zero chance for large scale party realignment or splits, and I can see substantial part of uncertainty splitting in that direction. Was there much reason in 1908 to suspect the great upset of 1912? There had been only one 5+% third party election in previous ~30 years, there was little indication that TR would run *against* Taft or that Debs would double his 1908/1904 level turnout. As a case of upset of smaller magnitude, Perot launched his 1992 campaign only after the primary season had started.
I would pick the general base rate as the best estimate that incorporates how often this kind of things may happen in US politics. And anyway, it still gives 80% chance for all 3rd candidates getting <5% of votes, which is the usual trend. 80% is often.
"9. Will China’s reach 100,000 official cases?
China's official case count on 12/31/20 was 95,963, so false."
Actually, depends. China publishes two different offical numbers. What you took is the number of symptomatic cases (which China, other than all other countries, enters into the official statistics websites). China also publishes the number of asymptomatic cases each day (positive test but no symptoms), but doesn't make them available in a nice and accessible format. If you count those, too, you are way above 100,000.
https://en.wikipedia.org/wiki/Statistics_of_the_COVID-19_pandemic_in_mainland_China
So the answer is both "true" and "false". Very much in the spirit of Yin and Yang.
Long thoughtful article by Siddhartha Mukherjee. "While the virus has ravaged rich nations, reported death rates in poorer ones remain relatively low. What probing this epidemiological mystery can tell us about global health."
https://www.newyorker.com/magazine/2021/03/01/why-does-the-pandemic-seem-to-be-hitting-some-countries-harder-than-others
I made my own predictions on Scott's questions. Assuming I've scored correctly with ln(p) - ln(.5), I managed a *3.34*!
1. Bay Area lockdown (eg restaurants closed) will be extended beyond June 15 20% -0.92
2. …until Election Day 10% -1.61
3. Fewer than 100,000 US coronavirus deaths 10% 1.61
4. Fewer than 300,000 US coronavirus deaths 70% -0.34
5. Fewer than 3 million US coronavirus deaths 90% 0.59
6. US has highest official death toll of any country 90% 0.59
7. US has highest death toll as per expert guesses of real numbers 80% 0.47
8. NYC widely considered worst-hit US city 80% 0.47
9. China’s (official) case number goes from its current 82,000 to 100,000 by the end of the year 70% -0.34
10. A coronavirus vaccine has been approved for general use and given to at least 10,000 people somewhere in the First World 20% -0.92
11. Best scientific consensus ends up being that hydroxychloroquine was significantly effective 20% 0.92
12. I personally will get coronavirus (as per my best guess if I had it; positive test not needed) 10% 1.61
13. Someone I am close to (housemate or close family member) will get coronavirus 30% 0.51
14. General consensus is that we (April 2020 US) were overreacting 10% 1.61
15. General consensus is that we (April 2020 US) were underreacting 30% 0.51
16. General consensus is that summer made coronavirus significantly less dangerous 10% -1.61
17. …and there is a catastrophic (50K+ US deaths, or more major lockdowns, after 1+ month without these things) 2nd wave in fall 30% -0.51
19. At least half of states send every voter a mail-in ballot in 2020 presidential election 10% 1.61
20. PredictIt is uncertain (less than 95%) who won the presidential election for more than 24 hours after Election Day 20% -0.92
"I will never bet against Zvi on anything. Last year he bet against me on what restaurant I would have dinner at, without knowing anything about my situation or food preferences, and won anyway."
I am so curious about this.