1. Will an existing, validated personality assessment tool or validated questions be used in the upcoming survey to assess personality?
2. If it became widely known that people with X (personality) trait are poor predictors (or at least widely known by prediction model participants), will those people self-select out of prediction markets, especially those associated with actual money (some people already decline to participate in polls and some people don't "gamble")?
3. Could it be that some personalities predict some things well, but other things poorly? Would that correlation be looked for?
Edit: Sorry, I see the contest rules are deliberately ambiguous ("probably Briar score"), probably to prevent me from doing exactly the sort of gaming of the system I was planning :) However just to say that on my own forecasting tournament Briar and log -loss produce quite different results some years
Might I suggest giving a slightly below-average score for skipped questions? Maybe 40th percentile or so? As someone who'll probably skip some questions but hopefully not too many I don't want to feel like I'm incentivized to skip questions I'm less confident on.
How do you plan to prevent the blind mode prizes from going to people who cheated by spending more than 5 minutes on research or looking at Manifold or Metaculus?
Does this question include Sweden and Finland as "new countries" or no? They've signed accession protocols so their membership is imminent, but they aren't strictly speaking members of NATO yet.
For new countries joining NATO, I assume that's beyond Finland and Sweden? I'm pretty sure they're not formal members yet so it might be good to clarify if they count.
7. Does this have to be an accident? As opposed to say an intentional strike, or sabotage? What about intentional mismanagement leading to a predictable but plausibly deniable "accident"? If accident just means nuclear related event leading to evacuation (due to the nuclear part as opposed to say some fighting/strikes in the area), I think my predictions are different.
Small request on the survey, based on looking at last year's — I don't remember ever taking a formal IQ test, and I don't have my SAT score, but I still have my ACT score on hand. Can we use ACT as an entry in that section? (If not, should I just leave the entire intelligence section blank?)
This should be fun! I entered your 2022 contest, and was inspired to host my own contest for friends that was derived from the questions you sent out + more fun/dumb cultural questions about sports, James Bond, etc to try to capture a category of friends who wouldn't typically try this kind of thing. Every few months I send out an email that provides a rundown of the questions that resolved, the questions in the news and the current leaderboard. I plan to do the same this year but make it more community-oriented by crowdsourcing at least a few questions from participants, maybe we'll have a t-shirt or something dumb like that. There isn't really a way to scale this without losing some specificity (it takes time and it's more fun amongst people who know each other), but I do think there's more buy-in knowing that it's a year-long thing, that your friends are doing it and that there's some minimal interaction with the content.
I wish there was some way to easily plug in the data I collect back into the data you all are collecting.
TW: Very Pedantic. The questions are written as questions, not statements, but the answers are probability percentages which (to my understanding) correspond with an affirmative statement. I find it a bit unintuitive but I assume this is by some convention I'm not aware of. The 2022 survey had questions in the form of a statement
E.g. Question 1. "Will A be B of C". The actual statement I am giving a probability for is instead "A will be B of C".
Last year's was fun, I took a look back and think I did alright. I was a bit too bullish on the GOP and the economy, but correctly bearish on Crypto
38. Applepay (Apple) already allows users to pay using USDC, a stablecoin. Would this resolve to true assuming it doesn't break the partnership?
50 and arguably 47 already resolved as true in 2022. For 50, Imagen Video (https://imagen.research.google/video/) is currently the best, but Meta has a similar model. For 47, early in the Ukraine - Russia war, there was a deepfake of Zelenskyy calling for Ukranian soldiers to surrender, and I find it highly probable someone was upset by this, though I can't name a specific person.
This is super cool. As a machine learning scientist I recommend cross entropy loss as a scoring mechanism. Cross entropy loss is a principled and mathematically pleasing way to evaluate probability estimates. Let x be the probability you assigned to the true eventual outcome. The cross entropy loss of this forecast is -log(x). So if you said x is 100% your loss is zero. If you said x is 0% your loss is infinite. (Perhaps why Scott didn't allow 0% or 100% predictions - if you're wrong then in theory no one would ever be able to trust you again.) Cross entropy loss is how all the best machine learning models are trained and it would be a great way to score this competition as well.
43. <i>Will a new version of COVID be substantially able to escape Omicron vaccines?
This question will resolve positive if in the opinion of the judges the scientific consensus is that getting all currently-recommended vaccines, including the two original vaccines and the Omicron booster, decreases risk of the new variant by less than 50%.</i>
This seems underspecified. What risk do you mean? The risk of being hospitalized? The risk of dying? The risk of contracting the virus? The risk of transmitting the virus? And do you mean absolute risk reduction or relative? If a new variant emerges that is far less lethal and results in far less hospitalizations, but is totally unaffected by the vaccine, did risk increase or decrease ?
This is a winner-takes-all game without punishment for bad guesses. Doesn’t this push us away from making our best-calibrated guess toward a more extreme higher-risk strategy?
Question 9 is very specific about nuclear weapons, then question 10 is exceptionally vague. It's not clear if an act of terrorism involving a dirty bomb would qualify for question 10 and this could have a significant effect on the answer. It's also not clear what "used" means in question 10. You could argue that technically, Russia is already "using" its nuclear weapons in war, since their existence and its threatening posture is limiting the Western response.
In question 49, an "average big tech company employee" probably works in Amazon warehouse fulfillment or Meta content moderation. This probably should be limited to software engineers.
Question 19 (Will the Supreme Court rule against Affirmative Action) needs much more clarification, in my opinion.
The range of outcomes that might be considered to resolve to true range from: "It is illegal to overtly and explicitly admit someone on the basis of their race, but it is perfectly fine to do so as long as you don't say you are doing it," to "any evidence whatsoever of preference on the basis of race alone is actionable."
Rather than going off a predefined list of predictions, it would be more fun to let people come up with their own unlikely predictions. You could then choose, let's say, 20 most interesting ones and put them up for a vote. The winner would be the person whose prediction was judged as least likely to occur by voters, while also being judged as "interesting" by at least 50% of voters and you.
For question 47: "The harm must come directly from the victim believing the deepfake..."
So, to clarify, this specifically means the VICTIM (person harmed) has to be the one believing the deepfake, not anyone else?
For instance, suppose Alice makes a deepfake video of Bob committing a crime (that Bob did not actually commit), shows this video to the police, the police arrest Bob, and Bob is convicted of the crime. Is it the case that this does not count as positive resolution, because the harm did not come from **Bob** believing the deepfake (it came from the police and jury believing the deepfake)?
Re question 13, on wars other than the Russia-Ukraine. If the Russia-Ukraine war escalates to WWIII and starts to involve many other participants, does it count as the other war or as the same war?
Oh this is fun, When can we talk about these questions, and resolutions... where the data comes from to decide. I feel that for some of the questions, the final result source might be more important than the question. I also feel like some of the questions need more details.
For question 14 I need units clarification. Is this 25 million for a year, or 25 million in one day, week.. forever or instantaneously?
So, please provide a method come 2023 of retrieving my submissions (presumably based on email address or bespoke ID code) so that I can compare them to the resolutions.
My strategy: bet on the "nothing really changes" side of each question with some unreasonably high probability. This doesn't maximise my expected score but might maximise my chances of winning some money.
My initial strategy was to vote 99 on everything but I think you deliberately structured the questions in such a way as to be resistant to this. In hindsight, I should've alternated between 1 and 99 instead.
Are we allowed multiple submissions? I want to see if random patterns can beat humans. I'd like to submit a coinflip attempt, where I flip a coin to decide each answer.
Inspired by this video [0] it would be interesting to see an answer inspired by an animal selecting some kind of food. Can prediction markets beat a dog?
From a game theoretic perspective, you maximize your expected score by answering with your true beliefs on probabilities, but maximize your chances of winning the contest by biasing towards overconfidence, because that increases the standard deviation of your score. Being the top scorer in a large group of people requires, high mean, high variance, and then getting lucky such that the variance turns out positive rather than negative. So even if some event genuinely has a 70% probability of happening (whatever that means) with the best information available, it may be worthwhile to guess 80% or 90% anyway, or even 99% under the assumption that if it doesn't happen you were going to lose anyway so might as well go all in.
This is significantly mitigated by having 50 questions. Someone who makes a habit of guessing 99% randomly is going to have too low of a chance and there aren't 2^50 people in the competition to drown it this way, but I would not at all be surprised to find that the winner is someone who does stuff like guessing 99% on what ought to be 90% questions and manages to get away with it, while more principled people end up dominating the high-but-not-top scores.
Great idea, however I am a bit worried about the following: 'If you skip, we’ll give you an “average” score (that is, you lose the average amount of points that forecasters lost on that question).'
Usually scores are not linear, so a few very bad predictions (e.g. 1% and 99% ones) can skew it quite a lot. I feel like it might be dangerous to skip questions even if you know nothing about it. It could be better to default to 50%.
Shall loss for skipped question be changed to min(current scheme loss, 50% prediction loss) in the next year?
For Full Mode (I just imagine this is a definite no for Blind), would you permit group entries if these were explicitly flagged as such? I'm really interested to see how groups perform in general compared with individuals, and I have a group I definitely want to do this with. My reading of the rules of Full Mode suggests this is allowed, because, in a sense, my group is just some evidence I've consulted for my entry, but it feels marginal in spirit so thought I'd ask. Would still do it if ineligible for prizes as a result, but don't want to mess stuff up by doing something you didn't expect.
Shouldn't there be a question like: "how many jelly beans are in the jar pictured above?"
The most important aspect of prediction "markets" is the wisdom of the crowds, which is basically sampling theory based on convenience sample - at best (or a confounding self-selected participation).
Many, here and elsewhere, seem to think it is about the monetization aspect or skin in the game. But to me this is a fallacy. First, skin in the game analysis would, to my way of thinking, require that we know about the wealth of the gambler. A $100 bet by a billionaire is not the same "skin" as the $100 bet by the pauper. Yet, so far as I can tell there is never any attempt to capture that information by the skin in the game folks.
Guess the jelly beans in the jar - free - random sample
Guess the jelly beans in the jar - free - self selection participation
Guess the jelly beans in the jar - de minimus entry fee - like raffle
Guess the jelly beans in the jar - entry fee as surrogate for confidence - bet as little or as much.
Guess the jelly beans in the jar - entry fee as surrogate for confidence where bet size is proportion of wealth and/or income.
Who does better? Which "average" result is better?
Maybe an obvious point, but most of these questions are very specific to the interests of the Bay Area rationalist community, such that giving clueful responses (especially in Blind Mode) will often be more dependent on familiarity with recent SpaceX and OpenAI press releases, hailing taxis in SF, how stablecoins are supposed to work, and similar niche topics than any kind of generalizable "forecasting accuracy" trait.
Accordingly, anyone using this dataset will have to take care to distinguish hypotheses like "Using LessWrong tends to make you less wrong" from the less exciting "rationalists are relatively up-to-date about the topics rationalists like to talk about".
There are a lot of half measures the US Supreme Court might take in the case on affirmative action; that one could probably use some preemptive clarification to avoid grey area resolution down the road.
There were a number of questions where my uncertainty was mostly driven by trying to guess what the evaluator's judgement would be, rather than the ground truth: 5, 38, 41, 43, 47, 49, 50. Some of these could be improved with more specifics in the resolution criteria, but for the most part I often felt like I needed more info about what the evaluator thought about similar questions in the past - was there a "cease fire" between Ukraine and Russia 2015-2019?
("The evaluator" would be Scott for most of these, but not for e.g. 41)
Apologies if this was clearly answered somewhere else, but if we provide our email address will our individual results/scores be sent to us after the completion of the contest?
If that's not planned, what would it take to make that happen?
OK I made my guesses. So I don't understand the scoring, for questions I didn't know... like all UK politics. I just put down 50%. Why is this the wrong (or right) thing to do.
The rational strategy is a high-variance approach aimed at getting the highest score out of anyone while also increasing the likelihood of getting a below average score. Even if this approach leads to a lower average score, it will be rewarded more than trying to make well-calibrated predictions that with a higher expected average score. This underminess the usefulness of such an exercise.
Do you expect Twitter to continue reporting net income and mDAU on a quarterly basis, even though they are a private company and have no obligation to publish quarterly reports?
> I’m planning to give out at least 4 x $500 prizes: one for winner of Blind Mode, one for winner of Full Mode
One potential issue with this is that playing to win could end up a little at odds with giving your best prediction. Ie. someone who expects to have a non top-tier performance by giving their best guesses would likely have a higher chance of winning by maximising their variance, even at the cost of accuracy, as an average score can't win, but by making overconfident guesses, they increase the chance they'll get lucky, even though their average expected score would be lower.
Admittedly, I don't think you can really practically solve that - ideally you'd want a payout proportional to score, rather than a winner-takes-all setup, but the logistics of that would be incredibly impractical (and might also encourage other forms of gaming like spamming duplicate accounts/entries). Probably not a major issue, since I think most will be playing for fun / aiming just for a good performance, rather than solely aiming for the prize.
On a tangent, that you can't predict anything, the World Cup final is being played right now. Argentina were leading 2-0 right up to the 80th minute, then Kylian Mbappé got two goals.
So now the match really has caught fire again, because whoever gets the next goal in the next ten minutes (if anyone does) is going to win this, and Messi may never get that elusive world cup winner's medal after all (after eighty minutes that it looked done and dusted) 🤣
Several people mention that the winner-take-all structure of the contest does not incentive reporting honest preferences. Does anyone know of an example of a forecasting contest where someone successfully exploited this? Maybe you should increase confidence, but I suspect most people would overdo it and sabotage themselves.
It seems to me that the more relevant strategy is to exploit the correlation between the questions. If you think there's a 30% chance of nuclear war, you should gamble on it actually happening. Put down 99 and analyze what it does to the rest of the questions. But if you think there's a 30% chance of nuclear war, you should spend a lot more than $500 optimizing your life around the risk. (I got the 30% figure from Tegmark, who did upend his life.) Far short of this, it might make sense to bet on Ukraine escalating or de-escalating, but that probably doesn't affect all the other questions as much.
Do you know any examples of people exploiting correlation between questions in forecasting contests?
Also: don't comment here on my comments there! I have disabled comments (a blessing for the comments: may השם bless and keep the comments ... far away from us), but will enable them on that post so you aren't tempted to reply with spoilers.
I did gauntlet mode, no internet and 30 seconds pondering max. Being non-USA will hurt with some answers but we'll see. I'm confident that I'll be the dark horse in this :D
General question: if any of the "did it happen" questions actually resolve in 2022 (eg 43 with BQ1.1, or say Meta surprises us with a "Quest Pro 2" on 12/25/22), will it resolve positively?
1-4. Are these point or duration questions?
4. The main train station in Zhaphorizia, Zhaphorizia-Live, including its rail yard, appears to be large enough (10km wide) that there could plausibly be Ukrainian control over the west side and Russian control over the East, and there is a platform at 4 km from the "0km" platform that is still labeled as Zhaphorizia-Live. Can you be more specific about what point must be controlled?
7. Is this at any point during the year, and if the evacuation is precautionary, and turns out to have been unnecessary, ie, people go back home before 1/1/24, will it resolve positively?
8. Does a fizzle (attempted to detonate a nuke but only detonates as a dirty bomb) resolve positively?
11. Which step at https://www.nato.int/cps/en/natolive/topics_49212.htm will cause this to resolve positively? After completion of 5 seems best to me, but during/after 6 or 7 seems reasonable as well. Asking because I think there's a reasonable chance 5 finishes late next year, not to be pedantic.
19. This is really, really vague. A ruling upholding the status quo could be interpreted as "against affirmative action". Unfortunately I don't have quite enough familiarity with the subject to know which precedents being overturned or ignored should count towards a positive resolution.
22. Is this a point, duration, or at-any-point question?
30. Will this be reevaluated when subsequent numbers come out a couple months later, or just based on preliminary numbers in January?
37. Ditto.
40. The first "orbital flight test" is currently intended to stop just short of "orbital velocity", and it will not complete an entire orbit. Technically it's still on an orbit, just a guaranteed decaying one; the perigee will be above the surface of the Earth, but at ~70km (https://twitter.com/planet4589/status/1411781300063813648), below the Kármán line) for mission simplicity, not because the vehicle will be incapable of getting the last 30 m/s. So there's 4 potential boundaries for resolution here: A. Getting into LEO (the Orbital Flight Test alone would not resolve positively), B. Any perigee above the Kármán Line will resolve positively (the OTF will only meet this if Starship gets unexpectedly fast, but still below LEO), C. Getting to the target perigee (within a reasonable deviation) or higher (only a nominal OTF or higher perigee will resolve positively), or D. getting into any orbit with a > sea level perigee (even a sub-nominal OTF will resolve as long as the perigee condition is met by the termination of the almost-orbital-insertion burn, even if Starship ends up deorbiting unpowered). Again asking because I think there's a reasonable chance only the OTF, and not subsequent flights, will take place before 2024.
48. This question asks about "Global Health Emergency"s, but gives the previous 3 "Public Health Emergency of International Concern"s as examples instead, while there have been several other "Global Health Emergency"s in 2022, and the odds of a GHE seem much higher than of a PHEIC.
I cannot answer Q13, the term "any other war" is very open-ended. If we're talking about literally any other war, then no, absolutely not. WW2 alone saw 16+ million military personnel die, just amongst the Allies, iirc.
Any other war in 2023? I'm not sure there's going to be one.
This seems like a good opportunity to see how Metacalculus and Manifold perform against an array of human experts + non-experts. It might make sense to pre-register a date and time (Midnight GMT on Feb 1st?) to consider their current prices as their answer for the purpose of the contest? (Or is there some better way of doing it, like looking at the average price over some time interval?)
Registering my own prediction... my bet is that the markets outperform the median participant, because the lowest-effort participants in Scott's context will likely make some wild guesses and are less likely to participate in the markets. But I bet they under-perform the top participants and act more as an averaging function across the guesses of people who bothered to participate in the markets (call it maybe 75th percentile performance in the total contest), because I don't think people more likely to be right will meaningfully correlate with people likely to spend more
Open question for anyone, not just Scott: is there a condition you can imagine which would cause you to update towards prediction markets being less important, less valuable, less relevant, etc?
For "Will any other war have more casualties than Russia-Ukraine?" does the "other war" have to be ongoing in 2023 (otherwise surely World War II wins)?
Does it have to *start* in 2023? Are the casualties being compared just 2023 casualties vs. 2023 casualties, or all casualties for that war? I think any of the Tigray War, Yemeni Civil War, Syrian Civil War, Somali Civil War, Boko Haram Insurgency, or War in Darfur could beat Ukraine War if it's ok for the war to not have started in 2023 and/or we don't compare 2023 vs 2023 only.
Are you able to send us an email (if we included it) with our responses? I was expecting we would receive some record of our guesses and I did not make a copy of them.
Comment/critique on specific questions: the Covid-19 statistics especially on China seem one that would be hard to have an accurate figure for at any point in time (except perhaps way into the future from independent investigations) and hence doesn’t seem fair or useful when assessing results.
As others have mentioned, question 13 is phrased extremely ambiguously. The rest seemed basically fine to me, although at least one will probably be ambiguous by the time it resolves (i.e. true Covid deaths - worldwide and in China - are going to be higher than the official count).
Wait, blind mode is closed? It says "until January 10" here – I took that to be inclusive. Did you already release the "anonymized versions of all Blind Mode predictions"?
1. Is there any way to get a copy of our submission before the end of 2023?
2. Will a “running tally” of events that have/have not occurred be kept visible of the course of the year? For instance, gpt-4 has been released; is there anywhere that that's marked as “has happened”?
1. Will an existing, validated personality assessment tool or validated questions be used in the upcoming survey to assess personality?
2. If it became widely known that people with X (personality) trait are poor predictors (or at least widely known by prediction model participants), will those people self-select out of prediction markets, especially those associated with actual money (some people already decline to participate in polls and some people don't "gamble")?
3. Could it be that some personalities predict some things well, but other things poorly? Would that correlation be looked for?
Your definition of "major US political figure" in #24 would exclude Donald Trump (or Mike Pence). Did you mean to do that?
How is the contest scored please?
Hugely excited to participate!
Edit: Sorry, I see the contest rules are deliberately ambiguous ("probably Briar score"), probably to prevent me from doing exactly the sort of gaming of the system I was planning :) However just to say that on my own forecasting tournament Briar and log -loss produce quite different results some years
was the 2022 one not official?
Might I suggest giving a slightly below-average score for skipped questions? Maybe 40th percentile or so? As someone who'll probably skip some questions but hopefully not too many I don't want to feel like I'm incentivized to skip questions I'm less confident on.
How do you plan to prevent the blind mode prizes from going to people who cheated by spending more than 5 minutes on research or looking at Manifold or Metaculus?
https://manifold.markets/IsaacKing/in-the-acx-2023-prediction-contest
"Will any new country join NATO?"
Does this question include Sweden and Finland as "new countries" or no? They've signed accession protocols so their membership is imminent, but they aren't strictly speaking members of NATO yet.
For new countries joining NATO, I assume that's beyond Finland and Sweden? I'm pretty sure they're not formal members yet so it might be good to clarify if they count.
7. Does this have to be an accident? As opposed to say an intentional strike, or sabotage? What about intentional mismanagement leading to a predictable but plausibly deniable "accident"? If accident just means nuclear related event leading to evacuation (due to the nuclear part as opposed to say some fighting/strikes in the area), I think my predictions are different.
Small request on the survey, based on looking at last year's — I don't remember ever taking a formal IQ test, and I don't have my SAT score, but I still have my ACT score on hand. Can we use ACT as an entry in that section? (If not, should I just leave the entire intelligence section blank?)
This should be fun! I entered your 2022 contest, and was inspired to host my own contest for friends that was derived from the questions you sent out + more fun/dumb cultural questions about sports, James Bond, etc to try to capture a category of friends who wouldn't typically try this kind of thing. Every few months I send out an email that provides a rundown of the questions that resolved, the questions in the news and the current leaderboard. I plan to do the same this year but make it more community-oriented by crowdsourcing at least a few questions from participants, maybe we'll have a t-shirt or something dumb like that. There isn't really a way to scale this without losing some specificity (it takes time and it's more fun amongst people who know each other), but I do think there's more buy-in knowing that it's a year-long thing, that your friends are doing it and that there's some minimal interaction with the content.
I wish there was some way to easily plug in the data I collect back into the data you all are collecting.
> For the forecasting questions, you just have to give a percent chance, from 1 - 99.
Since people are rounding to the nearest integer percentage, a response of 0 or 100 is not necessarily irrational, and I think you should allow it.
This is neat! I've made some meta-markets on how this contest will go.
https://manifold.markets/IsaacKing/in-the-acx-2023-prediction-contest-1b0e37bda1a1
https://manifold.markets/IsaacKing/will-any-of-the-events-in-the-acx-2
https://manifold.markets/IsaacKing/in-the-acx-2023-prediction-contest-d819b7d08e28
https://manifold.markets/IsaacKing/in-the-acx-2023-prediction-contest
https://manifold.markets/IsaacKing/in-the-acx-2023-prediction-contest-5228d3a003cc
Fun guessing game but I expect to be completely wrong on everything.
TW: Very Pedantic. The questions are written as questions, not statements, but the answers are probability percentages which (to my understanding) correspond with an affirmative statement. I find it a bit unintuitive but I assume this is by some convention I'm not aware of. The 2022 survey had questions in the form of a statement
E.g. Question 1. "Will A be B of C". The actual statement I am giving a probability for is instead "A will be B of C".
Last year's was fun, I took a look back and think I did alright. I was a bit too bullish on the GOP and the economy, but correctly bearish on Crypto
38. Applepay (Apple) already allows users to pay using USDC, a stablecoin. Would this resolve to true assuming it doesn't break the partnership?
50 and arguably 47 already resolved as true in 2022. For 50, Imagen Video (https://imagen.research.google/video/) is currently the best, but Meta has a similar model. For 47, early in the Ukraine - Russia war, there was a deepfake of Zelenskyy calling for Ukranian soldiers to surrender, and I find it highly probable someone was upset by this, though I can't name a specific person.
This is super cool. As a machine learning scientist I recommend cross entropy loss as a scoring mechanism. Cross entropy loss is a principled and mathematically pleasing way to evaluate probability estimates. Let x be the probability you assigned to the true eventual outcome. The cross entropy loss of this forecast is -log(x). So if you said x is 100% your loss is zero. If you said x is 0% your loss is infinite. (Perhaps why Scott didn't allow 0% or 100% predictions - if you're wrong then in theory no one would ever be able to trust you again.) Cross entropy loss is how all the best machine learning models are trained and it would be a great way to score this competition as well.
That was fun. Kinda. 5 min research did a lot to form my answers - in the few cases, I did so. :)
... - deleted not to violate blind mode -
43. <i>Will a new version of COVID be substantially able to escape Omicron vaccines?
This question will resolve positive if in the opinion of the judges the scientific consensus is that getting all currently-recommended vaccines, including the two original vaccines and the Omicron booster, decreases risk of the new variant by less than 50%.</i>
This seems underspecified. What risk do you mean? The risk of being hospitalized? The risk of dying? The risk of contracting the virus? The risk of transmitting the virus? And do you mean absolute risk reduction or relative? If a new variant emerges that is far less lethal and results in far less hospitalizations, but is totally unaffected by the vaccine, did risk increase or decrease ?
Wish some of this was specified better. Meta has an ar headset coming out too already. Does new Covid variant count as a new pandemic?
This is a winner-takes-all game without punishment for bad guesses. Doesn’t this push us away from making our best-calibrated guess toward a more extreme higher-risk strategy?
Am I supposed to try and win the contest or make the most honest/accurate predictions? They aren't the same thing.
Some pedantic nitpicking about questions:
Question 9 is very specific about nuclear weapons, then question 10 is exceptionally vague. It's not clear if an act of terrorism involving a dirty bomb would qualify for question 10 and this could have a significant effect on the answer. It's also not clear what "used" means in question 10. You could argue that technically, Russia is already "using" its nuclear weapons in war, since their existence and its threatening posture is limiting the Western response.
In question 49, an "average big tech company employee" probably works in Amazon warehouse fulfillment or Meta content moderation. This probably should be limited to software engineers.
Question 19 (Will the Supreme Court rule against Affirmative Action) needs much more clarification, in my opinion.
The range of outcomes that might be considered to resolve to true range from: "It is illegal to overtly and explicitly admit someone on the basis of their race, but it is perfectly fine to do so as long as you don't say you are doing it," to "any evidence whatsoever of preference on the basis of race alone is actionable."
Rather than going off a predefined list of predictions, it would be more fun to let people come up with their own unlikely predictions. You could then choose, let's say, 20 most interesting ones and put them up for a vote. The winner would be the person whose prediction was judged as least likely to occur by voters, while also being judged as "interesting" by at least 50% of voters and you.
Can someone explain (or link me to) some potential ways to score something like this?
I basically guessed most of the non-political questions. If I somehow win, either prediction is nonsense or God decided to briefly speak through me.
This probably won't be relevant, but by "midnight" do you mean 0000 hours on the specified day?
For question 47: "The harm must come directly from the victim believing the deepfake..."
So, to clarify, this specifically means the VICTIM (person harmed) has to be the one believing the deepfake, not anyone else?
For instance, suppose Alice makes a deepfake video of Bob committing a crime (that Bob did not actually commit), shows this video to the police, the police arrest Bob, and Bob is convicted of the crime. Is it the case that this does not count as positive resolution, because the harm did not come from **Bob** believing the deepfake (it came from the police and jury believing the deepfake)?
Re question 13, on wars other than the Russia-Ukraine. If the Russia-Ukraine war escalates to WWIII and starts to involve many other participants, does it count as the other war or as the same war?
Oh this is fun, When can we talk about these questions, and resolutions... where the data comes from to decide. I feel that for some of the questions, the final result source might be more important than the question. I also feel like some of the questions need more details.
For question 14 I need units clarification. Is this 25 million for a year, or 25 million in one day, week.. forever or instantaneously?
Decided to participate for the lolz. Answered most of the questions, but using my gut instinct and mostly binning into one of a few categories:
- 75 is "almost sure"
- 50 is "tossup"
- 30 is "probably not"
- 10 is "almost sure not"
Skipped the ones I don't have any background to answer even with a gut guess.
Meta-prediction--I'm unlikely to be in the top 10 forecasters. Maybe 10% chance?
(45) I may be misunderstanding, but is Google Glass not an example?
Whenever I take a test, I want to know how I did.
So, please provide a method come 2023 of retrieving my submissions (presumably based on email address or bespoke ID code) so that I can compare them to the resolutions.
My strategy: bet on the "nothing really changes" side of each question with some unreasonably high probability. This doesn't maximise my expected score but might maximise my chances of winning some money.
My initial strategy was to vote 99 on everything but I think you deliberately structured the questions in such a way as to be resistant to this. In hindsight, I should've alternated between 1 and 99 instead.
Are we allowed multiple submissions? I want to see if random patterns can beat humans. I'd like to submit a coinflip attempt, where I flip a coin to decide each answer.
Inspired by this video [0] it would be interesting to see an answer inspired by an animal selecting some kind of food. Can prediction markets beat a dog?
[0] https://youtu.be/USKD3vPD6ZA
From a game theoretic perspective, you maximize your expected score by answering with your true beliefs on probabilities, but maximize your chances of winning the contest by biasing towards overconfidence, because that increases the standard deviation of your score. Being the top scorer in a large group of people requires, high mean, high variance, and then getting lucky such that the variance turns out positive rather than negative. So even if some event genuinely has a 70% probability of happening (whatever that means) with the best information available, it may be worthwhile to guess 80% or 90% anyway, or even 99% under the assumption that if it doesn't happen you were going to lose anyway so might as well go all in.
This is significantly mitigated by having 50 questions. Someone who makes a habit of guessing 99% randomly is going to have too low of a chance and there aren't 2^50 people in the competition to drown it this way, but I would not at all be surprised to find that the winner is someone who does stuff like guessing 99% on what ought to be 90% questions and manages to get away with it, while more principled people end up dominating the high-but-not-top scores.
> Scoring will be through some proper scoring rule, probably Brier score
Shouldn’t this be established before the start of the contest? People who make different assumptions about this are playing different games, no?
Great idea, however I am a bit worried about the following: 'If you skip, we’ll give you an “average” score (that is, you lose the average amount of points that forecasters lost on that question).'
Usually scores are not linear, so a few very bad predictions (e.g. 1% and 99% ones) can skew it quite a lot. I feel like it might be dangerous to skip questions even if you know nothing about it. It could be better to default to 50%.
Shall loss for skipped question be changed to min(current scheme loss, 50% prediction loss) in the next year?
For Full Mode (I just imagine this is a definite no for Blind), would you permit group entries if these were explicitly flagged as such? I'm really interested to see how groups perform in general compared with individuals, and I have a group I definitely want to do this with. My reading of the rules of Full Mode suggests this is allowed, because, in a sense, my group is just some evidence I've consulted for my entry, but it feels marginal in spirit so thought I'd ask. Would still do it if ineligible for prizes as a result, but don't want to mess stuff up by doing something you didn't expect.
Shouldn't there be a question like: "how many jelly beans are in the jar pictured above?"
The most important aspect of prediction "markets" is the wisdom of the crowds, which is basically sampling theory based on convenience sample - at best (or a confounding self-selected participation).
Many, here and elsewhere, seem to think it is about the monetization aspect or skin in the game. But to me this is a fallacy. First, skin in the game analysis would, to my way of thinking, require that we know about the wealth of the gambler. A $100 bet by a billionaire is not the same "skin" as the $100 bet by the pauper. Yet, so far as I can tell there is never any attempt to capture that information by the skin in the game folks.
Guess the jelly beans in the jar - free - random sample
Guess the jelly beans in the jar - free - self selection participation
Guess the jelly beans in the jar - de minimus entry fee - like raffle
Guess the jelly beans in the jar - entry fee as surrogate for confidence - bet as little or as much.
Guess the jelly beans in the jar - entry fee as surrogate for confidence where bet size is proportion of wealth and/or income.
Who does better? Which "average" result is better?
Maybe an obvious point, but most of these questions are very specific to the interests of the Bay Area rationalist community, such that giving clueful responses (especially in Blind Mode) will often be more dependent on familiarity with recent SpaceX and OpenAI press releases, hailing taxis in SF, how stablecoins are supposed to work, and similar niche topics than any kind of generalizable "forecasting accuracy" trait.
Accordingly, anyone using this dataset will have to take care to distinguish hypotheses like "Using LessWrong tends to make you less wrong" from the less exciting "rationalists are relatively up-to-date about the topics rationalists like to talk about".
There are a lot of half measures the US Supreme Court might take in the case on affirmative action; that one could probably use some preemptive clarification to avoid grey area resolution down the road.
My 2023 prediction is that it's going to somehow suck even more than 2022 did.
And 2022 started with me working for a company that was, at the time, headquartered in Ukraine. Things went downhill from there.
*sigh*
There were a number of questions where my uncertainty was mostly driven by trying to guess what the evaluator's judgement would be, rather than the ground truth: 5, 38, 41, 43, 47, 49, 50. Some of these could be improved with more specifics in the resolution criteria, but for the most part I often felt like I needed more info about what the evaluator thought about similar questions in the past - was there a "cease fire" between Ukraine and Russia 2015-2019?
("The evaluator" would be Scott for most of these, but not for e.g. 41)
Apologies if this was clearly answered somewhere else, but if we provide our email address will our individual results/scores be sent to us after the completion of the contest?
If that's not planned, what would it take to make that happen?
I would argue that the Quest Pro is already an AR headset, as released by Meta.
If Quest 3 has color cameras and passthrough would that count?
OK I made my guesses. So I don't understand the scoring, for questions I didn't know... like all UK politics. I just put down 50%. Why is this the wrong (or right) thing to do.
The rational strategy is a high-variance approach aimed at getting the highest score out of anyone while also increasing the likelihood of getting a below average score. Even if this approach leads to a lower average score, it will be rewarded more than trying to make well-calibrated predictions that with a higher expected average score. This underminess the usefulness of such an exercise.
Do you expect Twitter to continue reporting net income and mDAU on a quarterly basis, even though they are a private company and have no obligation to publish quarterly reports?
> I’m planning to give out at least 4 x $500 prizes: one for winner of Blind Mode, one for winner of Full Mode
One potential issue with this is that playing to win could end up a little at odds with giving your best prediction. Ie. someone who expects to have a non top-tier performance by giving their best guesses would likely have a higher chance of winning by maximising their variance, even at the cost of accuracy, as an average score can't win, but by making overconfident guesses, they increase the chance they'll get lucky, even though their average expected score would be lower.
Admittedly, I don't think you can really practically solve that - ideally you'd want a payout proportional to score, rather than a winner-takes-all setup, but the logistics of that would be incredibly impractical (and might also encourage other forms of gaming like spamming duplicate accounts/entries). Probably not a major issue, since I think most will be playing for fun / aiming just for a good performance, rather than solely aiming for the prize.
Is there a way to download a transcript of how we answered?
Peter Robinson
On a tangent, that you can't predict anything, the World Cup final is being played right now. Argentina were leading 2-0 right up to the 80th minute, then Kylian Mbappé got two goals.
So now the match really has caught fire again, because whoever gets the next goal in the next ten minutes (if anyone does) is going to win this, and Messi may never get that elusive world cup winner's medal after all (after eighty minutes that it looked done and dusted) 🤣
Several people mention that the winner-take-all structure of the contest does not incentive reporting honest preferences. Does anyone know of an example of a forecasting contest where someone successfully exploited this? Maybe you should increase confidence, but I suspect most people would overdo it and sabotage themselves.
It seems to me that the more relevant strategy is to exploit the correlation between the questions. If you think there's a 30% chance of nuclear war, you should gamble on it actually happening. Put down 99 and analyze what it does to the rest of the questions. But if you think there's a 30% chance of nuclear war, you should spend a lot more than $500 optimizing your life around the risk. (I got the 30% figure from Tegmark, who did upend his life.) Far short of this, it might make sense to bet on Ukraine escalating or de-escalating, but that probably doesn't affect all the other questions as much.
Do you know any examples of people exploiting correlation between questions in forecasting contests?
Is it a percent yes, or percent no?
Can I change my prediction on whether Musk will still be head of twitter? lol
As a curmudgeonly blogger, I felt compelled to write a blog post expounding on my various predictions.
https://www.newslettr.com/p/the-2023-prediction-dispatch - don't click that link if you plan to enter in Blind Mode!
Also: don't comment here on my comments there! I have disabled comments (a blessing for the comments: may השם bless and keep the comments ... far away from us), but will enable them on that post so you aren't tempted to reply with spoilers.
6. Would a kick scooter count as a vehicle?
8. You mean worldwide, not just in Ukraine and not even just on land, right?
43. Risk of what? Infection? Any noticeable symptoms? Symptoms bad enough to be unable to work for ≥1 day? Hospitalization? Death?
I did gauntlet mode, no internet and 30 seconds pondering max. Being non-USA will hurt with some answers but we'll see. I'm confident that I'll be the dark horse in this :D
General question: if any of the "did it happen" questions actually resolve in 2022 (eg 43 with BQ1.1, or say Meta surprises us with a "Quest Pro 2" on 12/25/22), will it resolve positively?
1-4. Are these point or duration questions?
4. The main train station in Zhaphorizia, Zhaphorizia-Live, including its rail yard, appears to be large enough (10km wide) that there could plausibly be Ukrainian control over the west side and Russian control over the East, and there is a platform at 4 km from the "0km" platform that is still labeled as Zhaphorizia-Live. Can you be more specific about what point must be controlled?
7. Is this at any point during the year, and if the evacuation is precautionary, and turns out to have been unnecessary, ie, people go back home before 1/1/24, will it resolve positively?
8. Does a fizzle (attempted to detonate a nuke but only detonates as a dirty bomb) resolve positively?
11. Which step at https://www.nato.int/cps/en/natolive/topics_49212.htm will cause this to resolve positively? After completion of 5 seems best to me, but during/after 6 or 7 seems reasonable as well. Asking because I think there's a reasonable chance 5 finishes late next year, not to be pedantic.
19. This is really, really vague. A ruling upholding the status quo could be interpreted as "against affirmative action". Unfortunately I don't have quite enough familiarity with the subject to know which precedents being overturned or ignored should count towards a positive resolution.
22. Is this a point, duration, or at-any-point question?
30. Will this be reevaluated when subsequent numbers come out a couple months later, or just based on preliminary numbers in January?
37. Ditto.
40. The first "orbital flight test" is currently intended to stop just short of "orbital velocity", and it will not complete an entire orbit. Technically it's still on an orbit, just a guaranteed decaying one; the perigee will be above the surface of the Earth, but at ~70km (https://twitter.com/planet4589/status/1411781300063813648), below the Kármán line) for mission simplicity, not because the vehicle will be incapable of getting the last 30 m/s. So there's 4 potential boundaries for resolution here: A. Getting into LEO (the Orbital Flight Test alone would not resolve positively), B. Any perigee above the Kármán Line will resolve positively (the OTF will only meet this if Starship gets unexpectedly fast, but still below LEO), C. Getting to the target perigee (within a reasonable deviation) or higher (only a nominal OTF or higher perigee will resolve positively), or D. getting into any orbit with a > sea level perigee (even a sub-nominal OTF will resolve as long as the perigee condition is met by the termination of the almost-orbital-insertion burn, even if Starship ends up deorbiting unpowered). Again asking because I think there's a reasonable chance only the OTF, and not subsequent flights, will take place before 2024.
48. This question asks about "Global Health Emergency"s, but gives the previous 3 "Public Health Emergency of International Concern"s as examples instead, while there have been several other "Global Health Emergency"s in 2022, and the odds of a GHE seem much higher than of a PHEIC.
Thanks in advance.
Q.40, how do you define reaching orbit? Does it need to complete a full revolution around the Earth?
The problem with question 14 is it's not actually a question about if there will be a large covid outbreak in china, but if they will bother going through the trouble of recording and reporting it officially. Just today the government officially said they estimate that there were 37 Millon cases in a singular day. Even though this Is directly said from the CCP it doesn't count in this circumstance. Link to article https://www.bloomberg.com/news/articles/2022-12-23/china-estimates-covid-surge-is-infecting-37-million-people-a-day?srnd=premium-europe&leadSource=uverify%20wall
Does a North Korean nuclear test qualify for question 8, "Will a nuclear weapon be detonated (including tests and accidents)?"
> Will Twitter's average monetizable daily users be higher in 2023 than in 2022?
Which specific dates are being compared?
I meant to save my predictions for future reference, but I forgot to do so. Is there any way to access that?
I cannot answer Q13, the term "any other war" is very open-ended. If we're talking about literally any other war, then no, absolutely not. WW2 alone saw 16+ million military personnel die, just amongst the Allies, iirc.
Any other war in 2023? I'm not sure there's going to be one.
That leaves a huge range to work with.
This seems like a good opportunity to see how Metacalculus and Manifold perform against an array of human experts + non-experts. It might make sense to pre-register a date and time (Midnight GMT on Feb 1st?) to consider their current prices as their answer for the purpose of the contest? (Or is there some better way of doing it, like looking at the average price over some time interval?)
Registering my own prediction... my bet is that the markets outperform the median participant, because the lowest-effort participants in Scott's context will likely make some wild guesses and are less likely to participate in the markets. But I bet they under-perform the top participants and act more as an averaging function across the guesses of people who bothered to participate in the markets (call it maybe 75th percentile performance in the total contest), because I don't think people more likely to be right will meaningfully correlate with people likely to spend more
Open question for anyone, not just Scott: is there a condition you can imagine which would cause you to update towards prediction markets being less important, less valuable, less relevant, etc?
Is there a way to get a copy of my responses?
For "Will any other war have more casualties than Russia-Ukraine?" does the "other war" have to be ongoing in 2023 (otherwise surely World War II wins)?
Does it have to *start* in 2023? Are the casualties being compared just 2023 casualties vs. 2023 casualties, or all casualties for that war? I think any of the Tigray War, Yemeni Civil War, Syrian Civil War, Somali Civil War, Boko Haram Insurgency, or War in Darfur could beat Ukraine War if it's ok for the war to not have started in 2023 and/or we don't compare 2023 vs 2023 only.
Can you work with someone else for blind mode or no?
Are you able to send us an email (if we included it) with our responses? I was expecting we would receive some record of our guesses and I did not make a copy of them.
I have the same question as others- can we get a copy of our responses?
Comment/critique on specific questions: the Covid-19 statistics especially on China seem one that would be hard to have an accurate figure for at any point in time (except perhaps way into the future from independent investigations) and hence doesn’t seem fair or useful when assessing results.
As others have mentioned, question 13 is phrased extremely ambiguously. The rest seemed basically fine to me, although at least one will probably be ambiguous by the time it resolves (i.e. true Covid deaths - worldwide and in China - are going to be higher than the official count).
Wait, blind mode is closed? It says "until January 10" here – I took that to be inclusive. Did you already release the "anonymized versions of all Blind Mode predictions"?
(The form says blind mode will no longer count.)
1. Is there any way to get a copy of our submission before the end of 2023?
2. Will a “running tally” of events that have/have not occurred be kept visible of the course of the year? For instance, gpt-4 has been released; is there anywhere that that's marked as “has happened”?
I need to talk to experts in the world of 3Dprinting technology,the future of manufacturing,addictive manufacturing AM