A couple of weeks ago I ran a betting market predictions game night for the NYC rationalist weekly meetup (all bets should be settleable by the end of the night and the winner is whoever has the most net points). It is (a) a lot of fun and (b) incredibly chaotic, would strongly recommend more people do it. I wrote a post about how I set it up technically and what I'd do to run it smoother here https://shakeddown.wordpress.com/2022/08/04/what-i-learned-about-running-a-betting-market-game-night-contest/
You get some incredibly weird phenomena - I had one question that was "which question will have the most bet on it", which led to two other questions getting a ton of mana sunk into them as the people who'd bet on them tried to make their questions win out. One of these questions was "which answer will get the most points" (the other was "how many light years away is alpha centauri"), which was won by one guy suddenly sinking all his points into the runner-up just before it closed (and since it had a lot of points invested, he ended up winning the whole game by like a 2-1 margin over the runner-up thanks to this. This was also the guy whose day job is a Jane Street trader).
(Also several other fun attempted manipulations. It really is a great creativity generator).
The 55% chance of war with China by 2050 is honestly surprising. I mean, I perceive China as an extremely rational entity ( that is for a government, governments overall are far less rational than people) and I would assume that the disadvantages of waging a war, should be obvious enough that no rational agent would try it.
But then again, this was the assumption with Russia, that Putin would be rational, see that he does not benefit from starting a war, and not do it. And here we are now.
Re "I’m kind of confused what’s going on here", I would guess that while there's a lot invested, there's not many limit orders on the 'no invasion' side (because China might invade Taiwan any day, or there might be news that it's suddenly much more likely to do so, and then the probability spikes for real so your limit order is filled but it's now a bad investment). You don't want to leave open an offer on 'no invasion', but you'd happily buy 'no' if you see that the price is way too high. Thus one buy on 'yes' can change the probability a lot.
Does anyone else enjoy predicting what would happen in fictional worlds is the laws of narrative and plot were suddenly suspended? Like Palpatine says "execute order 66" and then suddenly Star Wars stops being a narrative universe and just proceeds in a way that is dictated by its own internal laws, but no tendency towards narrative arc- what happens?
The difficulty is that in some fantasy universes, the narrativity is written into the plot. In Star Wars, very explicitly and in Dune, implicitly, what happens is the outworking of a superhuman intellectual agency.
In 2016 on PredictIt, I saw one case, which seems like undeniable insider trading. I was on the RealClearPolitics poll market, and someone was buying large amounts of a 2c YES stock, 10 minutes minutes before closing (around midnight EST). A few minutes later, a new poll was added to RealClearPolitics, causing the correct answer to flip. Even with the $850 market cap, buying in at 2-5c, would mean $20k+ in winnings.
> [A]s far as I can tell this is only about who makes it to the finalist round, and then to some degree the usual criteria will start creeping in again.
Yeah, that's what it says.
I've been doing some of this prediction market stuff and have been pretty good at it. (I only predict things I have domain knowledge about. But I'm told that's not an issue.) And this sort of thing keeps me out. It keeps me out of a fair number of things where I know there's going to be the same old standard filter applied after all the fancy new ways of sorting. Layering an additional test on top of the standard one only selects for people already in the circle. So if I'm not in I expect I can beat them all and still lose on points to the Harvard PhD.
Another way to think of it: Imagine a normal fellowship takes 100 applications and selects 1 scholar based on normal academic criteria. They disqualify 60 of them and I'm in the 60 because I don't have a PhD. But they chose one of the 40. I won't apply. Imagine this fellowship takes 100 applications and uses prediction markets to narrow it down to 5 scholars. And then selects based on normal academic criteria. Even if I'm 100% confident I can get into the top five, even I'm 100% confident I will be number 1, my expectation is that I get into the top five and then don't get the scholarship. So my expectation is that both of these contests end with basically the same result. The only thing that shifts is possibly which of the 40 who would make it through the original filter would win. The only way it won't is if all 5 winners are not people otherwise qualified which is unlikely. That's something but not enough for me to feel like I'm actually in the running.
The net filter is at least somewhat equivalent to asking people for resumes, selecting the qualified ones and excluding everyone else, and then having them do a prediction market to decide who wins.
This is without getting into the tournament criticisms here which I broadly agree with:
If the MATH dataset is open source, is there no guarantee people aren't just overfitting the test set to meaninglessly pad their accuracy in ways that won't generalize out of sample? To run a real ML competition you would need a secret test set that the participants never get access to.
People reading this might come away with the impression that PredictIt was still mostly in compliance with the no action letter, since Scott chose to only highlight three of the nine requirements in the no action letter. Just to be clear, and as I suspect Scott is aware, PredictIt was in substantial violation of a number of the other requirements in the no action letter, such as the requirement to keep the subject matter of the markets to a narrow range of subjects and limits on overall revenue generation.
I am surprised about the question regarding pop songs being composed by AI; that is, I was assuming that they were already mostly composed by AI. Was I wrong ? Are there really a bunch of human composers still toiling in the pop mines -- or are humans just sitting at the top of the loop, clicking "next step" when they find a generated song that they like ?
"This seems underpriced, unless people expect whatever group manages the list to disqualify AI songs, or some kind of civilization-wide agreement that they don’t count. There’s already mediocre AI-produced music, and nothing stopping it from getting better with really simple scaling. Given that most songs are already composed by nameless corporate composers and not the pop stars themselves, I don’t think there will be any resistance to this."
Uh....
Pop music isn't what's best, it's what's pushed. It has to be good, but more importantly it has to be pushed, and it has to have the kind of cachet to be pushed.
So this is essentially a bet that the cool crowd of cool kids will decide it's cool to push computers as artists instead of exclusively pushing other (relatively talented and convincing and possibly interestingly insane) cool kids. Will "this was made by a computer, oooooh" be competitive with staring into the absurdly compelling eyes of the likes of Taylor Swift and Billie Eilish and Adam Levine?
You know I'm down on AI, but 43% seems perfectly priced to me.
>I’m kind of confused what’s going on here. Does someone keep throwing in money to push it up to 70-90%, and then other people keep buying it back down?
If this was for real money I'd suspect insider trading. After all, well before an invasion happened PRC generals and probably US/Taiwanese intelligence would know it was going to happen; you have to plan a D-Day-scale event well in advance. And there's a good amphibious window in a couple of months, so it's not incredibly implausible that such a plan is in motion.
...but it's not real money, so insider trading doesn't seem very sensible.
Has anybody ever figured out how to get thinking going about not widely expected events?
For example, in late August 2015, Angela Merkel opened the borders to a million Muslim refugees, an eye-opening event that helped lead to Brexit and Trump. Were any prediction markets tracking the chances of something like that before it happened, or did it not occur to anybody to ask about it?
Can someone explain how the bets with really long time horizons work?
"Will there be war with China before 2050" won't be resolved for 28 years for example, any money placed on that bet would have an opportunity cost something like 28 years of compound returns at say ~10%, about 14x. You'd need an absolutely huge edge to make that bet worthwhile.
I’m not sure I understand Nostalgebraist’s criticism you quoted. Won’t a “real money” prediction market just have the same biases as the play money ones except now the ability to be an influential player is limited to those who have sufficient money?
I guess the argument I expect here would be that the folks with money would be motivated to discover and fund superforecasters who may not otherwise be able to enter the market at a meaningful level. I’m not sure I totally buy that, since the place to discover a superforecaster would be a play-money market for nerds. If that already exists, then the only purpose of the real money market is to make investors money on a replica of what is already happening in the play-money market, and the quality of the predictions doesn’t improve.
I think foxes are more accurate than the hedgehogs in their ecosystem, but a functioning ecosystem needs a bunch of hedgehogs. Hedgehogs are the ones that generate the weird obsessions that foxes can then average out. But a community that only consists of foxes will just get bogged down in groupthink and conventional wisdom because they don’t have anyone pushing their Overton windows.
A lot of people assume foxes are the ones that *break out* of groupthink, but they do this by synthesizing the competing ideas of other groups. It takes a hedgehog to have the obsession with one weird idea to push that makes them see some possibilities, while the fox needs an overview of several hedgehog communities to get the estimates on these possibilities better than any individual community.
PredictIt and Polymarket are out of alignment on the 2024 GOP nomination market because PredictIt is shutting down in February 2023. Everyone now has either pulled their money, or are just gambling on how PredictIt will resolve open markets when they are forced to shut them prematurely.
The problem is that Predictit was shit. Most people on it made money not by holding a bet to resolution, but through volatility. So they were not saying "is this risk priced correctly" but rather "will other people think this risk is priced correctly" or "will future events likely change this price" - which seem similar but are not the same thing. And there was massive amounts of collusion and market manipulation.
Furthermore, the whole idea of prediction markets is generally misguided. The thought is that they serve to aggregate (and perhaps process) information. But if there is no information going in, then there is nothing to process. When there is no clear relationship between informational inputs and desired outputs then all you are measuring is bias and noise.
The best example of this was the Predicit cabinet post markets, which had terrible records at predicting the correct nominee (reasonably far in advance - not two minutes after the nominee's name was leaked to twitter). Predicit would add names as those names were floated in various Politico, WaPo, or NYtimes articles. Grifters would assemble positions and then coordinate rumor campaigns in an attempt to move the price. Some rando nobody expected would be nominated.
The clearest examples of Predictit violating the terms were the Trump tweets markets. How many times the president tweets is not a political event.
On a similar vein, an arbitrary average of the generic ballot polling is also not a political event.
Another pretty clear violation are multiple markets on the same topic (which party will win, who will win, will a specific candidate win).
The obvious solution is to fine Predictit, and then create specific rules about which markets are allowed. All markets outside of that purview are subject to review.
> Also, I wouldn’t have naively expected that, with $34,455 already invested, an extra $100 would almost double the chance, from 14% to 24%. Probably I should poke around at their algorithm some more until this makes sense to me.
This confusion tripped me up as well. Here's the explanation, insofar as I understand it: If you click on the three horizontal dots next to the M$ bet amount, you can open the "market info" panel, where you'll see that while the trade volume on the market is now M$39k (which makes sense given that 856 bets were made), the liquidity pool is only M$666. In other words, lots of people have participated in the market and then sold back their shares (or something), so the actual liquidity is small and any single transaction, buy or sell, can move the market by a lot.
In fact, this exact confusion is why I eventually stopped participating on the site. I'd put M$500 into the ">=50k Chinese Covid deaths by end of 2022" market (https://manifold.markets/MetaculusBot/will-cumulative-reported-deaths-fro), then eventually realized I'd made a bad bet and wanted to cut my losses, only to discover that my shares that I'd bought for M$500 at 70-80% odds were now only worth M$24 if I sold them all, even though the current market probability was still 21%.
Why? Because despite a market volume of M$4.9k, the liquidity pool is only M$105, so if I sell the shares I bought for M$500 I can only get a fraction of that M$105 (or something). Or maybe it would be more accurate to say that the problem is that the liquidity pool for this market is both small and apparently contains ~210 Yes shares vs. only ~26 No shares, so selling Yes shares is worth very little? Hell if I know.
So that's when I stopped using the site. I'd thought prediction markets were a straightforward way to profit from having better information/calibration. But then I learned that if you want to profit, this is ultimately a market, and so understanding the fine print and market mechanics (which I don't care about at all) matters at least as much, and probably much more, than having accurate beliefs.
It's particularly upsetting that the site shows you relatively meaningless information like trade volume, while hiding the important information (like liquidity) in an unnoticeable pop-up.
Calling Kalshi the monopoly is kind of rich. PredictIt absolutely had monopolistic pricing and a weird no-action letter no one else could compete with. Kalshi has a first mover advantage but not a monopoly. Anyone can — and will — go through actual regulatory channels to make competitive markets. Markets are a pretty slam dunk case for regulatory rule setting and oversight as well.
Also VPNs are cute, but the issue with non-US market use for US citizens *I*s *R*eally *S*omething else entirely….
This doesn't work as well as you think - financial markets tend to break when participants are optimizing for the probability of finishing in the top N, especially when N is small. If I were trying to maximize my probability of winning this tournament in a blended pool of rational and irrational agents, I would be:
1) taking way more risk than I would if my goal were simply to maximize my portfolio's returns
2) taking risks that give me different risk exposures from other participants, even if the arithmetic expectation isn't in my favor.
For example, if the true probability of an event happening is 90%, and various market participants bid it up from 50% to 88%, well, I am probably more likely to sell it than buy it at those prices. The intuition is that if the event resolves YES, I will have bought it at the worst price among all the people who bought YES, but if the event resolves NO then I will have sold it at the best price among the people who sold YES.
Requiring people to trade a variety of markets also doesn't do much - it's pretty easy for people to buy YES at 99% on a variety of already-basically-resolved markets to meet this requirement, especially when those markets will formally resolve quickly.
All in all I expect that this competition tests the skill of "how good are you at gaming formal systems" a lot more than it tests the skill of "how good are you at making well calibrated predictions about the world".
Source: I was a quantitative trader for 2 years and they made us play a bunch of games market trading games with solutions like these during training.
> I love Manifold as a company but I’m sometimes skeptical of what I see there - there are too many really low-liquidity markets, too many sources of extra play money, too many stupid “the probability on this market will be under 49% on August 31” troll questions, too little desperation keeping people on their toes. The Salem subtournament solves all those problems
I agree that most of those are problems. But it's worth calling out this one:
> too many stupid “the probability on this market will be under 49% on August 31” troll questions
Questions in that genre aren't problems. They're valuable sources of empirical insight into the general area of "Keynesian beauty contests", which are of considerable interest.
"You know a prediction market has really made it when its insane questions about Donald Trump start getting close to zero. Insight has almost made it."
Scott, as one of the internet's premier prediction market evangelists, you should be better than this. Even the market "This market resolves to true at the end of 2022" wouldn't get much lower than 2%. due to the risk-free rate.
PredictIt's New Zealand arm was already shut down years ago when some local anti-money laundering legislation was passed. I've disliked anti-money laundering laws as a concept ever since.
Hey Scott, I've been following and reading your blog for sometime, and I’m fairly interested in prediction markets as well! I am hopeful that regulatory issues on the US are solved and that the projects you mentioned can grow, not just those that work with fake money.
Have you heard about Polkamarkets? I don't recall you ever mentioning them. They are decentralized and autonomous, but the feature I like the most is that any person can create prediction markets on real-life events.
You can also make forecasts, and resolve markets. They’re built on multichain EVM, Moonriver and Moonbeam networks atm.
Not a very big community, they lack liquidity and volume atm but their app has an incredibly ux/ui, it’s very functional tbh. They’re using DeFI and NFTs in an interesting and entertaining way too
Max Martin ( https://en.wikipedia.org/wiki/Max_Martin ) isn't a household name the way the performers he wrote and produced music for are, but the only songwriters that have had more #1 Billboard hits than he did were Paul McCartney and John Lennon, who are kind of a special case...
You should use "it" as a pronoun for organizations rather than "they". Otherwise there's an inconsistency in verb conjugation: "CFTC is" but "they are". I know it's currently in vogue to refer to organizations in plural but it's grammatically incorrect and leads to such jarring inconsistencies.
Actually Predicit’s violations are more serious than I realized. The initially terms are for “contracts consisting of two sub markets regarding binary election outcomes and economic indicators”
> I think Metaculus’ defense would be that their aggregation method lets them weight the guesses of people with good track records more highly
I should have been clearer about this in my post, but I was mostly taking aim at what Metaculus calls the "Community Prediction" -- which is just a (recency-weighted) average of all the predictions, like a literal poll. (As opposed to the "Metaculus Prediction," which is weighted based on track record.)
The reason I focused so much on the Community Prediction is that, when people talk about Metaculus forecasts, they're almost always talking about the Community Prediction. For example, everyone talking about moves in Metaculus graphs is talking about the Community Prediction, since no one can even see the Metaculus Prediction until a question has closed (unless they spend "Tachyons" to get a sneak preview for a particular question).
I agree that doing track-record-weighted aggregation would resolve some of my concerns. But if you ask the question "does Metaculus do track-record-weighted aggregation?", the answer is almost always "not for the numbers we're discussing, no."
A couple of weeks ago I ran a betting market predictions game night for the NYC rationalist weekly meetup (all bets should be settleable by the end of the night and the winner is whoever has the most net points). It is (a) a lot of fun and (b) incredibly chaotic, would strongly recommend more people do it. I wrote a post about how I set it up technically and what I'd do to run it smoother here https://shakeddown.wordpress.com/2022/08/04/what-i-learned-about-running-a-betting-market-game-night-contest/
You get some incredibly weird phenomena - I had one question that was "which question will have the most bet on it", which led to two other questions getting a ton of mana sunk into them as the people who'd bet on them tried to make their questions win out. One of these questions was "which answer will get the most points" (the other was "how many light years away is alpha centauri"), which was won by one guy suddenly sinking all his points into the runner-up just before it closed (and since it had a lot of points invested, he ended up winning the whole game by like a 2-1 margin over the runner-up thanks to this. This was also the guy whose day job is a Jane Street trader).
(Also several other fun attempted manipulations. It really is a great creativity generator).
The 55% chance of war with China by 2050 is honestly surprising. I mean, I perceive China as an extremely rational entity ( that is for a government, governments overall are far less rational than people) and I would assume that the disadvantages of waging a war, should be obvious enough that no rational agent would try it.
But then again, this was the assumption with Russia, that Putin would be rational, see that he does not benefit from starting a war, and not do it. And here we are now.
Vicariously Predicting Noncompliance
The 10th Avatar of Vishnu is Kalki, not Kalshi.
M$1 on Manifold is one cent USD, so when you say there was $4000 bet on the AI company data breach question, that's actually USD $40.
Re "I’m kind of confused what’s going on here", I would guess that while there's a lot invested, there's not many limit orders on the 'no invasion' side (because China might invade Taiwan any day, or there might be news that it's suddenly much more likely to do so, and then the probability spikes for real so your limit order is filled but it's now a bad investment). You don't want to leave open an offer on 'no invasion', but you'd happily buy 'no' if you see that the price is way too high. Thus one buy on 'yes' can change the probability a lot.
Does anyone else enjoy predicting what would happen in fictional worlds is the laws of narrative and plot were suddenly suspended? Like Palpatine says "execute order 66" and then suddenly Star Wars stops being a narrative universe and just proceeds in a way that is dictated by its own internal laws, but no tendency towards narrative arc- what happens?
The difficulty is that in some fantasy universes, the narrativity is written into the plot. In Star Wars, very explicitly and in Dune, implicitly, what happens is the outworking of a superhuman intellectual agency.
Still, it's a fun exercise.
In 2016 on PredictIt, I saw one case, which seems like undeniable insider trading. I was on the RealClearPolitics poll market, and someone was buying large amounts of a 2c YES stock, 10 minutes minutes before closing (around midnight EST). A few minutes later, a new poll was added to RealClearPolitics, causing the correct answer to flip. Even with the $850 market cap, buying in at 2-5c, would mean $20k+ in winnings.
I can't speak to how commonly this occurred.
> [A]s far as I can tell this is only about who makes it to the finalist round, and then to some degree the usual criteria will start creeping in again.
Yeah, that's what it says.
I've been doing some of this prediction market stuff and have been pretty good at it. (I only predict things I have domain knowledge about. But I'm told that's not an issue.) And this sort of thing keeps me out. It keeps me out of a fair number of things where I know there's going to be the same old standard filter applied after all the fancy new ways of sorting. Layering an additional test on top of the standard one only selects for people already in the circle. So if I'm not in I expect I can beat them all and still lose on points to the Harvard PhD.
Another way to think of it: Imagine a normal fellowship takes 100 applications and selects 1 scholar based on normal academic criteria. They disqualify 60 of them and I'm in the 60 because I don't have a PhD. But they chose one of the 40. I won't apply. Imagine this fellowship takes 100 applications and uses prediction markets to narrow it down to 5 scholars. And then selects based on normal academic criteria. Even if I'm 100% confident I can get into the top five, even I'm 100% confident I will be number 1, my expectation is that I get into the top five and then don't get the scholarship. So my expectation is that both of these contests end with basically the same result. The only thing that shifts is possibly which of the 40 who would make it through the original filter would win. The only way it won't is if all 5 winners are not people otherwise qualified which is unlikely. That's something but not enough for me to feel like I'm actually in the running.
The net filter is at least somewhat equivalent to asking people for resumes, selecting the qualified ones and excluding everyone else, and then having them do a prediction market to decide who wins.
This is without getting into the tournament criticisms here which I broadly agree with:
https://www.cspicenter.com/p/introducing-the-salemcspi-forecasting/comment/8235817
PS: To be explicit I have no idea if I'd win. But my strong sense is that even if I did win I wouldn't get beyond being a finalist.
If the MATH dataset is open source, is there no guarantee people aren't just overfitting the test set to meaninglessly pad their accuracy in ways that won't generalize out of sample? To run a real ML competition you would need a secret test set that the participants never get access to.
> I still think the hedgehog was supposed to be the bad one.
Whatever you say Robotnik
People reading this might come away with the impression that PredictIt was still mostly in compliance with the no action letter, since Scott chose to only highlight three of the nine requirements in the no action letter. Just to be clear, and as I suspect Scott is aware, PredictIt was in substantial violation of a number of the other requirements in the no action letter, such as the requirement to keep the subject matter of the markets to a narrow range of subjects and limits on overall revenue generation.
I am surprised about the question regarding pop songs being composed by AI; that is, I was assuming that they were already mostly composed by AI. Was I wrong ? Are there really a bunch of human composers still toiling in the pop mines -- or are humans just sitting at the top of the loop, clicking "next step" when they find a generated song that they like ?
Are there any short term AI forecasting benchmarks I can make predictions on to see how well calibrated I am?
Why didn't predictit:
1. spend some of their zillions of fees on hiring former CFTC people and lobbying the CFTC to let them continue existing?
2. Make a big public fuss about the CFTC's corruption?
3. Move their operations outside the US and stop accepting US customers due to their Vexing Political Nannystate?
I am confused why they didn't do any of this, unless they are planning to relaunch the site under a different legal framework.
I don't think that trading has stoped on PredictIt.
Netpick, but UTA is the University of Texas Arlington. UT Austin is just UT.
"This seems underpriced, unless people expect whatever group manages the list to disqualify AI songs, or some kind of civilization-wide agreement that they don’t count. There’s already mediocre AI-produced music, and nothing stopping it from getting better with really simple scaling. Given that most songs are already composed by nameless corporate composers and not the pop stars themselves, I don’t think there will be any resistance to this."
Uh....
Pop music isn't what's best, it's what's pushed. It has to be good, but more importantly it has to be pushed, and it has to have the kind of cachet to be pushed.
So this is essentially a bet that the cool crowd of cool kids will decide it's cool to push computers as artists instead of exclusively pushing other (relatively talented and convincing and possibly interestingly insane) cool kids. Will "this was made by a computer, oooooh" be competitive with staring into the absurdly compelling eyes of the likes of Taylor Swift and Billie Eilish and Adam Levine?
You know I'm down on AI, but 43% seems perfectly priced to me.
Minor quibble:
> Also, isn’t Kalshi the apocalypse-avatar of Vishnu who’s supposed to kill everybody and purify the world for the next era? Seems concerning.
That's _Kalki_
I think the Predictit 2024 gop primary market will probably never resolve which would ruin the arbitrage.
>I’m kind of confused what’s going on here. Does someone keep throwing in money to push it up to 70-90%, and then other people keep buying it back down?
If this was for real money I'd suspect insider trading. After all, well before an invasion happened PRC generals and probably US/Taiwanese intelligence would know it was going to happen; you have to plan a D-Day-scale event well in advance. And there's a good amphibious window in a couple of months, so it's not incredibly implausible that such a plan is in motion.
...but it's not real money, so insider trading doesn't seem very sensible.
Has anybody ever figured out how to get thinking going about not widely expected events?
For example, in late August 2015, Angela Merkel opened the borders to a million Muslim refugees, an eye-opening event that helped lead to Brexit and Trump. Were any prediction markets tracking the chances of something like that before it happened, or did it not occur to anybody to ask about it?
Can someone explain how the bets with really long time horizons work?
"Will there be war with China before 2050" won't be resolved for 28 years for example, any money placed on that bet would have an opportunity cost something like 28 years of compound returns at say ~10%, about 14x. You'd need an absolutely huge edge to make that bet worthwhile.
More evidence that governments are organized crime rings with better PR.
I’m not sure I understand Nostalgebraist’s criticism you quoted. Won’t a “real money” prediction market just have the same biases as the play money ones except now the ability to be an influential player is limited to those who have sufficient money?
I guess the argument I expect here would be that the folks with money would be motivated to discover and fund superforecasters who may not otherwise be able to enter the market at a meaningful level. I’m not sure I totally buy that, since the place to discover a superforecaster would be a play-money market for nerds. If that already exists, then the only purpose of the real money market is to make investors money on a replica of what is already happening in the play-money market, and the quality of the predictions doesn’t improve.
On hedgehogs and “good” or “bad”:
I think foxes are more accurate than the hedgehogs in their ecosystem, but a functioning ecosystem needs a bunch of hedgehogs. Hedgehogs are the ones that generate the weird obsessions that foxes can then average out. But a community that only consists of foxes will just get bogged down in groupthink and conventional wisdom because they don’t have anyone pushing their Overton windows.
A lot of people assume foxes are the ones that *break out* of groupthink, but they do this by synthesizing the competing ideas of other groups. It takes a hedgehog to have the obsession with one weird idea to push that makes them see some possibilities, while the fox needs an overview of several hedgehog communities to get the estimates on these possibilities better than any individual community.
PredictIt and Polymarket are out of alignment on the 2024 GOP nomination market because PredictIt is shutting down in February 2023. Everyone now has either pulled their money, or are just gambling on how PredictIt will resolve open markets when they are forced to shut them prematurely.
The problem is that Predictit was shit. Most people on it made money not by holding a bet to resolution, but through volatility. So they were not saying "is this risk priced correctly" but rather "will other people think this risk is priced correctly" or "will future events likely change this price" - which seem similar but are not the same thing. And there was massive amounts of collusion and market manipulation.
Furthermore, the whole idea of prediction markets is generally misguided. The thought is that they serve to aggregate (and perhaps process) information. But if there is no information going in, then there is nothing to process. When there is no clear relationship between informational inputs and desired outputs then all you are measuring is bias and noise.
The best example of this was the Predicit cabinet post markets, which had terrible records at predicting the correct nominee (reasonably far in advance - not two minutes after the nominee's name was leaked to twitter). Predicit would add names as those names were floated in various Politico, WaPo, or NYtimes articles. Grifters would assemble positions and then coordinate rumor campaigns in an attempt to move the price. Some rando nobody expected would be nominated.
The clearest examples of Predictit violating the terms were the Trump tweets markets. How many times the president tweets is not a political event.
On a similar vein, an arbitrary average of the generic ballot polling is also not a political event.
Another pretty clear violation are multiple markets on the same topic (which party will win, who will win, will a specific candidate win).
The obvious solution is to fine Predictit, and then create specific rules about which markets are allowed. All markets outside of that purview are subject to review.
> Also, I wouldn’t have naively expected that, with $34,455 already invested, an extra $100 would almost double the chance, from 14% to 24%. Probably I should poke around at their algorithm some more until this makes sense to me.
This confusion tripped me up as well. Here's the explanation, insofar as I understand it: If you click on the three horizontal dots next to the M$ bet amount, you can open the "market info" panel, where you'll see that while the trade volume on the market is now M$39k (which makes sense given that 856 bets were made), the liquidity pool is only M$666. In other words, lots of people have participated in the market and then sold back their shares (or something), so the actual liquidity is small and any single transaction, buy or sell, can move the market by a lot.
In fact, this exact confusion is why I eventually stopped participating on the site. I'd put M$500 into the ">=50k Chinese Covid deaths by end of 2022" market (https://manifold.markets/MetaculusBot/will-cumulative-reported-deaths-fro), then eventually realized I'd made a bad bet and wanted to cut my losses, only to discover that my shares that I'd bought for M$500 at 70-80% odds were now only worth M$24 if I sold them all, even though the current market probability was still 21%.
Why? Because despite a market volume of M$4.9k, the liquidity pool is only M$105, so if I sell the shares I bought for M$500 I can only get a fraction of that M$105 (or something). Or maybe it would be more accurate to say that the problem is that the liquidity pool for this market is both small and apparently contains ~210 Yes shares vs. only ~26 No shares, so selling Yes shares is worth very little? Hell if I know.
So that's when I stopped using the site. I'd thought prediction markets were a straightforward way to profit from having better information/calibration. But then I learned that if you want to profit, this is ultimately a market, and so understanding the fine print and market mechanics (which I don't care about at all) matters at least as much, and probably much more, than having accurate beliefs.
It's particularly upsetting that the site shows you relatively meaningless information like trade volume, while hiding the important information (like liquidity) in an unnoticeable pop-up.
Calling Kalshi the monopoly is kind of rich. PredictIt absolutely had monopolistic pricing and a weird no-action letter no one else could compete with. Kalshi has a first mover advantage but not a monopoly. Anyone can — and will — go through actual regulatory channels to make competitive markets. Markets are a pretty slam dunk case for regulatory rule setting and oversight as well.
Also VPNs are cute, but the issue with non-US market use for US citizens *I*s *R*eally *S*omething else entirely….
Someone should lobby to transfer jurisdiction over prediction markets to a new regulatory agency.
I think the CSPI competition wasn't setup correctly for reasons well described in this comment from their announcement: https://www.cspicenter.com/p/introducing-the-salemcspi-forecasting/comment/8235817
///
This doesn't work as well as you think - financial markets tend to break when participants are optimizing for the probability of finishing in the top N, especially when N is small. If I were trying to maximize my probability of winning this tournament in a blended pool of rational and irrational agents, I would be:
1) taking way more risk than I would if my goal were simply to maximize my portfolio's returns
2) taking risks that give me different risk exposures from other participants, even if the arithmetic expectation isn't in my favor.
For example, if the true probability of an event happening is 90%, and various market participants bid it up from 50% to 88%, well, I am probably more likely to sell it than buy it at those prices. The intuition is that if the event resolves YES, I will have bought it at the worst price among all the people who bought YES, but if the event resolves NO then I will have sold it at the best price among the people who sold YES.
Requiring people to trade a variety of markets also doesn't do much - it's pretty easy for people to buy YES at 99% on a variety of already-basically-resolved markets to meet this requirement, especially when those markets will formally resolve quickly.
All in all I expect that this competition tests the skill of "how good are you at gaming formal systems" a lot more than it tests the skill of "how good are you at making well calibrated predictions about the world".
Source: I was a quantitative trader for 2 years and they made us play a bunch of games market trading games with solutions like these during training.
///
> I love Manifold as a company but I’m sometimes skeptical of what I see there - there are too many really low-liquidity markets, too many sources of extra play money, too many stupid “the probability on this market will be under 49% on August 31” troll questions, too little desperation keeping people on their toes. The Salem subtournament solves all those problems
I agree that most of those are problems. But it's worth calling out this one:
> too many stupid “the probability on this market will be under 49% on August 31” troll questions
Questions in that genre aren't problems. They're valuable sources of empirical insight into the general area of "Keynesian beauty contests", which are of considerable interest.
"You know a prediction market has really made it when its insane questions about Donald Trump start getting close to zero. Insight has almost made it."
Scott, as one of the internet's premier prediction market evangelists, you should be better than this. Even the market "This market resolves to true at the end of 2022" wouldn't get much lower than 2%. due to the risk-free rate.
This was all new to me, but I found it extremely informative and fascinating. Love your subliminal suggestions on how to use the foreign exchanges!
PredictIt's New Zealand arm was already shut down years ago when some local anti-money laundering legislation was passed. I've disliked anti-money laundering laws as a concept ever since.
Which law or laws allow Americans to gamble at a casino in e.g. Macao but not, while in e.g. Macao, bet real money on a non-US based website?
Hey Scott, I've been following and reading your blog for sometime, and I’m fairly interested in prediction markets as well! I am hopeful that regulatory issues on the US are solved and that the projects you mentioned can grow, not just those that work with fake money.
Have you heard about Polkamarkets? I don't recall you ever mentioning them. They are decentralized and autonomous, but the feature I like the most is that any person can create prediction markets on real-life events.
You can also make forecasts, and resolve markets. They’re built on multichain EVM, Moonriver and Moonbeam networks atm.
Not a very big community, they lack liquidity and volume atm but their app has an incredibly ux/ui, it’s very functional tbh. They’re using DeFI and NFTs in an interesting and entertaining way too
"Nameless corporate composers"?
Max Martin ( https://en.wikipedia.org/wiki/Max_Martin ) isn't a household name the way the performers he wrote and produced music for are, but the only songwriters that have had more #1 Billboard hits than he did were Paul McCartney and John Lennon, who are kind of a special case...
The 10th avatar of Vishnu is Kalki, not Kalshi: https://en.wikipedia.org/wiki/Kalki. Kalshi is an Arabic word meaning everything: https://www.forbes.com/sites/frederickdaso/2019/04/03/kalshi-an-mit-betting-startup-is-allowing-people-to-bet-on-anything/
You should use "it" as a pronoun for organizations rather than "they". Otherwise there's an inconsistency in verb conjugation: "CFTC is" but "they are". I know it's currently in vogue to refer to organizations in plural but it's grammatically incorrect and leads to such jarring inconsistencies.
Actually Predicit’s violations are more serious than I realized. The initially terms are for “contracts consisting of two sub markets regarding binary election outcomes and economic indicators”
Most of predictit’s contracts do not comply.
Smarkets.com is another non-US option.
> I think Metaculus’ defense would be that their aggregation method lets them weight the guesses of people with good track records more highly
I should have been clearer about this in my post, but I was mostly taking aim at what Metaculus calls the "Community Prediction" -- which is just a (recency-weighted) average of all the predictions, like a literal poll. (As opposed to the "Metaculus Prediction," which is weighted based on track record.)
The reason I focused so much on the Community Prediction is that, when people talk about Metaculus forecasts, they're almost always talking about the Community Prediction. For example, everyone talking about moves in Metaculus graphs is talking about the Community Prediction, since no one can even see the Metaculus Prediction until a question has closed (unless they spend "Tachyons" to get a sneak preview for a particular question).
I agree that doing track-record-weighted aggregation would resolve some of my concerns. But if you ask the question "does Metaculus do track-record-weighted aggregation?", the answer is almost always "not for the numbers we're discussing, no."