154 Comments

One more: "The replacement administrator for Astral Codex Ten Discord identifies as female" is finally resolved. It was my honour to participate in zeroth and final stages of the selection conclave!

https://manifold.markets/Honourary/the-replacement-administrator-for-a

Expand full comment
founding

> This is crazy and over-optimistic, right?

Hahaha thankfully, nobody's ever asked to see my brier score before deciding whether I'm worthy of starting a startup. (spoiler alert, it's not that great)

I often think about this piece by Duncan Sabien: https://medium.com/@ThingMaker/reliability-prophets-and-kings-64aa0488d620. Essentially, if you make a statement about the future, there are two ways the statement could come true. Either you can be a _prophet_ aka superforecaster who is really good at modeling the universe; or you can be a _king_ aka highly agentic person who is good at making things happen.

I identify much more strongly with the latter, and I imagine most founders do as well~

Expand full comment

Small correction:

“Consensus” is the crypto conference. “ConsenSys” is a crypto company, working on Ethereum projects and led by Joe Lubin.

They’re unrelated, except that ConsenSys employees attend and speak at Consensus.

I’m curious how ConsenSys got on your radar to cause this slip.

Expand full comment

The Manifold valuation question might be high because as currently worded, it's impossible to win by betting against it.

Also I wonder how many of the AI tasks will suddenly stop counting as AGI once they're achieved.

Expand full comment

500 million recorded cases of covid sounds low to me. Of course there may be twice (or more) as many unrecorded.

The 2% are crazy, yes. Omicron had an r value close to Measles and monkey pox is much harder to transmit.

Lamda is impressive, arguably it passes the Turing test, although in a slightly uncanny valley style. The analysis of the broken mirror was great, the story about the owl facing down a Lion was the lamest story ever.

That said, I’ve never believed that passing the Turing test proves intelligence. It’s a necessary but not sufficient condition.

Expand full comment

I'd note that humans aren't capable of consistently writing 10k lines of bug-free code from natural-language specifications. Certainly not without testing.

Expand full comment

Am I crazy, or are the Musk Vs. Marcus decision criteria insane? Very few people could achieve all five, and I posit still less than half could do even three. Further, "work as a cook in a kitchen" seems wrong: that feels very similar to self-driving AI, and few people would accept self-driving as an indicator of AGI.

I would start with asking:

* What criteria would the vast majority of people meet, that current AI does not?

* What are some examples of interesting novel ideas, and what are ways we can prompt humans to provide some?

* What sort of human behaviors rely on a world model? How could we ask an AI to demonstrate those behaviors? ( I do think the novel / movie criteria fit this)

* How do humans generally describe qualia? How can we prompt an AI to describe it's own qualia in a convincing way? (the way a machine feels should be necessarily different from how a human does)

Expand full comment

Who on earth thinks that non-native-born-US-citizen Elon Musk will be the 2024 Republican presidential nominee?

Expand full comment

I don't think that LAMBDA did a good job with Les Miserables. The prompt asks about the book. LAMBDA's response is about the musical.

LAMBDA: "Fantine is being mistreated by her supervisor at the factory and yet doesn’t have anywhere to go, either to another job, or to someone who can help her. That shows the injustice of her suffering. ... She is trapped in her circumstances and has no possible way to get out of them, without risking everything."

This is a weird notion of justice. Justice is supposed to be impartial, but LAMBDA is concerned that her supervisor didn't take her particular circumstances into account. But maybe that's the book's notion of justice. Let's see what it says:

Les Miserables: "Fantine had been at the factory for more than a year, when, one morning, the superintendent of the workroom handed her fifty francs from the mayor, told her that she was no longer employed in the shop, and requested her, in the mayor’s name, to leave the neighborhood. This was the very month when the Thénardiers, after having demanded twelve francs instead of six, had just exacted fifteen francs instead of twelve. Fantine was overwhelmed. She could not leave the neighborhood; she was in debt for her rent and furniture. Fifty francs was not sufficient to cancel this debt. She stammered a few supplicating words. The superintendent ordered her to leave the shop on the instant. Besides, Fantine was only a moderately good workwoman. Overcome with shame, even more than with despair, she quitted the shop, and returned to her room. So her fault was now known to every one. She no longer felt strong enough to say a word. She was advised to see the mayor; she did not dare. The mayor had given her fifty francs because he was good, and had dismissed her because he was just. She bowed before the decision. ... But M. Madeleine had heard nothing of all this. Life is full of just such combinations of events. M. Madeleine was in the habit of almost never entering the women’s workroom. At the head of this room he had placed an elderly spinster, whom the priest had provided for him, and he had full confidence in this superintendent, - a truly respectable person, firm, equitable, upright, full of the charity which consists in giving, but not having in the same degree that charity which consists in understanding and in forgiving. M. Madeleine relied wholly on her. The best men are often obliged to delegate their authority. It was with this full power, and the conviction that she was doing right, that the superintendent had instituted the suit, judged, condemned, and executed Fantine."

The musical has a male superintendent who sexually harasses her and then dismisses her cruelly. The book has a female superintendent who dismisses her with severance pay. The book explicitly says that Fantine considered the decision to be just.

This is one instance of the musical completely rewriting the central theme of Les Miserables. The musical is a call for liberty for people who are unjustly suffering. The book is a call for compassion for people who are justly suffering. The theme isn't justice and injustice. It's justice and mercy.

It's not surprising that a text predictor would talk about the musical. A lot more people have seen the musical than have read the book. The training set probably even includes people who claim to be talking about the book, but have only seen the musical. LAMBDA has read the book, but clearly has not understood it.

Expand full comment

I made some markets on Manifold for predicting the plot of Stranger Things S4 volume 2 (comes out on July 1), here is one for who will die first https://manifold.markets/mcdog/stranger-things-s4-who-will-die-fir . I personally think it's the most fun use of prediction markets this month, but so far there hasn't been a lot of use, so I guess come and have the fun with me

Expand full comment

I'm fairly certain Elon Musk doesn't qualify as a US presidential nominee.

Expand full comment

> Does Metaculus say this because it’s true, or because there will always be a few crazy people entering very large numbers without modeling anything carefully? I’m not sure. How would you test that?

It probably has to be “collect 1000 examples of 1% likelihood Metaculus predictions and see how well calibrated they are”, right? (Or whatever N a competent statistician would pick to power the test appropriately).

Expand full comment

Caruso is a smart guy, successful high-end developer, and USC board member influencing some important fixes to university scandals. He’ll need a big part of the Hispanic vote to win, facing a black woman Democrat.

Expand full comment

About the prediction that 84% that Putin will remain the president of Russia that never changes.

There used to be a meme in Russian internet that if you search "84% of Russians" (in Russian), you'll get all kinds of survey results where 84% support Putin, trust the TV, believe in God, don't speak English, etc etc. Assumption being that 84% is a convenient number that the manufacturers of fake surveys like to put next to the "correct" answer. Right now, Google says that 84% of Russians "consider themselves happy" and (independently) "trust Russian army". This is not a coincidence, of course, as per the usual rule.

Expand full comment

Did anyone predict that Musk wouldn't end up buying twitter? What are the odds looking like now?

I asked about this in the hidden open thread and it's possible that no one predicted that the deal might not happen.

Expand full comment

"This is encouraging, but a 2% chance of >500 million cases (there have been about 500 million recorded COVID infections total) is still very bad. Does Metaculus say this because it’s true, or because there will always be a few crazy people entering very large numbers without modeling anything carefully? I’m not sure. How would you test that?"

One thing you could do is to pick a handful of the best Metaculus forecasters and pay(?) them to make careful forecasts on that question, with special attention to getting the tails right.

That would tell you a lot about whether these fat tails are from "a few crazy people entering very large numbers without modeling anything carefully", and it would provide some less definitive information about how seriously to take these tails forecasts & whether they're well-calibrated.

Expand full comment

When I lived in Los Angeles, Rick Caruso was definitely a known local figure. If you've spent much time in Los Angeles, he's the developer behind The Grove, and I believe The Americana in Glendale, which really set the tone as to what a "mall" is in post-2000 USA. As someone who hates malls, these spaces are actually totally fine as public spaces, and even have cutesy urbanist touches that people like. It's hard to predict how someone like him fares against a partisan political figure in a non-partisan election.

Expand full comment
Jun 14, 2022·edited Jun 14, 2022

>Well darn, even though this superficially changes nothing I think it prevents me from using this as an example of prediction markets being self-correcting to outside interference ever again.

Worse than not being self-correcting, the incentive to manipulate outcomes becomes greater the less likely that outcome was predicted since there is more money on the table when odds are long, which also means a manipulator has a motive not only to hide their actions but to actively deceive the other participants in the opposite direction.

Prediction markets, with their discrete, time-limited results, are much less like financial markets than they are like sports betting markets, which have always been susceptible to having results fixed by the bettors. Major professional sports are hard to fix today simply because players are rewarded so much for playing well gamblers can’t afford to pay them to play less-well. Modern-day fixing targets are usually the (closely observed) refs. Major sports also have career-ending penalties imposed against player/ref-manipulators, sanctions prediction markets lack.

The sad truth might be that heavy market regulations may be necessary to keep prediction markets useful, which may in turn make them impractical.

Expand full comment
Jun 14, 2022·edited Jun 14, 2022

It doesn't really make sense to bet against Marcus because in a world with AGI you won't have much use for the money.

Expand full comment

"Does Metaculus say this because it’s true, or because there will always be a few crazy people entering very large numbers without modeling anything carefully? I’m not sure. How would you test that?"

I actually think it has more to do with how distributions get entered into the system, and how finicky and relatively low-sensitivity the scoring is to these tails. (I'd be more excited for a binary question "will there be over 500k cases of monkeypox," which is not far out enough in the tails to end up as 99%, and would calibrate the other curve.

Expand full comment

> ...a 2% chance of >500 million cases...

I notice I am confused. It's quite possible I don't understand how this market works, but I wouldn't have thought it was structured in such a way that it would give you a *probability* for "over 500 million cases".

Do you really mean that a "true" probability of anything other than 2% would imply a violation of the efficient market hypothesis? i.e. that the market is set up such that, if 2% is the wrong probability for "over 500 million cases", and I know it's wrong, I can bet against that probability for that specific event, and make money in expectation, and correct the market in the process, even if I know *nothing else* about the probability distribution of cases?

Or do you actually mean "2% of the bets are on over 500 million cases"? Which I'm pretty confident is not the same thing. I believe that would be more like saying "2% of people answered 'yes' on our poll" than "the market cleared when the price of 'yes' was two cents".

Expand full comment

> Kiev-centric

#KievNotKyiv is pretty much accepted across the West right now.

Expand full comment

RE: The AGI Test. I can only do one thing, work as a cook in a kitchen, and I am a 10X software engineer. Bug-Free code has never been written before so this seems like a great goal, but not a true test of intelligence.

Expand full comment

Pretty cool! I would add this "Futuur" market, it's with several bets and with a good liquidity too:

"Who will be elected President of Brazil in 2022?"

https://futuur.com/q/137153/who-will-be-elected-president-of-brazil-in-2022

Expand full comment

So nobody has yet commented on the stained glass praying mantis? To me, it's the most beautiful mantic monday article picture to date.

Expand full comment

Re: "How would you test [crazy people entering large numbers]?"

Shouldn't this show up in the aggregate stats for outlier conditions? So what percentage of 1% predictions actually come true? It should be 1% if things are well calibrated, but the aforementioned crazy people should push that number down. The more predictions the market has, the more power you should have for smaller percentages. That will make you increasingly sensitive to smaller and smaller populations of crazies.

Expand full comment

(1) Wait, they're predicting monkeypox deaths? I thought it was supposed to be generally harmless? Or was that just "let's not panic the public about a new death plague, especially as Covid hasn't gone away yet" public health messaging?

"Is monkeypox deadly?

The Congo Basin variety of monkeypox can have death rates up to 10% of those infected. But the good news is that is not what we are dealing with. The current outbreak is caused by the West African variety, which is far less deadly (less than 1% fatality rate). No people with confirmed cases have died thus far.

We are in the early stages of understanding this outbreak. No doubt the situation will evolve and so will our understanding of how it is spreading and how to contain it."

(2) "Read a novel and answer complicated questions about eg the themes (existing language models can do this with pre-digested novels, eg LAMDA talking about Les Miserables here"

I was completely unimpressed with the LaMDA answer about Les Miserables as it read exactly like a model answer scooped out of SparkNotes or other student aids. Nothing to indicate the thing even knew what it was talking about.

(3) Wait, part deux: Joe Biden as nominee in 2024? He'll be 82 then, what was all that talk about Trump being too old and too unhealthy to run first time round? The "is he/isn't he" debates over cognitive decline/senility will only get worse. Trump will be 79 then, which is also pushing it a bit, but it would be the same age as Biden is now (more or less).

I admit, I'm torn about President Newsom. On the one hand, he'd be nothing more than a mannequin in office, and should you elect presidents based on "well he has great hair"? On the other hand, he'd be a mannequin in office, and with nothing more to occupy him than "must visit my coiffeur", how much harm could he do to the country?

Expand full comment

> “all” or “most” of the first AGI is based on deep learning

The "based" is doing a lot of work.

I could imagine a case where we just take current deep-learning and throw more compute at it and we get AGI and that would certainly qualify.

But what about a paradigm shift which uses deep learning optionally/optimally/necessarily as an underlying building block.

It would be like claiming modern medicine is based on the 4 humours model of medicine because we care a lot about blood these days. There are even hematologists who specialize in it!

Expand full comment

Metaculus has 68% on AGI existing by 2030 but only 2% on humans going extinct by 2100. That seems way too optimistic about AI alignment.

Expand full comment
founding

> Warcasting

Any form of deal, armistice or treaty that is not extremely close to the status quo of that moment requires trust. I don't think there's enough trust between those two parties to fill a sherry glass, let alone do complex land swaps. And you can't go and put neutral peacekeeping troops in there either, because there aren't any.

Expand full comment
Jun 15, 2022·edited Jun 15, 2022

Marcus' criteria for true AGI would rule out most humans. Then again, perhaps the threshold for acknowledging AGI should be higher. But still, that includes a good bit of robotics in there.

Expand full comment
Jun 16, 2022·edited Jun 16, 2022

>This is encouraging, but a 2% chance of >500 million cases (there have been about 500 million recorded COVID infections total) is still very bad. Does Metaculus say this because it’s true, or because there will always be a few crazy people entering very large numbers without modeling anything carefully?

It is probably due to nothing other than the way one inputs distributions in metaculus. It is actually quite hard/impossible to get a reasonably-shaped distribution that has most of the support where one wants it. Thus, for a lot of questions, the tails on the predicted distributions tend to be fatter than they should be.

Expand full comment

Regarding Scott's remarks on Metaculus — "or because there will always be a few crazy people entering very large numbers without modeling anything carefully?" — Considering that standard epidemiological models have failed to predict the course and timing of the six plus SARS2 waves, the crazy people explanation is probably the most likely.

Expand full comment

I think the right way (maybe) to interpret the presidentical prediction markets when you have one (to put it lightly) extremely well known politician vs a field of rivals is not "trump is winning!" but "67% chance the nominee is not trump." Obviously for him not to be the nominee someone else has to be, but at the moment the field is split between (relative) unknowns and obviously in the actual primary eventually all the not-trump probabilities will be reflected in a fewer number of politicians, potentially one.

Expand full comment