Doesn't the "new multiple choice questions" thing ruin that entry for statistical purposes? Is there a way to identify responses before/after the date they were added?
I'm too lazy to do anything with the data myself, I mean, just curious!
Thank you for sharing this! We might use as one of the example data sets for course I teach.
Really struck by the racial breakdown here — 87% white. This is a very different breakdown from (e.g.) a standard tech school, where White and Asian-American are usually at similar ratios.
Would be fun to compare (where possible) some of these statistics to the GSA. I think the GSA provides enough information to produce a sample that matches the ACX gender/race/age demographics.
It's difficult to understand the results of the questions based on a linear scale: it only shows the theme of the question as a title and then a bar graph showing the percentage of users who chose each option. For example: a graph titled "Global Warming" with five bars labeled 1, 2, 3, 4, and 5.
-Scott's audience is somewhat whiter, and *much* more male than would be expected by geographic demographics
-Functionally no transmen, more enbies than transwomen (Scott might want to put an option for "transmasc"/"transfem" next time?)
-Sexual orientation about as expected
-Way more married readers than I was expecting - I'm around the median (not modal) age, and I feel like very few of my peers are married
-Slightly less poly than expected? Though "no preference" basically translates to "poly by nature" as far as I'm concerned.
-~40% tech, only 1% psych. Scott is truly the techie's psych blog.
-Initially I was shocked at the number of people with only a high school diploma, before I realized that included all the younger college kids.
-The fact that 19% of respondents were "not done" with education and only ~13% of the total don't have a BA/BS means near majority of current students are postgrads.
-Virtue ethics had a surprisingly strong showing for such a consequentialist blog
-I can't tell if 75% is a low or high number of lurkers, since I would expect most lurkers to not respond at all, and lurker numbers are usually astonishingly high to me
-The new title of the "Political Spectrum" question makes it totally unclear what those numbers mean. IIRC it was Left < Center < Right ?
-Strong showing for Social Democrats; Liberals and Libertarians about as expected.
-Once again, with the questions omitted it's a little unclear what the numerical political opinions are measuring
-HBD is a bell curve lol
-Very middle class, statistically surprising number of upper-middle class, both in childhood and readership.
-I'm actually surprised at how few readers are depressed, especially compared to anxiety
-Less autistic than I expected - but I suppose I'm biased by selection effects in my personal life and it's still really high (and "Autism" is more specific/stigmatized than "on the spectrum")
-Ah, way more people have considered suicide than are currently depressed. "It gets better", indeed.
-Life satisfaction pretty high, surprised that romantic satisfaction is even higher (but I suppose I shouldn't be, considering the number of married individuals)
The main things I'm surprised of: lots of European-style progressivism (e.g. "Social democratic" on "Political Affiliation", "Not registered" in "American Parties", 1 on "Global Warming", so few "No" on "COVID vaccine", so few people commuting by car, so many people who don't even own one) and yet so many married people -- these don't usually exactly go hand in hand.
I am procrastinating my real work... a few other numbers:
1. filtering out extremes, people are bad at the distance Paris-Moscow; average is 4452 km (median guess is 3000). True distance is 2486 km. This is surprising to me; when I do wisdom of the crowd estimates people are often extremely good — within 10% on interesting questions for a group of > 100.
2. when you ask them to rethink it, they get worse (5231 km mean; median doesn't change).
3. again filtering out trolls, mean IQ (for the population answering both IQ and SAT) is 140 (median 140, too). This is mildly inconsistent with the reported SAT Math scores for the IQ-answering population (mean 725; median 750) which should be higher.
After seeing other people's guesses on the question about the distance between Paris and Moscow, I am feeling marginally better about my grasp of European geography and my ability to convert between miles and kilometers. I looked it up afterwards and I had overshot on my initial guess by about a thousand km, if I recall correctly.
Can't wait to see what the intention behind that pair of questions was! I'm guessing it was something to do with confidence in one's estimations.
Even though I knew it would be uncommon, for some reason I was surprised to see that the number of FTM transgender readers who took the survey is so low. Hello to the other 22 of us!
What the hell? Either this is normal (which is bad), this is normal for ACX types (which is less bad; do we grow up in unusually pedophilic communities), a fluke (which is an unlikely) or some lizardman's constant kind of deal. I'm forgetting other options, because, again, what the hell?
Seeds of Science, a journal publishing speculative and non-traditional scientific articles (and ACX grant winner), would like to offer itself as a peer-reviewed publishing platform for any analyses using the ACX reader survey data. Visit the website (theseedsofscience.org) to learn more or contact us at info@theseedsofscience.org.
13% of people not having an internal monologue is 10-100x higher than I would have expected. I thought the concept was a joke/insult about lacking self awareness or a (temporary) goal in meditation.
Would it be possible to add the aphantasia type questions again next year please to do crosstabs with the inner monologue question?
I assume that those are the two most common ways of thinking (for people with both senses) and am wondering if lacking one increased the chance you need the other.
I'm also curious about what percentage of people lacking an inner monologue also have aphantasia for audio. I personally don't think visually 99% of the time but am still capable of it when I want to. Smell/touch/taste can be remembered but I have no idea how I'd reason with them.
I'd also assume someone without an inner monologue would be lonelier when alone and might feel the need to be more extroverted as a result, so some more specific questions about people's social lives to tease that out would be helpful.
Final analysis for me tonight: I was struck by Scott's inclusion of a "suicide" question. At the risk of stating the obvious, please, if you are actively suicidal, contact someone. If you can't contact someone, go to the nearest emergency room. I lost a friend to sucide a few years ago; we had fallen out of touch and perhaps he felt he couldn't call me; I wish he had.
Here's a demographic breakdown from the Scott data. The modal respondant on the survey is white, cis-male, and heterosexual; the attempt rate for this "base" group is 4%. For this analysis I'll restrict to 30-and-under, so that we don't have too much age confounding.
Here are the relative risk ratios for other groups (e.g., x2 means a factor of two higher than the base group).
Trans: x5.6
Cis-female: x1.7
Non-white: x1.8
Non-white and non-male: x2.8
Non-straight: x3.2
I did not find strong effects for IQ. If we restrict to "I have attempted it in the past, and wish the attempt had succeeded", the results equalize a little. The rate for the base group is now 0.7%.
Trans: x4.2
Cis-female: x0.93
Non-white: x1.1
Non-white and non-male: x2.7
Non-straight: x2.2
Please, if you are in this group, seek care. A good therapist can be powerful. So can a good friend.
I had been very curious about those results, so great to see them.
There is a couple of things I find puzzling. The most current: shouldn't episodic memory be much better, given the IQ and SAT scores? Or are those not related?
I noticed that pretty much all ACX readers are in STEM fields, especially CS. As someone in the humanities (writing and literature), I can see why that is. I do wish there were more rationalist humanities types, but I understand why an ideology based on rationality would not appeal to people that study more emotional and unquantifiable subjects.
Interesting results...on most of these I am closer to the average here than I would have thought (certainly compared to the Worldwide or even Western World average)… There seem to be quite a few groups that are overrepresented among ACX readers compared to the Global or even Western World population...The following struck me the most:
Countries (Anglosphere, Scandinavia + Central Europe, Israel) . Anglosphere is obvious , and Scandinavia + Israel make sense because of either widespread English-Language proficiency in the former and diaspora/immigrants in the latter (I would think).
Racially the readership of ACX is less diverse than the US population, but not that different from the Western World at large (probably, since most European countries do not consider "racial" categories in their censuses). Compared to the US average, Black is heavily underrepresented and Asian overrepresented... I assume this is because of job patterns by race/ethnicity, and the type of readers this blog attracts (see job section in the results)?.
LGBT individuals also seem to be overrepresented in this blog. Also Atheists and Agnostics...
Very interesting results, I wish more blogs and/or topical websites would do these.
I am really baffled by circumcision question. Is this an American thing THAT much? In Russia this is extremely associated with religion (and widely laughed upon; for context, not only am I not circumcised, I don't think I know - in person - anyone who is! Not that people would freely discuss this...), but the religion question makes it necessary that many of these are "secular" circumcisions. Like, about half of the survey takers are circumcised (and the other half includes females!).
I'm curious as to how to explain the 33% (highest percent) of participants working in computers (practical). How related is it to the topics mentioned in ACX, the writing style, referrals etc?
It’s pretty remarkable how positive people are towards meetups. I think for the majority of online communities the consensus would be “I never want to meet any of you!”
I was curious about the two Moscow-Paris questions, so I'll scoop Scott on those. For the analysis, I restricted to the 6378 people who answered both questions.
Before I started the analysis, I suspected that for a question like this, you should take the geometric mean (GM) instead of the arithmetic mean (AM). In other words, the data will make more sense on a log scale. Indeed:
- The true answer is 2,486km.
- The arithmetic mean of all estimates is very bad. For the first estimates the AM is 7088km, for the second estimates it is 9331km, and for first and second estimates together it is 8210.
- The geometric mean of all answers is pretty good. For the first estimates the GM is 2,722, for the second estimates it is 2961, for first+second it is 2,839. That is only 9% / 19% / 14% from the truth.
Now to the interesting part: If you have only access to yourself, should you a) trust your first guess, b) trust your second guess, or c) trust the GM of your two guesses? In my book review on Consciousness and the Brain [1], I have mentioned a paper [2] which claims that you can get a better estimate by taking the mean of your two answers. I didn't have too much trust in it, so let's see:
For each estimate I computed a factor F >= 1 by which the estimate was off. So I computed the quotient "estimate/truth" if the estimate was larger than the truth, and computed "truth/estimate" otherwise. This is equivalent to the distance from the truth if we convert the data to a log scale. (The case distinction is because "the distance" is the absolute value of the difference.) The result:
- The first estimate was off by a factor 1.815. (This means that the GM of all those factors was 1.815)
- The second estimate was off by a factor 1.901.
- The GM was off by a factor 1.791.
To look at it another way, I removed the 75 answer where both estimates were equal, and asked:
- How often was the first estimate better than the second: in 53.3% of the cases.
- How often was the GM better than the first estimate: in 52.8% of the cases.
- How often was the GM better than the second estimate: in 60.0% of the cases.
So what is the conclusion? First of all, the second estimate was clearly worse than the first. My partner said before the analysis that naively we should expect the opposite: when you answer the second question, you have thought twice about the problem (and possibly harder the second time), so you have considered more information for your second estimate. Shouldn't this improve your answer?
My best explanation is that the second question asked you to imagine that your first answer was off by a non-trivial amount. This might give you a wrong bias from whatever correct reasoning you had. But the paper [2] also found that the second estimate was much worse for their questions, which were probably not phrased like this. And they had the same effect when there were three weeks between the two questions. I am not sure why the second estimate is so much worse.
But coming back to our ACX question: even though the second answer is not really good, the GM of both estimates is still slightly better than the first one, though the advantage is small. For a random person, the probability that the GM is better than the first answer is slightly higher than 50%. The factor by which you are off is better for the GM, but only by a small amount.
Overall, I could reproduce the conclusion from [2], though the effect looks pretty small to me, while it was huge in [2]. This could be because they used a very different type of analysis (arithmetic mean + mean square error). Mean square errors punish outliers a lot, so it may help the mean to shine.
Still, it's remarkable that you can take the (much worse) second guess to improve your first one. Apparently, you can gain a little bit from harvesting the wisdom of your inner crowd, even if your second guess on its own is less accurate.
"I strongly identify as Jewish and go to synagogue regularly; if asked "What is your religion?", I would definitely say Jewish. However, "the religion you believe" is the wrong way to describe my relationship to Judaism (and the wrong way to describe many people's relationship to their religion, esp. non-Western religions), since none of my beliefs about the world are based on my Judaism. (E.g. as previously indicated, I am agnostic.) So the correct answer to the question "What religion do you believe in?" might, in my case, be "none". For the purposes of your data analysis, please feel free to code this answer as either "Jewish", "Mixed/other", or blank, depending on what you actually meant by the question."
is a pretty Jewish (or at least, secular Ashkenazi) thing to say?
So I managed to find a total of one Alt-Right transsexual in the database. Assuming it wasn't someone trolling - admittedly a very big if - I would very much like to read an interview with that person and how they reconcile their political views with their sexual identity.
Scott: May I repost some of those COVID survey results on my weekly COVID update on Twitter? I'll credit you and ACX—or not—as you wish. What's interesting to me, is the number of COVID virgins. I've argued in the past that 25% of the nation's population has caught COVID yet. But without updated serological data it's hard to prove.
Country: There are 40 from "Netherlands" and another 20 from "The Netherlands". Last time I looked, these were all Dutch. ;) You may skip the option "The N." next time.
Kids: Hard to read, but if got it right one needs just 6 kids to become the lone Genghis Khan of ACX. And: Elon Musk did not take the survey.
Pairwise linear regressions on all the satisfaction questions:
Job vs. life: slope=.54, r^2=.36
Social vs. life: slope=.55, r^2=.36
Romantic vs. life: slope=.37, r^2=.28
Social vs. job: slope=.4, r^2=.16
Romantic vs. job: slope=.18, r^2=.06
Social vs. romantic: slope=.36, r^2=.23
All fantastically significant. Successful people are successful, but you could probably use this to prioritize at the margin: if you want to be happy, focus on getting a satisfying job and social life and don't worry as much about romance. If you want a better love life, improving your social activity is more useful than being happier at work.
* I wonder what can be the reasons why homosexuals have higher median IQ/SAT scores (the effect is rather small though, and the IQ/SAT responses are sparse and have outliers);
* "Political Spectrum" has many strong correlations, but most of them are obvious. One non-obvious one is a mildly positive correlation with "Children" (maybe it is because of age, but is still counterintuitive);
* "Children" also correlates positively with "BMI" (also likely because of age). However, contrary to the popular stereotype, "BMI" does not correlate with "Romantic Satisfaction", which is quite interesting.
* Again, contrary to stereotypes, "STEM" doesn't correlate with "IQ" (or even correlates mildly negatively). Though this can be due to the biased selection or poor quality of "IQ" responses.
When you click on a comment (on the date) to go a dedicated page for that comment, there's a text field there, but if you submit a comment there, it ends up at the top level and not as a reply to the comment whose page you are on. Probably the reason for the many stray top-level comments. Sry can't really find a substack issue report contact.
I think it would be interesting to look into the emails people used and see if any groupings show up. Such as those who use years or numbers in their email address or silly phrases or those who put school and work emails in. Or the demographics of gmail vs yahoo vs Hotmail and the like. I think only Scott would be able to perform that analysis due to having to know full emails.
On a hopefully less controversial front than my committed theist remark, I'm also surprised at how few people dream in B&W. My dreams have always been in black and white and really really hazy. Guess I'm one of the lucky/unlucky few - definitely jealous of all of you who get these awesome vivid dreams every night while I'm stuck with something that reminds me of playing video games from my youth where everything seems barely tangible.
Finally got around to going through the results and just remembered how annoyed I got at the "your country's covid response: too lax or too strict?" question. In the UK the covid response was too strict *because* it was too lax. In other words, they waited too long to act, so when they did they were forced to take extreme actions.
Some countries dealt with the pandemic in a relatively competent way, and others in a relatively incompetent way. Lax/strict isn't the most important axis.
Doesn't the "new multiple choice questions" thing ruin that entry for statistical purposes? Is there a way to identify responses before/after the date they were added?
I'm too lazy to do anything with the data myself, I mean, just curious!
Thank you for sharing this! We might use as one of the example data sets for course I teach.
Really struck by the racial breakdown here — 87% white. This is a very different breakdown from (e.g.) a standard tech school, where White and Asian-American are usually at similar ratios.
Would be fun to compare (where possible) some of these statistics to the GSA. I think the GSA provides enough information to produce a sample that matches the ACX gender/race/age demographics.
It's difficult to understand the results of the questions based on a linear scale: it only shows the theme of the question as a title and then a bar graph showing the percentage of users who chose each option. For example: a graph titled "Global Warming" with five bars labeled 1, 2, 3, 4, and 5.
God bless you for this occasional validation of the lurker class. We are legion.
The IQ results were very impressive. Well done all.
From what i remember the readership seems to have become more left-wing compared to previous surveys.
Welcome to the 0.6% of us who comment at least once a week
Brief thoughts, skimming through:
-Scott's audience is somewhat whiter, and *much* more male than would be expected by geographic demographics
-Functionally no transmen, more enbies than transwomen (Scott might want to put an option for "transmasc"/"transfem" next time?)
-Sexual orientation about as expected
-Way more married readers than I was expecting - I'm around the median (not modal) age, and I feel like very few of my peers are married
-Slightly less poly than expected? Though "no preference" basically translates to "poly by nature" as far as I'm concerned.
-~40% tech, only 1% psych. Scott is truly the techie's psych blog.
-Initially I was shocked at the number of people with only a high school diploma, before I realized that included all the younger college kids.
-The fact that 19% of respondents were "not done" with education and only ~13% of the total don't have a BA/BS means near majority of current students are postgrads.
-Virtue ethics had a surprisingly strong showing for such a consequentialist blog
-I can't tell if 75% is a low or high number of lurkers, since I would expect most lurkers to not respond at all, and lurker numbers are usually astonishingly high to me
-The new title of the "Political Spectrum" question makes it totally unclear what those numbers mean. IIRC it was Left < Center < Right ?
-Strong showing for Social Democrats; Liberals and Libertarians about as expected.
-Once again, with the questions omitted it's a little unclear what the numerical political opinions are measuring
-HBD is a bell curve lol
-Very middle class, statistically surprising number of upper-middle class, both in childhood and readership.
-I'm actually surprised at how few readers are depressed, especially compared to anxiety
-Less autistic than I expected - but I suppose I'm biased by selection effects in my personal life and it's still really high (and "Autism" is more specific/stigmatized than "on the spectrum")
-Ah, way more people have considered suicide than are currently depressed. "It gets better", indeed.
-Life satisfaction pretty high, surprised that romantic satisfaction is even higher (but I suppose I shouldn't be, considering the number of married individuals)
As a committed theist, I'm shocked at how few there are here. I had to double check the color coding.
Surprised by how high some states rank in terms of readership. Tennessee more than New York for example, very interesting.
The main things I'm surprised of: lots of European-style progressivism (e.g. "Social democratic" on "Political Affiliation", "Not registered" in "American Parties", 1 on "Global Warming", so few "No" on "COVID vaccine", so few people commuting by car, so many people who don't even own one) and yet so many married people -- these don't usually exactly go hand in hand.
A tip for analysis involving country: Czechia and Czech Republic should be merged into one (that makes it in total 27 people in total :-D ).
It makes me feel a lot better about my computer use habits that 19.3% of you also have more than 50 browser tabs open.
I am procrastinating my real work... a few other numbers:
1. filtering out extremes, people are bad at the distance Paris-Moscow; average is 4452 km (median guess is 3000). True distance is 2486 km. This is surprising to me; when I do wisdom of the crowd estimates people are often extremely good — within 10% on interesting questions for a group of > 100.
2. when you ask them to rethink it, they get worse (5231 km mean; median doesn't change).
3. again filtering out trolls, mean IQ (for the population answering both IQ and SAT) is 140 (median 140, too). This is mildly inconsistent with the reported SAT Math scores for the IQ-answering population (mean 725; median 750) which should be higher.
After seeing other people's guesses on the question about the distance between Paris and Moscow, I am feeling marginally better about my grasp of European geography and my ability to convert between miles and kilometers. I looked it up afterwards and I had overshot on my initial guess by about a thousand km, if I recall correctly.
Can't wait to see what the intention behind that pair of questions was! I'm guessing it was something to do with confidence in one's estimations.
Even though I knew it would be uncommon, for some reason I was surprised to see that the number of FTM transgender readers who took the survey is so low. Hello to the other 22 of us!
"which of the following forms of trauma did you experience as a young child, ie before the age of 10?
[...]
Rape (ie nonconsensualsexual intercourse) [:] 1.8%"
What the hell? Either this is normal (which is bad), this is normal for ACX types (which is less bad; do we grow up in unusually pedophilic communities), a fluke (which is an unlikely) or some lizardman's constant kind of deal. I'm forgetting other options, because, again, what the hell?
I quickly checked if richer people in the sample are more satisfied with life than the average responder and they do seem to be.
edit: I added graphs for IQ to Income and IQ to Life Satisfaction.
https://twitter.com/Tenoke_/status/1616553455212179456
Seeds of Science, a journal publishing speculative and non-traditional scientific articles (and ACX grant winner), would like to offer itself as a peer-reviewed publishing platform for any analyses using the ACX reader survey data. Visit the website (theseedsofscience.org) to learn more or contact us at info@theseedsofscience.org.
No human biodiversity question? Awww.
I love that the AI question has, in order, responses of
1
.1
.01
.001
.00001
.000001
.0000000001
.0000000000001
Your preliminary summary would be more useful if the 1-5 etc. responses were better explained.
13% of people not having an internal monologue is 10-100x higher than I would have expected. I thought the concept was a joke/insult about lacking self awareness or a (temporary) goal in meditation.
Would it be possible to add the aphantasia type questions again next year please to do crosstabs with the inner monologue question?
I assume that those are the two most common ways of thinking (for people with both senses) and am wondering if lacking one increased the chance you need the other.
I'm also curious about what percentage of people lacking an inner monologue also have aphantasia for audio. I personally don't think visually 99% of the time but am still capable of it when I want to. Smell/touch/taste can be remembered but I have no idea how I'd reason with them.
I'd also assume someone without an inner monologue would be lonelier when alone and might feel the need to be more extroverted as a result, so some more specific questions about people's social lives to tease that out would be helpful.
Final analysis for me tonight: I was struck by Scott's inclusion of a "suicide" question. At the risk of stating the obvious, please, if you are actively suicidal, contact someone. If you can't contact someone, go to the nearest emergency room. I lost a friend to sucide a few years ago; we had fallen out of touch and perhaps he felt he couldn't call me; I wish he had.
Here's a demographic breakdown from the Scott data. The modal respondant on the survey is white, cis-male, and heterosexual; the attempt rate for this "base" group is 4%. For this analysis I'll restrict to 30-and-under, so that we don't have too much age confounding.
Here are the relative risk ratios for other groups (e.g., x2 means a factor of two higher than the base group).
Trans: x5.6
Cis-female: x1.7
Non-white: x1.8
Non-white and non-male: x2.8
Non-straight: x3.2
I did not find strong effects for IQ. If we restrict to "I have attempted it in the past, and wish the attempt had succeeded", the results equalize a little. The rate for the base group is now 0.7%.
Trans: x4.2
Cis-female: x0.93
Non-white: x1.1
Non-white and non-male: x2.7
Non-straight: x2.2
Please, if you are in this group, seek care. A good therapist can be powerful. So can a good friend.
I had been very curious about those results, so great to see them.
There is a couple of things I find puzzling. The most current: shouldn't episodic memory be much better, given the IQ and SAT scores? Or are those not related?
I noticed that pretty much all ACX readers are in STEM fields, especially CS. As someone in the humanities (writing and literature), I can see why that is. I do wish there were more rationalist humanities types, but I understand why an ideology based on rationality would not appeal to people that study more emotional and unquantifiable subjects.
I don’t think I made it through the survey- I thought why? What is this all about exactly ? I drifted off
Interesting results...on most of these I am closer to the average here than I would have thought (certainly compared to the Worldwide or even Western World average)… There seem to be quite a few groups that are overrepresented among ACX readers compared to the Global or even Western World population...The following struck me the most:
Countries (Anglosphere, Scandinavia + Central Europe, Israel) . Anglosphere is obvious , and Scandinavia + Israel make sense because of either widespread English-Language proficiency in the former and diaspora/immigrants in the latter (I would think).
Racially the readership of ACX is less diverse than the US population, but not that different from the Western World at large (probably, since most European countries do not consider "racial" categories in their censuses). Compared to the US average, Black is heavily underrepresented and Asian overrepresented... I assume this is because of job patterns by race/ethnicity, and the type of readers this blog attracts (see job section in the results)?.
LGBT individuals also seem to be overrepresented in this blog. Also Atheists and Agnostics...
Very interesting results, I wish more blogs and/or topical websites would do these.
I am really baffled by circumcision question. Is this an American thing THAT much? In Russia this is extremely associated with religion (and widely laughed upon; for context, not only am I not circumcised, I don't think I know - in person - anyone who is! Not that people would freely discuss this...), but the religion question makes it necessary that many of these are "secular" circumcisions. Like, about half of the survey takers are circumcised (and the other half includes females!).
I'm curious as to how to explain the 33% (highest percent) of participants working in computers (practical). How related is it to the topics mentioned in ACX, the writing style, referrals etc?
It’s pretty remarkable how positive people are towards meetups. I think for the majority of online communities the consensus would be “I never want to meet any of you!”
Shocked by the suicide data. And surprised that I felt sad to see it.
I was curious about the two Moscow-Paris questions, so I'll scoop Scott on those. For the analysis, I restricted to the 6378 people who answered both questions.
Before I started the analysis, I suspected that for a question like this, you should take the geometric mean (GM) instead of the arithmetic mean (AM). In other words, the data will make more sense on a log scale. Indeed:
- The true answer is 2,486km.
- The arithmetic mean of all estimates is very bad. For the first estimates the AM is 7088km, for the second estimates it is 9331km, and for first and second estimates together it is 8210.
- The geometric mean of all answers is pretty good. For the first estimates the GM is 2,722, for the second estimates it is 2961, for first+second it is 2,839. That is only 9% / 19% / 14% from the truth.
Now to the interesting part: If you have only access to yourself, should you a) trust your first guess, b) trust your second guess, or c) trust the GM of your two guesses? In my book review on Consciousness and the Brain [1], I have mentioned a paper [2] which claims that you can get a better estimate by taking the mean of your two answers. I didn't have too much trust in it, so let's see:
For each estimate I computed a factor F >= 1 by which the estimate was off. So I computed the quotient "estimate/truth" if the estimate was larger than the truth, and computed "truth/estimate" otherwise. This is equivalent to the distance from the truth if we convert the data to a log scale. (The case distinction is because "the distance" is the absolute value of the difference.) The result:
- The first estimate was off by a factor 1.815. (This means that the GM of all those factors was 1.815)
- The second estimate was off by a factor 1.901.
- The GM was off by a factor 1.791.
To look at it another way, I removed the 75 answer where both estimates were equal, and asked:
- How often was the first estimate better than the second: in 53.3% of the cases.
- How often was the GM better than the first estimate: in 52.8% of the cases.
- How often was the GM better than the second estimate: in 60.0% of the cases.
So what is the conclusion? First of all, the second estimate was clearly worse than the first. My partner said before the analysis that naively we should expect the opposite: when you answer the second question, you have thought twice about the problem (and possibly harder the second time), so you have considered more information for your second estimate. Shouldn't this improve your answer?
My best explanation is that the second question asked you to imagine that your first answer was off by a non-trivial amount. This might give you a wrong bias from whatever correct reasoning you had. But the paper [2] also found that the second estimate was much worse for their questions, which were probably not phrased like this. And they had the same effect when there were three weeks between the two questions. I am not sure why the second estimate is so much worse.
But coming back to our ACX question: even though the second answer is not really good, the GM of both estimates is still slightly better than the first one, though the advantage is small. For a random person, the probability that the GM is better than the first answer is slightly higher than 50%. The factor by which you are off is better for the GM, but only by a small amount.
Overall, I could reproduce the conclusion from [2], though the effect looks pretty small to me, while it was huge in [2]. This could be because they used a very different type of analysis (arithmetic mean + mean square error). Mean square errors punish outliers a lot, so it may help the mean to shine.
Still, it's remarkable that you can take the (much worse) second guess to improve your first one. Apparently, you can gain a little bit from harvesting the wisdom of your inner crowd, even if your second guess on its own is less accurate.
[1] https://astralcodexten.substack.com/p/your-book-review-consciousness-and
[2] https://journals.sagepub.com/doi/10.1111/j.1467-9280.2008.02136.x (paywall, but accessible through sci-hub if that is legal for you)
As a minor aside, does anyone think
"I strongly identify as Jewish and go to synagogue regularly; if asked "What is your religion?", I would definitely say Jewish. However, "the religion you believe" is the wrong way to describe my relationship to Judaism (and the wrong way to describe many people's relationship to their religion, esp. non-Western religions), since none of my beliefs about the world are based on my Judaism. (E.g. as previously indicated, I am agnostic.) So the correct answer to the question "What religion do you believe in?" might, in my case, be "none". For the purposes of your data analysis, please feel free to code this answer as either "Jewish", "Mixed/other", or blank, depending on what you actually meant by the question."
is a pretty Jewish (or at least, secular Ashkenazi) thing to say?
So I managed to find a total of one Alt-Right transsexual in the database. Assuming it wasn't someone trolling - admittedly a very big if - I would very much like to read an interview with that person and how they reconcile their political views with their sexual identity.
Kinda funny how there are more readers living in Denmark(39, population 5.9 million) than are black.(26, population >1 billion)
Scott: May I repost some of those COVID survey results on my weekly COVID update on Twitter? I'll credit you and ACX—or not—as you wish. What's interesting to me, is the number of COVID virgins. I've argued in the past that 25% of the nation's population has caught COVID yet. But without updated serological data it's hard to prove.
Country: There are 40 from "Netherlands" and another 20 from "The Netherlands". Last time I looked, these were all Dutch. ;) You may skip the option "The N." next time.
Kids: Hard to read, but if got it right one needs just 6 kids to become the lone Genghis Khan of ACX. And: Elon Musk did not take the survey.
Pairwise linear regressions on all the satisfaction questions:
Job vs. life: slope=.54, r^2=.36
Social vs. life: slope=.55, r^2=.36
Romantic vs. life: slope=.37, r^2=.28
Social vs. job: slope=.4, r^2=.16
Romantic vs. job: slope=.18, r^2=.06
Social vs. romantic: slope=.36, r^2=.23
All fantastically significant. Successful people are successful, but you could probably use this to prioritize at the margin: if you want to be happy, focus on getting a satisfying job and social life and don't worry as much about romance. If you want a better love life, improving your social activity is more useful than being happier at work.
Thanks for posting the data! Here is t-SNE visualization of responses (surprisingly, there are no well-defined clusters, the space is rather continuous): https://enryu43.github.io/misc/acx_2022_survey_tsne.html
Some unexpected correlations:
* I wonder what can be the reasons why homosexuals have higher median IQ/SAT scores (the effect is rather small though, and the IQ/SAT responses are sparse and have outliers);
* "Political Spectrum" has many strong correlations, but most of them are obvious. One non-obvious one is a mildly positive correlation with "Children" (maybe it is because of age, but is still counterintuitive);
* "Children" also correlates positively with "BMI" (also likely because of age). However, contrary to the popular stereotype, "BMI" does not correlate with "Romantic Satisfaction", which is quite interesting.
* Again, contrary to stereotypes, "STEM" doesn't correlate with "IQ" (or even correlates mildly negatively). Though this can be due to the biased selection or poor quality of "IQ" responses.
The children chart doesn’t need fractions on the x axis unless king Solomon comes back as a zombie
When you click on a comment (on the date) to go a dedicated page for that comment, there's a text field there, but if you submit a comment there, it ends up at the top level and not as a reply to the comment whose page you are on. Probably the reason for the many stray top-level comments. Sry can't really find a substack issue report contact.
I think it would be interesting to look into the emails people used and see if any groupings show up. Such as those who use years or numbers in their email address or silly phrases or those who put school and work emails in. Or the demographics of gmail vs yahoo vs Hotmail and the like. I think only Scott would be able to perform that analysis due to having to know full emails.
On a hopefully less controversial front than my committed theist remark, I'm also surprised at how few people dream in B&W. My dreams have always been in black and white and really really hazy. Guess I'm one of the lucky/unlucky few - definitely jealous of all of you who get these awesome vivid dreams every night while I'm stuck with something that reminds me of playing video games from my youth where everything seems barely tangible.
Finally got around to going through the results and just remembered how annoyed I got at the "your country's covid response: too lax or too strict?" question. In the UK the covid response was too strict *because* it was too lax. In other words, they waited too long to act, so when they did they were forced to take extreme actions.
Some countries dealt with the pandemic in a relatively competent way, and others in a relatively incompetent way. Lax/strict isn't the most important axis.