91 Comments

If your link / draft is meant to be accessible to those *not* running surveys, I don't think it's working. I get "DON'T TAKE THIS YET! THIS IS JUST AN EXAMPLE! This survey is no longer accepting responses."

Expand full comment

The phrasing of

"Computers (other academic, computer science)

Computers (practical: IT, programming, etc.)"

is a little confusing, I initially read "academic, computer science" as giving two alternatives (either or), and then was confused when I saw "programming" in the next bullet. it might be clearer to just remove the comma.

Expand full comment

> Please keep this user ID saved - I might ask for it again on future ACX surveys so we can chain them together!

Your answer

I kinda doubt anyone’s gonna do this

Expand full comment

* For the psychological conditions, there're a few where I want to answer both "Yes, I have a relative with a diagnosis of this" and "Yes, I think I have this but haven't been formally diagnosed." Please let me select both options or provide some guidance for which I should pick.

* "Age" currently accepts negative and decimal answers. Not sure if you want to bother fixing this, though.

* I'm surprised Poland and Finland make it to the ten most popular countries - is there a particular reason?

* You can probably get "State" to be a dropdown if you want to bother. But if you do, please remember to include the District of Columbia. (And maybe Puerto Rico, the Virgin Islands, and American Samoa if you really want to be comprehensive.)

* Thank you for disambiguating the direction of transgender options.

Expand full comment

I know that nobody likes a whiner but could there be a Retired choice for ‘Job’?

I mean I’m doing okay and all but Independently Wealthy is overstating things.

I’ve always shared the Homemaking with my wife but…

And Unemployed just makes me feel like a deadbeat.

Expand full comment

Should the libertarian option on your generic demographic survey read "minimal/no redistribution of wealth" rather than "minimal/no distribution of wealth"?

Expand full comment

Scott, your definition for "alt-right" in the draft survey would encompass 90 million members of the Sino-Leninist CCP, given current PRC national policies and ambitions!

Expand full comment

I sent a survey in last Friday (7/9) on motivations during the pandemic. It was very hastily done and if it is not up to par for inclusion that is fine with me. However, I am curious about what the results would be, so if you didn't get it, should I post a link here?

Expand full comment

Don't mind me, just stress-testing your survey form. Good form allowing negative responses in number fields; will make it easier for you to calculate the lizardman's constant.

Expand full comment

Hi Scott, mine didn't get in so I guess my email wasn't coming through. Here is the link to the survey:

https://forms.gle/hDgEZUdRvkeyH4AR9

Expand full comment

On the depression question it might make sense to distinguish between people who are currently depressed and people who have been depressed. (Assuming thats not part of the definition of clinical depression in which case it would probably be good to explain what exactly clinical depression is)

Expand full comment

Let the nitpicking begin! On the sexual orientation question, I'm never sure which option to pick - is asexuality an orientation as distinct from a subset within primary attraction? And I do see it tending to be lumped in under "queer" which is not what I identify with or as at all. If we're sticking with "gay, straight, bi or other", just be aware some of the "other" may be "cis het but not actively so"?

The politics is difficult, as well: I'm not a UK Conservative but I can't identify "like the Democratic Party - socially liberal multiculturalism".

Well, we all can't have everything we want.

Expand full comment

I feel like "high taxes, traditional values" is missing as a political alignment. It doesn't really fit *me*, but it's not an inconsistent position or anything, and the existing categories don't really cover it.

"Within two generations" seems somewhat ambiguous. Are first cousins counted? Great-aunts/great-uncles?

"Drug addiction" should probably also exclude caffeine (technically all the methylxanthines, but I imagine anyone who knows that cocoa and tea have slightly-different alkaloids knows that they behave substantially as caffeine).

Expand full comment

For Gender, which currently has five options (CM, CF, TM, TF, Other) - I would appreciate the option of "Non-Binary" (or "Transgender Non-Binary"), which seem to be about 30% of transgender people.

Sexual Orientation and Relationship Status could include "Asexual" and "Aromantic".

Work Status seems like it should allow multiple answers. A person can be both self-employed and work for profit, or both a student and a non-profit volunteer.

Profession seems to have high resolution for sciences but very low resolution for anything else. Maybe you should just include the top 20 most common professions in the world / in the US.

Under Length of Time, "How long have you been reading SSC?" - maybe mention ACX since it's the new website's name?

In Political Interest, I think this scale should be 1...10 just like the previous one, instead of 1..5.

In Depression, Anxiety, OCD, etc.... there are currently four options (diagnosed, undiagnosed, in family, neither of the above). I think this should be split into two questions each - one for yourself (diagnosed, undiagnosed, previously afflicted, none) and one for your family (in extended family, not in extended family (not including yourself)).

Expand full comment

I periodic request I make: ask the user *when they are at the end of the survey* if they want their information publicly available or not.

I recall the survey asking at the front, people filling it out, and forgetting while leaving personally identifiable information somewhere along the way, and people would email to try to undo their entry.

At the beginning, mention that you'll ask at the end, and to consider that while they answer. Then, at the end, ask them to give their final answer.

Expand full comment

Possible suggestions:

--For age, you might want to ask "what is the year of your birth" and force it to accept a year (I think you can do that, if not it can at least be validated to an intenger or number)

--For state, you should make it a dropdown box unless you have an intern you don't particularly like that is responsible for cleaning up your data.

--For work status, you might break unemployed out into "Unemployed and looking for work" vs "unemployed and not looking for work"

--Degree - I know the standard is always to ask what is the highest degree earned, but I've always thought there should also be a question for highest degree worked toward but not completed. A person with an undergrad =/= a person who was a PhD candidate for a few years then decided to change tracks for whatever reason.

--Your Religious VIews is jarring to those of us in the tribe that is actually "theistic" or "diest" because usually don't call ourselves that. It would be like running a survey and asking if you are "libertarian" or "committed statist."

--Services attended in the last month is probably an important question as well

-- "Length of Time How long have you been reading SSC?" copy paste error or subtle messaging here?

--I'm just be a picky libertarian but you might want to replace "liberal" with "progressive" if your are going to have international respondants and/or picky libertarians taking this survey

Expand full comment

Could you include a box for IQ? I had this weird idea (after hearing about the large fraction of LGBT students at some elite private college.) That there might be a correlation between IQ and being part of LGBT? (Maybe this study has already been done?) Is there too large a scatter in self-reported IQ? I ask as a cis hetero male, with slightly above average IQ, but probably below average for this group. Or do people find this question offensive? In which case it's not that important to me.

Expand full comment

It feels like you're really going to want to stress test every survey with some subset of the most pedantic, irritating, nit-picky data scientists in your audience group. Otherwise, most of the "interesting" results from the surveys are just going to be p-hacking or caused by confounders.

On this survey, for example, the option "I have family members (within two generations) with this condition" would seem to be totally pointless. It has two major problems:

The first is that "family" is ambiguous. I would guess that this means "blood-relative" and therefore omit my spouse and my sister through adoption, but you'll likely find that the interpretation of "family" is correlated with many of the other breakdowns on the survey, particularly the cultural ones. This may be a far bigger issue than you think, because parental mental illness could easily be a non-trivial reason for adoption, and if this is the case, then the way that people who were themselves adopted interpret "family" will bias your data.

The second, bigger problem is that there is a huge information gap here that is also going to be heavily correlated with many of the other breakdowns. You are not going to be able to determine the relative heritability of mental illness in people of east Asian descent unless religious conservative families in Pakistan tell their kids about granny's schizophrenia at the same rate as liberal families in California. This is going to compound the first problem, because people generally have extremely accurate information about the mental health of their spouse, who is not interesting, but far less accurate information about their ancestral blood relatives who are. This is going to add a ton of noise.

Almost any conclusion you draw from this is going to be akin to drawing constellations in the stars.

Expand full comment

Suggestions for the demographic survey:

1) Add "NEET" (not in education, employment, or training) as an option for the question "In what area do you currently work or study?"

2) The religious denomination question contains "Other" in both the last and second-to-last options. If you want to keep the last option (wherein people can write what they mean by "Other"), then it should be removed from the second-to-last option.

3) Among doctoral degrees (ie PhD and MD), which is higher? (Perhaps this is obvious to those who hold both PhDs and MDs, in which case I guess ignore this suggestion.)

4) Considering the results of previous surveys, you will likely get lots of people who see their politics as orthogonal to the classic Left-Right political spectrum. That question could perhaps be restructured to say something like "...and if you do not fall on the classic spectrum, inidcate where on the classic spectrum you find least objectionable?"

5) When asking people about political affiliation, are you interested in the respondent's affiliation relative to their countrymen, or in some more absolute sense? If in relation to the politics of their country, then lumping Republicans and Tories together makes sense (they occupy a similar cultural niche in each country), but if in a more absolute sense, Tories are more similar to Democrats than to Republicans on most policy questions.

6) On the mental illness questions, clarify what "within two generations" means -- for instance, are nieces and nephews / aunts and uncles within two generations?

7) On the mental illness questions, "I have a formal diagnosis" and "I have family members with this condition" can both be true. Similarly, "I think I might but have not been diagnosed" and "I have family member with this condition" can also both be true.

Expand full comment

Can you make the race question accept multiple answers or add a two or more races option, akin to the US Census? I find it mildly frustrating and potentially misleading to have to pick a single race as a mixed race person, particularly if this is supposed to be a proxy for genetics and/or culture.

Expand full comment

For the mental illness questions, should there be an option like, "Diagnosed in the past but currently in remission"?

Expand full comment

Obviously, the subsurveys aren't from you, but this doesn't make sense:

"K on business (targeting people in tech)"

I suspect it's targeting generic businesspeople, who probably couldn't code their way out of a wet paper bag, but happen to be doing their vulture capital/MBA stock manipulation/etc. in a company that produces some kind of tech product.

Or it's the far too common attempt to understand how many tech people abandon tech in order to become (mis)managers and similar. (This is always presented as positive, but I disagree.)

Of course I'd love it to be about e.g. the attitudes and reactions of those actually doing tech to those who are instead doing "business", with or without prior experience and training doing tech.

Expand full comment

Currently imagining taking a 2 hour demographic survey with every single persons suggested questions and variations included

Expand full comment

The country thing is weird. I'm a Canadian who has lived in the US since 1992. I'd prefer to see country of citizenship and country of residence as separate questions. Phrased as it is, I'll identify with Canada and then (ignoring the "if American") also fill in the state in which I currently reside, producing what will appear to be incoherent data. Meanwhile American expats won't be able to answer the state question, even though you tell them to.

I don't "identify with" any race. This may be because of "white privilege" (I'm not picked on about race, so I can afford to be race blind.) But the idea of identifying with race isn't part of my personal reality. (The distinction between English and French Canadians is much more salient.)

Ditto with regard to identifying with a gender. (To the extent I group myself with similar people on gender-like grounds, it would be those who share the experience of being raised as girls.)

Are you next going to ask me what hair colour category I identify with, or what height category?

Race and gender are things other people ascribe to me, and then treat me according to their beliefs about "people like that".

Yes, I realize a lot of people do base their sense of self very much on their race and/or gender, and not always on a race/gender which would be obvious to a stranger judging based on their appearance.

OTOH, I've no idea how to square this circle. Whatever you ask, someone won't fit.

Expand full comment

Suggestions:

- Like others said, add nonbinary and asexual options (or at least make the "other" a write-in); make the race/ethnicity question checkboxable.

- Do people with ancestry from other parts of Asia (e.g., southeast Asia (like Vietnam), Philippines, Malaysia, Singapore) round to the nearest of "Asian (East Asian)" or "Asian (Indian subcontinent)"? This seems rather imprecise. Why split them up in the first place? If you're going to keep them separate, consider adding an "other Asian" option.

- Profession: What about students who aren't really studying their intended fields yet (e.g., most high-schoolers, many lower-division undergrads)? Should they select their intended fields, "other" (presumably white-collar), or what?

- "How long have you been reading SSC?": Is this length of time counted from "stumbled across link to SSC, read a couple of posts, left, and forgot about it for three years", from "started checking website to read new posts every few weeks", or from "reads almost every post written by Scott, and way too many comment threads, and should probably find a hobby other than reading blog posts"?

Expand full comment

You specify a variety of denominations for Christianity, but none for Judaism - you could ask practicing/Non-practicing, or include any of Orthodox/Conservative/Reform/Reconstructionist/Humanist as options - I don't know if people would be interested in this, just a thought.

Expand full comment

Good grief, you list "Philosophy" and "Art" (distinct from "Media") as anticipated professions among the readers, but have reduced the entire service sector to blue or white collar "other?"

You do realize that people with philosophy and art degrees are far more likely to be working in a service industry than as full-time philosophers and artists, right?

So maybe toss in "service industry," if not "food/hospitality" and/or "consumer services," and/or perhaps "building/construction?"

Expand full comment

"- A on biostasis/cryonics"

What does the letter here mean?

Expand full comment

Not sure what "formal education" means in this context (just high school? all planned education in a formal setting?) and what extra information does it provide over the previous question

Expand full comment

Neither of my initials is on the list. I'm giving you edit privs on my survey, so you'll get an email with a link to it.

Expand full comment

Seconding the below comment regarding the mental health questions. I don't know who counts as "in my family" - is it supposed to be genetic/raised with/currently living with?

Maybe this breakdown is more clear:

1. I have a formal diagnosis for myself

2. I suspect I have this

3. I don't know

4. I know I don't have this

Separate question on Relatives:

1. I know that a close genetic relative (at least 1/4 shared genes) has this

2. I don't know if my close genetic relatives have this

3. I'm very sure that none of my close genetic relatives has this.

Or smush the latter 3 in with #3 and make it one question with 7 options.

That makes it clear that you are interested in the survey takers mental diagnosis, or likelihood thereof.

Expand full comment

I think that your demographics survey question

"With which of these political descriptions do you most identify?"

needs an answer for people in the Social Justice movement, since it's the most-important political movement in the US today. The only option you've got that would even be acceptable to them is "socially democratic", which is hardly relevant to their political concerns (which are primarily critical race theory, post-colonialism, and gender identity).

Expand full comment