In my job as a consultant in the game industry I ran some large scale surveys and designed surveys campaigns with all sorts of security/validation concerns, in case you have any technical questions about survey software and deployment that goes beyond google forms. Seems like you have a plan for that already though so fee free to ignore.
Recently I've been rereading some of SSC 2019 adversarial collaborations, and I was so impressed with how much work the collaborators put into them. I agree that they may not be as entertaining to read as book reviews, but they serve a very important role. I feel that Scott gave up on them too soon. It would great to see them again once in a while.
I want to express how happy I am to see this post. As a huge enthusiast of SSC-user generated content, I love seeing Scott involve the community in projects like these.
Scott, I know during the book review contest you missed some submissions to their authors’ anguish. I assume you have learned your lesson and already solved whatever the failure mode was that caused that problem. But if you need a volunteer to help with the administrative work keeping tabs on all the submitted requests, I could be such a volunteer.
You should try to replicate the colour perception - depression finding, or maybe do it with a tone-deafness test since readers would be less primed for that, or a sherik grey scale eye test if you want to stick to a visual perception test.
Is it also okay to suggest studies that are not Google-Docs based? I would like to test whether suggestibility to a visual illusion is linked to depression and/or anxiety symptoms, but I don´t think that this will work in Google Docs. Rather I would do that in an online study builder (e.g. https://lab.js.org/).
Okay, I'm ROFL at this because didn't Neike in the Links thread joke about why aren't we a cult? The Rightful Caliph is now putting us to work for the good of the Family so here we go! 🤣
Second, I hope there will be a "if you're not in the USA" option, because it's frustrating to be asked questions that are specifically USA-centric. On the other hand, if the survey takers want USA answers only, making that clear will avoid a lot of confusion.
Third, the race/gender/whatsit options are going to be pecked to death with "you didn't include this/you put that under the other heading and it should have gone here" but again, this is what you must expect on here.
Fourth, yes dammit, I love taking random surveys online so sign me up!
I would propose having the users email themselves their user id, in a way they could locate it later, and then have them use it again in future surveys, in subsequent years. Panel data can answer many questions that a single survey can't.
Note that this is too short a timeline for many university-based researchers to get IRB/ethics approval to have you include their questions, which means some university-based researchers who send to you might later have to withdraw or be uncomfortable even asking without IRB approval. A solution to consider is extending the deadline.
I sent an email with two suggested surveys, one of which I have finished the form for and one of which I am still working on. Can you confirm that you have received this email?
A suggestion on offering to link to studies that derive from your survey: set conditions on 'good' studies, including negative results and list what that would include (pre-register, etc, etc)
Good idea and strategy for execution. It might be helpful to keep in mind that close to half of early drafts of questions designed by professional measurement specialists fail validation, usually on one of a small number of technical criteria. Anyone who cares about the validity of their findings will want to get a measurement scientist involved. In turn, this will entail validation studies on small samples. Short of that and you are likely to have interesting responses to discuss but little more.
In my job as a consultant in the game industry I ran some large scale surveys and designed surveys campaigns with all sorts of security/validation concerns, in case you have any technical questions about survey software and deployment that goes beyond google forms. Seems like you have a plan for that already though so fee free to ignore.
I look forward to Infowars writing another shitty story about whatever unflattering tendencies the survey uncovers.
I have an offtopic comment:
Recently I've been rereading some of SSC 2019 adversarial collaborations, and I was so impressed with how much work the collaborators put into them. I agree that they may not be as entertaining to read as book reviews, but they serve a very important role. I feel that Scott gave up on them too soon. It would great to see them again once in a while.
I want to express how happy I am to see this post. As a huge enthusiast of SSC-user generated content, I love seeing Scott involve the community in projects like these.
Scott, I know during the book review contest you missed some submissions to their authors’ anguish. I assume you have learned your lesson and already solved whatever the failure mode was that caused that problem. But if you need a volunteer to help with the administrative work keeping tabs on all the submitted requests, I could be such a volunteer.
You should try to replicate the colour perception - depression finding, or maybe do it with a tone-deafness test since readers would be less primed for that, or a sherik grey scale eye test if you want to stick to a visual perception test.
Is it also okay to suggest studies that are not Google-Docs based? I would like to test whether suggestibility to a visual illusion is linked to depression and/or anxiety symptoms, but I don´t think that this will work in Google Docs. Rather I would do that in an online study builder (e.g. https://lab.js.org/).
Okay, I'm ROFL at this because didn't Neike in the Links thread joke about why aren't we a cult? The Rightful Caliph is now putting us to work for the good of the Family so here we go! 🤣
Second, I hope there will be a "if you're not in the USA" option, because it's frustrating to be asked questions that are specifically USA-centric. On the other hand, if the survey takers want USA answers only, making that clear will avoid a lot of confusion.
Third, the race/gender/whatsit options are going to be pecked to death with "you didn't include this/you put that under the other heading and it should have gone here" but again, this is what you must expect on here.
Fourth, yes dammit, I love taking random surveys online so sign me up!
I would be interested to see how responses vary based on subscriber status, could you include that along with the demographic questions?
Also, will data from the survey be publicly available (conditional on participant consent to be in the publicly available dataset)?
I would propose having the users email themselves their user id, in a way they could locate it later, and then have them use it again in future surveys, in subsequent years. Panel data can answer many questions that a single survey can't.
Are you going to post the Daniel Ingram meditation survey? That sounds interesting.
Note that this is too short a timeline for many university-based researchers to get IRB/ethics approval to have you include their questions, which means some university-based researchers who send to you might later have to withdraw or be uncomfortable even asking without IRB approval. A solution to consider is extending the deadline.
I sent an email with two suggested surveys, one of which I have finished the form for and one of which I am still working on. Can you confirm that you have received this email?
A suggestion on offering to link to studies that derive from your survey: set conditions on 'good' studies, including negative results and list what that would include (pre-register, etc, etc)
Good idea and strategy for execution. It might be helpful to keep in mind that close to half of early drafts of questions designed by professional measurement specialists fail validation, usually on one of a small number of technical criteria. Anyone who cares about the validity of their findings will want to get a measurement scientist involved. In turn, this will entail validation studies on small samples. Short of that and you are likely to have interesting responses to discuss but little more.
Thanks !