You are viewing a single comment's thread.

view the rest of the comments →


[–] Fact_Checking_Alien 0 points 1 points (+1|-0) ago  (edited ago)

That's actually quite good. Bear in mind you're dealing with a lot of variables when conducting a study, and often times without funding you're left with convenience samples or whatever is at hand, usually that means college students. This is a problem with all research involving humans, whether it be neurological research, psychology, and especially sociological papers.

Imagine, for a moment, that you're a researcher. You have a hypothesis you want to test, and the only grants or funding available amounts to a few hundred to a few thousand dollars. On top of spending however long trying to find likeminded people who are ultimately volunteering their time for this study beyond their normal work hours (a lot of researchers also have to be lecturers), but you have no funding for travel, no funding to pay for any archival fees if your work requires it, no funding to even pay for (in the case of survey-related work) any company that does survey calling. All you have is a piddly amount of funding and the best you can manage is to, maybe, get your own students to participate in the survey for extra credit. Honestly most research is in this boat, and some hypotheses that are ultimately true end up failing due to reliance on convenience samples too.

There are tons and tons of problems with needing to rely on convenience samples, especially students. Not least of which an unknown number of students figuring out what it is that you're studying, or due to their education severely biasing their own notions of what the survey is about, which in turn radically alters the results. Or, even worse, in very difficult studies like affect misattribution, for whatever reason your sample does misattribute a given bias but another sample doesn't. This doesn't mean a given idea for a given mechanism of affect misattribution is wrong, but might mean there's a different mechanism altogether to explain why one group has this subtle bias and another does not (education being one possible factor).

In the end, being a researcher is miserable. Very few researchers get lucky enough to end up with huge research grants or huge research facilities that have the resources able to get non-convenience samples, let alone travel expenses or the volunteer effort necessary (as when one attempts to study clinical subjects). Then there are plenty of times where, as in clinical samples or concerning any company, you have contracts they require you to sign to omit information. Or even not publish if there are certain results. Or maybe they just tell you to fuck off due to fear of potentially bad PR. For example, there are very very few experiments like the Rosenhan experiment. Not only is it incredibly difficult to reliably put up an act, and extreme personal risk of being involuntarily committed, as well as health risks of being medicated when you don't need it, but all institutions will do their damndest to ensure you get buried in lawsuits or smear PR campaigns.

What I'm trying to get across is that the reasons behind why reproduction tends to be so poor in so many fields are very complex. It's not as if 95% of researchers have much in the way of access, and so the overwhelming majority of the time you're slaving away using samples you know aren't good just in the hopes of getting funding to get good samples. And if you, like every other researcher, roll those dice and your studies come up with nothing? Enjoy mediocrity purely by chance. Because that's what it is. You're applying the same rules as everyone else and whoever "wins" that game didn't get there, most of the time, by actual genius. It's pure luck whether your convenience sample is ultimately confirmed, which in turn will get you that funding to do research without convenience samples.

Edit: And here's one study in neurology that was quite well known in its day - - because no matter what you think of a given science, whether it's "hard" or "soft" science, there are tons of problems at every single step of the process. This includes medical research

This is not a problem with the "soft" sciences. It's a problem with all of science.