• OK, it's on.
  • Please note that many, many Email Addresses used for spam, are not accepted at registration. Select a respectable Free email.
  • Done now. Domine miserere nobis.

Failed reproducibility of psychological research

Fukyo

blurb blurb
Local time
Today 9:31 PM
Joined
Jan 4, 2009
Messages
4,288
-->
This thread will get even less replies than r4ch3l's "introverted face", but Science magazine published a report testing 100 scientific studies for reproducibility and found over 60% did not cut it.

This is a blog post discussing it and here is the full paper. As a lazy information ingesting netizen I haven't bothered to read it but I figure at least one person may be interested in it. This is not the first time such issues arise and I've been aware of them for a while reading posts on the (used to be quite interesting) Neuro Skeptic blog.

This has interesting implications.
 

Miss spelt

Banned
Local time
Today 1:31 PM
Joined
Sep 3, 2015
Messages
202
-->
Science magazine published a report testing 100 scientific studies for reproducibility and found over 60% did not cut it.

What they actually said was (essentially) there isn't really a single indicator which sufficiently describes replication success, but they came up with a few that appear to be fairly valid. Using this "best guess" indicator, they found that reproducibility is an important factor in the field of scientific research and should be given more attention in general before we accept information as accurate.

:cat:
 

Glaerhaidh

straightedgy
Local time
Today 9:31 PM
Joined
Nov 21, 2013
Messages
4,373
-->
Location
someplace windswept
The open science framework page is quite interesting.
Note: Updated control condition
task
and instructions as well as
sample size information 4 June 2013
Approved Replication Protocol
Study 1 from Schooler & Engstler
-
Schooler (1990)
Article Citation:
Schooler, J. W., & Engstler
-
Schooler, T. Y. (1990). Verbal
overshadowing of visual memories: Some things are better left unsaid. Cognitive
Psychology, 22, 36
-
71.
Rationale:
Prior to the original finding of verbal overshadowing, most memory research
suggested that any rehearsal of to
-
be
-
remembered materials would enhance recall
of those materials. The original verbal overshadowing result was both theoretically
important and surprising because it showed that verbally rehearsing an experienced
event impaired memory for visual details from that event. The finding suggested that
eyewitness recollection might be impaired by asking witnesses to describe what they
saw
, a result with both practical and theoretical importance. Over the ensuing years,
the study’s original author (Jonathan Schooler) has tried to reproduce that finding,
and the measured effect sizes were substantially smaller than those of the original
stud
y

the effect seems to be more temperamental than initially thought. Despite
receiving more than 500 citations since it was first published in 1990, few other
laboratories have attempted direct replications of the crucial first study. Moreover,
Schooler has
argued that the reduced size of the effect in his attempts to repeat the
experiment reflects an active mechanism that has led to a reduced effect size, the
so
-
called “decline” effect. His writing about the reduced effect size of this result has
received e
xtensive coverage in journals and the popular media, including
commentaries in
Nature
and a feature article in the
New Yorker
. The effect itself
appears regularly in cognitive psychology textbooks as well. Given the uncertainty
about the size of the effect
, direct replication of the original study by multiple
laboratories will help determine the robustness of the interfering effects of verbal
rehearsal on recognition of visual materials.
Materials:
The authors of the original study have made the original
study materials
available. These include the critical video, test image, instructions, and filler task.
Design:
Subjects view a test video of a bank robbery and immediately afterward they
spend 5 minutes either describing the robber (experimental conditio
n) or listing
states and state capitals (control condition). They then complete an unrelated filler
task for 20 minutes (an easy crossword puzzle). Finally, they view an image
depicting 8 people and select the one who was in the bank robbery video and rate
their confidence in their selection.
Critical result:
The critical test is a Chi
-
Square comparing the proportion of people in
each condition who correctly identify the bank robber. The original study found worse
recognition in the experimental condition
than in the control condition (effect size of r
= .260).


Protocol Requirements
Sample size:
n

100 total, with n

50 in each of the two conditions.
N
o
te
that
this is
the minimum allowabl
e sample size. Laboratories
should
include as many
participants as possible.
Sample demographics:

Undergraduate subject pool or equivalent

Age range 18
-
25 years

Between 20% and 80% female,

Given that the perpetrator depicted in the original video is White, subjects shoul
d
also be White. The verbal overshadowing effect is though
t to be weaker with other
-
race faces than own
-
race faces (Fallshore & Schooler, 1995).

Must be able to understand the instructions and have vision adequate to watch the
video and see the images
Testing location:

In
-
person testing (may not be conducted on
line)

Subjects must not be able to see or hear each other when responding or when
viewing stimuli

Subjects may be tested individually or in small groups, provided they cannot
communicate during the study.

The study may not be conducted in a classroom setti
ng.
Experimenters:

Any trained research assistant, postdoctoral researcher, or faculty person

Experimenter should have experience collecting experimental psychology data and
interacting with subjects.

No special expertise is necessary to conduct the study

The experimenter does not need to be blind to condition assignment (as that would
be impractical)
Data collection:

The video may be shown on a computer display, on a television, or projected on a
screen.

Confidence ratings and written descriptions can
be made on paper or on a computer.

Subjects should be blind to the hypothesis about verbal overshadowing and should
be unaware of any conditions other than their own (i.e., subjects in the
experimental condition should not know that there is a control co
ndition, and vice
versa).


Note: Updated control condition
task
and instructions as well as
sample size information 4 June 2013

Subjects should be randomly assigned to the experimental and control condition with
the constraint that approximately equal numbers of subjects are assigned to each
condition.

Subjects should not know they are participating in a
recognition memory task.

The signup for the study should describe it as a study of perception and memory.
Data from a subject may be excluded if:

They did not follow instructions on the experimental or control task

They did not complete all tasks

The Expe
rimenter incorrectly administered the task or instructions

Data may only be excluded prior to examining the recognition task performance and
should be based on factors unrelated to the outcome measures.

Any excluded data and the reason for exclusion should
be identified and included in
the data sets.
Required Data Analyses:

Compute a Chi
-
Square comparing percentage of correct identification in the
experimental and control condition

Compute a Chi
-
Square comparing the ratio of the percentage selecting the wr
ong
face (misidentification) to the percentage indicating “not present” across the
experimental and control conditions.

Compute a 2 (Condition) x 2 (Correct vs. incorrect/miss) ANOVA on confidence
ratings
Procedure:
1)
Subjects are recruited to participate
in a study of memory and perception
2)
Subjects are randomly assigned to the experimental condition or the control
condition.
3)
Subjects are told: “This experiment consists of several tasks. First, please pay close
attention to the following video”
4)
Subjects vie
w a 30
-
second video depicting a bank robbery.
5)
Subjects receive different instructions depending on their condition assignment:
Experimental Condition
: “Please describe the appearance of the bank robber in as
much detail as possible. It is important that y
ou attempt to describe all of his
different facial features. Please write down everything that you can think of
regarding the bank robber’s appearance. It is important that you try to describe
him for the full 5 minutes”
Control Condition
: “Please name
as many countries
and their capitals
as you can

6)
After 3 minutes, each group should receive the following reminder:
Experimental Condition
: “Please continue describing every detail of the bank
robber. It is important that you provide as full a description as possible”
Control Conditi
on
: “Please continue to list
countries
and their capitals. It is
important that you continue this task for the full five minutes.”


7)
After 5 minutes of writing/typing, all subjects should spend 20 minutes working on
the provided crossword puzzle. Each participant should be given a printed copy
of
the puzzle.
8)
Subjects view the lineup of 8 faces and identify the one they saw in the robbery
video or report that it wasn’t present. They should read/hear the following
instructions: “Next you will see an lineup with 8 faces. Please identify the
individ
ual in the line up who you believe was the bank robber in the video you
watched earlier. If you do not believe the bank robber is present please indicate
‘not present’”
9)
If the lineup task is computerized, the images are numbered 1
-
8 to allow a keyboard
r
esponse and the last sentence of the
instructions in #8 should add the following:
“...please indicate ‘not present’ by pressing '9'. Press ‘space’ to view the image.”
10)
Subjects rate their confidence in their selection. They should be giving the
following in
structions: “Please indicate your confidence in your selection from the
lineup on a scale from 1 (guessing) to 7 (certain).”
After registration you have access to past research results, materials reports and so on. God, that one was so boring.

Psych research, diet research, it's usually shrug worthy from the beginning, whatever new findings they may have, most of them not applicable, not informative or inconclusive.
I wish to stay clear from soft sciences because of that.

A partly evident truth is that very little effort is spent on replicating and expanding research results that no one cares about.
Or there's this problem when research topics may seem optimistic while it's very difficult to tell if there will be a valuable result, if any.
 

Brontosaurie

Banned
Local time
Today 9:31 PM
Joined
Dec 4, 2010
Messages
5,646
-->
The problem lies mostly in the acceptance of social psychology as an academic discipline. I'm not all that familiar with how research funding works but as far as i've understood it, tax money is even being used to make this lousy research in this "science" that precludes scientific validity by its very definition.

Social psychology has several glaring issues:

* Social psychology is bordering on the humanities. There is an affinity in terms of legacy, norms, tropes, culture, ideological motives and work ethics.
* Social psychology likes surveys. Surveys are, mildly put, problematic. Actually, surveys are completely useless informational red herring tail-chasing mirages. No real data can be obtained from surveys. I expect to see a groundbreaking analytical proof on this in the next few decades.
* Social psychology often likes to be qualitative rather than quantitative. This is unscientific. There is such a thing as qualitative research, sure. But it's not a part of any scientific process. It's part of the openness that ideally should be maintained by any and every person who wishes to partake in the production of knowledge. There is no method to this openness - that would defeat its purpose. Yet social psychologists sell "observational research" as science. It's both boring and useless. Science is boring and useful, openness is fun and useless, social science tries to be both only to get its wings burned like old Sisyphus the sun-flying dude.
 

Sinny91

Banned
Local time
Today 8:31 PM
Joined
May 16, 2015
Messages
6,299
-->
Location
Birmingham, UK
Social science studies society which is an ever moving mechanism.

But apart from that, most mainstream efforts in social science/psychology is tainted towards certain agendas.

Money being the most obvious.
 

Pyropyro

Magos Biologis
Local time
Tomorrow 4:31 AM
Joined
Feb 3, 2012
Messages
4,044
-->
Location
Philippines

QuickTwist

Spiritual "Woo"
Local time
Today 2:31 PM
Joined
Jan 24, 2013
Messages
7,182
-->
Location
...

Brontosaurie

Banned
Local time
Today 9:31 PM
Joined
Dec 4, 2010
Messages
5,646
-->
Fuck, I'm way behind the curve.

"Behind the curve" is an obsolete and misleading expression, since in fact there is - at least in a majority of relevant cases - no actual curve present. Science should strive for precision, as should its language. The routine use of metaphorical phrases risks clouding our judgment by implicitly granting emphasis and authority to dubious intuitive conjecture. For this reason, we suggest replacing "behind the curve" with the more accurate "spin-lagging in the fractal dimension of superimposed psychophysical approximation".

While this criticism may seem like stifling pedantry, such a hasty rejection fails to appreciate the cascade effects of conceptual formation.

Now i feel like mac'n'cheese.
 

QuickTwist

Spiritual "Woo"
Local time
Today 2:31 PM
Joined
Jan 24, 2013
Messages
7,182
-->
Location
...
"Behind the curve" is an obsolete and misleading expression, since in fact there is - at least in a majority of relevant cases - no actual curve present. Science should strive for precision, as should its language. The routine use of metaphorical phrases risks clouding our judgment by implicitly granting emphasis and authority to dubious intuitive conjecture. For this reason, we suggest replacing "behind the curve" with the more accurate "spin-lagging in the fractal dimension of superimposed psychophysical approximation".

While this criticism may seem like stifling pedantry, such a hasty rejection fails to appreciate the cascade effects of conceptual formation.

Now i feel like mac'n'cheese.

I would totally be making clam chowder joke right now if I didn't think you were being silly.
 

QuickTwist

Spiritual "Woo"
Local time
Today 2:31 PM
Joined
Jan 24, 2013
Messages
7,182
-->
Location
...

QuickTwist

Spiritual "Woo"
Local time
Today 2:31 PM
Joined
Jan 24, 2013
Messages
7,182
-->
Location
...
I guess the main takeaway from that clip was: "vomit". Are you saying my post above was verbal vomit? How dare you!!!!!!!!!!!!

Does this condiment really cause such intestinal unrest though?

I was making a jest. No need to feel offended friend.
 

QuickTwist

Spiritual "Woo"
Local time
Today 2:31 PM
Joined
Jan 24, 2013
Messages
7,182
-->
Location
...
Top Bottom