An object lesson in scientific rigour: Read the article (not just the title).

Clinical Psychologists have been citing a study by Stein and Lambert (1995), apparently in support of the validity of two tier remuneration, in their recent comments on the RAPS website. Other readers had concerns about the study, so, of course, we took a close look. That’s when things got a bit awkward.

Let’s unpack all this slowly and calmly.  It appears that these mighty ‘evidence-based practitioners’ of the APS Clinical College are content to rely on ONE, rather moth-eaten, American meta-analysis from 24 years ago, including 36 studies going back to 1970.  Yes, 1970 – check Table 1. There are another two from 1971.  In case some people have missed this, an essential component of scientific research is replication.  In scientific circles, a discovery has to be independently replicated by other researchers before it is accepted as a valid finding. But the clinicals are content to rely on just one reference. 

Given the APS’ resources, why have they never, in all these years, commissioned a rigorous study of the effectiveness of psychological treatment? They are lobbying for access to millions of dollars of public funds – shouldn’t they prove that their interventions do help clients? Furthermore, it would be fairly simple to test whether more training really did change client outcomes.

Anyway, we digress; back to Stein and Lambert (1995) and their meta-analysis.  What is a meta-analysis?  It’s a systematic combination of several studies into a single large study in order to increase the power of the research.  Meta-analysis cannot correct for poor design or sample bias in the original studies. Although the studies that they included have recently been described as “..36 well controlled studies”, Stein and Lambert were very critical of the poor standard of research that they found. It’s clear that the ones they did include would be best described as ‘least worst’ rather than ‘good’. Lack of random assignment of clients to therapists rules out cause and effect conclusions.  They included the following vital caveat:

It is important to note that the present review of the literature does not focus on differences in therapy outcomes among degreed professionals possessing different levels of experience (e.g., number of postdegree hours of psychotherapy) or differences in outcomes that are due to type of degree (MS versus PhD or MD). Adequate studies on these latter topics have not been conducted to date.  (p. 187)

In other words, Stein and Lambert (1995) stated plainly they could NOT test whether holding a higher degree in the area made you a better psychologist, because the research was simply not available.

What were the treatment outcomes, you ask? Well, since you ask, we’ll tell you. The treatment outcome was whether clients completed treatment or terminated prematurely. We’re not talking about clients attaining insight or living their best life – just whether they stuck it out or not.

The studies that Stein and Lambert used typically compared treatment outcomes obtained by psychologists versus those of ‘nurses, teachers or other paraprofessionals’ or ‘trained community volunteers’.  Yes.  It’s true. Qualified psychologists can produce “ a modest but fairly consistent effect size..” (p.192) when compared with volunteers! A little underwhelming, but good to know.

The outcome of all this work by Stein and Lambert is five recommendations for Future Training and Research (p.193).  For the present purposes, their most important recommendations are numbers one and three:

1. A substantial challenge to graduate training programs in psychology, psychiatry, and social work remains. Programs have yet to systematically demonstrate (a) whether the skills they teach relate directly to year-by-year increases in the successful number or quality of therapy outcomes among the patients of trainees or (b) that specific didactic or practicum experiences affect dropout rates over time. 

3. A critical unaddressed question pertains to the true utility of the extensive training found in most PhD and PsyD programs, (vs. MA-MS program) for conducting psychotherapy. Does extensive training relate to more effective therapy with more serious or difficult problems (e.g., severe anorexia nervosa), and the ability to work with diverse caseloads (e.g., marital problems, anxiety disorders, etc.)?

In 1995 graduate training programs had not shown whether the skills they taught increased ‘successful therapy outcomes’ – have they managed to do this by now?  The clinicals are silent.  Stein and Lambert directly questioned whether more training produces more effective therapy for clients with serious problems.  The central argument of the APS Green Paper is that more extensive training of clinical psychologists qualifies them and them alone to deal with ‘serious’ level clients.  Do they have any current Australian evidence that this is true?  

So what on earth has happened here? How did an article about the complete lack of evidence for requiring a higher degree become the major plank for the APS’ insistence that only those with Master’s degrees and above can work with severely ill clients? We put it down to the common habit of reading an article title and hoping that it supports your point of view. Stein and Lambert’s article is entitled:

Graduate Training in Psychotherapy: Are Therapy Outcomes Enhanced?

The clinical college are hoping that the answer will be ‘Yes’ and haven’t bothered to read on and find that the answer is NO.


Stein, D M and Lambert M J. (1995). Graduate Training in Psychotherapy: Are Therapy Outcomes Enhanced? J. Cons. & Clin. Psychol., 63, (2), pp 182 – 196.