Second Evaluation of the Pirkis et al., 2011 Study

Following constant criticism by clinical psychologists of the Pirkis, et al. (2011) Evaluation of the Better Access initiative which found no differences in treatment outcomes between clinically endorsed and generalist psychologists, Reform APS requested a second evaluation of the study from Dr Felicity Allen, who has a strong history of working in the research sector and is a former board member of the APS.  Dr Allen was a senior academic at Monash University for 20 years and specialised in teaching methodology and multivariate statistics.

The Pirkis, et al. study was previously evaluated by Professor Mark B. Andersen, who found it to be sound and robust. Prof. Anderson also conducted post hoc analyses of the data presented in Pirkis, et al. and was astounded to find no significant differences in outcome between clinical and non-clinical psychologists. The small differences in outcomes with the DASS and the K-10 are so small as to be meaningless and within the margin of measurement error, showing further support for the study.

Dr Allen’s recent evaluation shown below also found that the Pirkis, et al. study was very strong using a robust methodology which shows that psychological intervention under the Better Access scheme offers excellent chances for patients to improve. It shows that there were virtually no differences between the two types of psychology clinicians in terms of their outcomes on the chosen measures.

Evaluation of a study: F.C.L Allen, PhD. M.A.P.S.

I was approached by RAPS to give my professional opinion of a study of the impact of treatment under Better Access by three groups of treatment providers.

Overall comment: This is an excellent study using a strong methodology which shows that psychological intervention under the Better Access scheme offers excellent chances for patients to improve. While the authors did not test this directly, sufficient information is provided to show that there were virtually no differences between the two types of psychology clinicians in terms of their outcomes on the chosen measures.

The study:  Pirkis, J., Ftanaou, M., Williamson, M., Machlin, A., Spittal, M.J., Bassilios, M., Harris, B. (2011). Australia’s Better Access initiative: An evaluation. The Australian and New Zealand Journal of Psychiatry, 45, 726-739.

Characteristics of the study, methodology and analysis: This is a large (for psychology) study containing two samples (2420 highly experienced providers) and 883 patients using Better Access, mainly suffering depression/anxiety (90%+).

Sampling: Stratified random

Methodology: Mixed model, Pre/post-treatment. (Note: No control group is required in this study design)

Response rate of patients: Not available

Response rate of clinicians: Clinical psychologists (8%), Registered psychologists (8%), GPs (3%)

Measures: K-10, DASS-21.

Major outcomes: Statistically significant and clinically meaningful improvements in the psychological distress of the patients.

Significant predictors of improvement:

  •  higher pre-treatment K-10 scores (the only significant predictor of improvement for those recruited by clinical psychologists);
  •  in addition to K-10 scores, treatment completion and region predicted improvement for patients recruited by registered psychologists;
  •  in addition to the above three, number of sessions and previous health care predicted improvement among those recruited by GPs.

Omissions: a direct test of the effects of different types of providers on outcomes. This has been partly corrected by post hoc analyses on Table 5. See below (from Anderson, M. Information supplied to me by RAPS).

Clinical psychologists’ clients’ improvements in DASS and K-10 scores (Cohen’s d effect size for dependent means) pre- to post- treatment (from Table 5, p. 734 in Pirkis et al.)

  • DASS Depression 1.04
  • DASS Anxiety .82
  • DASS Stress 1.04
  • K-10 1.22

Generalist psychologists’ clients’ improvements in DASS and K-10 scores (Cohen’s d effect size for dependent means) pre to post treatment

  • DASS Depression 1.00
  • DASS Anxiety .87
  • DASS Stress 1.06
  • K-10 1.20

By conventional behavioural sciences research, all of these effect sizes (for treatment) are in the large to very large range. The small differences between clinical and generalist clients’ outcomes on these psychometrics are so small that they are meaningless.

F.C.L. Allen, PhD, M.A.Ps.S.

RAPS stands by its policy that only a one-tier system can be justified and will be sending these results to the Health Minister, and Shadow Mental Health Minister.