AlphaPlus’ Director of Research, Andrew Boyle, shares his thoughts on working with professional bodies.

AlphaPlus has a fair claim to be the leading independent assessment agency in the United Kingdom. Our team includes a range of experts in the field, and we carry out projects across the whole spectrum of educational assessment, from primary school tests to postgraduate professional examinations.

Assessment work in schools’ examinations and national tests is well known. There are UK awarding organisations that do large amounts of insightful and cutting-edge research. The vocational qualifications sector also has a long tradition and a clear understanding of the rationale for this particular style of assessment.

The practices and concerns of professional assessment are rather less well known, certainly outside the confines of specialists in the field. Despite this, the professional assessment sector is vibrant and has to deal with a range of substantial challenges. AlphaPlus works with a considerable number of organisations in this sector and does some interesting and worthwhile work.

Firstly, examinations authorities need to set their pass marks accurately, work out who has passed or failed, usually according to broad performance criteria, and issue accurate results lists to their candidates. AlphaPlus supports a number of professional awarding bodies (especially medical royal colleges) with such results determination work. Our highly skilled statisticians use the statistical programming environment R to derive exams results for clients. Using a programmed approach such as this, we can provide results accurately, promptly and with the sophisticated analyses that clients require. It is generally preferable to use a programmed approach such as this, as precision and replicability of results are much enhanced compared to simply using drop down menus in an ad hoc manner.

If we are setting pass marks, we are talking about standards setting. There are many esoteric (and sometimes slightly bizarre) standards setting methods. AlphaPlus advises professional examinations agencies about the best standards setting method for their particular context, and the particular type of questions in their examinations. We carry out standards setting analysis for clients – both advising on best practice, and doing the calculations for methods such as Angoff, borderline regression and so on. We also work with bodies to move from standards setting to standards maintaining; we say that, the first time you run a new exam, you do standards setting, but on subsequent occasions, you link back to that initially set standard using statistical techniques. We are experts at Rasch modelling and linking designs that are pre-requisites for such exercises.

As well as the detailed psychometrics, we carry out a varied range of other work for professional examinations agencies. For example, if a medical royal college is considering moving its examinations to onscreen, we can advise. Such consultancy might involve helping the agency to set up a robust procurement process. We might also help the body to understand how to use its new software to generate the types of assessment tasks that it requires.

We also do some diverse evaluations into professional training and qualifications systems. We have worked with professional bodies to develop multiple regression models; this work has allowed them to understand any background factors that appear to predict achievement in their exams. This analysis provides such bodies with an objective view as to whether they are fulfilling their obligations under equalities legislation.

Our experience is not limited to medical bodies; we have also carried out evaluations in the legal sector. We advised the Solicitors Regulation Authority (SRA) as they set up their new Solicitors Qualifying Examination. Several of our reports have been published on SRA’s website. SRA benefitted from the advice of senior assessment experts as they considered the complex issues involved in a major innovation of this sort. Currently, we are working with the Bar Standards Board to evaluate their Future Bar Training (FBT) system.

So, working in professional assessment is rewarding and interesting. The levels of technicality are high, and direct and indirect clients can be demanding (candidates in professional exams will tell you if they think their results are wrong!). But the work is worthwhile; UK professions are strong international ‘brands’ – many overseas professionals are keen to seek UK certification. AlphaPlus’ work as a centre of assessment excellence enhances professional bodies’ ability to be credible and defensible in this vibrant environment.