The recruitment of participants in a fair and responsible way is a specialist activity, and is led by a team at AlphaPlus who also support recruitment of participants for training and evaluation projects.  Examples of work over the last 12 months, include:

• pre-testing of high stakes assessments for primary and secondary school learners in the UK
• pre-testing adaptive assessments for the national assessment programme in Wales
• piloting assessments for a new secular Moral Education programme for school pupils in the UAE

This article will focus on the challenges encountered when delivering large-scale trials of new assessment materials in schools.

Purpose of trialling

What is the purpose of a ‘trial’ in the world of assessment? Principally, it is to test new assessment materials in a representative and secure environment, and to produce trustworthy data to help ensure that the tests are valid and reliable. This stage is “where the rubber hits the road” – where the heavy investment of time and money in test creation is proved to be effective (or not), and to underpin essential pre-publication adjustments and calibration.  The evidence from trials is also increasingly shared with educators who demand to know how the tests they use are being created. In order to build valid and reliable assessments, it is vital to pre-test the materials with a sufficiently large and balanced cohort of learners (typically up to 1000 learners, from diverse backgrounds, attempting every question), so that the data can be used for revisions to the item bank before tests go live. 

Communicating with schools

In order to deliver successful trials, there are typically some challenges to contend with. Firstly, it is crucial to recruit sufficient schools and learners to achieve a valid and reliable sample, and this often includes schools of certain types – large and small, private and public, rural and urban, etc.   Whatever the sampling framework, recruiting schools entails engaging with school contacts (Headteachers/Assistant Heads; Heads of Numeracy/Literacy; Digital Champions), persuading them to participate, and ensuring they receive clear guidance materials to run trials effectively and with a minimum of disruption to their busy schedules.   Securing the school leaders’ confidence that we’re trustworthy, businesslike, understand their challenges, and are focused on minimising their hassle is critical, and our team’s specialist educational knowledge is key to this.

We offer ongoing support to schools throughout the trialling process, with an emphasis on treating all participants in an ethical and respectful way. It is important to emphasize the benefits to a school of participating in a given trial, both as a diagnostic tool for staff to measure learners’ progress, and to help learners and teachers become familiar with style and content of assessments. In some cases, participation may be incentivised (or occasionally, such as with some national tests, mandatory).  

We use a bespoke CRM (Customer Relationship Management) database, which contains details of all schools and colleges in the UK. This is constantly updated with details of the key staff members and nominated contacts within institutions, with a record of each point of contact. The trialling team can therefore contact specific individuals, while complying with our ethics policy by not bombarding a school with requests. The CRM database is a major asset, alongside the strong interpersonal (and linguistic) skills of the Trials Team in building trust and credibility with school stakeholders across multiple projects.

After a school has been recruited for a trial, we offer ongoing support throughout the process, working with nominated members of staff to ensure that the trial runs smoothly. This often involves the provision of a helpdesk to deal with queries and concerns throughout the trialling window.

Administrators

The trialling process may be administered either by a teacher-administrator within a school, or by an external Test Administrator provided by AlphaPlus. Many higher stakes trials require an external administrator to ensure a fair and secure testing environment, so we must ensure that a well-qualified pool of DBS-certified Test Administrators (and markers) are recruited and trained to follow protocols.  

This is where the AlphaPlus Associate Database comes into play, containing many experienced Test Administrators who have delivered trials in schools across the UK.

Paper-based or online

If a trial involves paper-based assessments, we have to manage the logistics of printing and distribution, ensuring confidential materials are delivered securely and on time, which entails liaising with professional printing and distribution partners and courier services.

If a trial is to be administered online, we work with schools to ensure that they have appropriate technological infrastructure and provision for all learners. There is a huge variation in the technological capability of schools (broadband connectivity, compatible devices, staff competence), as well as the reliability of testing platforms and web user interfaces.

Other factors

Unsurprisingly, it can be quite a challenge to negotiate all of these requirements while keeping within the prescribed budget! It can also be tough to predict how long a given recruitment campaign will take, as there are many variables, dependencies and project-specific factors, such as the age of learners, language medium, geographical or demographic differences, as well as considering the type of trial that is being undertaken, and what exactly is required from schools.

The current crisis and school closures have meant that all trials scheduled during the summer term have been postponed. This may present some interesting opportunities in terms of running online trials during lockdown, initially on a small scale. There will of course be challenges to ensure that a sufficient volume of valid and reliable data is produced; we will work with schools to provide options in these unprecedented times!

To learn more about AlphaPlus’ trialling capabilities, please email us at .