Ecctis was commissioned to carry out an independent evaluation and benchmarking of the Oxford Test of English Advanced, an English language proficiency test developed by Oxford University Press (OUP). The Oxford Test of English Advanced is an adaptive computer-based test that assesses listening, reading, speaking, and writing skills. The test assesses language from CEFR levels B2-C1 and is delivered and taken online at Approved Test Centres. Test items include input texts, situations, and examples likely to be found in real-life settings. The target demographic is those who wish to attend Further and Higher Education study programmes in English medium education settings or to seek employment.

Upon completion of the Oxford Test of English Advanced, candidates receive a report of a standardised score from a scale of 0 to 170 and a corresponding CEFR level.

Receptive skills (listening and reading) are assessed through an adaptive computer-based system. The system uses an algorithm to select items from a pre-designed item bank and adapts the selection of items according to the student’s successive progression throughout the test. A score is calculated via the algorithm at the end of the receptive skills sections. Productive skills (speaking and writing), on the other hand, are randomised linear tests, and questions are selected at random, rather than through an adaptive algorithm. Once students have completed the productive skills sections of the test, their answers are marked by examiners against a mark scheme designed to align performance to intended CEFR level. The overall score is calculated as an average of the scores obtained in each of the four modules (speaking, writing, reading, and listening), and based on this test takers are awarded the corresponding CEFR level. Test takers who score 110 or below are not awarded a CEFR band but are given an indicative ‘Below B2’.

This study employed Ecctis’ established methodology for the benchmarking of English language proficiency tests, informed by the Council of Europe 2009 Manual1 and the 2020 Companion Volume.2 This encompassed a desk-based review of Oxford Test of English Advanced test specifications, assessment procedures, test items, test-taker samples, and relevant marking criteria and scoring rubrics, as well as an on-site visit to Oxford University Press where the test was taken multiple times under different conditions. Analyses of test items were presented to an internal panel for peer review of the CEFR level findings made.

In order to undertake the evaluation, a number of documents were reviewed, including administration and quality assurance documentation, reading, listening, speaking and writing test papers, assessor handbooks, marking and scoring criteria, and anonymised test-taker samples.

Overall, the quality assurance processes reviewed by Ecctis were sufficient to support the CEFR comparability findings detailed in the key findings of this report.

Overall, Ecctis’ review of test construct, individual assessment tasks, input texts, test-taker samples, and mark schemes found the Oxford Test of English Advanced to provide a sound assessment of English language competency in listening, reading, speaking and writing at CEFR levels B2 to C1.

The listening section of the Oxford Test of English Advanced is approximately 40 minutes in duration and consists of four different parts. The first part consists of discrete tasks requiring the candidate to listen to a short monologue or dialogue and select one out of three text or image options as an answer to a question about the audio. The second task involves a gap-fill where candidates are presented with a text (which is a shortened, rephrased version of the audio), which has gaps that candidates must fill based on what they hear in the audio. The third part includes a long dialogue that emulates experts speaking and discussing different opinions about a topic, and the task involves matching opinions to speakers. Finally, the fourth part consists of multiple-choice questions of three options where candidates listen to short monologues and dialogues and choose the correct answer.

The input audio texts are appropriate for the intended candidate demographic and were found to be suitable for candidates between CEFR levels B2-C1.

The listening items sampled in the Oxford Test of English Advanced targeted a wide range of CEFR oral comprehension competencies, and included: overall oral comprehension, understanding audio media and recordings, understanding conversations between other people, understanding as a member of a live audience, and vocabulary range.

The reading section is approximately 40 minutes and is made up of four parts. The first part consists of multiple-choice questions of three options based on a short input text that includes texts based on real-life situations such as disputes with a landlord, receiving text messages, and reading announcements as well as academic journal extracts and adapted authentic literary extracts. Part two provides three input texts written by three different people, and candidates must associate different stated opinions with the right person. The input text of the third part of the reading test is a longer text, adapted from an authentic source, that has missing sentences. Candidates choose from a selection of sentences, of which there is one extra and which does not fit in the text, and allocate it to the right gap. Finally, part four consists of an input text, also adapted from an authentic source, and candidates must answer a suite of questions related to the text. The reading input texts were found to be appropriate for the target candidate demographic and were found to be suitable for candidates between CEFR levels B2-C1.

The reading items sampled in the Oxford Test of English Advanced targeted a wide range of CEFR reading comprehension competencies and included: overall reading comprehension, reading correspondence, reading for information and argument, identifying cues and inferring, and vocabulary range.

The writing section of the Oxford Test of English Advanced lasts 50 minutes and consists of two parts. In the first part, candidates must write an essay of about 220-280 words based on a provided essay question and prompts for information and arguments that should be included. In the second part, candidates are provided with two texts (a textbook extract and a lecture transcription extract) that explain a specific phenomenon or topic. Candidates must synthesise the main points into a summary of 80-100 words. The writing test is marked against criteria which assess task fulfilment, organisation, grammar, and lexis. The writing tasks and marking criteria were found to assess CEFR levels B2-C1.

The writing items sampled in the Oxford Test of English Advanced targeted a range of CEFR written production skills and included: overall written production, mediating a text, reports and essays, sociolinguistic appropriateness, and grammatical accuracy.

The Oxford Test of English Advanced speaking section includes five tasks and lasts approximately 15 minutes. The first task includes an interview involving introductions and six questions on everyday topics. The second task requires the candidate to leave a voice note responding to a situation that requires diplomacy. The third task is a summary exercise where candidates must summarise the main ideas expressed by two different speakers (provided in audio input). The final task requires candidates to put forward arguments as if in a debate, and then respond to pre-written follow-up questions relating to the debate. The exam involves the candidate receiving computer-based prompts and pre-recorded audio input and does not involve spontaneous interaction with an examiner. The speaking test is marked against a criterion which assesses task fulfilment, pronunciation and fluency, grammar, and lexis.

The speaking tasks and marking criteria were found to assess CEFR levels B2-C1. The speaking items sampled in the Oxford Test of English Advanced targeted a range of CEFR oral production skills and included: overall oral production, mediating a text, addressing audiences, sociolinguistic appropriateness, fluency, sustained monologue: putting a case, and general linguistic range