Summer Meeting 2018

At the Summer Meeting, we tried to pull together key aspects of this year’s topic: ‘Marking criteria, rubrics and grading scales used to assess speaking and writing in a foreign language’. We started by recalling the main ideas that emerged from previous meetings and then we stimulated further our reflection with a presentation by Dr Kamilah Jooganah, who shared her experience and insights on the implementation of Grade-Based Assessment at Nottingham Trent University (click on the title to see the slides):
The transition to grade-based marking for assessing student work at university: Institutional change and challenge, Kamilah Jooganah, CQSD
The use of a percentage scale to mark student assessment at university is widespread across the Higher Education sector. Notwithstanding this, it has been argued that the use of numbers to make qualitative judgements about student work is based on flawed assumptions and reveals little in terms of student learning (Dalzeil, 1998; Rust, 2011; Yorke, 2011). In other words, the use of percentages is not fit-for-purpose and what is needed, as others would argue (e.g. Dalzeil, 1998), is a cultural shift in how we assess student performance. However, when introducing an improved assessment tool across an institution, contradictions within the tool itself can prevent this cultural and conceptual shift.
This paper discusses the introduction of Grade-Based Assessment (GBA) across a Higher Education Institution. This new assessment tool aimed to enable better and more transparent judgements of student achievement, and effect changes to assessment practices to foreground the enhancement of student learning

From the very first meeting in the autumn term (see ‘Autumn Meeting 2017’ post), it emerged both from some presentations and discussion, that analytic marking systems, although widely used to guarantee transparency in marking and promote self-regulated learning, suffer from some drawbacks. The main one is the difficulty of measuring students’ complex performance on a linear scale, in which equal numerical increments should correspond to equal increments on the underlying elements (i.e., quality of a student’s work). Other issues brought to light were the anchoring of the scale at the top and the bottom of the matrix, especially for lower levels of language development; and the use of terms like ‘adequate’, ‘good’, ‘excellent’ or ‘outstanding’. As the Summer Meeting was the last one of the academic year and the last one to discuss the topic of rubrics for foreign language assessment, we gave ourselves the task of outlining some shared guiding principles that had emerged from the practice and theory presented and discussed at our meetings. Some basic principles, which could guide the development and use of rubrics in our institutions were recorded on flip chart sheets during group discussions at tables:

  • Rubrics should be linked to the learning outcomes of the module in a clear and transparent way
  • Rubrics should be understandable and meaningful to students
  • Assessors should make efforts to share their ‘tacit knowledge’ of standards with students by engaging them in marking exercises and  discussions of rubrics
  • Verbal feedback is effective and appreciated by students, and can help them to understand marking schemes better
  • A more holistic approach to evaluation and marking can be useful. Marking is not a mechanical or a mathematical process
  • Students’ oral or written work can be ranked using a holistic approach before being related to rubrics to arrive at a mark
  • The grading scales of rubrics should be anchored at the upper end to realistic and achievable descriptors
  • Rubrics should be regarded as a ‘work in progress’ that assessors adjust and refine through practice and reflection
  • It is beneficial to compare and discuss marking schemes with colleagues in our own institution and across institutions

A second presentation by Rodney Coombridge helped us to see how the work of the LT CoP this year relates to the wider context of the UoR Curriculum Framework. Rodney’s engaging presentation stimulated our self-reflection and gave us the opportunity to recognise in the Framework many principles that underpin our professional practice. However, we also acknowledged that more could be done to enhance the assessment culture in our institutions and to shift the balance from ‘assessment of learning’ towards ‘assessment for learning’. In the Curriculum Framework, the pedagogical principles relating to assessment are as follows: 
“Assessment is for learning. It is carefully planned across a programme. It contributes directly to learning and skill development; it is authentic, varied and proportionate. There is an appropriate balance between formative and summative assessment, and formative assessment prepares students well for summative assessment. Where possible, students are provided with some choice in assessment methods. Feedback on assessment feeds forward; it is regular, accessible, thorough, and timely”.

For assessment and feedback in relation to the Curriculum Framework see also:
Assessment and the Curriculum Framework
Engage in Feedback 

Spring Meeting 2018

The aim of the Spring Meeting was to discuss some ideas brought up in the presentations at the meeting in October. We discussed some core aspects of the debate around the use of marking criteria, grading scales and rubrics. We started by reading together through a list of core points and then commented on these and shared ideas at each table. These are the points we discussed:

  • Rubrics increase the transparency and reliability of assessment
  • Rubrics foster autonomy and self-regulation in students
  • “(In order) to share standards in higher education, there has been an overemphasis on detailing criteria and levels. Using explicit criteria cannot capture all the different aspects of quality”
  • “Limits to the extent that standards can be articulated explicitly must be recognised” (HEA, 2012. A Marked Improvement. Transforming assessment in HE)
  • Grading complex performance requires professional judgements more than measurement. Grading decisions are holistic (Yorke, 2011).
  • Assessors work backwards: a) holistic judgement, b) awarding of marks to criteria, c) justified grade decision (Bloxham, Boyd & Orr, 2011; Brooks, 2012)
  • Criteria and standards can be better communicated by discussing exemplars (e.g., annotated exemplars can illustrate what a ‘wide and precise’ vocabulary means compared to ‘simple vocabulary’)
  • Comparative judgement (‘which essay is better’?) plays to human strengths by requiring markers to compare two things, rather than make an absolute judgement
  • Comparative judgement reduces teacher workload and can provide high levels of reliability for large sets of essays (e.g., across campuses – UoR and UoRM – or even across universities)
  • Comparative judgement can also promote teacher development. When using comparative judgement, teachers judge a wide range of student work

Autumn Meeting 2017

In 2017-2018, we widened participation by inviting colleagues with an interest in foreign language pedagogy to join us in termly meetings. The purpose of the first meeting, held on 13th November 2017, was to spark a discussion on a topic that is currently highly controversial: marking criteria, rubrics and grading scales used to assess speaking and writing in a foreign language, which is the theme we chose for this year. Three short presentations served as a springboard for some initial discussion, which was carried over to the Spring Term Meeting. Below are brief summaries of the presentations and links to  the slides:

What can I say? An overview of the new GCSE speaking assessment, Sarah Marston, IoE
An insight into how GCSE pupils are graded against the new 1-9 grade criteria in the speaking component of the GCSE.  An explanation of how their spoken language skills are tested, which skills, grammar and vocabulary are they expected to know, which topics are now included and how does the exam link to the National Curriculum.  The presentation also outlines how the current expectations differ to what our current MFL under-graduates experienced and what we should now expect the MFL undergraduates of 2020 to be able to do.  Draft examples of exam specifications and mark schemes are discussed.

What use are band descriptors?, Rob Playfair, IFP, ISLI
Focussing on descriptors for writing assessment, I discuss my experience in both Singapore and on the PSE course at Reading grappling with descriptors to a) help student progress and b) produce reliable scores. After outlining the problems I have faced, I discuss my embryonic thoughts on possible supplements to descriptors, based mainly on the work of Daisy Christodoulou:

‘Standards’ and ‘marking criteria’: honest attempts at impossible tasks? , Rita Balestrini, MLES
Despite their limitations, analytic marking systems have become widely accepted in Higher Education. Marking schemes that make use of rubrics are commonly used to increase the transparency of the assessment process and help students to understand what is expected from them. Unfortunately, neither is necessarily the case. Are we perhaps asking too much of marking criteria? How can we enhance the process of assessing writing and speaking skills in foreign languages? In this presentation, I draw on recent literature in the area of assessment and feedback in higher education to discuss these questions.