Spring Meeting 2019

At the Spring meeting we took forward our work by systematically examining the descriptors of ‘scholarship’ found in two different sets of role profiles: the UoR Academic and Research role profiles and those included in the Teaching and Scholarship career path of the national library of Academic role profiles. Critical reflection and discussion of these descriptors helped us to focus on what we see as key aspects of any possible definition of ‘scholarship’, not limited to Language Teaching and Learning.
 
It was generally agreed that scholarship activities should be considered essential at all stages of career development and should not be associated only with higher levels of teaching experience and expertise. Some colleagues identified three basic elements of scholarship which can be expected of all teachers from the outset of their careers: the ability to draw on relevant academic knowledge and research, a reflective approach towards one’s own pedagogic practice, and a responsible attitude towards one’s own learning. This led to considering:
1) the potential overlap of ‘Continuing Professional Development’ activities with ‘scholarship activities’ and 2) the opposition between ‘consulting’ and ‘producing’ scholarship and research to inform teaching.
 
The boundary between scholarship and research was discussed. It was pointed out that often the descriptors for scholarship are indistinguishable from those for research, for example, when they indicate the expectation to conduct pedagogic research, present findings at conferences and publish in academic journals. However, it was noted that pedagogic research that does not count towards the Research Excellence Framework is not usually incentivised and supported in the same way as disciplinary research, for example by providing mentoring schemes or by allocating adequate time in workloads.
 
Various ideas to facilitate the development of research skills were suggested. One option would be to offer teaching focused academics the chance to be part of a group aiming to develop knowledge of research methods and approaches; to provide information about opportunities for research funding; and to support the grant application process.
 
Another option suggested by some participants was to set up a ‘buddy system’, for example, pairing a research-intensive member of staff with a teaching-focused member of staff for a defined period of time. It was felt that this collaboration could help bridge the gap between ‘research’ and ‘practice’ and ‘theory’ and ‘application of theory’. A researcher in language learning and teaching would bring knowledge of current research trends, gaps and needs along with experience in securing grants. The teaching-focused academic would have a direct knowledge of learning dynamics and challenges in Higher Education classroom settings, with direct access to gathering data. They would also be in a good position to suggest potential research questions related to language learning in the HE context. However, the pairing of a researcher into a foreign culture with a language teaching-focused academic could also offer relevant and interesting collaborations.
 
Our discussion at the Spring meeting can be viewed as an effort to respond to the first of the ‘Recommendations for institutions seeking to recognise the Scholarship of Teaching and Learning (SoTL)’ in the HEA study, which we set as our point of departure at the Autumn Meeting 2018 (see previous post): ‘SoTL needs to be discussed and made explicit as a concept to generate some  institutional consensus on its usefulness to enhance practices’ (Definining and supporting the Scholarship of Teaching and Learning: A sector-wide study. Executive summary, 2016: 8).

Autumn Meeting 2018

The first meeting of the academic year 2018-19 was devoted to introducing the new theme we chose to focus on, ‘The Scholarship of Language Learning and Teaching and its influence on students’ learning’, and to starting a preliminary exchange of ideas on what ‘scholarship’ means for us.

The notion of Scholarship of Teaching and Learning (SoTL) has evolved in the last decades, but it still lacks clarity and there is no strong agreement on its definition. There is also the need to reflect on different levels of scholarship within discipline-specific contexts. Although at first glance this topic might seem too ‘theoretical’ to be the focus for a LT CoP, we decided that in fact it was relevant because whether or not our practice is reflective and informed by theory and ‘research’, has a direct impact on students’ learning.

With this in mind, we started with group discussion at tables on what we mean by ‘scholarship’ in general and in relation to our local contexts, before considering a study by the HEA on the SoTL:

Defining and supporting the Scholarship of Teaching and Learning (SoTL): A sector-wide study (HEA, 2016) 

After some reflection, we decided to make the first recommendation of the study the focus of our activities this year:

‘SoTL needs to be discussed and made explicit as a concept to generate some institutional consensus on its usefulness to enhance practices’ (Executive summary, p. 8).

We also realised that some of the findings of the HEA study were particularly relevant for our professional practice and so we used them to generate initial discussion. These are listed below:

  • Most institutions, regardless of their mission group, associate SoTL with publications and ‘REF-able’ profiles (ibid.: p.5)
  • There is a tendency to employ assessment criteria that are associated with research (especially research that is disseminated in peer-reviewed journals) (ibid.: p. 5) 
  • The concept of student involvement and partnership in SoTL is especially challenging for students, depending on social and cultural background and capital (ibid.: p.6)
  • The definition of SoTL is also problematic for students, particularly as ‘scholarship’ is often associated with bursaries (ibid.: p.6)

We also considered the section on the review of recent SoTL literature to see what the major areas of interest are and how these have shifted. In the course of time the focus has been on:

  • Individual practice
  • Enhancing student learning
  • Importance of peer-review and building communities of practice in SoTL inquiry
  • How SoTL works within different disciplines
  • Promotion of research-informed teaching
  • Promotion of undergraduate research, and student engagement (SE) in disciplinary or SoTL research
  • SoTL as an institutional tool for strategically developing excellence, and linking to career planning, promotion and recognition                                                                        (ibid.: p. 4)

This work will be taken forward at the Spring Meeting.

Summer Meeting 2018

At the Summer Meeting, we tried to pull together key aspects of this year’s topic: ‘Marking criteria, rubrics and grading scales used to assess speaking and writing in a foreign language’. We started by recalling the main ideas that emerged from previous meetings and then we stimulated further our reflection with a presentation by Dr Kamilah Jooganah, who shared her experience and insights on the implementation of Grade-Based Assessment at Nottingham Trent University (click on the title to see the slides):
The transition to grade-based marking for assessing student work at university: Institutional change and challenge, Kamilah Jooganah, CQSD
The use of a percentage scale to mark student assessment at university is widespread across the Higher Education sector. Notwithstanding this, it has been argued that the use of numbers to make qualitative judgements about student work is based on flawed assumptions and reveals little in terms of student learning (Dalzeil, 1998; Rust, 2011; Yorke, 2011). In other words, the use of percentages is not fit-for-purpose and what is needed, as others would argue (e.g. Dalzeil, 1998), is a cultural shift in how we assess student performance. However, when introducing an improved assessment tool across an institution, contradictions within the tool itself can prevent this cultural and conceptual shift.
This paper discusses the introduction of Grade-Based Assessment (GBA) across a Higher Education Institution. This new assessment tool aimed to enable better and more transparent judgements of student achievement, and effect changes to assessment practices to foreground the enhancement of student learning

From the very first meeting in the autumn term (see ‘Autumn Meeting 2017’ post), it emerged both from some presentations and discussion, that analytic marking systems, although widely used to guarantee transparency in marking and promote self-regulated learning, suffer from some drawbacks. The main one is the difficulty of measuring students’ complex performance on a linear scale, in which equal numerical increments should correspond to equal increments on the underlying elements (i.e., quality of a student’s work). Other issues brought to light were the anchoring of the scale at the top and the bottom of the matrix, especially for lower levels of language development; and the use of terms like ‘adequate’, ‘good’, ‘excellent’ or ‘outstanding’. As the Summer Meeting was the last one of the academic year and the last one to discuss the topic of rubrics for foreign language assessment, we gave ourselves the task of outlining some shared guiding principles that had emerged from the practice and theory presented and discussed at our meetings. Some basic principles, which could guide the development and use of rubrics in our institutions were recorded on flip chart sheets during group discussions at tables:

  • Rubrics should be linked to the learning outcomes of the module in a clear and transparent way
  • Rubrics should be understandable and meaningful to students
  • Assessors should make efforts to share their ‘tacit knowledge’ of standards with students by engaging them in marking exercises and  discussions of rubrics
  • Verbal feedback is effective and appreciated by students, and can help them to understand marking schemes better
  • A more holistic approach to evaluation and marking can be useful. Marking is not a mechanical or a mathematical process
  • Students’ oral or written work can be ranked using a holistic approach before being related to rubrics to arrive at a mark
  • The grading scales of rubrics should be anchored at the upper end to realistic and achievable descriptors
  • Rubrics should be regarded as a ‘work in progress’ that assessors adjust and refine through practice and reflection
  • It is beneficial to compare and discuss marking schemes with colleagues in our own institution and across institutions

A second presentation by Rodney Coombridge helped us to see how the work of the LT CoP this year relates to the wider context of the UoR Curriculum Framework. Rodney’s engaging presentation stimulated our self-reflection and gave us the opportunity to recognise in the Framework many principles that underpin our professional practice. However, we also acknowledged that more could be done to enhance the assessment culture in our institutions and to shift the balance from ‘assessment of learning’ towards ‘assessment for learning’. In the Curriculum Framework, the pedagogical principles relating to assessment are as follows: 
“Assessment is for learning. It is carefully planned across a programme. It contributes directly to learning and skill development; it is authentic, varied and proportionate. There is an appropriate balance between formative and summative assessment, and formative assessment prepares students well for summative assessment. Where possible, students are provided with some choice in assessment methods. Feedback on assessment feeds forward; it is regular, accessible, thorough, and timely”.

For assessment and feedback in relation to the Curriculum Framework see also:
Assessment and the Curriculum Framework
Engage in Feedback 

Spring Meeting 2018

The aim of the Spring Meeting was to discuss some ideas brought up in the presentations at the meeting in October. We discussed some core aspects of the debate around the use of marking criteria, grading scales and rubrics. We started by reading together through a list of core points and then commented on these and shared ideas at each table. These are the points we discussed:

  • Rubrics increase the transparency and reliability of assessment
  • Rubrics foster autonomy and self-regulation in students
  • “(In order) to share standards in higher education, there has been an overemphasis on detailing criteria and levels. Using explicit criteria cannot capture all the different aspects of quality”
  • “Limits to the extent that standards can be articulated explicitly must be recognised” (HEA, 2012. A Marked Improvement. Transforming assessment in HE)
  • Grading complex performance requires professional judgements more than measurement. Grading decisions are holistic (Yorke, 2011).
  • Assessors work backwards: a) holistic judgement, b) awarding of marks to criteria, c) justified grade decision (Bloxham, Boyd & Orr, 2011; Brooks, 2012)
  • Criteria and standards can be better communicated by discussing exemplars (e.g., annotated exemplars can illustrate what a ‘wide and precise’ vocabulary means compared to ‘simple vocabulary’)
  • Comparative judgement (‘which essay is better’?) plays to human strengths by requiring markers to compare two things, rather than make an absolute judgement
  • Comparative judgement reduces teacher workload and can provide high levels of reliability for large sets of essays (e.g., across campuses – UoR and UoRM – or even across universities)
  • Comparative judgement can also promote teacher development. When using comparative judgement, teachers judge a wide range of student work

Autumn Meeting 2017

In 2017-2018, we widened participation by inviting colleagues with an interest in foreign language pedagogy to join us in termly meetings. The purpose of the first meeting, held on 13th November 2017, was to spark a discussion on a topic that is currently highly controversial: marking criteria, rubrics and grading scales used to assess speaking and writing in a foreign language, which is the theme we chose for this year. Three short presentations served as a springboard for some initial discussion, which was carried over to the Spring Term Meeting. Below are brief summaries of the presentations and links to  the slides:

What can I say? An overview of the new GCSE speaking assessment, Sarah Marston, IoE
An insight into how GCSE pupils are graded against the new 1-9 grade criteria in the speaking component of the GCSE.  An explanation of how their spoken language skills are tested, which skills, grammar and vocabulary are they expected to know, which topics are now included and how does the exam link to the National Curriculum.  The presentation also outlines how the current expectations differ to what our current MFL under-graduates experienced and what we should now expect the MFL undergraduates of 2020 to be able to do.  Draft examples of exam specifications and mark schemes are discussed.

What use are band descriptors?, Rob Playfair, IFP, ISLI
Focussing on descriptors for writing assessment, I discuss my experience in both Singapore and on the PSE course at Reading grappling with descriptors to a) help student progress and b) produce reliable scores. After outlining the problems I have faced, I discuss my embryonic thoughts on possible supplements to descriptors, based mainly on the work of Daisy Christodoulou: https://thewingtoheaven.wordpress.com/2017/01/03/making-good-progress-the-future-of-assessment-for-learning/.

‘Standards’ and ‘marking criteria’: honest attempts at impossible tasks? , Rita Balestrini, MLES
Despite their limitations, analytic marking systems have become widely accepted in Higher Education. Marking schemes that make use of rubrics are commonly used to increase the transparency of the assessment process and help students to understand what is expected from them. Unfortunately, neither is necessarily the case. Are we perhaps asking too much of marking criteria? How can we enhance the process of assessing writing and speaking skills in foreign languages? In this presentation, I draw on recent literature in the area of assessment and feedback in higher education to discuss these questions.