Text mapping | An alternative approach to designing listening tests

Designing listening tests Rita Green.jpg

Earlier this year, I attended a course organised by the Hornby Regional School on designing communicative language assessments in Bangladesh. The course was taught by Dr. Rita Green, from Lancaster University, who is a research leader in the field of language assessments. My biggest take-away from the course was an alternative approach to designing listening tests called text mapping. Text mapping is a technique that Dr. Green conceived as a way of addressing some of the issues test designers experience when they select items from a listening text for a test. In January, when I was at the course, the technique was literally hot off the press and her new book Designing Listening Tests had just been published.

Text mapping questions prevailing practices for selecting items in a sound file for a test. Here’s what I normally do and perhaps you do something similar.  I usually skim the transcript to get a sense of the text and maybe write a gist listening question and then read it again to come up with some listening for specific information questions. I might then listen to the clip to ensure that the accent or speed isn’t too challenging for the target learners.

Dr Green challenged this practice and these two quotes she cited drove the point home:

A transcript and the speech it represents are not the same thing, the original is a richer, contextualized, communicative event.

Lynch, 2010

Life doesn’t come with a tapescript.

Helgesen, 2008

Text mapping attempts to address this gap in how we deal with listening texts. But, before we get on to the actual process, it’s important to distinguish between Specific Information & Important Details (SIID) and Main Ideas and Supporting Details (MISD). I think in teacher training, when we refer to these two listening strategies using the oft-used terms, listening for specific information and listening for detailed understanding, we inadvertently obfuscate what they really are. Dr Green differentiated the two in a way that was very easy to understand.

SIID requires selective listening. We listen for information such as dates, times, places, names, prices, percentages, numbers, measurements, acronyms, addresses, URLs, adjectives and nouns.

MISD requires careful listening. We listen for ideas, examples, reasons, clauses (nouns + verbs), descriptions, explanations, causes, evidence, opinions, conclusions, recommendations and points.

Text mapping can be used for gist, SIID and MISD but I’m going to describe the process for SIID which is what I experienced at the course and subsequently tried out on some unsuspecting colleagues.

1. Prep

Choose a level appropriate audio clip and organise a quiet room with good quality speakers. The text mappers you assemble should not have heard the clip before.  The clip should be short (approximately 30 seconds)

You will need to prepare an Excel sheet with SIID from the clip along with the time stamps of individual items which means you will need to text map the sound file yourself.

2. Briefing 

You need at least three text mappers to ensure validity. A larger pool will increase validity. Explain to the text mappers that they are going to be listening for Specific Information and Important Details. You may need to ICQ this to ensure that all the text mappers are on the same page about what constitutes SIID. SIID is usually not more than one or two words.

3. Listening to the sound file

Play the clip only once and ask the text mapper to listen for SIID. They must not make any notes during this time. When the clip finishes, ask the text mappers to write down SIID. The clip is played only once because Dr. Green suggested that over exposure could lead to too many items being identified.

4. Text mapping 

Ask the text mappers to tell you the SIID they wrote down. Enter these into an Excel worksheet. Poll the group to see who else got this SIID and maintain a tally. If you have variations in the response because they only heard a part of it or misheard it, record these as separate entries. After you’ve finished eliciting these responses, copy paste the time stamps that you’d prepared earlier. You’re likely to get items that are not SIID. A simple test is to check if the information being offered has a noun and verb in which case it is MISD and not SIID.

There may be variations with numerals because in real life we tend to write down numbers immediately or ask for them to be repeated. The test designer will need to keep this mind when selecting an item which has achieved consensus through a number of variations such as Room No. 4045, 4045, 4054, 4055 etc.

The text mappers might not give you items chronologically which is alright. You’ll just need to reorder them so that they appear sequentially in the worksheet.  You’ll also need to be strict about disallowing any  responses that were not written down. I experienced this with my colleagues when several said “Oh I remembered that but I didn’t write it down.”

5. Analysis 

Look at the SIID that a majority of the text mappers were able to identify. These are the items you ought to be testing. However, there are some things to bear in mind. Items at the very beginning of the clip should be disregarded even if you reach consensus (consensus means at least two thirds of the text mappers have identified it) with it because a test candidate may miss it merely because she is orienting herself to the clip in the first few seconds. Additionally, if two items appear within four to six seconds of each others, we ought to test one but not the other. Items should be evenly distributed through the sound file. It’s also important that all items test the same kind of listening behaviour – in this case selective listening for SIID.

6. Writing the test

The next step is to design questions using the items that were identified.

Reflections on text mapping

Here’s one that a colleague and I worked on with a sound file on making a hotel reservation. By text mapping a sound file, you have a systematic approach for identifying what you ought to test as we did with this file. The fact that you are listening to the file as opposed to reading a transcript facilitates the selection of  more authentic items  i.e., that reflect how we receive and process information in real situations. Selecting items from a transcript (and this often happens with me) may result in the testing of obscure items which we may not even register in a real life context.

Text mapping.png

When we ran this exercise with a group of our colleagues, we faced some resistance to the concept. The main bone of contention was that we were testing memory instead of listening skills. I think the clip we selected (at 2 min 10 seconds) was far too long. I recall Dr. Green using really short clips with us (around 20-30 seconds). In a Google Preview of her book, I also recall seeing something about chunking the clip for MISD and allowing text mappers to make notes while listening for SIID with longer clips. Unfortunately, those chapters are no longer available online.

However, our colleagues came around when they saw the extent to which there was consensus for the items outlined in yellow in the preceding table and interestingly this coincided with an earlier round of text mapping with another group of text mappers.

I’m still a little uncertain about the relationship between the text mappers who are selected and the items that are identified through consensus. Text mapping as a process is designed not just for test designers but also to empower teachers to work collaboratively to design meaningful tests.  Therefore, wouldn’t the items selected depend on the language proficiency level of the text mappers? I suspect that in a monolingual English-speaking environment, the results of text mapping may be different than one where English is not the L1 like I experienced in Bangladesh. Further, what kind of impact does this have on item selection from the learner’s perspective, taking into consideration their own language proficiency. While theoretically, a sound file at B1 should have all of its items at B1 but in reality, this may not be the case.

These unanswered questions not withstanding, text mapping is a useful alternative to the somewhat random way in which listening tests are currently constructed. If you try out text mapping, do let me know about your experiences in the comments section.

Dr Rita Green.jpg

No prizes for guessing who in this triad is Dr. Green!

References

  • Green, S. Designing Listening Tests: A Practical Approach. Palgrave Macmillan: 2017.
  • Helgesen in Wilson, J. J. How to teach listening. Pearson: 2008.
  • Lynch, D. Teaching Second Language Listening. OUP: 2010.

Many thanks to Azania Thomas for creating the text mapping sheet that I’ve used in this blog.

Dr. Green’s book is unfortunately really expensive (as interesting ELT books tend to be). You can read a preview here. It includes some relevant chapters on text mapping for gist and issues with listening texts and working with authentic sound files.

Empowering teachers through CPD | IATEFL 2017 Plenary summary

Maggioli IATEFL

The full title of this talk is ‘Empowering teachers through continued professional development: frameworks, practices and promises’. The speaker, Gabriel Diaz Maggioli is from Uruguay and delivered this year’s opening plenary. 

Maggioli kicked off his talk with a quote from Dennis Sparks about professional development lacking focus/effectiveness which I suspect comes from Designing Powerful Professional Development for Teachers and Principals. He stated that he wanted to probe what had been going on in CPD in the last 20 years. He felt professional development was oscillating between individual development and the institutional development which is more about advancing the community. He perceived the former as piecemeal and the latter as more systemic. He went on to suggest that top down reforms usually don’t involve the teachers who need to be ‘fixed’ (he used this odd word often, perhaps on purpose) and as a result are not successful and that a better way to think about development and change is a learning community which is a group of individuals who come together because they have some mutual interests.

Maggioli described two diametrically different roles for teachers within professional development programmes: the teacher as a technician who just implements policy and the teacher as a transformative intellectual who propels the development of learning communities. He argued that professional development that was effective drew on targeted professional expertise (based on teacher needs), adopted structured peer support, and provided opportunities for reflecting on why something worked or didn’t work. For these reasons, professional development, he suggested, ought to be done in-house.

Professional development is inquiry oriented learning sustained over time. It requires the the use of tools and protocols that help create coherence, sustain learning and make evidence collection manageable. My favourite quotable quote was when Maggioli suggested that professional development should be done WITH teachers not TO teachers. In his own research, he discovered that teachers listed surfing the web to find ideas and free webinars as their top two professional development channels.  From the institutional perspective, there was only mandated INSETT in the list of responses but these weren’t focused on teacher needs and were often disconnected from the reality of the classroom. Respondents described INSETT as having the following faults: no follow up, too much talking very little doing, too short, a low level, and a lack of access to resources to apply this learning.

Maggioli portrayed traditional professional development as untimely and not tailored to the career stages of teachers because teachers are seen as having to perform a function which explains the manufacturing-style one size fits all approach. He spoke about he doesn’t perceive career stages as a continuum but a loop and that every time we move into a new role or context, we kick-start the loop again. He suggested that teachers need time, resources and support and that any kind of CPD in the absence of these three things was futile.

He then shared some approaches to CPD.

  • Mirror coaching – Teacher-initiated; a peer takes notes during your lesson. You are accessing your own behaviour from someone else’s eyes. Maggioli stated that this wasn’t the same as watching a recording.
  • Co-teaching: You and your co-teacher model behaviours and learn from each other. expert coach – not from  deficit perspective (easier said than done).
  • Expert coaching: The teacher is coached by someone who is acknowledged as an expert in the area that they’re getting feedback on.
  • Study group: A teacher shares something she did in class with her peers. They then ask her questions which records. The group then goes on a coffee break during which time the teacher prepares to answer questions which is what happens when everyone comes back from the break.
  • Collaborative action research: groups of teachers who plan and implement interventions.
  • Exploratory action research
  • Lesson study: This technique apparently comes from Japan.
  • Learning circles: Ad hoc professional development meetings that follow a structured process.

Learning cirlces

  • Mentoring
  • Professional portfolio

These activities sit within a simple but interesting framework for raising teacher awareness and also identifying those within the learning community who can share, coach and/or mentor:

Maggioli finally ended by exhorting the audience to commit to some actions:

While I don’t think there was anything new or revolutionary in what Maggioli was suggesting, it is I suppose food for thought given how often we talk about CPD, extol CPD frameworks and construct CPD plans. To what extent are these frameworks and activities effective and do they adopt a deficit approach to development?

Dennis Sparks’ book is available as a free download for educators from this link.

IATEFL 2017

Go wide & then narrow | Reflections on life, career & the future

Go wide and then narrow.jpg

Go wide and then narrow. 

I know I’ve been a wee bit quiet for a while.  Mark Armstrong was perhaps a little hasty in suggesting I was hyper-prolific.  I’ve had so very much on my plate since the beginning of the year. I came back from a holiday in Cambodia to an insane amount of work. Because I work across so many contexts, projects, and organisations, I was spread very thin; to the point of losing my sanity. I suddenly realised how stupid I’d been. I wasted years at university studying economics instead of linguistics because I’d thought I would need a career that would get me job. And here I was doing a rinse and repeat. I was delivering corporate training that I didn’t enjoy. I suddenly found myself teaching things that made no sense to me (some faff called team dynamics and some other crud about leadership). I was being bullied by the organisations I worked with. And yes it all paid really well but money as you well know isn’t everything.

The breadth of activities was useful in giving me insight into the bigger picture and making my approach more multi-disciplinary. However, ELT is where I want to be. It may seem narrow but this is the field I find most fulfilling.

Go shallow and then deep 

Regardless of what folks might say about ELT practices not being backed by adequate research, this is, in my opinion, perhaps the only field in the wide spectrum of learning-related disciplines that actually attempts to base its practices on research and tries to (truly) consider the learner experience. If you’re cynical about ELT, you ought to see the crud that passes for learning in corporate contexts. I can barely hide my sneer when I see the words ‘centre of excellence’ on a business card. An overwhelming majority of what’s on offer is thoroughly underwhelming: the latest digital learning (crappy presentations converted into elearning), on-the-go mobile learning (same crappy presentations in a mobile format), and engaging instructor-led training (crappy presentations being talked through by a Charlie who ought to have his lips sown).

Although I have worked across different ELT contexts (General English, Business English, ESP, EAP and teacher training), I feel like I’ve barely skimmed the surface. I haven’t done any formal study since my DELTA three years ago. Teacher training and materials writing are areas I want to explore more deeply.  Young learners and refugees are contexts I want to work in. I’d also like to get more involved with professional associations and other ELT collectives and churn out some papers and maybe even an ebook or two. In other words, I gotta dig deeper.

I’ve recently been approved to train CELTA courses as an Assistant Tutor, a move that will help me consolidate my portfolio of activities and facilitate the depth that I’m looking for.

Go fast and then slow

This past year has been frantic. I’ve been spread so thin that I’ve hardly had any time for myself, let alone for people I enjoy spending time with. I didn’t go the theatre, barely got any exercise, didn’t go trekking during the monsoon, didn’t participate in eclectic workshops, didn’t read as much, didn’t learn any new languages, didn’t meet new people … and the list goes on. I even spent my birthday writing training materials, albeit at a beautiful beach side hotel in Goa.

When I quit a full time job, I thought I’d have more time, not less, to do the things I love. It turns out that freelancing, especially when you have niche skills in a market full of people with generic offerings, is just a volley of clients and institutions putting pressure on you to over-commit. As a result, you end up in the perpetual fast lane and half the time you don’t know if you’re coming or going.

I just have a couple more contractual obligations to complete and then I’m taking the first exit. I’m really looking forward to 2017-18 which is going to be filled with meaningful projects, and at least half as much time with family, friends and doing things other than work.

Whereabouts are you now? Are you going wide, shallow and fast? What will it take for you to go narrow, deep and slow?

Image attribution: Dive! | Robbie Sproule | Flickr | CC by 2.0

My very first plenary presentation | Reflections

adi-rajan

Last week I co-presented a talk with a colleague at the Learning for a Sustainable Future – Teacher Conference in Delhi. It was my very first plenary presentation and there were hundreds of attendees. We were presenting our initial findings from piloting a new assessment approach (with behaviour libraries within a smartphone app) in a project we are working on in the south of India.

Here are some quick reflections with presenting to a large audience in this format for the first time.

WWW (What went well)

  • We went after someone who did a fairly high-level talk on the assessment approach  we were piloting. This really helped our presentation because many of the attendees came to me later and said that it wasn’t till they heard our experiences from the field that it all made sense to them.
  • We incorporated a task for the teachers to try their hand at crafting their own criteria, which helped make the session a bit more engaging.
  • We’d done a shorter version of the same presentation for a group of policymakers the previous day which helped us anticipate questions.
  • We constructed our presentation in a way that was of value to both teachers who are familiar with approaches to assessing non-academic skills as well as those who were completely new to the topic.

EBI (Even better if) 

  • We made it less impersonal. I reckon we kept it a bit business-like. We could have throw in some humour and perhaps taken advantage of the fact there were two of us on stage and engaged in a more natural dialogue rather than “you take this slide, I’ll take this one.”
  • We expanded on our experiences and shared more anecdotes because the audience seemed to respond to stories far more than factual information.
  • We made it even more relevant to the audience who were mostly K12 teachers. This was a bit challenging because we were merely reporting from our project which looks at assessment from the perspective of project outcomes rather than tracking progress of individual learners (which is of course what teachers are interested in). It may have been worth exploring how the teachers could have used the same assessment approach with their own learners.

During a subsequent talk, I noticed that the erudite looking woman sitting in front of me had dropped her notepad. When I was passing it back to her, she suddenly realized that I looked familiar and she said “great presentation” and then added “I shouldn’t say this, I’m a teacher after all, but you have a sexy voice.” Nonetheless, I’m dreading the prospect of watching myself (uggggghhhh) whenever the video gets uploaded to YouTube.

A Malvika by any other name (preferably Monica)

Names.jpg

I read this article on how stereotyped ethnic names can sadly be a barrier to workplace entry and was reminded of a course I designed earlier this year. It was for a client who was going to purchase the materials from me. When they reviewed the workbook, they asked me to change all the names to ones that were familiar to people in the Philippines because they were planning on running the program in Manila. So I changed the names to the names of people I worked with on a short stint in the Philippines.

When I resubmitted the materials to my client, they got back to me with a concern that the names would sound too foreign to learners in India because they planned to run the module in both countries. I suggested having two versions. They made noises about standardisation and asked me to incorporate ‘globally acceptable’ names. I tried to put up a fight but I had to finally give in. The final straw was when they told me that they were also planning to launch the program in the US and that the names would need to be globally acceptable to Indians, Americans, Filipinos and anyone else who’d happen to be around.

I changed the names in the text to ones that I kinda thought would be culture and country agnostic (although that’s a fairly erroneous line of thinking in a multicultural, globalised world)

Male names 

  • Omar
  • Jay
  • Ray

Female names

  • Alisha
  • Anita
  • Mira
  • Melita
  • Monica
  • Tanya
  • Teena
  • Tara

I couldn’t come up with any others. I ended up using Jay in four different texts. I was wondering if anyone else has faced a similar situation. Also what names would you add to this ‘globally acceptable’ list?

Image attribution: O inmost wind of living ecstasy… by haRee | Flickr | CC BY-NC 2.0

Pronunciation as protest | A thought experiment

Emotion.jpg

I was intrigued by this recent NY Times article about two newly elected members of the Hong Kong legislature and their anti-China protests during their swearing-in ceremony. What’s particularly fascinating is how both of them pronounced China as /ˈtʃiː.nə/ , and how it instantly infuriated Beijing (this despite all sorts of other apparently anti-Beijing activities happening at this oath-taking event). Sixtus, the young man in this video, later blamed it on his poor English accent, and did so in perfectly fluent English. The Chinese government, it seems, perceives  the aberrant pronunciation as a slur from the time of the Japanese occupation.

 

This is the first time I’m hearing of pronunciation being used to mark protest. I am , however, familiar with the sentiment, because I’ve been doing something similar subconsciously for a while. A lot of Indian place names have been officially renamed over the last two decades to make them sound (allegedly) more Indian. Like a lot of people, I use the old Anglicised pronunciation out of habit, but never with any kind of consistency. I have met language chauvinists who’ve corrected me subtly reformulating my pronunciation or explicitly pointing out my dirty elitist, colonial ways.

I now use the Anglicised pronunciation intentionally, even with people who I reckon it’ll provoke. I think it’s great that place names are pronounced in ways that reflect the culture that shaped it in the first place. But what I take issue with is the empty populism of politicians who fritter away public money that could have been spent on more pressing needs like health, education, sanitation, and hunger – yes hunger – on meaningless name changes and all the associated costs that entails.

So if you live in India, here’s a thought experiment for you. Over the course of a fortnight, keep a record of which of the following you use and in what situations.

Bombay or Mumbai

Madras or Chennai

Poona or Pune

Calcutta or Kolkata

Bangalore or Bengaluru

Cochin or Kochi

At the end of the fortnight, analyse the results. Do you for example use Bombay consistently with your friends but Mumbai at work? Do you (like a lot of people I know) use Bangalore and Cochin all the time but can’t bring yourself to say Madras?  Is it because Chennai is a totally new name and not just a different pronunciation? If you haven’t made a clean break from the old names to the new ones, what’s your reason for favouring some from the old lot and others from the new one? Or are you, like me, using pronunciation as a form of low-level protest?

If you’re not from India, I’m curious about whether you have any parallels in your own culture or region where you feel pressured to pronounce a word in a certain way and the impetus to rebel.

My evolving relationship with coursebooks

Coursebooks.jpg

Everyone seems to be debating coursebooks again or perhaps it’s an issue that never in fact left the spotlight. There seems to be somewhat of a consensus among bloggers that course books are evil and ought to be eschewed. If that’s the case, I’ve succumbed to the dark side.

I left a full time job in early 2014 and started working independently in July 2014 when I’d completed the DELTA. Since 2014, I have been required to use course books mandatorily twice; during the DELTA with New Headway and last year when I taught a semester at a university with New English File. Except for these two occasions, I’ve had autonomy in making decisions about which materials to use on my courses. Have a look at the number of course books I’ve used between 2014 and 2016. Do you see a pattern?

course-books

There is no doubt that bespoke materials drawn from the learner’s context are more effective. This holds good in general contexts and is particularly true of business contexts. So why am I increasingly relying on coursebooks? Unfortunately, most of my clients refuse to pay for the time spent on designing customised materials. They tell me to use what I already have. In the early days, I accommodated this objection and invested unpaid time and effort in producing bespoke materials, naively telling myself that enhanced learner experiences and outcomes justified this small sacrifice I had to make. Small, however, was an understatement. To design a two day workshop, I’d have to spend approximately five days developing the materials. This is actually well below learning & development industry averages which estimate four to six hours of development for each hour of instructor led training (I’m not sure what the parallel ELT figures are).  So, I’d end up spending seven days on this project (which probably doesn’t even take into account the time spent on need analysis and client meetings) but only get paid for two.

I refuse to be enslaved by a cycle of unpaid drudgery. Therefore, until I find a better solution … long live the course book.

Post-script: Funnily enough, what’s stopped me from embracing coursebooks with more fervor is their lack of availability. Adult ELT coursebooks are rarely used in India. The big publishers mostly cater to the K12 segment and I frequently run into situations where the distributor doesn’t have enough copies or has had to sift through warehouses across the country to put an order together.

Image attribution: I think I do by eltpics | Flickr | CC BY-NC 2.0