Text mapping | An alternative approach to designing listening tests

Designing listening tests Rita Green.jpg

Earlier this year, I attended a course organised by the Hornby Regional School on designing communicative language assessments in Bangladesh. The course was taught by Dr. Rita Green, from Lancaster University, who is a research leader in the field of language assessments. My biggest take-away from the course was an alternative approach to designing listening tests called text mapping. Text mapping is a technique that Dr. Green conceived as a way of addressing some of the issues test designers experience when they select items from a listening text for a test. In January, when I was at the course, the technique was literally hot off the press and her new book Designing Listening Tests had just been published.

Text mapping questions prevailing practices for selecting items in a sound file for a test. Here’s what I normally do and perhaps you do something similar.  I usually skim the transcript to get a sense of the text and maybe write a gist listening question and then read it again to come up with some listening for specific information questions. I might then listen to the clip to ensure that the accent or speed isn’t too challenging for the target learners.

Dr Green challenged this practice and these two quotes she cited drove the point home:

A transcript and the speech it represents are not the same thing, the original is a richer, contextualized, communicative event.

Lynch, 2010

Life doesn’t come with a tapescript.

Helgesen, 2008

Text mapping attempts to address this gap in how we deal with listening texts. But, before we get on to the actual process, it’s important to distinguish between Specific Information & Important Details (SIID) and Main Ideas and Supporting Details (MISD). I think in teacher training, when we refer to these two listening strategies using the oft-used terms, listening for specific information and listening for detailed understanding, we inadvertently obfuscate what they really are. Dr Green differentiated the two in a way that was very easy to understand.

SIID requires selective listening. We listen for information such as dates, times, places, names, prices, percentages, numbers, measurements, acronyms, addresses, URLs, adjectives and nouns.

MISD requires careful listening. We listen for ideas, examples, reasons, clauses (nouns + verbs), descriptions, explanations, causes, evidence, opinions, conclusions, recommendations and points.

Text mapping can be used for gist, SIID and MISD but I’m going to describe the process for SIID which is what I experienced at the course and subsequently tried out on some unsuspecting colleagues.

1. Prep

Choose a level appropriate audio clip and organise a quiet room with good quality speakers. The text mappers you assemble should not have heard the clip before.  The clip should be short (approximately 30 seconds)

You will need to prepare an Excel sheet with SIID from the clip along with the time stamps of individual items which means you will need to text map the sound file yourself.

2. Briefing 

You need at least three text mappers to ensure validity. A larger pool will increase validity. Explain to the text mappers that they are going to be listening for Specific Information and Important Details. You may need to ICQ this to ensure that all the text mappers are on the same page about what constitutes SIID. SIID is usually not more than one or two words.

3. Listening to the sound file

Play the clip only once and ask the text mapper to listen for SIID. They must not make any notes during this time. When the clip finishes, ask the text mappers to write down SIID. The clip is played only once because Dr. Green suggested that over exposure could lead to too many items being identified.

4. Text mapping 

Ask the text mappers to tell you the SIID they wrote down. Enter these into an Excel worksheet. Poll the group to see who else got this SIID and maintain a tally. If you have variations in the response because they only heard a part of it or misheard it, record these as separate entries. After you’ve finished eliciting these responses, copy paste the time stamps that you’d prepared earlier. You’re likely to get items that are not SIID. A simple test is to check if the information being offered has a noun and verb in which case it is MISD and not SIID.

There may be variations with numerals because in real life we tend to write down numbers immediately or ask for them to be repeated. The test designer will need to keep this mind when selecting an item which has achieved consensus through a number of variations such as Room No. 4045, 4045, 4054, 4055 etc.

The text mappers might not give you items chronologically which is alright. You’ll just need to reorder them so that they appear sequentially in the worksheet.  You’ll also need to be strict about disallowing any  responses that were not written down. I experienced this with my colleagues when several said “Oh I remembered that but I didn’t write it down.”

5. Analysis 

Look at the SIID that a majority of the text mappers were able to identify. These are the items you ought to be testing. However, there are some things to bear in mind. Items at the very beginning of the clip should be disregarded even if you reach consensus (consensus means at least two thirds of the text mappers have identified it) with it because a test candidate may miss it merely because she is orienting herself to the clip in the first few seconds. Additionally, if two items appear within four to six seconds of each others, we ought to test one but not the other. Items should be evenly distributed through the sound file. It’s also important that all items test the same kind of listening behaviour – in this case selective listening for SIID.

6. Writing the test

The next step is to design questions using the items that were identified.

Reflections on text mapping

Here’s one that a colleague and I worked on with a sound file on making a hotel reservation. By text mapping a sound file, you have a systematic approach for identifying what you ought to test as we did with this file. The fact that you are listening to the file as opposed to reading a transcript facilitates the selection of  more authentic items  i.e., that reflect how we receive and process information in real situations. Selecting items from a transcript (and this often happens with me) may result in the testing of obscure items which we may not even register in a real life context.

Text mapping.png

When we ran this exercise with a group of our colleagues, we faced some resistance to the concept. The main bone of contention was that we were testing memory instead of listening skills. I think the clip we selected (at 2 min 10 seconds) was far too long. I recall Dr. Green using really short clips with us (around 20-30 seconds). In a Google Preview of her book, I also recall seeing something about chunking the clip for MISD and allowing text mappers to make notes while listening for SIID with longer clips. Unfortunately, those chapters are no longer available online.

However, our colleagues came around when they saw the extent to which there was consensus for the items outlined in yellow in the preceding table and interestingly this coincided with an earlier round of text mapping with another group of text mappers.

I’m still a little uncertain about the relationship between the text mappers who are selected and the items that are identified through consensus. Text mapping as a process is designed not just for test designers but also to empower teachers to work collaboratively to design meaningful tests.  Therefore, wouldn’t the items selected depend on the language proficiency level of the text mappers? I suspect that in a monolingual English-speaking environment, the results of text mapping may be different than one where English is not the L1 like I experienced in Bangladesh. Further, what kind of impact does this have on item selection from the learner’s perspective, taking into consideration their own language proficiency. While theoretically, a sound file at B1 should have all of its items at B1 but in reality, this may not be the case.

These unanswered questions not withstanding, text mapping is a useful alternative to the somewhat random way in which listening tests are currently constructed. If you try out text mapping, do let me know about your experiences in the comments section.

Dr Rita Green.jpg

No prizes for guessing who in this triad is Dr. Green!

References

  • Green, S. Designing Listening Tests: A Practical Approach. Palgrave Macmillan: 2017.
  • Helgesen in Wilson, J. J. How to teach listening. Pearson: 2008.
  • Lynch, D. Teaching Second Language Listening. OUP: 2010.

Many thanks to Azania Thomas for creating the text mapping sheet that I’ve used in this blog.

Dr. Green’s book is unfortunately really expensive (as interesting ELT books tend to be). You can read a preview here. It includes some relevant chapters on text mapping for gist and issues with listening texts and working with authentic sound files.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s