This talk had a lot of promise but unfortunately didn’t live up to it. I suspect partly because the presentation lacked specificity (perhaps for proprietary reasons) and partly because in actual implementation, the idea of taking a test on a mobile device isn’t revolutionary at all, it just replicates test formats from a computer with some adjustments for mobile UI.
The speaker, Adrian RapRaper and Sean McDonald from ClarityEnglish and TELC explained that this pilot was in response to the challenge of providing a placement test to 2000 students at Asia University in Taiwan. They had to work with a smaller screen size, and move away from true or false and MCQs but ensure reliability and validity. They suggested that new devices enabled new types of interactions and new items which you can’t do on paper including cross-skills testing. They were quite vague about these new test types and only explained one in detail: moving a word into a right place in a sentence, which is a fairly conventional digital activity and not all that uncommon in assessments. They described a reading test where candidates can look at the text and questions by flicking between the two but again this was described quite superficially.
Much of the presentation dealt with the security issues that come with using mobile devices such as the fact that candidates could go into Google if they’re using their own device. They suggested that the two ways of dealing with this is through a time limit and through question type. They claimed that the item type that involves moving a word into the right place in the sentence is fairly cheat proof. I really doubt that – Google can show you sentence patterns with a general search.
The way it works is that the candidates download the assessment app before they come to the test centre. At the centre, they are given a code to access the test. In case of poor connectivity, the app puts answers into a queue similar to how Whatsapp operates but the upload sizes are only about 23kb which is really tiny. The other issues they touched on were more interesting such whether the type of device has any impact on performance, the latest iPhone vs. an older smartphone. They are are collecting data about this in their pilot. They don’t currently offer gap fills or writing item types because of the problems associated with typing: auto-correct and the default language on the device which might not be English.