Is the new Victorian learning difficulties screening assessment any good?14 Replies
From the end of this year, all children starting school in this state will be screened for learning difficulties.
The Minister’s media release says, “…the improved process will be the first step in the early detection of learning difficulties, so that children could get the support they need sooner” and that the policy means “properly supporting students with additional learning needs so they get the opportunity at a great education.” (Does poor grammar in the Minister for Education’s media releases make you feel a bit despairing, too?)
A useful free online document called Selecting Screening Instruments explains why school entry screening is a worthwhile thing to do: “The goal of universal, early reading screening is to identify children at risk of future failure before that failure actually occurs. By doing so, we create the opportunity to intervene early when we are most likely to be more effective and efficient. Therefore, the key to effective screening is maximizing the ability to predict future difficulties”.
What are the features of a good early reading screening test?
A good early reading screening test measures things that matter over time (i.e. is valid and predicts well) and gives consistent results (i.e. is reliable).
It sorts out the kids who are going to be fine from the kids who are going to struggle, without too many false positives (worrying about kids who are actually going to be fine) or false negatives (failing to identify kids who are actually going to struggle).
It has norms which allow any child’s performance to be compared with the typical performance of their peers.
What are the best predictors of reading and writing difficulties?
The best predictors at school entry of difficulty with reading and writing the following year (Grade 1) are phonemic awareness and letter knowledge.
However, the best predictors of persistent and severe reading and writing difficulties (Grade 3 and beyond) are oral vocabulary and rapid automatised naming.
Good screening tests take into account how children’s learning changes over time, and can identify not only the kids likely to take a while to get off the ground, but also the few who are likely to need extra help in the long-term.
High-quality literacy screening tests
There are a number of high-quality standardised tests suitable for use in screening for learning difficulties at school entry, such as these US tests:
- Predictive Assessment of Reading (the best screener, according to Selecting Screening Instruments)
- The RAN/RAS
I find the UK Children’s Test of Nonword Repetition useful too, although it’s not a comprehensive screener. It takes four minutes, has norms for ages 4-8, is simple enough for even preschoolers, and doesn’t disadvantage children whose home language isn’t English or children who’ve hardly seen a book and don’t know what a rhyme is. Poor non-word repetition is quite a good predictor of language and reading difficulties.
Homemade is only good if you know what you’re doing
I’m a bit mystified as to why our education department has chosen not to buy an off-the-shelf, valid, reliable, norm-referenced screening tool or tools for use with Victorian children, and has chosen instead to improve its existing English Online Interview.
Our state government is still smarting from couple of recent, massive DIY technology fiascos: the education department’s Ultranet (see “Corruption-fighting body to investigate botched $180 million Ultranet project for schools”) and our public transport system’s Myki card (see “Outsmarted: Victoria pays the price”). So you’d think they’d be wary of commissioning something new when they could be buying something already known to work well elsewhere.
I can’t find any statistics to back up the education department’s website statement that the existing English Online interview is “a powerful online tool for assessing the English skills of students in Years Prep to 2”.
Until it’s finalised and researched extensively, there won’t be any statistics showing how validly the revised version predicts future reading and/or writing performance, how accurately it pulls out the “worry about” kids and not kids who’ll be fine, or whether it allows us to compare one child’s performance with that of their peers in a meaningful way.
A closer look at the current English Online Interview
Only Victorian education department employees can access the English Online Interview. However, website information indicates that the assessment for school beginners currently involves a child:
- Having a conversation with a teacher, which the teacher evaluates using rubrics.
- Retelling a story, again evaluated via rubric.
- Answering questions about the story just retold. The website video says at least half the questions are easy for most students.
- Writing their name, drawing a picture about the story they’ve just heard and retold, writing about their picture, and then reading what they have written to the teacher. The website video about this task says, “The teacher scores how relevant the student’s writing is, based on what the student says their writing tells you. The teacher also records how easy it is to match what the student has read to what the student says. The teacher then looks at the student’s writing to see if there are any identifiable words. Most children at the start of Prep aren’t writing yet, however there are a very small number who can actually write a recognisable sentence with punctuation”.
- Identifying and generating rhyming words. The Grammar Obsessive in me wasn’t sure whether to laugh or cry when, in the demonstration video, the teacher asks the child, “does bed and shed rhyme?”
- Giving a name or sound for upper and lower case letters of the alphabet. About two-thirds of students are reported to be able to get at least five letters right.
- Matching words with the same first sound (about half the children are said to succeed) and last sound (very few succeed).
- Using a highly repetitive, predictable book e.g. starting at the front, turning pages in order, identifying a word or letter, listening to the teacher read the first two pages then finishing off the story (the video says this may be done by making it up from the pictures if necessary) then answering a comprehension question, assisted by the pictures.
- Attempting to read another, more difficult text (but most children can’t).
As previously said, phonemic awareness and letter knowledge are the things which best predict reading problems in Grade 1, and oral vocabulary and rapid automatised naming are the things which predict problems in Grade 3 or later.
The English Online Interview currently includes tasks requiring phonemic awareness and letter knowledge, but there seems to be no research reported on the tasks used, at least in the public domain.
There aren’t any tasks in the current English Online Interview assessing oral vocabulary or rapid automatised naming, so these would need to be added, but unless they add tasks that have already been researched and found to be valid, reliable and accurate, we won’t know how good they are until this research is done.
Perhaps some of the other tasks on this assessment are useful to teachers for other purposes, though I’m not sure why statistically robust, research-based tools like the CELF-4 Screening test aren’t used to screen oral language, or whether children identified as having language difficulties are routinely referred to a Speech Pathologist. Children with Severe Language Disorders seem to get no extra funding any more, and Speech Pathologists in schools are like hen’s teeth, so however they do it, teachers must find identifying children with significant oral language difficulties rather a thankless task.
I’m not sure why children just starting school are being asked to write a sentence, or read a book that’s too hard for them. Trying to spot gifted and hot-housed kids?
I also wonder how such an assessment might be improved to the point where it meets the criteria for a good school entry learning difficulties screening tool (valid, reliable, accurate, normed). It seems quite subjective and descriptive, and to not even report statistics, let alone attempt to be statistically robust.
I know tests cost money, but not accurately identifying kids with learning difficulties currently costs us a lot more in the long run. After Toupee-gate, our education bureaucracy needs to put some serious work into rebuilding our confidence that it has its spending priorities right.
Getting teachers to administer long assessments which are not as informative or well-targeted as they might be also has an opportunity cost.
Screen-then-intervene or intervene-then-screen?
In the UK, all school beginners are supposed to get explicit, systematic synthetic phonics teaching from the first week of school, then they test to see who’s not catching on after a few months, and provide them with extra teaching.
Children start school with a vast range of language and (pre) literacy skills. Some of them don’t speak English. Some can’t say long sentences. Some aren’t read stories because their parents are illiterate. Others can already read and write at school entry, and engage you in complex, polysyllabic, vast-vocabulary conversation for hours. Lucky them.
In the UK, everyone’s taught how to sound out words and all the main spelling patterns, in small, fast steps. Yes, some children already know a lot about this, but it’s better for them to be given an opportunity to shine than go too fast for some children, leaving them behind and miserable.
Children’s learning of the things they have been taught is then assessed, using tests like the PERA. Children across the country must also do the Phonics Screening Check. Strugglers identified on these tests are referred for additional intervention.
The UK system seems a better way to minimise the singling-out of strugglers, but really I don’t mind which system we use, as long as we abandon our current wait-to-fail system.
If our government wants us to have confidence in the new screening system, it needs to put a lot more information about the screening tool’s validity and reliability, accuracy and norms in the public domain.
Until that’s available, if I were a Principal, I’d be looking for a statistically robust screening tool to help my teachers focus their reading and writing intervention where it’s needed most. I guess it goes without saying that I’d also require all early years and special needs staff to have explicit systematic synthetic phonics training and resources. There’s no point identifying problems you don’t know how to solve.