I’ve been a little obsessed with Corwin’s Visible Learning MetaX since Matthew Mays shared it at a conference last month. I’ve been skeptical of Hattie’s work in the past, and to some extent, I still am. Recall that Hattie’s work centers around distilling all of the research around education, and analyzing meta-analyses to draw conclusions about what works in education. When you keep summarizing and distilling research, you lose a lot of the nuance and detail. That can lead to oversimplification and conclusions that don’t necessarily apply in every context.

You know this from looking at Google maps. If you zoom out, you can see a much bigger area. But you lose the details. In some contexts, that’s good. But it can also get you into trouble.

The Visible Learning tool tries to combat this by introducing a couple metrics for filtering the Hattie data. You can filter by the number of studies included, as well as the total number of students in those studies. While it doesn’t address the structure of the studies themselves, it at least lets you know how much research there is to back up a particular conclusion. For each influence, an overall confidence rating (1-5) is included that takes this data into account.

The site also allows you to filter on effect size. Effect sizes vary from -1.0 to 1.4. Negative numbers mean a negative effect. Generally, a strategy or practice is considered to have a significant effect if the effect size is 0.40 or grater. The Visible Learning tool translates this in to a number of impact categories, from “potential to considerably accelerate” to “likely to have a negative impact.” That helps people using the data get beyond the numbers and skip right to the conclusions.

A normal person, then, would look for an effect size higher than 0.4, with more than 250 studies looking at a 200,000 or more students. In all this data, what has the biggest impact on student achievement?

You can look that up if you want. I’m not a normal person. I immediately went the other way. What are the factors that have the strongest negative impact on learning? What are the things that we should work really hard to STOP doing?

I looked for a negative effect size, and found 30 factors that negatively impact student learning. Then, because I’m a tech person, I filtered that to just show the Technology domain. Are there any technology factors that have a negative impact on student learning? Why yes. Yes, there are:

• Presence of Mobile Phones has an effect size of -0.34.
• Screen Time has an effect size of -0.29.
• FaceTime and Social Media has an effect size of -0.07.

This data would suggest that limiting student access to mobile phones, social media, and devices in general should positively impact student learning. That doesn’t look good for 1:1 programs, online learning strategies, and goals of ubiquitous access to technology for learning.

But looking at the confidence levels, a different picture emerges. The confidence levels for these factors are all 2 or 3. The body of research includes 1,850 meta-analyses comprising more than 108,000 studies looking at more than 300 million students. But these conclusions are based on a very small number of students in only a handful of studies. So while it would be fair to say “there is some evidence to suggest that students having mobile phones and no limits on screen time are less successful than those without these distractions,” that’s a far cry from “we should ban cell phones.”

So what are the negative influences on student academic success? What are the things we should be working hard to eliminate? Here are the biggest ones, with a confidence level of 4 or 5:

• Illness (-0.44): Chronic illnesses, such as iodine deficiency or asthma, or chronic physical diseases, such as cerebral palsy and spina bifida. Also includes diabetes, asthma, brain injury, cleft lip, or sickle cell disease.
• Student Anxiety (-0.36): An emotional state or trait that is variably related to academic performance, potentially influencing that performance positively or negatively. The most critical aspect is the coping strategies that children can be taught to deal with anxiety.
• Mobility (-0.30): School mobility refers to the frequency of such moves among students in a particular classroom, school, or district. This includes those with parent(s) in the military, and many forms of itineracy.
• Surface Motivation and Approach (-0.13): A student’s surface motivations as he or she approaches learning can be contrasted with deep, intrinsic motivations. For instance, a student might be motivated to learn to achieve a particular grade in order to participate in a sports program, to gain sufficient facts and content to pass a test, or to look good in front of their peers.

So how do we try to reduce these? Working in schools, there’s not a lot we can do about illness or mobility. We can try to limit the effects that those factors have on students. We can be purposeful about welcoming and caring for new students, and making the transitions easier. We can be proactive about our approaches to chronic illness, and the inevitable accommodations and absenteeism that go along with those.

But there’s a lot we can do with the other two. Anxiety is an enormous problem, as our students pick up on the high expectations we have for them, and the pressure for perfection takes root. While I wouldn’t advocate for lowering standards or eliminating accountability, creating a culture of discovery, experimentation, and creativity can help combat the idea that there is one best answer, and our students are competing to find it first.

And that brings us to surface motivation. Its effect is much smaller than the others, but it’s still there. Why are your students doing their work? Looking back to Schlechty, students have three reasons for doing the things we ask them to do. They do their work to please a teacher that they have a strong relationship with. They do their work because they want the grade that is promised for being compliant. Or they do their work because they’re interested in the work itself. The more we can get them into this last category, the more engaged they’re going to be. And there’s some evidence to suggest that better learning outcomes will result.

That’s not to say that there are no technology factors. But one of the things that has become incredibly apparent in the last two years is that school, more than anything else, is about taking care of people. Technology is helpful, but it’s not the center of learning. Let’s focus more on the factors that can make a big difference in student academic success.