what data actually matters and are we paying attention to it?

I gave this talk to a small group at the ICOT Conference in 2013.

To post the talk now in 2025 seems a bit odd, but I think it still holds up. It is of it's time, in that I reference the early days of Network for Learning (N4L). I did go on to work to work at N4L 2015-2017, and the final talk I gave in that role was this presentation to staff at AUT.

This is the core of the talk - call it a re-constructed version - but I ended with this small section about DJ Earworm. I've since lost the slide deck, so the images are a selections from the Google Photo library. Most of the links below are via the Web Archive as well - which I guess is another data point in how tenuous our digitized memories are.

Over the past few years, I've been drawn to a David Foster Wallace speech, that he gave to Kenyon College in 2005. It's known simply as "This is Water". He opens with this story:

There are these two young fish swimming along and they happen to meet an older fish swimming the other way, who nods at them and says "Morning, boys. How's the water?"

And the two young fish swim on for a bit, and then eventually one of them looks over at the other and goes "What the hell is water?"

It's a beautiful, almost mournful speech, woven through with analogies and soul, that is direct and clear about what it is that matters, and the power of the individual in controlling or paying attention to what matters.

To quote further:

This, I submit, is the freedom of a real education, of learning how to be well-adjusted. You get to consciously decide what has meaning and what doesn't. You get to decide what to worship...

It's the starting point for my thinking and for this talk today. In my talk today, I'd like to share several examples from my experiences, that have been shaped by the ethos of Wallace's talk.

In particular, what it means to notice data, to pay attention to how data shapes us, and what that means for the choices we might make.

1. Of being aware, and paying attention to what matters.

I'd like to start with the Walt Disney Company. It is a multi-billion dollar global organization, with a portfolio of services and products, ranging from TV channels, publishing, online and interactive media, movie production, numerous parks and resorts, cruise liners, property and financial assets. Their big brands include the Muppets, Lucasfilm, Marvel Comics, ABC and ESPN television stations and the Disneyland resorts.

Disney does user experience very well, it's stated mission is simple: " to make people happy"

But I'd like to talk about the lines.

Because like any modern theme park or event centre, at Disneyland, the line becomes one of the dominant features.

Lines are usually negative experiences.

If you're not aware of lines, the psychology and science behind line design, you'd be surprised at how nuanced and developed a field of study it is.

A New York Times article from August of 2012 discusses this, and on of the key points about line design is that:

" the psychology of queuing is often more important than the statistics of the wait itself,”

Disney pays very close attention to lines - and does a specific number of things that matter.

1. They give people relevant information (wait times) - but they overestimate those wait times, so that guests are "pleasantly surprised" when they get to the ride ahead of schedule.
2. They hide their lines, snaking them around buildings, through trees or buildings. The lines are designed to be invisible, or if not invisible, to at least blend in with the surrounding environment.
3. They build distraction into their lines. You can listen to Buzz Lightyear's pre-flight briefing, pull on ropes while waiting for the Indiana Jones ride, or be entertained by a customs-style droid at the Star Tours experience.
4. They tailor how one is able to engage in the lines. Single Rider allows a single person to bypass the main queue and get to the front to fill up an single seats on a ride. Fastpass, allows guests to book their ride time - then return to get a preferential entry to the queue.

This tailoring of the line experience is not without flaws - namely the perception of fairness for those waiting in the standard queue - but they reflect the ability of Disney to focus on ways to improve and enable the user experience.

In 2010, Disney redesigned their systems, to allow more control of their resort experience and the environment,

Ride capacity is determined in part by analyzing hotel reservations, flight bookings and historic attendance data. Satellites provide minute-by-minute weather analysis. Mobile Magic, a $1.99 app, allows visitors to type in “Sleeping Beauty” and receive directions to where that princess (or at least a costumed stand-in) is signing autographs.

In 2013 Disney is introducing a vacation management system called MyMagic+

Visitors will wear rubber bracelets encoded with credit card information, snapping up corn dogs and Mickey Mouse ears with a tap of the wrist.

It's part of a global project estimated to cost between $800million to $1billion. The aim is to make Disney's theme parks less daunting and more amenable to modern consumer behavior. Disney is betting that happier guests will spend more money.

“If we can enhance the experience, more people will spend more of their leisure time with us,”

Can you see the upsides?

Can you see the negatives?

The company already collects data to use in future sales campaigns, but parts of MyMagic+ will allow Disney for the first time to track guest behavior in minute detail.

Did you buy a balloon?

What attractions did you ride and when?

Did you shake Goofy’s hand, but snub Snow White?

If you fully use MyMagic+, databases will be watching, allowing Disney to refine its offerings and customize its marketing messages.

Disney pay attention to detail, and that's not without the negative sides. However, they pay attention to what matters in and for their business, the data that matters and they act accordingly. Everything is designed and built around meeting that vision, to make people happy. They use real data to monitor and manage those guests, and they have specific plans to act on that data.

Arguably, this all an example of a system that pays attention to the data that matters.

Now lets have a look at a not so positive example.

2. Of paying attention to data - but missing what matters.

I've just become a Dad for the second time. We had a hospital birth, at Wellington hospital and it was suitably professional and well managed. But ... it was an interesting experience for a range of reasons.

I just mentioned the vision of Disneyland, here are the vision and values of the Wellington Hospital, as per their webpage:

And in the maternity ward, here are some examples of the visual environment that surround those who come into the system, and which reflect that vision.

When you stop and consider why the guests are coming in to this system - why are they there?

It's a maternity ward, they're there to give birth, to bring an human being into the world. It's a time filled with stress, hope, fear, sweat, pain, elation and relief. It should be a place that supports those emotions, and makes that experience memorable.

Now of course the act of bringing a human being into the world is amazing.

But I reflected that none of the signage makes one bit of difference to those coming in to the system to use it. It alienates and distracts them at best.

All of these signs are for the benefit of those who work within the system. They're reminders, self-imposed plaudits, and directed AT those who come into the system.

Keep in mind it was my second time, and I remembered where to find ice for our jug of water, and where the bathrooms were, and I was able to walk up and down with my daughter and avoid any of the issues identified. Additionally, I read English - so I had an advantage - although my font nerdery was highly offended.

Remember I talked about guests at Disneyland, that's a specific part of their language - they don't have "customers" - they have guests.

Language matters.

I'm not saying we should turn our hospitals into theme parks that resemble Disneyland, but I think it's valid to question the data points that the hospital is using to shape its environment - and is it data that matters to the people who are there to use that environment?

3. Education - on paying attention to a system that desires data - but isn't really sure what data matters

Lastly, let's have a look at something a lot closer to most of us - education and data.

In education we love data, we collect it in great lumps of the stuff. We collate, and analyze and sort and celebrate it. We upload and download it, we print it and publish it. Governments and ministries love data, and want it all to the be clean and tidy and meaningful.

Like the hospital example, we often forget that our data gathering only captures a representative slice of an actual human and we can miss the point of the exercise.

To demonstrate this I want to continue with a brief discussion on one particular aspect of the NZ education sector, that some of you may be aware of, namely, the Network For Learning. As said I've been invited to be part of a working group of sector people, to give some advice to those tasked with creating this Network for Learning. So my opinions expressed here, have already, in part, been heard.

The Network for Learning has this vision statement:

"Our overall objective is to make a significant contribution to improving educational outcomes with a vision of unleashing learner potential.

N4L’s focus is to transform schools to technology-based learning environments. We aim to deliver what schools need in connectivity, content and dynamic “learning-to-learn” online services that are safe, easy, relevant and affordable. These services will enable and encourage collaborative, interactive, fast-paced, learner-focused education, reflecting the needs of individual students and teachers."

What that means in reality is still up for grabs. And debate. And discussion.

I understand from the initial working group meeting, that the aim is that no school will get less than what they already have with regards to internet provision and services. That N4L will be a subsidized internet service for a set number of years, before potentially becoming a subscription service.

The expectation is that it will then "unleash learner potential" across our publicly funded education system.

But will it? How will we know?

Before we go any further - let's look at a few other details that shape our defaults:

Decentralised schooling:
The NZ education system is highly decentralized, and Tomorrows Schools has allowed, I believe, many schools to do amazing things, for their students. Those amazing things have come about as a result of dedicated leaders, teachers and communities who share a vision of what is best for their students. Those same people have chosen tools to collect, analyze and utilize the data that's matters most to their students. That matters most to the parents of those students. That matters most, and actually makes a difference to those students.

Public Accountability:
Political pressure for accountability for all of that public expenditure is coming to bear - and as has been seen around the world, it's captured in terms like "national standards" "No Child Left Behind", "Michelle Rhee", "NAPLAN" and "Global Education Reform Movement".

All of these methods use data gathering tools, which are often fixed points to measure and quantify the value of the education system. Keep in mind this data measures "educational achievement", and that data matters mostly to those who build the system, and who want to be seen as having done something for education. It's mainly a political exercise, not an educational one.

21st Century Learning:
This is a favorite of many in the education sector. We want to present ourselves as future focused. And we do this by choosing tools and devices that look and sound like the future. To make ourselves sound like we've got a handle on the future I guess - because we're leading learning.

We've pimped video projectors and IWBs, we heralded e-portfolios and online parent portals, and now we’ve got smartphones and iPads and tablets. All of them touted as revolutions in learning. And to model being professionals, we go to conferences and follow hashtags, and build wikis to share apps and howtos and FAQs – and we call it PD and educamps and tweetups. All of that being data that we gather to justify the expenditure, and the implementation of all of this 21st Century stuff.

So we have these 3 forces. A decentralized education system, a political demand to centralize and manage education delivery, and an almost pathologic desire in the sector to invest in shiny technology.

Into this mix comes a Network for Learning, a politically useful tool, to say "We're doing big things in education" - a way to deliver educational achievement across our sector, a way to make our sector look like the 21st Century.

But what are the implications of this Network for Learning? Here's some points to consider.

The services and support within a nationally led "Network for Learning" are tools that will demand change and shift in many individual school processes. Sometimes in hardware and infrastructure, sometimes in workflow processes, sometimes in who you've hired to be teaching, or which IT support company you have.

Remember, N4L will be a centralized system being overlaid onto a decentralized school model, and it's economically impossible to tailor that system to every single schools needs. So there will be changes within schools, and there will lots of learning in how technology can be implemented.

What happens when we've streamlined the student identity management so well, and all schools are connected via a national network, that it's now a simple matter to administer a national test in reading, writing and maths - and have those results fed back to the Ministry of Education automatically?

Where will we, as a teaching profession stand then?

When this government, after dropping $400million on this network, mandates the use these services, these websites and these assessment tools, despite the fact that these services may not include any tools that we’ve decided are actually more beneficial to student learning needs – will we be blaming the technology we accepted?

How will we critique or honestly evaluate the thing we demanded?

The Parliamentary Inquiry into 21st Century Learning has recommended:

".. that the Government consider the benefits of implementing and operating a single system for core ICT services in schools, including identity and access management, a student management system, a learning management system, e-portfolio, e-asTTle and e-Admin systems such as ENROL"

A single system - which is handy from the perspective of delivering educational outcomes. But may or may not be the thing that matters most to the schools within that system, who are creating learning experiences and conversations on a daily basis. And it may or may not mean a thing to the students within that school.

4. On checking our defaults:

At what point does our inability as educators to pay attention to the data and the connections that matter, totally undermine any claim to the moral high ground as leaders of teaching and learning?

When will we start quietly blaming ourselves for not paying attention to the bigger picture policy issues around delivering public education to society?

I think we as a sector have spent so much time talking about technology as the future of education, and having that as the starting point of the conversation, that we’ve forgotten that the unsexy, deep understanding of what teaching and learning is, doesn’t actually require shiny technology.

It requires taking the time and making the effort to create personal, honest, focused relationships with young people. To show them. To learn alongside them. To give them social skills, work skills, thinking skills. To model what it is to be a learner. To give them respect and a place to stand and be heard.

That conversation needs to be about what online, connected and collaborative learning looks like, and is like for your students and for your teachers.

Right now I see most of the effort going into arguing about acronyms and which tool to use. We should be having rich, messy, sometimes inconclusive conversations about what critical, key things we want our students to learn.

Those conversations should be based on what we see in front of us every day, namely our students and their needs.

And out of those conversations, comes a choice in what digital tools and technologies to enhance and further the conversation.

Instead we’re focusing on the shiny tool that looks like it’s from the future. Something like the N4L for example.

One of the targets in the Public Achievement Information project, of which the N4L is a part, is this timeline (in 2025 recovered from the Internet Archive with the simple filename "BLUETimeline")

2017: 100% of schools are reporting high quality achievement information.

I've always wondered what happens if after 5 years of massive investment in data gathering and moderation, professional development and network infrastructure, what happens if we have 100% of schools with high quality achievement information - telling us what we already know now.

We are the future.
Our choices shape the future.
We should hold ourselves responsible for that future.
We should pay attention to that.

I'd like to finish here by with one of the closing statements from David Foster Wallace's address.

Our own present culture has harnessed these forces in ways that have yielded extraordinary wealth and comfort and personal freedom. The freedom all to be lords of our tiny skull-sized kingdoms, alone at the centre of all creation. ... The really important kind of freedom involves attention and awareness and discipline, and being able truly to care about other people and to sacrifice for them over and over in myriad petty, unsexy ways every day.

I believe as educators our enduring challenge will be checking our defaults, checking the data and the forces that enforce those defaults - and gently reminding ourselves....

"This is water."