Episode 10 · Built for Someone Else
Ninety-seven percent for whom?
Every technology imagines someone. Dr. Philip Hickman on the default user, what happens to everyone outside that picture when AI makes decisions that matter, and the five BUILT Questions any leader can ask.
6:29 · Download MP3
In this episode
- What the default user is, and when falling outside it turns from an inconvenience into a barrier
- The strongest case for the machines, and the assumption hiding inside it
- The BUILT Questions: Builders, Users, Inputs, Limits, Trace
- Why you don't need to be an engineer to govern AI, only the person who asks better questions than the sales deck answers
- A five-question check to run on one system you already use
The book: Built for Someone Else Try The BUILT Questions scorecard
Transcript
Every technology imagines someone. A door imagines a person tall enough to reach the handle. A form imagines someone with one address and one last name. And the question that keeps me up is simple. What happens to everybody the technology didn't imagine?
I'm Dr. Philip Hickman, and this is The Human Advantage.
Let me give you two people from the opening of the book. They're fictional. I invented them to show a pattern more clearly than any single case can.
Renee is applying for a claims analyst job, work she did well for six years. The interview is one-way. No interviewer. A recorded woman in a blazer asks a question, a timer starts, and Renee has ninety seconds to answer into the camera. The video has no captions. Renee is Deaf. She reads the woman's lips until the woman glances down at her notes. Eleven days later an automated email describes her as below benchmark on engagement and suggests she work on her listening skills.
Jalen is a pre-med sophomore sitting down to an online chemistry midterm in his dorm room. The proctoring software wants a picture of his face, and the screen says, face not detected, please improve lighting. He pulls his desk lamp closer and closer, until the bulb is a hand's width from his cheek. The software finds him, loses him, finds him. He takes the exam squinting into a light bulb and gets flagged fourteen times for being absent from the frame. Jalen is Black. His roommate, who's white, took the same exam an hour earlier at the same desk under the same ceiling light. He never touched the lamp.
Nobody in those stories set out to hurt anybody. That's the point.
Early in any design process, engineers picture the person who'll use the thing. At first, that person is just a sketch. A pile of assumptions about bodies, voices, lighting, and language. I call that sketch the default user.
Most of the time, defaults are harmless, even necessary. You can't design for every human at once. The trouble starts when two things happen together. First, the system starts making decisions that matter. Who gets an interview. Who gets flagged for cheating. Who gets believed. Second, the people outside the default turn out not to be a random slice of humanity. They're often the same people who've been outside the default before. People with darker skin, for whom cameras were calibrated last. Deaf and disabled people, whose bodies and voices got treated as exceptions to handle later. People who speak a dialect or a second language.
So here's the idea this whole book is built on. When AI decides who gets flagged, hired, admitted, or believed, falling outside the default stops being an inconvenience. It becomes a barrier. And because that barrier is built into software that runs millions of times a day, it can be more consistent than any single biased person could ever be.
Now let me make the best case against my own concern, because you'll hear it, and much of it is right. People are biased decision makers too. And a biased algorithm is, in one important way, easier to fix than a biased person. You can test a program on thousands of cases in an afternoon. You can open it up and change it. You can't do that with a hiring manager's gut.
I believe that. I build AI tools myself. But notice what the argument assumes. It assumes someone checks. It assumes the data to check with exists. And it assumes that when something goes wrong, the person who was harmed can find out and do something about it.
So the question is never AI or no AI. It's this. Built by whom, for whom, and with what recourse?
To make that usable in a real meeting, I break it into five smaller questions. I call them the BUILT Questions.
B is Builders. Who designed it, and who was missing from the room?
U is Users. Who was the imagined user, and who falls outside that picture?
I is Inputs. Whose lives are in the data, and whose are absent or distorted?
L is Limits. Where does it fail, and on whom was it tested?
T is Trace. Can the person affected see, challenge, and correct the decision?
Run Renee's interview through them and the problem gets specific fast. The builders may never have tested with a Deaf candidate. The imagined user hears. If speech recognition was involved, the inputs came mostly from hearing speakers. The limits were never reported by disability. And the trace ended at a link with no human to contact and no way to say, you measured the wrong thing.
Notice what those questions don't do. They don't tell you whether to buy a product or ban it. They tell you what you need to know before you decide. And you don't have to be an engineer to ask them. You have to be the person in the room who asks better questions than the sales deck answers.
I know the buying side of that table. As a superintendent and a principal, I lived with the vendor's deadline and the board's excitement, and I know how rarely a contract says anything about who the software was tested on. And as a builder, I've had to ask these same five questions about my own work. I won't pretend I always get them right.
So picture a vendor presentation. Slide eleven says ninety-seven percent accuracy. Everyone nods. And one person raises a hand and asks, ninety-seven percent for whom?
The room goes quiet the way rooms do when someone has asked the right question. That's the whole book, really. One person, finally in the room, asking who it was built for.
Here's your one thing for this week. Don't start with a new purchase. Start with something already running. Pick one system in your school, your company, or your office that screens, scores, ranks, or verifies people. A résumé screener. A proctoring tool. An early warning flag. Write the five questions on one page. Builders, Users, Inputs, Limits, Trace. Then see how many you can actually answer. Mark each one green, yellow, or red. And if you get a red on Limits or Trace, treat that as a reason to pause and go get the answers in writing.
If this helped, the book is Built for Someone Else. I'll see you next time.