They say Artificial Intelligence is the future. I prefer to say, thinking about the data, that Artificial Intelligence is the result.
And every result, before it’s a dish, has been cooked. We often hear the same question, asked with intensity: “When do we launch the pilot?”. And I, having seen this film several times already, always think the same thing: before launching anything, show me the data. Because there, in that uncomfortable question, almost everything that comes afterwards is decided.
A Data Driven AI project isn’t born in the algorithm. It’s born in the pantry.
Without good raw ingredients there’s no good dish
You can have the best chef on the market, the most expensive model, the most sophisticated architecture, but if the raw ingredients are rotten, the dish comes out rotten. No sauce can disguise it. And with data it’s exactly the same: if the data doesn’t work, nothing works. Not the model, not the pilot, not the pretty presentation you’re going to give to the board. Preparing the data isn’t a minor task you delegate to the intern of the day while the serious folks discuss strategy.
Preparing the data is the strategy. And it’s prepared on three fronts, like someone setting up a kitchen before service.
Volume. You need enough quantity. Three photos of the fridge and the promise that we’ll go shopping as we go won’t cut it. If there’s no volume, there’s no learning, and if there’s no learning, what you have isn’t Artificial Intelligence, it’s an expensive calculator with a very nice interface.
Structure. The data has to be ordered, labelled, with meaning. A warehouse without shelves isn’t a warehouse, it’s a junk room. And nobody in their right mind cooks inside a junk room.
Quality and cleansing. This is where people get tired before they even start, because cleansing is boring, it doesn’t show up in any LinkedIn photo and nobody applauds you for it. But it’s the step that separates a serious project from a fairground experiment. You have to remove duplicates, correct errors, eliminate the noise. Clean, clean and clean again.
Three fronts. None of them optional.
The eighty and the twenty: the recipe you should follow and that nobody follows
Here comes the part almost everyone skips, because they’re in a hurry, because the committee wants to see the pilot on Friday, because we’ll adjust it later. And it’s precisely the part that blows up in your face later on.
Of all the data you have available for a project, the rule is simple: eighty percent is applied to the project, you train with it, you build with it.
The remaining twenty percent isn’t touched. It’s not looked at, not used, not slipped in a little to save time. That twenty percent is kept aside, as historical data, intact, for a single purpose: testing.
And here’s the sentence I’d like to tattoo on the wall of every project room: testing isn’t the end, it’s the pilot’s obligation.
Because a pilot that isn’t tested with data it has never seen before proves nothing. It can give you wonderful results, beautiful charts, numbers that make the committee fall in love. And still be hallucinating. Making up answers with the same confidence as a bad student reciting a lesson they haven’t learned, but who sounds convincing.
The twenty percent is the surprise exam. It’s the question the model hasn’t seen in the notes. If it answers well, there’s confidence. If not, better to know now, in the kitchen, and not at the client’s table.
The conclusion, brief, as the house demands
There are no shortcuts. The data is prepared, it’s cared for, and a part is set aside to distrust yourself.
Whoever skips that part isn’t doing Artificial Intelligence. They’re improvising with an apron on.