After $2 billion, 4,000 GPUs, and three years of development, researchers confirm model “fits data roughly the way a line would.”
Pictured: nothing. The scatterplot shaped like a boomerang was, per legal, not to be reproduced.
PALO ALTO, CA—Leading AI researchers expressed shock Thursday after unveiling GRADIENT-PRIME, a linear regression model trained on 11 petabytes of data at a cost of $2 billion, and discovering that it produces a straight line.
“We were absolutely certain that with enough compute, it would become nonlinear,” said Dr. Alan Whitcombe, principal investigator at the Institute for Advanced Predictive Sciences, gesturing at a slide of a scatterplot shaped roughly like a boomerang. “We gave it every advantage. Distributed training. Custom silicon. A dedicated cooling facility. And it just kept being a line.”
The model, which has 14 coefficients, took 26 months to train on a cluster that draws as much power as Reno. Engineers confirmed that its final weights are identical, to six decimal places, to the ones obtained by a graduate student in 40 milliseconds using a closed-form solution.
“We knew about the normal equations,” admitted Whitcombe. “But those felt, you know, small. We wanted to be part of something bigger.”
Early benchmarks were promising in the sense that the model performed extremely well on data that was already roughly linear. On data that was not, GRADIENT-PRIME confidently drew a straight line through it and reported an R-squared of 0.31 with what observers described as “unearned serenity.”
Trouble escalated when the model was tasked with predicting housing prices in a market with a well-documented nonlinear relationship between square footage and price. GRADIENT-PRIME projected that a 40,000-square-foot home would cost negative $3 million, meaning the seller would owe the buyer a boat.
“That’s extrapolation. Technically the model is doing exactly what it was designed to do. Forever. In both directions.”
“That’s extrapolation,” said the team’s lead engineer, wiping his forehead. “Technically the model is doing exactly what it was designed to do. Forever. In both directions.”
Attempts to fix the issue reportedly included adding more data, adding more GPUs, and, at one point, adding a second variable that was just the first variable again. “We called it a feature interaction,” said one researcher. “It was not.”
Meanwhile, a rival team at a competing lab announced an XGBoost model that outperformed GRADIENT-PRIME on every metric using a laptop and one afternoon. The lab’s CEO immediately responded by announcing GRADIENT-PRIME-2, which would be “twice as large, and, we believe, twice as linear.”
Statisticians were not surprised. “Residual plots exist,” said Dr. Priya Ramanathan, a state university professor whose grant for $30,000 was declined last year. “You plot the residuals. If they look like a smile, the model is wrong. I keep saying this. I have it on a mug.”
Company leadership has remained upbeat. In an all-hands meeting, the CEO reassured employees that the model’s limitations were “a data problem, not a model problem,” before being reminded that the data had been collected at a cost of $600 million.
Investors have since poured an additional $8 billion into the project, citing the model’s “extremely stable behavior” and “total absence of overfitting,” which the company’s own chief scientist described as “the one thing we did right, entirely by accident, because the model can’t fit anything.”
At press time, GRADIENT-PRIME had been deployed in production, where it was predicting that every trend would continue in the same direction forever, and receiving strong reviews from executives.