Somewhere in your browser tabs right now, a conversation is open with ChatGPT or Gemini or Claude, and you've described your 1,800-square-foot ranch in Columbus with its single-pane windows and furnace from 2003, and you've asked the question that every homeowner considering a renovation eventually types into a chat window: What's the best bang for my buck? The chatbot gave you a confident answer, structured in numbered steps, with estimated savings and a payback period calculated to the month, sounding very much like an engineer who had studied your house.
It had not.
Researchers funded by the National Science Foundation tested six leading AI chatbots, ChatGPT o3, DeepSeek R1, Grok 3, Gemini 2.0, Meta's Llama 3.2, and Claude 3.7, on exactly this kind of question across 400 real American homes drawn from the National Renewable Energy Laboratory's ResStock dataset, which models 550,000 representative residences spanning 49 states, three housing types, and construction vintages from pre-1940 to 2010. No fine-tuning, no domain-specific training, just the same general-purpose AI that millions of homeowners are already consulting for renovation decisions that will cost them $5,000 to $25,000.
Carbon Math: Surprisingly Competent
Ask an AI chatbot which retrofit package will cut your home's carbon emissions the most, and it performs remarkably well, well enough that you might mistake competence for expertise. Across all six models, the study found 39.9% Top-1 accuracy, meaning the chatbot picked the single best carbon-reducing option about two in five times, and when the researchers broadened the criteria to the top five options, accuracy climbed to 79.9%, with ChatGPT o3 hitting 92.8% and Gemini 2.0 leading at Top-1 with 54.5%.
By any reasonable standard, that is useful. An AI chatbot can narrow a homeowner's carbon-reduction options to a strong shortlist, not perfect but a legitimate starting point for a conversation with a contractor or energy auditor about which heat pump, insulation level, or electrification package delivers the biggest environmental benefit for a specific building in a specific climate zone.
Payback Math: Confidently Wrong
Then the researchers asked the question homeowners actually care about.
Which retrofit pays for itself fastest? Accuracy collapsed to 11.0% at Top-1 across all six models, with the best-performing model on this question, DeepSeek R1, getting it right only 14.3% of the time, and Gemini 2.0, which had led the carbon task, managing a dismal 6.5%. Even under the most generous Top-5 standard, where the chatbot gets credit if the right answer appears anywhere in its top five recommendations, overall accuracy reached only 34.8%, which means that two-thirds of the time, the correct payback-optimal retrofit didn't even crack the model's shortlist.
The failure mode was consistent across all six models and instructive in what it reveals about how large language models reason about money. They repeatedly confused the cheapest retrofit with the one that pays back fastest. A $4,000 insulation-only upgrade might cost less than a $22,000 heat pump package, but if the heat pump saves $2,800 a year on energy bills while the insulation saves $400, the heat pump's payback period is eight years versus ten, shorter despite the higher sticker price, because payback is a ratio of cost to savings, not a measure of cost alone. The chatbots saw a small number and called it efficient.
No Two Chatbots Agree
A homeowner asking ChatGPT, Gemini, and Claude the same question about the same house would get three different recommendations, a problem the study quantified using Fleiss' Kappa, a statistical measure where positive values indicate agreement. Selection-based agreement across all six models came back negative, meaning these chatbots agreed on which specific retrofit to recommend less often than six dice rolls would.
ChatGPT o3 and Grok 3 showed the highest pairwise alignment with each other, but that consistency didn't translate to accuracy: Grok 3 posted the worst carbon-reduction scores in the group, dragged down by an idiosyncratic tendency to overweight heating fuel type while ignoring building envelope characteristics that physics-based models correctly identify as the dominant drivers of residential energy consumption. Your answer depends on which chatbot you asked. Switching between them won't converge on truth.
What the Chatbots See but Don't Use
All six models rated "usage level," how much energy a household actually consumes based on occupant behavior, at roughly 2.5% importance when asked to rank which factors matter for retrofit decisions, then none of them incorporated it into their actual reasoning chains. Physics-based tools like EnergyPlus, which the researchers used as their accuracy benchmark, compute heat and mass balance equations for every thermal zone using every input parameter systematically. When you tell EnergyPlus your family runs the dryer twice a day and keeps the thermostat at 68, that information hits the calculation directly. A chatbot might note it, call it important, and then build its recommendation around county name and construction vintage instead.
What This Means for Your Renovation Budget
The 16 retrofit packages in the study ranged from insulation-only upgrades to full electrification with high-efficiency heat pumps, with costs spanning from a few thousand dollars for envelope improvements to over $20,000 for comprehensive packages including air-source heat pumps rated at SEER 24, heat pump water heaters, and full appliance electrification. At those price points, getting the payback period wrong by a factor of three can mean the difference between a retrofit that makes financial sense over a seven-year ownership horizon and one that never pays back before you sell.
A better version of this technology exists in prototype: fine-tuned models grounded in building-specific data, retrieval-augmented systems that pull from validated cost databases, hybrid tools that let an AI interpret your description and hand off to a physics engine for the math. None of these are what you get when you open a browser tab and type your question.
When to Use Them Anyway
A professional energy audit costs $300 to $800 and requires scheduling an inspector who may be booked three weeks out, whereas the chatbot is free and available at 2 AM when you're staring at your January utility bill wondering whether replacing that furnace is worth the disruption. For carbon reduction screening, where 80% accuracy in the top five is genuinely helpful as an initial filter, the chatbot earns its place: ask it which retrofits cut emissions, read its reasoning, and use that shortlist to have a more informed conversation with a contractor who can quote real costs in your market.
But do not ask it when you'll get your money back. A free lookup on NREL's database gives you unit costs by retrofit type, and a BPI-certified energy auditor with a blower door and infrared camera can tell you where your house actually leaks and what fixing it will actually save, as opposed to where a language model infers it probably leaks based on the year it was built and the county it sits in.
What This Study Did Not Test
The researchers used national average retrofit costs rather than regional pricing, which can vary 30% to 50% depending on labor markets, and they did not incorporate local utility rate structures, available rebates, or occupant preferences, all of which significantly alter real-world payback calculations. The models reflect early 2025 capabilities; newer versions fine-tuned against building-science datasets or integrated with retrieval from cost databases may improve. And the standardized prompts, necessary for reproducibility, don't capture how a homeowner iterates with a chatbot over multiple exchanges, refining questions and pushing back on vague answers.
None of these caveats rescue the core finding. An 89% error rate on the question most homeowners actually ask is not a prompt engineering problem, and it now has a number attached to it, peer-reviewed and replicable, sitting quietly in a journal while millions of people type their renovation questions into a chat window that will answer with absolute confidence and get the money part wrong nine times out of ten.