Choosing a Product Without AI in It

Updated 6 min read
  • ai pm
  • prioritisation
  • product strategy
  • side projects
  • decision making
On this page, 6 sections

I picked a side project with no model in it. Here is the scoring that got me there, and the row where reasoning overruled the score.

The shortlist, first pass

At the start of September I worked out what to build next. One product, built solo, on free hosting, with real people using it inside a month. The research ended with three candidates in rank order.

First was a Spec-to-Eval Kit: paste a feature spec with acceptance criteria, get back a stratified eval set (happy path, edge, adversarial), a rubric, and a run across two or three models with pass rate and cost per task. Second was a Review Miner for Indian D2C: upload a CSV of product reviews, get themes ranked by their pull on one-star versus five-star ratings, with Hinglish and regional-language reviews handled rather than dropped. Third was an Address Lint with a return-to-origin risk overlay for Indian checkouts, deterministic at its core.

Two of those three had a model at the centre. They were also the two I most wanted to build. That should have been the first warning.

Five criteria, and why each one existed

The scoring was deliberately boring. Every candidate took a one-to-five score on five criteria, and only ideas scoring four or better on One-shot fit and Not saturated made the shortlist.

  • Gap fill asked whether the idea covered a hole in my existing roster, which leans on developer infrastructure and personal productivity and is thin on anything a growth team would open twice.
  • One-shot fit asked whether it sat inside the build envelope: no auth, no billing, a single deployable.
  • Users in 4 weeks asked whether I already had a channel to real users through a small personal network, because a live URL with a few dozen users beats a polished demo with none.
  • Defensible insight asked whether there was a non-obvious product decision I could be interviewed on.
  • Not saturated asked whether the research had actually found incumbents doing exactly this.

Notice what is missing. Nothing rewards a product for being technically interesting, or for using a model. Every criterion asks whether the thing gets finished, gets used, and can be defended out loud.

What changed when the AI premium came off

Then I added two constraints: no model inside the product, and free-tier hosting only. I re-scored on the same five criteria. Both LLM-centred candidates dropped out entirely, and the reasons were not ideological.

The first is evaluation cost, paid in my own hours. The Review Miner's whole value was theme quality on mixed-language text, and proving that meant hand-labelling a hundred reviews, thirty of them Hinglish or regional. My note on that line was blunt: human hours, no way around it.

The second is non-determinism. The address linter needed no model for its core, because a rule engine over the public India Post pincode directory is checkable and repeatable. Where a model was optional, it had to be evaluated against the deterministic path to earn its place. A rule that fails always fails the same way, which is a feature when one person supports the thing.

The third is data access. The Review Miner had no data of its own: it needed a public dataset, and employer data was out without written permission.

The fourth is trust, and it decided the question. The research flagged that reviewers are tired of candidates who ship a stack of AI-generated apps and cannot explain why the choices were right, and that a chat-with-your-X wrapper is the archetype of what gets discounted. My own risk note on the eval kit said it plainly: perceived as a prompt wrapper if the rubric method is thin. Adding a model raised the bar rather than lowering it.

The winner has no model in it

The pick is a Cohort and Retention Studio. Drop an orders CSV into the browser and get cohort retention curves, repeat purchase rate, time to second order, lifetime value by first product, and a free-shipping threshold what-if. Nothing is uploaded to a server. The intended users are growth people and founders at brands on WooCommerce, Amazon, Dukaan or Shopify Basic, and the gap check came back open and verified: Lifetimely, Peel and Polar all need a Shopify install and charge hundreds of dollars a month, with no free CSV-only alternative found. It has a wow moment on the user's own data, a monthly reason to return, and a privacy line I expect operators to care about (untested).

Update, 30 Sep 2026: the build is written up as a case study. It is ready for a local pilot, with no hosted pilot verified and no users yet, so the one-month users goal above was missed.

Before and after

Totals are the sum of the five criteria, out of 25.

IdeaFirst passAfter the re-score
Spec-to-Eval Kit1st, 23Out, LLM at the core
Review Miner for Indian D2C2nd, 21Out, LLM at the core
Address Lint + RTO Risk3rd, 201st at 22, then struck the same day
Eval Workbench, no LLMnot scored2nd at 20
Cohort and repeat-purchase analysernot scored3rd at 18, then the flagship pick

The third row is the honest part. The rescored winner did not survive the day: I argued myself out of it within hours. The risk band is a network-data problem incumbents win by construction, the decision point lives inside checkout where a standalone tool cannot reach, and usage would be occasional and bulk, with no retention signal. Scoring got it wrong, and reasoning corrected the score.

What this means for an AI PM

None of this is a retreat from AI. I still think the eval harness is the real artefact of an LLM feature, and I have written about why the harness comes before the prompt. Choosing a model is a decision with a cost curve, a labelling bill, a failure distribution and a trust discount attached, and a PM who cannot say no to a model cannot really say yes to one either.

The tell is simple. If you can describe the output as a rule, write the rule. Retention curves are arithmetic. Address defects are lookups. Reaching for a model there buys a demo and sells the thing people came for: an answer they can check.

Knowing when a model is the wrong tool is part of the job, not a step back from it.

Score the boring criteria first, before you fall in love. Build the version a reviewer can verify. Then, if a model beats the rules on the same labelled set, ship it and publish the comparison. More of how I think about this lives on my AI PM page.

Dhruv Singhal

Dhruv Singhal has about a year of product experience across internships, most recently as a product intern on the growth team at The Sleep Company (Jul to Oct 2026). He builds small, tested products and writes about AI evaluation, retention analytics and product judgment.