Building in public: why FounderFlow grades its own confidence instead of just guessing
One decision I keep coming back to while building FounderFlow: an AI that sounds confident but is wrong is more dangerous than one that admits it doesn't know.
Early versions just gave one clean-sounding answer for everything: "this deal is at risk," "this email needs a reply today." It felt impressive in demos. Then I started using it across my own three businesses and caught it being confidently wrong more than once, which is a much worse failure mode than being unhelpfully vague.
So now every insight in FounderFlow gets tagged: Verified (we have hard data), Very Likely (strong signal, not certain), Needs Review (you should look at this yourself), or Monitor Only (early signal, don't act yet).
It's a small UI change but it forces the product to be honest about what it actually knows, and it's made me trust my own tool more, which says something.
For anyone else shipping AI features: have you had to walk back a "confident" AI answer after it turned out to be wrong? How did you handle it?
Nemo & Anna
Create IoT devices that don’t exist yet — no firmware, with built-in safety.
Comments (4)
This is something we dealt with building SaaS Hive too. We have an AI crawlability audit (Unhid) that tells founders if their site is readable by AI. Early on the temptation was to give a clean pass/fail. But the reality is more nuanced than that. A site can be 73% crawlable and that number matters more than a green checkmark. The moment we started showing the actual score instead of a simple yes/no, founders trusted the results more and actually took action. Confidence grading changes behavior. Good call building it in early.
Wow... I need my boss at my full-time job to integrate this into the platform we use at work! He built an MVP and we were all stuck having to train the agents. After seeing it give so many wrong answers, I had to confront the agent on its BS, and its reply was, and I quote, "Oh yes, I did make that up... my bad!" So I had to have it store to memory to check that the answer will be a verifiable fact before responding.
Olga, the 73% vs pass/fail example is such a good parallel — a binary result hides exactly the information someone needs to act on. That's basically the same lesson that pushed us toward confidence tiers instead of one clean answer. Curious whether showing the actual score changed how founders prioritized fixes, or just how much they trusted the audit itself?
Ashley, "Oh yes, I did make that up... my bad!" is both hilarious and exactly the failure mode I'm terrified of. Having it verify against facts before responding is a great instinct. Would love to hear how it's going once your boss's team gets it integrated — that's a great real-world stress test for confidence grading.
Sign in to comment or upvote.