News Agent, running an AI-built project as a non-developer

The short version

Result

Scoring the AI against my own verdicts on 46 articles found problems no test caught. I closed it after a handful of sessions.

What I did

Directed AI agents building a news reader, and set the spending caps, the stop buttons and what needed my approval.

The call

I closed it once I knew what I could control and check without reading the code, and what I couldn't.

I wanted a calmer way to read news, a short briefing drawn from sources I chose myself and low on distress.

Mostly, though, I wanted to find out whether I could run an AI-built software project as a product owner without being able to read most of the code, and where my limits were. The news reader was the test case.

AI coding agents wrote all the code. I kept the decisions a product owner shouldn't hand over, such as what the system could spend, what data could leave my machine and when to stop.

We wrote the rules down before building anything. Money, privacy and anything running unattended needed my yes. Technical choices were left to the AI, because I couldn't evaluate them and saw no use in pretending to review them.

1. 01

   A spending cap

   At most $15 a month and 25 cents per run. The system checked the cap before every call to the paid AI service, not after.

2. 02

   Stop buttons

   Every part could be switched off from one settings page, and I tested that they did stop things instead of only saying so.

3. 03

   Nothing ran by itself

   The system only worked when I started it. Automatic scheduling was built, but the switch stayed off, because leaving it on was my decision to make.

4. 04

   A fixed test it could not change

   46 articles I had judged myself, saved so they could not be edited. Every new version of the AI was marked against the same 46.

It collected articles from six news sources, had an AI judge each one on how distressing and how important it was, ranked them, and produced a briefing I could read in the browser and give feedback on. Around it sat a settings panel for budgets and controls, a monitoring dashboard, over a thousand automated tests and automatic publishing of updates to a small server at home.

I understood what every part was for, but I couldn't have built any of it or read most of the code.

The only way I could judge the AI's work was to compare its judgement with mine. So I recorded my own verdicts on 46 articles, and the system scored the AI against them. That comparison was worth more than anything else in the project.

The AI thought everything was important. Of 46 articles it marked one as unimportant, where I had marked seventeen. Its instructions described in detail what "distressing" meant, but never what "unimportant" looked like. Once that was fixed, we agreed twice as often. None of the automated tests caught this, and I only found it through the comparison.

The measuring stick was too blunt. The same comparison, run twice with nothing changed, moved seven points on its own, so a new version scoring two points higher told me nothing. With only 46 articles, a single article could also push a result past or under pass marks like the 90% agreement I had set. The fix was judging more articles, and every one of them cost my own time.

A free check settled a decision. A free AI model, tested against the same 46 articles, rated one I had marked as severe as completely harmless. That one answer was enough to rule it out, and finding out cost nothing.

I chose to stop after a handful of working sessions spread over ten days, with nothing broken or half built.

The product only got good by learning my taste, and my taste shifts with my mood and the day, so teaching it meant judging article after article for a news reader I realised I didn't need. I had also answered the question I started with. I can set the limits, check the spending, test the stop buttons and measure the AI's judgement against my own, but I can't verify the code itself.