News Agent, running an AI-built project as a non-developer

The short version

The problem

I wanted to learn to run and check an AI system as a non-developer, on something real.

What I did

Directed AI agents building a news reader. I set the spending caps, made the decisions, and scored the AI's judgement against my own on 46 articles.

What it was worth

The comparison found problems nothing else could see. A handful of sessions, learning kept.

The call

Closed it on purpose. An evaluated no is a result, not a failure.

I wanted a calmer way to read news. A short briefing, low on distress, drawn from sources I chose myself.

But the news reader was never the real goal. I wanted to find out whether I could run an AI-built software project as a product owner, without being able to read most of the code. What I could control, what I could check, and where my limits were. The news reader was the test case.

AI coding agents wrote all the code. I made the decisions a product owner should not hand over. What the system was allowed to spend, what data could leave my machine, what needed my approval, and when to stop.

We wrote the rules down before building anything. Money, privacy and anything running unattended needed my yes. Technical choices were the AI's to make, because I could not evaluate them anyway, and pretending to review them would have been theatre.

1. 01

   A spending cap

   At most $15 a month and 25 cents per run. The system checked the cap before every call to the paid AI service, not after.

2. 02

   Stop buttons

   Every part could be switched off from one settings page, and I tested that they actually stopped things rather than just saying so.

3. 03

   Nothing ran by itself

   The system only worked when I started it. Automatic scheduling was built, but the switch stayed off, because leaving it on was my decision to make.

4. 04

   A fixed test it could not change

   46 articles I had judged myself, saved so they could not be edited. Every new version of the AI was marked against the same 46.

A working product. It collected articles from six news sources, had an AI judge each one on how distressing and how important it was, ranked them, and produced a briefing I could read in the browser and give feedback on. Around it sat a settings panel for budgets and controls, a monitoring dashboard, over a thousand automated tests, and automatic publishing of updates to a small server at home.

I understood what every part was for. I could not have built any of it, and I could not read most of the code. That gap is what the rest of this page is about.

No design time went into any of these screens. The panel was an instrument, and the hours went into what it measured.

The only way I could judge the AI's work was to compare its judgement with mine. So I read 46 articles and recorded my own verdicts, and the system scored the AI against me. That comparison turned out to be worth more than everything else in the project.

The AI thought everything was important. Out of 46 articles, it marked one as unimportant. I had marked seventeen. The instructions we gave it described in detail what "distressing" meant, but never what "unimportant" looked like, so it treated everything as newsworthy. Once the instructions were fixed, we agreed twice as often. No test caught this. Only the comparison did.

Small differences mean nothing. The same comparison, run twice with nothing changed, moved seven points on its own. That taught me to stop reacting to small movements. If a new version scores two points better, that is not improvement, that is chance.

46 opinions were not enough. I had set pass marks like 90% agreement. With only 46 articles, one article could swing the result past or under the mark, so a fail by one article told me nothing. My measuring stick was too blunt for the rules I had set. The honest fix was judging more articles, and every one cost my own time.

A cheap check settled a real decision. A free AI model was tested against the same 46 articles. It rated an article I had marked as severe as completely harmless. One answer like that was enough to say no, and finding it out cost nothing.

I stopped after a handful of working sessions, spread over ten days with gaps where life took over. On purpose. Nothing was broken and nothing was left half-built.

Two honest reasons. The product's quality depended on it learning my taste, and my taste shifts with my mood and the day, so teaching it meant judging article after article, indefinitely, for a news reader I realised I did not need. And I had answered the question I actually started with. I can run a project like this. I can set the limits, check the spending, test the stop buttons, and measure the AI's judgement against my own. What I cannot do is verify the code itself, and knowing exactly where that line sits is worth more to me than the product was.