Newsbin re-synced, rethought

A few weeks ago I wrote about tearing out how Newsbin syncs and rebuilding it around a core idea: sync less. Only the things you actually decided need to travel between your devices, your feeds, folders, tags, and what you’ve read or saved. Articles, I argued, don’t need to sync at all, because every device already knows how to fetch them from the same feeds.

I built exactly that. It’s finished, and it goes out soon in Newsbin 2.2. And in the building, I found out late in the game that I was about half right.

What the rebuild got right

The foundation held. Newsbin no longer syncs through SwiftData’s automatic CloudKit mirroring. SwiftData still runs the local database on every device, exactly as before. What changed is the pipe to iCloud: it runs on CKSyncEngine now, with its own zone and conflict rules I wrote down before I wrote the code.

Leaving the built-in mirror behind is what made everything after it possible, and it fixed a whole category of problems at the root:

I decide what syncs. The old mirror tried to copy the entire database up to iCloud, every article included, which is where most of the volume and most of the conflicts came from. The new engine carries only the things you actually chose.
I decide how conflicts resolve, in rules I can point at. The mirror settled disagreements however it saw fit, which is why a read article could quietly reappear as unread. Now read beats unread, your latest reorder wins, and the outcome is written down instead of guessed.
Duplicates are impossible instead of merely discouraged. Every record is keyed to the content it represents, so the same feed or article can’t land twice, no matter which device saw it first.
New versions won’t wedge old ones. Every record carries a version stamp from day one, so I can evolve the data later without stranding a device that hasn’t updated yet. Under the old mirror, adding a single field could stall sync until the whole schema was deployed to iCloud.
You can finally see it working. There’s a Sync Now button that means it, and a status readout that tells you when it last synced and whether anything’s pending, rather than a black box you wait on and hope.

It’s built to roll out the careful way too. The old mirroring and the new engine run side by side through the switchover, so your devices stay in sync no matter which version each one is on, and then the old system goes quiet. No forced-update cliff, no flag day. That’s the part I was most nervous about, and it’s the part I’m most confident in now.

The wall I hit was the unread count

Then I started living with it, and my devices disagreed about a number.

My iPad sat at 110 unread and would not clear. I’d read everything in sight on the Mac, and the iPad still insisted there were 110 things I hadn’t seen. When I dug in, the reason was almost funny: those 110 articles were ones the Mac had never pulled in the first place. The iPad had them, the Mac didn’t, so no amount of “I read this” could travel between them, because you can’t mark an article read on a device that never had it.

This is the flaw in “articles are derivable.” It’s true that each device can independently fetch the same feeds. It is not true that they do it at the same moment. One device refreshes at 8am, another was asleep until noon, a feed drops ten items and picks up nine. The sets drift. And read state can only ever describe articles a device is actually holding.

The fix that wasn’t enough

My first move was the one I felt could solve the issue in a clever way, and I still like it. Instead of syncing a read-or-unread flag for every single article, I sync a single tiny record per feed that says “everything published before this moment is read, and I decided that at this time.” I called it a read watermark. One little record stands in for a thousand.

It works well. Marking a feed read on one device quietly catches up every other device, cheaply, without shipping a record per story. If all I cared about was read state, I’d have stopped there.

But a watermark tells your devices what you’ve read. It doesn’t give them the articles themselves. So the iPad still showed a different number than the Mac, because the underlying lists were different. I’d fixed the intent and left the symptom sitting right there on the app icon.

Why the number is the product

Here’s where I changed my mind. A badge you can’t trust is worse than no badge at all. If your Mac says 11 and your phone says 10, you stop believing either of them, and once you stop trusting the count you start second-guessing the whole app. People would revolt if Mail did this, and they’d be right to.

So I did the thing my last post said I wouldn’t. Articles sync again.

How articles sync without bringing back the mess

The entire reason I didn’t want to sync articles was volume and conflict. Thousands of records a week, each one a thing to push, pull, and reconcile forever. So the article sync I actually built is deliberately narrow, and it keeps almost all of the savings that started this whole project:

Only the light fields travel. The headline, the link, the dates, a short excerpt. The full article body stays out of it by default.
Deletions never cross the wire. Each device retires its own old news on its own schedule, so a sync can never be the thing that empties your library. The highest-volume record type is the one I trust the least with deletes.
Every article’s ID is derived from its own link. The same story can’t land twice no matter which device saw it first, or in what order they talk to each other.

That’s enough to make the counts agree, which was the whole point, without dragging back the churn.

And the body, once you actually need it

One more thing surfaced once articles synced. If you keep full-text articles turned off, or you’re offline, a synced headline had nothing behind it. Nothing to read, nothing for the on-device summary to chew on. So the article body travels too now, but on the strictest terms I could give it: once, never merged, size-capped, and stripped back out of the sync cache the moment it’s stored so it can’t bloat anything. A synced article behaves like one you fetched yourself.

The safety net I’m most glad I built

I’ll be honest about one thing, because it’s the part that made me a better engineer this month. Somewhere in testing, a bad sync tried to take a large bite out of a library on one of my own devices. It’s a sobering thing to watch.

So the engine now refuses, on principle, to delete a large share of anything in a single pass. And a device will never delete your data just because another device told it to. Deletes are the one operation you can’t undo, so they get the most suspicion in the entire system. If something ever looks like a mass wipe, the engine treats it as a bug and declines, and I’d rather explain a stubborn leftover than a missing library.

What I actually learned

I was too dogmatic about one word. Derivable is not the same as identical. It’s fine to tell yourself “the device can recompute this” for most data. But for a number you glance at a hundred times a day, right-now-identical is the actual feature, and I’d quietly optimized for elegance instead of for the thing you see.

The good news is the foundation from the last post is exactly what let me change course without starting over. Syncing your decisions was always right. I just underestimated how much “the same list, right now” is one of the decisions you expect the app to keep.

It all lands next week in Newsbin 2.2. The counts match. There’s a button that tells you what sync is doing. And I get to keep being straight with you about the parts I got wrong on the way there.

A huge thanks to my TestFlight testers!