CelinQ Insights · No. 58

Restoring yesterday's server while today's EA laptop still exists

The hard part of disaster recovery for a local-first model is not taking the backup. It is what happens the moment a restored server meets a laptop that never lost anything.

Disaster recovery exercises tend to be written and rehearsed around a single, clean scenario: the server dies, the backup is restored, the service comes back, everyone breathes out. That scenario is worth rehearsing, and CelinQ supports exactly that path for its default SQLite deployment through an online backup that produces a consistent, independently openable copy without taking the server offline. But it describes only half of what actually happens in a local-first architecture, and the half it leaves out is the more interesting one. In this architecture, the server is never the only place the model lives. Every architect's laptop holds a genuinely complete local repository, one that keeps working, keeps accepting edits, and keeps queuing changes regardless of what is happening to the server. So the real disaster-recovery scenario is not "the server died and we restored it." It is "the server died, we restored it from last night's backup, and meanwhile a dozen laptops that never noticed the outage have spent the day producing work the restored server has never heard of." That is a fundamentally different, and considerably more interesting, problem.

Why this situation is specific to local-first architecture

It is worth being precise about why this scenario barely exists for a conventional centralised Enterprise Architect deployment, because the contrast explains why the problem deserves its own article rather than a footnote to ordinary backup procedure. In a shared-database setup, when the server goes down, nobody can work — there is no local copy standing in for the shared repository, so the outage stops everyone simultaneously, and a restore simply picks the whole team back up from wherever the backup left off. Painful, but conceptually simple: one timeline, interrupted and then resumed. A local-first architecture deliberately removes that single point of failure, and removing it is precisely what makes the platform resilient to network problems, server maintenance, and outages in the first place, as covered in the piece on offline collaboration. The cost of that resilience is that the server's timeline and each laptop's timeline can now genuinely diverge, not just briefly during a network hiccup but for hours, if the server has actually gone down and been restored from an older point while the laptops kept going. The very feature that makes an outage a non-event for the working architect is what makes the recovery from that outage a genuinely harder engineering problem than "restore and resume."

Consider the shape of it concretely. The server crashes at two in the afternoon. The most recent backup was taken at midnight. Between midnight and the crash, fourteen hours of legitimate, committed work happened — pushed, accepted, synchronised out to other clients, entirely real. Restoring the midnight backup brings the server back, but it brings it back fourteen hours behind its own recent history, and every one of those fourteen hours of committed revisions is gone from the server's perspective, even though several laptops in the fleet still hold local copies of a repository that included them, built on top of a server history the restored server no longer recognises as its own.

Why blind replay is the dangerous instinct

The obvious-seeming fix is also the wrong one, and it is worth explaining exactly why, because the instinct is a reasonable one on the surface. If a laptop's synchronisation agent simply resumes where it left off — pushing its queued local changes to the server the moment a connection reappears, exactly as it would after any ordinary network interruption — the server will accept those pushes as ordinary incremental changesets built on top of a revision history that, from the laptop's point of view, is completely normal. The server has no way to distinguish "a client catching up after a brief network blip" from "a client whose entire notion of the shared history predates a restore that erased part of that history," because both situations look identical at the level of an individual push: here is a changeset, here is the revision it claims to be based on, apply it.

The danger is not abstract. If the laptop's queued changes were built on top of revisions the restore just discarded, blindly replaying them risks reintroducing a version of events the server no longer has any record of, silently, without anyone realising the server's history and the laptop's history had ever diverged. Worse, if two different laptops each hold slightly different fourteen-hour tails built on the same now-discarded base, blind replay from both could apply them in whatever order they happen to arrive, producing a result that depends on race conditions in reconnection timing rather than on anything resembling a coherent decision about which version of the truth should win. This is precisely the "last write wins by accident of timing" failure mode this whole series argues against, and it is worse here than in an ordinary Fusion conflict, because an ordinary conflict at least has a consistent base both sides agree on. A restore breaks that shared base outright.

An ordinary sync conflict is two people disagreeing about what happened next. A stale-server-after-restore conflict is two participants who no longer agree on what the recent past even was. Treating it as the first kind of problem, and reconciling it automatically, is how a restore turns into a second, quieter data-loss event on top of the one the backup was meant to fix.

Workspace epochs: making the discontinuity visible instead of invisible

The mechanism CelinQ uses to prevent blind replay is deliberately simple, because the goal is not to solve the reconciliation automatically — that is a decision with real stakes that deserves a human's attention — but to make the discontinuity impossible to miss. Every workspace carries an epoch: a counter that starts at one and is bumped, explicitly and only, by an administrator, specifically after a restore or any other operation that resets the workspace's canonical history to a point other than where it naturally was. A client checks its last-known epoch against the server's current epoch as part of its ordinary synchronisation handshake. When the two match, nothing changes; synchronisation proceeds exactly as it always does, because as far as the epoch mechanism can tell, the server's history is continuous with what the client last saw. When they do not match, the client has just learned something important through a single, unambiguous integer: the canonical history it thought it was talking to no longer exists in the form it remembers, and blind-syncing on top of that assumption would be exactly the mistake described above.

This is a genuinely shipped mechanism, not an aspiration — the server exposes an endpoint to read a workspace's current epoch and a separate, administrator-gated endpoint to bump it, and every bump is written to the audit log with the acting administrator and timestamp, so a workspace's restore history is itself part of the permanent record rather than something reconstructed from memory afterwards. The epoch mechanism itself deserves the same precision: it is a signal, not a workflow. What it guarantees is that a stale client cannot silently blend its history back into a server that has moved on without warning. What it does not yet do, as a fully built, guided client-side experience, is walk an architect step by step through comparing their laptop's fourteen hours of orphaned work against the restored server's state and selectively deciding what to keep. That richer recovery experience — a dedicated view where the discontinuous local work can be inspected side by side with the restored canonical state, and specific changes selectively rebased onto the new history rather than accepted or discarded wholesale — is the direction the epoch mechanism is deliberately designed to enable, not a guided interface that exists today. The signal exists and is enforced; the polished workflow built on top of it is a natural and specific next step, and it deserves to be described that way rather than implied to already be a finished screen.

What actually happens today when a client hits a mismatched epoch

The current behaviour deserves to be stated concretely rather than left abstract. When a client's push arrives against a mismatched epoch, the server's job is to refuse to treat that push as an ordinary continuation of its history — not to silently apply it and not to silently discard it, but to make the mismatch visible so a human decision governs what happens next, in exactly the same spirit as every other place in CelinQ where an ambiguous situation is surfaced rather than guessed at. In practice, this means the fourteen hours of local work on that laptop are not lost. They remain exactly where they always were: recorded in the laptop's own local repository and in its own queued changesets, fully intact, fully inspectable, exportable, and available to be manually reviewed and reapplied by an administrator or the architect themselves, working directly with the specific elements involved rather than trusting an automatic merge across a boundary the system itself has flagged as discontinuous. This is slower and more manual than the ordinary Fusion path, deliberately so. A restore is a rare, high-stakes event, and the honest position is that it deserves a human looking directly at what is being reconciled, not an automatic process quietly making that judgement on their behalf.

Comparing restored state against local state, and what "rebasing" actually means here

The practical recovery process, done manually today and the intended shape of a future guided workflow, follows a specific sequence worth spelling out, because "reconcile the two" is not specific enough to act on. First, establish exactly where the server's history was cut — the revision list makes this visible directly, showing the last revision the restored backup actually contains, which is the same view described in the piece on the server dashboard as the ground truth behind every summary metric. Second, identify which of the stale client's queued or already-applied local changes were built on revisions at or before that cut point, versus which were built on top of the fourteen hours that the restore discarded — the former are safe, because they describe the same past the restored server still remembers; the latter are the ones that need a deliberate decision. Third, for each change in that second group, decide explicitly: does this specific edit still make sense against the restored state, in which case it can be reapplied as a fresh change against the current canonical history — the sense in which "rebasing" applies here, taking a change whose original base no longer exists and reapplying its intent on top of the base that does — or does it depend on other work from those same fourteen hours that the restore erased, in which case reapplying it in isolation would be wrong and it needs to wait, be redone from scratch, or simply be accepted as genuinely lost, which is the honest outcome a restore sometimes produces and no reconciliation mechanism can paper over.

That last possibility deserves to be said plainly rather than avoided. A restore recovers the server to a specific point in time. Work that happened after that point, on the server's own timeline, is gone unless a client's local copy independently preserved it. CelinQ's contribution to this situation is not a promise that nothing is ever lost after a restore — no backup-and-restore system anywhere can promise that, and any vendor who claims otherwise is not being straightforward about what a backup actually is. Its contribution is narrower and more defensible: nothing is lost silently. The epoch mechanism guarantees that a laptop holding fourteen hours of orphaned work will be stopped and flagged rather than quietly folded back in as if nothing had happened, and that the laptop's own copy of that work survives locally, untouched, for exactly as long as it takes a human to decide what to do with it.

A familiar shape: this is the Rescue Zone problem, at server scale

Architects who have already read about the Rescue Zone — the mechanism that preserves offline work when it collides with a concurrent delete rather than either silently resurrecting the deleted material or silently discarding the offline work — will recognise the shape of this problem immediately, because it is structurally the same dilemma at a different scale. In both cases, two honest accounts of reality have diverged: someone's local work says one thing happened, the canonical record says something else, and both are, in their own context, true. In both cases, the wrong instinct is to pick a winner automatically, because either choice destroys something real. And in both cases, the right instinct is the same: make the divergence visible, preserve every version of the truth exactly as it stood, and put the actual decision in front of a person who can see both sides rather than guessing from timing. The Rescue Zone applies that instinct to an individual package colliding with a delete. Workspace epochs apply the identical instinct to an entire server's history colliding with a restore. Neither mechanism resolves the underlying disagreement automatically, and neither should, because "automatically" is precisely how quiet, undetected data loss happens in systems less careful about preserving both sides of a genuine divergence.

This consistency of approach is not an accident and it is worth naming, because it is one of the more reassuring things about evaluating a platform like this as a whole rather than feature by feature. A tool that handled small-scale conflicts carefully but reverted to "restore and hope for the best" the moment the scale grew to an entire server would be telling you that its careful conflict handling was a feature bolted on for the common case rather than a genuine design principle. The fact that the same discipline — detect divergence explicitly, preserve everything, escalate to a human — shows up identically at the scale of one package and at the scale of an entire workspace's history is a much stronger signal about how the rest of the platform is likely to behave in situations this article has not covered at all.

Where this fits against the wider centralised-versus-local-first trade-off

Honesty requires saying plainly that this entire problem — a server's timeline and a laptop's timeline genuinely diverging during an outage — simply does not arise in a purely centralised deployment, rather than only pointing at what local-first buys you in exchange. A team on a single shared database restores from backup and everyone, without exception, resumes from that exact restored point, because nobody had anywhere else to keep working during the outage. There is no epoch to bump because there is no second timeline to reconcile against. That simplicity is real, and for some organisations, particularly ones with limited operational capacity to run the kind of DR testing this article recommends, it is a genuinely reasonable reason to prefer a centralised architecture, a trade-off examined in full in the piece on centralised consistency versus local-first availability. What local-first buys in exchange is that an outage stops being a team-wide event at all — the fourteen hours of work this article has been describing were fourteen hours of real, useful, uninterrupted modelling that would simply not have happened on a centralised system, where the same server crash would have stopped every architect cold rather than only complicating the eventual recovery for the small number of laptops that happened to keep working through it.

Auditability across the whole event

Every part of this sequence leaves a trace, and that traceability is not incidental — it is the entire reason the epoch mechanism was built as an explicit, administrator-gated action rather than something the server infers automatically. The backup itself, the restore, the epoch bump, and every subsequent recovery action taken by an administrator are recorded in the audit log with actor and timestamp, alongside the ordinary revision history that already records who changed what and when. Months later, if anyone asks exactly what happened during a particular outage — which laptops were affected, how long the discontinuity lasted, which changes were manually reapplied and which were judged genuinely unrecoverable — the answer does not depend on anyone's memory of a stressful afternoon. It is sitting in the audit log and the revision history, in the same durable, ordered form as every other governance record this series has described. For the public-sector and regulated environments this platform is often deployed into, that auditability is not a nice-to-have. It is frequently the difference between a disaster-recovery event that can be explained to an oversight body afterwards and one that cannot.

Testing this before you need it

None of this is worth much if the first time an organisation encounters a mismatched epoch is during an actual emergency. Disaster-recovery testing for a local-first deployment should deliberately include this specific scenario, not just the simpler "restore the server and confirm it comes back" check that most DR runbooks already cover. A realistic test takes an online backup, lets a test client accumulate some genuine changes afterwards, restores the backup to a fresh instance, bumps the epoch, and confirms that reconnecting the stale client is refused as an ordinary continuation rather than silently accepted — and, just as importantly, confirms that the administrator restoring the workspace can actually find the information described above: the cut point in the revision history, the specific changes a stale client is holding, and a clear path to reviewing them. This is the same discipline good disaster-recovery practice already demands of any system holding an organisation's critical records, applied here to the specific and less obvious failure mode a local-first architecture introduces. Running this once, deliberately, before it is ever needed for real, is what turns the epoch mechanism from a piece of engineering nobody has exercised into a procedure an administrator has actually walked through and trusts. The same discipline applies here as in any disaster-recovery practice worth the name: the value of a backup is not proven by taking it. It is proven by successfully restoring from it under realistic conditions, including the messy, specific condition of a laptop that never went down at all.

Honest limits

The honest state of this mechanism deserves restating plainly, rather than letting the reader assume more than what has actually been built. The epoch signal — a mismatch that a client can detect and that stops it from blind-syncing across a discontinuous history — is real, enforced server-side, and audited. What is not yet a finished, polished experience is the guided client-side recovery workflow: a dedicated view that walks an architect through comparing their orphaned local work against the restored canonical state and selectively rebasing it, changeset by changeset, without requiring an administrator to work through the revision history and the client's local queue by hand. Today, that comparison and reapplication is a manual, careful process supported by the revision history and audit log rather than a single guided screen, and any organisation evaluating this for a genuinely disaster-recovery-critical deployment should ask directly how much of that process is automated versus how much still depends on an administrator's own diligence, because the honest answer today leans toward the latter.

The PostgreSQL backup path deserves the same honesty. Where the default SQLite deployment offers a direct, tested online-backup action, PostgreSQL-backed deployments are expected to rely on standard database tooling — pg_dump or WAL archiving — configured and operated by the organisation's own database administration practice, rather than an equivalent single-click action inside CelinQ itself; the server is explicit about this rather than pretending an unsupported operation succeeded. None of this diminishes the value of what does exist: a server that refuses, by design, to let a restore silently overwrite work it never had a chance to see, and an audit trail that makes the entire event reconstructable afterwards. But it is exactly the kind of platform where overselling maturity would be the worst possible mistake, because the people evaluating it are the ones who will be relying on it during the one afternoon a year when everything actually goes wrong.