Persistence and migration

The observation log is the source of truth. Everything else is a cache, and saying so is what makes migration tractable.

What is stored

Two of these choices carry the design.

BankEntry stores the tree, not the features. Trees are the source of truth; renders and $\varphi$ are re-derived on import. That is what lets the feature extractor change without invalidating a saved bank, and it is why restoring a large session costs real work rather than being instant.

The log stores raw $\varphi$ by name. Not standardized, and not by index. Both halves of that matter, below.

A profile is the log plus its standardizer

pub struct Profile {
    pub log: ObservationLog,
    pub standardizer: Option<Standardizer>,
}

$\theta$ is only meaningful relative to the standardization that produced it, so the two persist together or not at all. A log without its standardizer is a set of numbers whose units have been lost.

The posterior itself is not in a profile. It does not need to be: it is recomputable from these two, and shipping a fitted model would mean shipping something that could disagree with the evidence it was fitted from.

A saved session does keep it, as a cache: a refit costs seconds on a slow laptop, and a restore that refitted held the first pair's sounds behind it. It is installed only when the log and the standardizer came back exactly as they were saved: a migration, a repair or a merged session sets it aside, and the session is fitted again after it loads. So it never disagrees with the evidence it is restored beside. A session saved before it was kept loads the same way, with no posterior until that fit.

Names, not indices

FitSet::build projects a stored log onto the current feature names, matching on the name. The rule is same name ⇒ same coordinate, and anything unmatched is left at the new standardizer’s mean, which standardizes to zero and means “this vote says nothing about that axis”.

That is the honest imputation, and it is why by-name storage is worth the bytes. By index, a feature-set change would silently re-interpret every historical vote: coordinate 12 was held_centroid_std yesterday and is mod_density today, and every vote ever cast would now be a claim about a different thing.

Three kinds of change, and the one that fails silently

Dropped coordinate. size and n_mix were removed to break exact linear dependencies. Drop them from the migration too; nothing is lost that was ever usable.

Changed units. These have to be converted, because a value silently carried across a unit change is worse than a dropped one: it is evidence pointing the wrong way. The conversions applied when the audio features moved to the log axis:

Renamed coordinate, the silent failure. When n_delay became n_time, by-name matching would have found no n_time in any historical row and imputed it at the mean for every vote ever cast. That reads as “this player has no opinion about delays” rather than as a rename, and nothing anywhere would have reported a problem.

RENAMES carries the value across, and in this case it is exact rather than a convenience: n_time counts delays and granulators, and no observation predating that wave can contain a granulator, so the old n_delay count is the new coordinate’s value for every row being migrated.

That reasoning is worth copying for the next rename. A rename table entry is only exact if the new coordinate’s extra contributors could not have been present in the old data.

Schema 1 → raw φ

The oldest logs stored standardized $\varphi$ over a 30-coordinate feature set with no names.

Recoverable, because the profile persisted the standardizer alongside it:

$$\varphi_{\text{raw}} = z \cdot s + \mu$$

inverts the transform exactly, and the schema-1 coordinate order is known and fixed (SCHEMA1_NAMES). Then the unit conversions above apply.

This is the concrete payoff of persisting the standardizer with the log: a legacy log plus the standardizer it was written under is the raw data, just encoded. Without the standardizer those votes would be unrecoverable.

Forward compatibility in the small

Individual fields use #[serde(default)] where a default is honest:

Each of those is a case where the default is correct history, not merely a value that parses. That is the bar for adding one: if the default would misrepresent what an old file meant, it needs a migration instead.

Where the browser keeps it

IndexedDB, under the page’s origin. No account, no server, nothing transmitted.

Consequences worth stating in a reference: the hosted build and a locally-served copy are different origins and do not share storage; clearing site data destroys the session; and there is no server-side copy to recover from. The only backup of the taste is a downloaded profile (⌘K’s Download your taste command).

Restore goes sound by sound

Restoring re-measures the saved bank, which is the single most expensive thing the app does on load. It is one entry at a time in bank order (import_session_deferred_v2 → bank_absorb per entry → restore_finish), with fill_progress (“recalling 12 of 40 sounds…”) after each:

• On the farm while a farm worker is ready, as the initial fill runs. See The web runtime. The first eight entries (FARM_AUDIO_AHEAD) are rendered with their audio whatever the cache holds; the rest are read from it where it has their rows.
• In the engine worker otherwise (?farm=0, a machine too small for a farm, or none ready yet), with a yield to the message queue between entries, so the bar moves and requests are answered while it runs. Each entry is read from the render cache when it has the entry's row, and rendered and written back otherwise. A farm worker that reports ready later takes the rest.

The cache has a row for what a farm worker rendered and for what an earlier restore rendered in the engine worker, and for nothing else (see what writes a row), so a sound that joined the bank in play is rendered again by the next restore.

In the engine worker it was one synchronous call (import_session_checked) that re-rendered every entry, read nothing from the cache and posted nothing until it returned: 13.6 s for 40 sounds on one unchanging line in Firefox on an M-series laptop, and minutes on a CPU four or five times slower. That call is kept only for a binary without the deferred surface. Whichever path runs, the bank comes back as import_state builds it, in its order (deferred_restore_equals_import_state).

The persistent render cache

$\varphi$ is a pure function of $(\text{term}, \text{spec})$, which is the determinism contract, so a featurization stored once can be replayed instead of re-rendered. Without that, every reload re-renders the whole bank from nothing: the app’s 40 sounds at ~0.5 s each, for numbers the machine computed yesterday.

Farm workers consult an IndexedDB store (auracle-renders) before rendering and write back every render they make. A restore the engine worker runs itself, with no farm worker ready, reads and writes the same rows under the same keys (bankPass in worker.js), so a session restored with no farm comes back from the store the next time, and so does one whose farm wrote its rows. The engine reports the hit rate per wave, and per restore it runs itself, into the app’s own log.

What writes a row

Only those two: a farm worker's render (a fill's draw, a restore's entry, a guess's candidate), and a render the engine worker's own restore makes. What the engine worker measures anywhere else goes into its memo and no further: a fill with no farm (fill_step), a generation's or ⚡'s children (walks return their child, not a row), a kept edit, an opened patch file, a preset taken in. Those are rendered again by the next restore, which writes their rows then. A row for each would make every return a read; it is not written today.

The engine worker opens the store once at boot, before any farm worker is handed the phrase: it creates the store on a first visit and stamps it with the namespace (below), so the farm workers’ own opens only read it. When each farm worker created and stamped it, their writes queued behind one another, and a first visit at six workers waited about 1.5 s longer for its first sounds. The farm waits two seconds at most for the stamp, which takes about 50 ms; past that, its workers open the store themselves, as they did before.

The key is not enough

render_key addresses $(\text{term}, \text{spec})$, which is everything $\varphi$ depends on given a fixed featurizer. It hashes the inputs, and a change to the normalizer or to a descriptor’s formula is a change to the function: the same key would then name a different measurement.

RENDER_EPOCH (3, in auracle-features’ cache.rs) is that missing coordinate for the featurizer. QUIVER_DSP_VERSION (0.4.0, beside it) is the same coordinate for the DSP library every render calls into: a quiver release can change a sample with no line of Auracle changing, as 0.4.0 did for Pluck and Delay. cache_namespace combines the three: the epoch, the quiver version, and a hash of the phrase spec without its audition clips. A clip is in the render_key of a patch that listens instead (audition clips), so a new clip never clears the store. A namespace mismatch orphans every stored row at once, which is the only correct granularity: a cache whose invalidation is anything less than total will one day serve a number from a featurizer that no longer exists. Bump the epoch on any change to a $\varphi$ coordinate, to loudness normalization (including PEAK_CEILING and TARGET_LUFS), to the vetting thresholds, or to the compiler’s term → module mapping. Move QUIVER_DSP_VERSION with the quiver dependency: a test reads Cargo.lock and fails while they disagree. When in doubt, bump: the cost is one cold boot.

Every row is stored under a key that begins with its namespace (farm_key: the namespace, then render_key), so a row written under another namespace is never a hit. The store’s own stamp is not enough for that: it is checked only when the store is opened, and a tab still running an older build goes on writing its rows after a newer tab has cleared and re-stamped it.

A hit is checked rather than trusted: pre_featurized re-derives the content address from the tree the engine holds at that index and drops the row if it disagrees.

Two deliberate limits

Cached rows carry $\varphi$ without samples, so a job that asked for audio still renders. Serving it a row would move the saving onto the first patches the player actually auditions, which is exactly where wantAudio exists to avoid it. Its row is still written: a restore with no farm asks for no audio, and reads it.

Eviction is “clear everything” past a row cap, which is crude on purpose: an LRU needs an access-time write on every hit, turning the cheap path into a write, and what is being protected is a disk quota rather than a working set.

It lives in the farm worker rather than in the engine’s runFarm loop, whose absorb cursor, re-issue watchdog, and speculative-work handling must not acquire asynchrony. A cache hit is simply a job that returns fast. The engine worker’s own restore reads its rows in one transaction before its first entry. It waits for the stamp as a crew does (renderStoreReady, at most RENDER_STAMP_MS), then for the open and the read, at most RENDER_STAMP_MS again, so up to twice that in all. A store that opens after that is closed unread, and every entry is rendered and none written back. One that opens in time but whose rows come later costs the entries measured before they come: those are rendered, and the rest are read.

Pins live engine-side

Candidate::pinned and BankEntry::pinned, not a UI-side set.

The engine is what evicts, so the engine must be what knows about exemptions. Holding pins in the UI beside the stars would rebuild exactly the split that made the stars-are-saves bug possible.

Capped at pool_size / 4 (Engine::pin_cap) so the pool can never be pinned solid. That state has no honest report, because it surfaces as insert_candidate returning None, which the app reports as children that did not rate above the sounds they would replace (3 were bred, but none rated above the sounds they would replace).

Kept as new, protected until a pick

Candidate::unjudged and BankEntry::unjudged: the other engine-side exemption, and not a pin. commit_edit (keep as new, and a shared patch imported through it) sets it on the sound it admits, after recording the comparison that kept it, which judges the original: the pick that keeps a sound is not a judgment of it. It clears the first time the sound is one of the two in a recorded pick: record_duel_as (every dealt pair, the warm start's, and a later kept edit's comparison with it as the original) clears both sides, and record_tree_duel (a PERFORM offer heard and answered) clears any member whose tree is either side; the app's perform_record also clears the sound in hand as it is in the bank, since PERFORM plays it with its controls moved (Engine::mark_judged). A cut (record_keep(false)) is an answer about the sound and clears it too; stars leave it set.

A PERFORM answer is recorded a window after it is given (eight seconds for a Take, so DON'T COUNT IT can drop it), and in that window the player can take the offer onto the bench and keep it as new. So the answer carries asOf, the newest pool id the page had shown when it was given, and record_tree_duel_as_of and mark_judged clear only members with an id at most asOf (ids are issued in order). A sound kept after the answer was given is not judged by it. The page's knowledge can only lag the engine's, which errs toward keeping a protection a little longer.

The rule is Candidate::kept() (pinned || unjudged), and with what the engine holds for the moment (a ⚡ seed in flight, the sounds on EVOLVE's table: Engine::spared) it is the one insert_candidate and eviction_order_by both apply, so retiring(), may_replace(), refine_finish, a ⚡ child's trim and a preset's insert all pass over it alike. (may_replace() passes over the table only for what a generation's end would take now, and lists the sounds the table could send it to as well: refinement.) It is not charged to pin_cap and the app shows no mark for it. The table is not saved: the app holds the pair it shows and the pair of a pick in its undo window (set_on_table), and a restore holds nothing.

Bounded at pool_size / 4 (Engine::unjudged_cap) on the flags of unsaved members: a keep that would put one more past the cap clears the oldest (lowest id; ids are issued in order), which stays in the pool as an ordinary member. A saved member is protected by its pin and is not counted; unsaving it bounds the flags again. With pins capped the same, at least half of a pool of four or more is never kept (below four, the caps' floor of one each can protect more than half). Every eviction also passes over any ⚡ seed in flight and the sounds on the table (at most ON_TABLE_MAX, four), and an insert over its own protect too (the sound a kept edit was made from). That still leaves an insert a member to evict in any pool of thirteen or more. In the app's 40 it leaves at least 14, more than the ten a generation's walks can put the pool over, so there an insert always lands and a generation's end always brings the pool back to size. A restore applies the cap again, for a file written under a larger one.