Memory & large data.
Single-cell steps can need many times more memory than the file on disk. Oreumics estimates that before it starts, so a long run fails at second zero with an explanation instead of being killed by the operating system twenty minutes in.
What happens before a node runs
- 01
Predict
Oreumics estimates the peak memory for this node on this data. Each node type has its own model — a step that materialises a dense copy is priced differently from one that reads a single matrix — and since v0.25.0 a node is charged only for the matrix slots it actually reads, not for every layer the dataset holds. - 02
Compare
The estimate is checked against what is actually free right now, minus a reserve that keeps the machine responsive and stays above the point where the watchdog would have to intervene. - 03
Warn or refuse
A tight fit shows Memory pressure expected, which you can accept; a step that clearly will not fit shows Likely out of memory with what is holding the memory and the specific settings that would reduce it. A near miss is re-measured for up to twelve seconds before the app gives up, because readings fluctuate and a run short by a fraction of a percent should not be refused on a momentary sample. - 04
Watch
While the node runs, memory is monitored so a runaway step is stopped rather than taking the app down. When the run completes, the measured peak makes the next estimate for that node more accurate.
When a run is refused
The dialog names the node, the data shape, what it needs, what is free, and a list of concrete actions. You can always press Run Anyway — the estimate is a prediction, not a lock — but if the prediction is right the watchdog aborts the task mid-run, so read the suggestions first. In rough order of how often each is the real answer:
The message mentions a dense expression matrix
Re-run the loader node.
A matrix that is mostly zeros but stored densely costs many times the RAM of the same data stored sparsely, and every step downstream inherits that cost. Since v0.25.0 loaders convert dense-on-disk matrices to sparse on load, so re-running the loader is usually the whole fix. This was the usual cause of “30k cells fail on a 32 GB machine”.
The refused node is Scale
Leave “Scale HVG Only” on, or run Highly Variable Genes first.
Scale then keeps roughly 2,000 genes instead of all of them, and PCA uses only those anyway. The full gene set stays in .raw, so DEG and gene plots are unaffected.
The refused node is Scale and HVG is already on
Turn off “Zero Center”.
Centring is what forces a sparse matrix dense — without it Scale stays sparse. Results do differ, so this is a deliberate trade rather than a free win.
The refused node is PCA, Neighbors or Harmony on a wide matrix
Run Highly Variable Genes upstream.
Peak memory scales with the number of genes, and these steps do not benefit from the tens of thousands of low-variance ones.
The dataset is very large
Subset Data first.
Confirm the workflow is right on a slice, then run the whole thing once. It is also the honest answer when the data genuinely exceeds the machine.
Free RAM is low but the dataset is modest
Close other applications — browsers first.
The ceiling is what is free right now, not what the machine has installed. A browser with many tabs can easily be the difference.
The refusal is marginal and you are on Balanced
Switch the profile to Low memory and retry.
Low memory keeps a smaller reserve for the rest of the system, so a run that was refused by a small margin often passes. It also runs fewer heavy steps at once.
Performance profiles
The memory chip in the toolbar opens a menu with four choices. They move the ceiling and the reserve together — a more permissive profile lets a run claim more, at the cost of leaving the rest of the machine less headroom.
| Profile | Use it when |
|---|---|
| Auto (recommended) | Always, unless you have a reason. Chooses Low memory or Balanced from the machine’s total and currently free RAM — and deliberately does not choose Performance by itself, because claiming most of a workstation should be a decision you make. |
| Low memory | Smaller machines, or when you need the rest of the system to stay responsive while an analysis runs. Fewer heavy steps run at once. |
| Balanced | The standard profile on machines with room. |
| Performance (64 GB+) | A dedicated analysis machine you are willing to hand over. Raises the ceiling and runs more heavy steps in parallel; leaves less for everything else. |
The same menu has Export memory diagnostics. If a refusal looks wrong, that file is what to send us — it contains the estimate, the readings behind it and the profile in force.
Disk, not just RAM
While a workflow runs, Oreumics writes checkpoints to a disk cache so that heavy steps can run in isolated worker processes and results survive a crash. The cache has a quota, but since v0.25.0 the quota is a cleanup target rather than a hard wall: stale files are reclaimed to get back under it, and a live checkpoint the current run needs is admitted even when the cache is full. The app also keeps a reserve of free disk space so it can never fill your drive; that reserve scales with the space you actually have.
How many cells can I analyse?
We do not publish a cell count for a given amount of RAM, because the honest answer depends on more than the number of cells: how the matrix is stored (sparse or dense), how many genes survive filtering, which steps you run, and what else is running on the machine. Two datasets with identical dimensions can differ several-fold. The estimator in the app answers this question for your data, which is why it exists — run the first node and it will tell you where you stand.
One structural note that does generalise: keeping the matrix sparse for as long as possible is worth more than any other single choice. That is what the loader, Scale HVG Only and Zero Center are all really about.
What changed in v0.25.0
Runs that fit are no longer refused
- Nodes pay only for the matrices they touch; the admission check used to charge every node for every layer a dataset holds.
- The disk cache quota is a cleanup target, not a wall — it could previously block a live checkpoint and dead-end a run at Highly Variable Genes.
- Matrices stored dense on disk are converted to sparse on load; Normalize and Log1p share buffers instead of copying.
- Refusal messages say in plain language what is holding the memory and what to do about it (v0.24.0), and a near miss is re-measured before giving up.
Related
- Preprocessing nodes — where
hvg_onlyandzero_centerlive. - Input / Output nodes — Subset Data, and the loaders that decide sparse versus dense.
- System requirements.
Read next