Three Tissues, One Config

← Single-Cell Biology · scGPT hyperparameter transfer across tissues

A bone-marrow-tuned hyperparameter search for scGPT gets reused, unmodified, on a lung tumor, a head-and-neck tumor, and a panel of healthy blood. Each one answers differently.

The question

scGPT is a single-cell foundation model: a transformer pretrained on tens of millions of cells, then fine-tuned per-dataset to integrate new single-cell measurements — reconciling different batches, donors, and modalities (RNA plus surface-protein) into one coherent map of a tissue.

A 2026 Bioinformatics paper (Tay, Le & Chua) ran a proper Bayesian hyperparameter search for scGPT's fine-tuning step and found it mattered: the tuned configuration clearly beat scGPT's own defaults on a public bone-marrow CITE-seq benchmark. That's a real result — but it was only ever tested on one tissue, from one benchmark, that the search itself was tuned against.

The open question this project asks: does that specific tuned configuration — unmodified, dropped as-is into a new context — still hold up? Or does every new tissue effectively need its own expensive re-optimization, and the published tuning is only ever valid for the exact data it was tuned on?

The candidate came from cross-checking three independent AI “deep research” runs against each other, then independently verifying the paper was real, the code was real, and the claim reproduced — before spending any compute chasing the actual question.

Method, in one paragraph

First, the paper's own claim gets reproduced independently on its own benchmark (bone marrow). Then, for each of three new datasets, scGPT gets fine-tuned two different ways: frozen transfer, the bone-marrow-tuned settings applied as-is with zero changes, versus dataset-specific optimum, a fresh small hyperparameter search (same search space as the source paper, ~12 trials) run on that dataset alone. Both arms get evaluated three ways — biological conservation (does it still separate real cell types), batch mixing (does it avoid over-separating cells just because they came from different samples), and rare cell-type recovery (does it still find genuinely uncommon cell types, not just the common ones) — each run three times with different random seeds, since a single run can't tell a real effect from noise.

Regret = (fresh search's score) − (frozen transfer's score), averaged across seeds, per metric. A pre-registered threshold of regret < 5% on all three metrics is the bar for “transfers well” — decided before any results existed, precisely so the bar couldn't be quietly moved to fit whatever came out.

Held-out evaluation cells never appear in either fine-tuning run — the same leakage discipline applies to every arm. Ground-truth cell-type labels always come from each study's own independent annotations, never from the model's own output, so “integration quality” can't become circular.

The datasets

All four datasets pair RNA measurements with an antibody-derived protein panel on the same cells (CITE-seq) — the exact multimodal format the source paper's fine-tuning task requires. That combination turns out to be concentrated almost entirely in immune-cell research, for structural reasons (antibody panels are built for immune surface markers) — not a gap in searching. A planned developmental-tissue arm was dropped early for exactly this reason; a planned melanoma arm got further, then was dropped after direct inspection showed it was a single cultured cell line with no real batch variable and no cell-type diversity to test two of the three metrics against. Both dead ends are worth naming, since a published dataset search is never as clean as the datasets that made the final cut.

Every QC figure computed directly from each dataset's real raw counts, using the identical method across all four — not taken from any paper's own reported numbers.

Reproducing the baseline claim

Before touching any new tissue, the source paper's own headline result gets reproduced independently, on its own benchmark, from scratch: default scGPT settings versus the paper's Bayesian-tuned configuration, on the bone-marrow CITE-seq dataset the paper itself used.

Tuned beats default on both metrics, independently reproduced. That's the premise the rest of this project stands on — and the reason it was worth spending real compute asking whether it generalizes.

Three regret tables

Lung tumor 1 exception

Leader et al. 2021, Cancer Cell · NSCLC, 17,608 cells, 8 patients · GSE154826

Frozen transfer

Dataset-specific optimum

scGPT's own cell embeddings, projected to 2D (UMAP) and colored by each cell's independently-annotated real cell type, seed 0. Broadly the same structure either way — consistent with the near-zero avg_bio regret above; the rare-cell-type gap that does show up doesn't read as an obvious visual difference at this scale.

Real GPU-hours: 1.66 across 11 completed search trials (a hyperopt bookkeeping quirk left a 12th trial slot as an empty stub every search run in this project hit — harmless, but worth reporting as 11, not 12). Best config found: nlayers=7, lr≈0.000345, epochs=30, batch_size=32 — close to the source paper's own bone-marrow-tuned config, an independent sanity check that the search itself works. The rare-cell-type regret is real, not noise: every one of the fresh-search arm's three seeds (0.355–0.371) beat every one of the frozen arm's three seeds (0.287–0.326) — a clean, non-overlapping gap, just barely over the pre-registered line rather than a clear-cut failure.

HNSCC tumor + lymph node 1 exception

Rahim et al. 2023, Cell · head & neck cancer, 20,488 cells, 3 patients · GSE212797

Frozen transfer

Dataset-specific optimum

Same projection, HNSCC, seed 0. This is the pair where the regret table's biggest exception (avg_bio, +6.06%) lives — the fresh-search embedding shows visibly tighter, more separated cell-type clusters than the frozen one, a rare case where the quantitative regret is also visible by eye.

Real GPU-hours: 3.94 across 11 completed trials. Best config: nlayers=6, lr≈0.000166, epochs=20, batch_size=16. This is the mirror image of the lung-tumor result: biological conservation, not rare-cell-type recovery, is the exception here — and a cleaner one, with zero overlap across all three seed pairs (frozen 0.477–0.503 vs. optimum 0.536–0.565). Two caveats worth carrying alongside the number: HNSCC has only 3 patients, and one of them alone supplies 60–84% of several of the rare cell types the “passes” verdict above depends on — a real reason to trust that pass a little less than the other arms' rare-cell-type results. And HNSCC has the shallowest RNA capture of the three tissues (700 genes/cell, vs. 935 and 2,041) — a plausible, though untested, explanation for why biological conservation specifically is the axis that struggles here.

PBMC, healthy blood 0 exceptions

Hao et al. 2021, Cell · 8 healthy donors, 24,998 cells · GSE164378

Frozen transfer

Dataset-specific optimum

Same projection, PBMC, seed 0. The cleanest pair of the three — consistent with a clean pass on all three metrics, the two embeddings look nearly interchangeable, right down to the fine substructure within the large T-cell and monocyte clusters.

Real GPU-hours: 6.22 across 11 completed trials (the most expensive search of the three — the largest dataset, and the winning config landed on the priciest corner of the search space: nlayers=6, lr≈0.000708, epochs=40, batch_size=32). The iLISI regret is a real, consistent signal (zero overlap across seed pairs) rather than noise, but small enough to stay well clear of the threshold either way. With 8 real donors, rare-cell-type representation is healthy throughout — no single donor drives the result the way one patient did for HNSCC.

Three different outcomes

“Transfers well, with one small caveat” does not repeat. Each tumor dataset has exactly one real exception — on a different metric each time — and the non-tumor dataset has none.

That's a stronger, more useful result than any single arm alone would have been. If lung tumor and HNSCC had both failed the same metric, the honest conclusion would be a single, specific weak point in the bone-marrow-tuned config — something to patch once and stop worrying about. Instead, the failure axis moves: rare-cell-type recovery for lung tumor, biological conservation for HNSCC, nothing at all for PBMC. That pattern looks more like hyperparameter transfer quality is a property of the specific dataset you're transferring to, not a fixed weakness in the source configuration.

The most counter-intuitive finding sits in the dataset table above, not the regret tables: PBMC has the lowest overlap between its own genes and scGPT's pretrained vocabulary of the three tissues (69.6%, against 88.7% and 97.9%) — and it's the dataset that transferred best. That number was double-checked before trusting it (a genuine worry was that PBMC's gene list had accidentally picked up duplicate-symbol artifacts during processing that would deflate the overlap number for no biological reason — ruled out directly; the affected genes are a real, small immunoglobulin gene family, not a data bug). The likely real explanation is mundane: PBMC's RNA reference is the full, uncurated genome-wide 10x gene panel, while the tumor datasets' gene lists had already been trimmed to more common, better-annotated genes by each study's own pipeline. Whatever the mechanism, the result argues against a tidy “the model already knows your genes, so it'll transfer fine” story — the opposite pattern held here.

And the most basic finding is worth stating plainly, since it wasn't obvious going in: reusing a bone-marrow-tuned configuration on healthy blood worked better than reusing it on either tumor. Tumor tissue, not the biological distance between blood and bone marrow, looks like the harder transfer case — which makes sense in hindsight (tumors are exactly the setting where cell-state heterogeneity and technical variability both spike), but wasn't the assumption this project started with.

Limitations, stated plainly

• n=1 tissue-pair per arm, three arms total. Each regret table is one bone-marrow-to-target comparison, not a distribution over many targets. The three-arm pattern above is suggestive, not a statistical guarantee that a fourth tissue wouldn't repeat one of these failure modes, or invent a new one.
• The 5% threshold is a round, pre-registered convenience number, not derived from a power calculation or a measured noise floor. It was fixed before any results existed specifically so it couldn't be moved after the fact — but it's still an arbitrary line, and results a point or two either side of it (lung tumor's +5.32%, HNSCC's +1.33%) deserve reading in that light, not as a hard pass/fail physical law.
• Rare-cell-type ground truth is thin by construction. A cell type that's rare enough to test “can the model still find it” is, by definition, the type with the fewest cells and the least annotation redundancy — any per-cell labeling noise in the original studies disproportionately hits exactly the metric being scored.
• The bone-marrow reference dataset is the most class-imbalanced of the four (64.2% in one cell type, the most extreme of any dataset here) and has the fewest donors tied with HNSCC (3). The original tuned config was never selected in an environment that demanded much rare-type resolution in the first place — worth remembering when a “regret” shows up on exactly that axis.
• Same 8-patient/8-donor cohort trains and evaluates both arms, for lung tumor and PBMC (3 for HNSCC). A quirk specific to that one cohort could in principle affect both the frozen and fresh-search arms symmetrically without showing up as noise across seeds — seeds vary model init and data shuffling, not which real people the cells came from.

What actually broke

In the spirit of being transparent about how independent research like this actually goes: getting real numbers out of three fine-tuning runs meant finding and fixing a string of genuine bugs, not just waiting for GPUs. None of these are exotic — they're the ordinary friction of running real code against real data — but each one would have silently produced wrong or missing results if left alone.

Data, code & licensing

Every dataset and tool used here is either fully public or permissively licensed, and this write-up was checked against each one's actual terms before publishing — not assumed. All notebooks, results, and the full technical log are public at github.com/kaysagit/scgpt-hyperparameter-transfer (MIT license).

One nuance worth being explicit about: GSE164378's raw sequencing reads are dbGaP-controlled for patient privacy — this project never touched them. Everything used here (processed count matrices, per-cell metadata) is hosted directly and openly on GEO's own public FTP, exactly the files the depositors intended for downstream reanalysis. NCBI's own policy places no restriction on secondary use or publication of publicly deposited GEO data.

Citations

• Tay, J. K., Le, D. T., & Chua, R. L. “Bayesian hyperparameter optimisation for scGPT fine-tuning.” Bioinformatics, 2026. DOI: 10.1093/bioinformatics/btag374. Code & hyperparameters: github.com/daren642/scGPT_multiomic_tuning
• Cui, H. et al. “scGPT: toward building a foundation model for single-cell multi-omics using generative AI.” Nature Methods, 2024. github.com/bowang-lab/scGPT
• Leader, A. M. et al. “Single-cell analysis of human non-small cell lung cancer lesions refines tumor classification and patient stratification.” Cancer Cell, 2021. DOI: 10.1016/j.ccell.2021.10.009. GEO GSE154826
• Rahim, M. K. et al. “Dynamic CD8+ T cell responses to cancer immunotherapy in human regional lymph nodes are disrupted by metastasis.” Cell, 2023. GEO GSE212797
• Hao, Y. et al. “Integrated analysis of multimodal single-cell data.” Cell, 2021. DOI: 10.1016/j.cell.2021.04.048. GEO GSE164378