Making sense of single-cell atlases, and how well what's learned from one transfers to another.
A 2026 Bioinformatics paper found that Bayesian-optimized fine-tuning settings meaningfully improve scGPT — a single-cell foundation model — at integrating multi-omic data, but only tested this on bone-marrow blood cells. Does that same already-tuned configuration still work when reused, unmodified, on a different tissue, or does it need its own expensive re-optimization? Surfaced by cross-checking three independent AI “deep research” runs against each other for genuinely open questions.
Read the full write-up → — all three regret tables, the UMAP figures, and the data behind them. Notebooks and results are public at github.com/kaysagit/scgpt-hyperparameter-transfer.
| Data | Source paper's own released bone-marrow (BMMC, CITE-seq RNA+protein) benchmark and winning hyperparameters, decoded directly from its published code. Three independent target datasets, each verified real and correctly formatted, then built into a clean dataset: a non-small-cell lung cancer CITE-seq atlas (17,608 cells, 8 patients), a head & neck cancer tumor+lymph-node CITE-seq dataset (20,488 cells, 3 patients — reached after a planned candidate, melanoma, turned out structurally unsuitable), and a healthy-donor PBMC reference (24,998 cells, 8 donors). |
|---|---|
| Method | Fine-tune scGPT on each target dataset two ways: (a) the bone-marrow-derived settings, frozen and reused as-is, vs. (b) a fresh, small hyperparameter search run on that tissue — then compare integration quality (biological conservation, batch-effect removal, rare-cell-type recovery) between the two. |
| Compute & cost | Free Colab T4 for verification and data-prep runs; Colab Pro/Vast.ai GPU for the fine-tuning searches · €0–15 spent so far |
| Deliverable | A “regret” table per dataset — how much integration quality is lost by reusing someone else's already-tuned settings instead of paying for a fresh search on the new data. Three datasets in: each fails (or doesn't) in its own way — two tumor datasets each have exactly one real exception, on two different metrics, while the healthy-blood dataset has none — the strongest sign yet that transfer quality is dataset-dependent rather than one universal rule. |
| Baseline to beat | Not an external benchmark — the comparison is internal (reused config vs. a fresh search on the same data). |
| Pitfall watch | Ground-truth cell-type labels must come from each dataset's own independent annotations, never the model's own output, or the “integration quality” score becomes circular. Held-out evaluation cells are kept completely separate from every fine-tuning run. A dataset's own patient count matters too — a metric that "transfers well" can still be riding on one patient's cells if there are only a few in the cohort (found and flagged for the 3-patient HNSCC dataset; confirmed not an issue for the 8-donor PBMC dataset). |
| Risk | Low — all three target datasets are verified and all three fine-tuning results are in and independently checked; main remaining work is the final write-up. |
How well a molecular signal — DNA methylation — tracks the number that supposedly matters least in biology and most everywhere else: age.
In 2013, Steve Horvath showed that DNA methylation levels at a specific set of 353 CpG sites predict a person's chronological age to within about 3.6 years — one of the most independently-replicated results in aging biology, and the foundation the whole later family of "epigenetic clock" biomarkers (Hannum, PhenoAge, GrimAge) built on. This project checks that result directly: take Horvath's own published coefficients, unmodified, and score them against a real cohort that was never used to build the clock, to see how close the prediction actually lands.

Scored zero-shot (no retraining) against 656 real whole-blood samples from an independent cohort (Hannum et al. 2013): MAE 4.77 years, r=0.918, comfortably clearing the sanity floor of 11.91 years (always predicting the mean). The gap versus Horvath's own reported 3.6-year MAE is expected, not a concern — this cohort was used to build a different clock (Hannum's), never Horvath's, so this is genuine cross-cohort external validation rather than an in-sample number.
| Data | Horvath's own published 353-CpG clock coefficients (via the methylclock R/Bioconductor package). Validation cohort: GSE40279 (Hannum et al. 2013), 656 real whole-blood samples, Illumina 450K methylation array, fully public GEO data. |
|---|---|
| Method | Apply Horvath's already-published, externally-trained coefficients directly to an independent cohort's methylation data — zero-shot, no retraining or fitting on this data at all — then compare predicted vs. real chronological age. |
| Compute & cost | Local CPU only (R/Bioconductor) · €0 spent — the only real cost was environment setup time, not compute |
| Deliverable | Mean absolute error against Horvath's own ~3.6-year benchmark, a predicted-vs-actual scatter, and a batch-leakage check. Result: MAE 4.77 years, r=0.918 — replicates. |
| Baseline to beat | Horvath's own ~3.6-year MAE is the number to match. Sanity floor: beat "always predict the training set's mean age" (11.91 years here) — cleared by a wide margin. |
| Pitfall watch | GEO methylation cohorts are pooled from separate processing batches; a model can look deceptively accurate if batch identity happens to correlate with age. Checked directly: none of this cohort's 9 plates is confined to a narrow age range. |
| Risk | Low — extremely well-documented ground truth, and the result is already in and checked. |
When a tumor's DNA mutates, the resulting altered proteins can get displayed on the cell surface for the immune system to recognize — a displayed mutant fragment is a neoantigen, and it's the basis of most modern cancer immunotherapy. Which mutations actually produce ones the immune system notices is still one of the field's open questions.
Chromophobe renal cell carcinoma (KICH) is a rare, understudied kidney cancer with a distinctive genomic quirk: relatively few point mutations, but a disproportionate share of small insertions/deletions (indels). An indel that shifts the DNA reading frame produces a stretch of protein sequence the immune system has never seen — and frameshift-derived neoantigens are reported in the literature as unusually immunogenic. Nobody had pointed a modern immunogenicity-aware scorer (not just a binding-affinity predictor) at this specific cancer type before. This project builds that catalog from scratch and tests whether that expectation holds up here.
The original angle — benchmarking immunogenicity predictors against each other — turned out to be oversaturated by well-resourced 2023–2026 papers already doing exactly that; this project instead points an established scorer (PRIME2.1) at real, previously-unanalyzed patient data. One real, disclosed limitation runs through every result below: true per-patient HLA genotypes turned out to be gated behind restricted-access archives everywhere checked, so every candidate is scored against a fixed population-representative panel of common HLA-I types, not each patient's real genotype.
Correction, 2026-08-18: the finding originally published here (frameshift candidates score less immunogenic than missense, p≈3×10−18) was retracted after external review caught a pseudoreplication bug: the significance test ran on all sliding-window candidate peptides as independent observations, but frameshift mutations mechanically generate ~2.7× more overlapping peptide-windows per event than missense mutations, inflating the p-value asymmetrically. Re-run correctly at the mutation level (one representative peptide per mutation, n=66 frameshift vs. 865 missense): no significant difference, p=0.54. Fixing the same bug in the burden metric used for survival analysis had the opposite effect — it turned a previously-dismissed result into the project's one real finding, below.
Headline result (corrected): immunogenicity-weighted neoantigen burden associates with worse survival in this cohort (Cox HR=2.14, p=0.00073; log-rank p=0.0092 — the two tests now agree, and the result survives removing the single largest-burden patient). The originally-published mutation-type comparison and the immune-infiltration correlation are both honest null results; only the survival association held up under a corrected, non-inflated analysis. Caveat carried forward plainly: this rests on only 65 patients and 9 observed deaths (8 without the outlier) — a real, worth-following-up signal, not a definitively established one.
| Data | TCGA-KICH somatic mutation calls (GDC's current per-patient pipeline, 66 patients), RNA-seq gene expression and clinical/survival data (GDC, all open-access), reference protein and coding sequences (Ensembl REST API). |
|---|---|
| Method | Custom mutation→peptide pipeline (missense: direct substitution against the reference protein; frameshift: real CDS-coordinate translation to the new stop codon) scored against a fixed 10-allele HLA-I population panel with PRIME2.1 + MixMHCpred3.0, since true per-patient HLA genotypes are not openly accessible for this cohort. |
| Compute & cost | Fully local CPU · €0 spent |
| Deliverable | The first immunogenicity-aware, mutation-type-stratified neoantigen catalog for this cancer type. Corrected headline (2026-08-18): immunogenicity-weighted burden associates with worse survival (Cox & log-rank agree, FDR-surviving, outlier-robust) — the mutation-type comparison originally published here did not survive a corrected, non-pseudoreplicated test and is now an honest null result. |
| Baseline to beat | No existing baseline for this exact analysis — that's the point. Original sanity check (frameshift scores more immunogenic than missense) came back null once tested correctly at the mutation level, not one direction or the other. |
| Pitfall watch | Scored against a population HLA panel, not real per-patient genotypes (five sources checked, all gated). A real pseudoreplication bug (sliding-window peptides treated as independent observations) was caught by external review 2026-08-18 and fixed — see the tech log. Multiple-testing correction re-applied across the corrected claims. |
| Risk | The surviving survival finding rests on only 65 patients / 9 deaths (8 without the largest-burden outlier) — reported as a real, hypothesis-generating signal, not a definitively established one. Mutation-type and immune-infiltration results are both honest nulls. |
A direct follow-on to the KICH project: after that project's original mutation-type finding turned out pseudoreplicated and null once corrected — at only 66 frameshift mutations, too few to detect anything but a large effect — this project reruns the identical corrected pipeline (PRIME2.1 + MixMHCpred3.0, same fixed 10-allele HLA-I panel, mutation-level aggregation built in from the start) on a cohort chosen specifically to have the statistical power KICH lacked: the MSI-High subset of TCGA-STAD (gastric adenocarcinoma), a classic mismatch-repair-deficient hypermutator with indels concentrated in mononucleotide-repeat slippage — a deliberately different mutational mechanism than KICH's own non-MSI indel skew.
80 MSI-High patients identified via two independent open sources (cBioPortal's MSIsensor score and TCGA's own published molecular subtype), cross-validated against each other. Real data confirmed via GDC's current mutation pipeline: 23,808 frameshift indels across all 80 patients (vs. KICH's 88 across 43/66) — every single patient has both mutation types, unlike KICH's 35% with zero frameshift mutations, making a per-patient paired comparison possible for the first time.
Where this stands: the question has split in two — the headline half is resolved, a narrower half is under active test. An equal-N subsampling check (2026-09-02) removed the window-count confound by construction: for every mutation, sample a fixed number of candidate windows before taking the best score, so both arms feed the same number of draws into the best-of-N. Across every configuration the naive effect collapses — rank-biserial r falls from 0.14 to |r| ≤ 0.05 — and the cleanest apples-to-apples comparison (frameshift mutations whose window count genuinely overlaps missense's, both subsampled to a common N) is a flat null: p=0.66. So at the level of a whole mutation event, a frameshift does not make its best candidate neoantigen meaningfully more or less immunogenic than a missense one, once frameshift stops getting more attempts at the target. That confirms the KICH project's corrected null at ~240× the frameshift sample size; the p=7×10−141 was almost entirely the multiplicity artifact.
What remains is a per-peptide question: holding the number of attempts exactly equal, is a frameshift-derived peptide itself scored differently? Compared strictly within bins of identical candidate-window count — and deduplicated against the MSI-High habit of producing the same neo-ORF peptide at recurrent slippage hotspots — this is not a flat null: it changes sign with neo-ORF length. Frameshift scores worse for short novel reading frames and better for long, well-expressed ones; in the bin matching a normal full-length missense mutation, frameshift is favored at p=9×10−12. Whether that is real per-peptide biology or an artifact of PRIME's %Rank scale at its extremes was checked with a structure-aware second opinion: ColabFold structures for the 302 contested peptide-MHC complexes, re-scored by ImmunoStruct (a 2025 model built on predicted structure and biochemistry, not just sequence, trained independently of PRIME). The result does not corroborate PRIME's signal — flat null (p=0.91, rank-biserial r=−0.008; a feature-sensitivity variant agrees). It isn't simply noise: ImmunoStruct's own scores still correlate with PRIME's %Rank within this set (Spearman=−0.19, p=7×10−4) — it specifically doesn't see a class-level frameshift effect where PRIME sees a large one. Honest read: the per-peptide question is now unresolved rather than confirmed reversed-to-null; PRIME's signal alone looks meaningfully weaker under an independent, structure-aware model. The available ImmunoStruct checkpoint is infectious-disease-trained rather than cancer-trained, so a stronger cancer-specific comparative check (folding the wild-type peptides too) is the natural next step, not yet run.
| Data | TCGA-STAD MSI-High somatic mutation calls (GDC's current per-patient pipeline, 80 patients), RNA-seq gene expression (GDC, 76/80 patients), reference proteome/CDS (bulk Ensembl FASTA, release 114). MSI-High status from two independent open cBioPortal sources (MSIsensor score, TCGA molecular subtype). |
|---|---|
| Method | Same corrected pipeline as the KICH project: custom mutation→peptide generation, PRIME2.1 + MixMHCpred3.0 against the identical fixed 10-allele HLA-I population panel, mutation-level aggregation (not peptide-level) from the start. Window-count confound controlled by equal-N subsampling. Follow-up structure-aware check on the contested subset: ColabFold peptide-MHC structures re-scored by ImmunoStruct. |
| Compute & cost | Local CPU for the full scoring pipeline (€0); Colab T4 (ColabFold + ImmunoStruct) for the ~300-complex structure-aware check — first cloud spend on this project, small, within the €50–100 budget. |
| Deliverable | A properly-powered test of whether frameshift-derived neoantigens differ in immunogenicity from missense-derived ones. Event-level answer reached (null, at ~240× KICH's power). The per-peptide sub-question — a neo-ORF-length-dependent effect in PRIME alone — did not survive an independent structure-aware re-score; now unresolved rather than confirmed either direction, pending a cancer-trained comparative check. |
| Baseline to beat | KICH's own corrected null result (p=0.54, underpowered) — this cohort has ~240x more frameshift mutations, so a real answer either direction would be informative. |
| Pitfall watch | The pseudoreplication class of bug from KICH is designed out here. A second multiplicity confound (unequal candidate-window counts per mutation between arms) was found and resolved via equal-N subsampling. Recurrent MSI-High hotspot peptides are a third pseudoreplication trap, now deduplicated in the per-peptide analysis. |
| Risk | Low on the event-level conclusion — a stable null under the strongest available control that matches an independent cohort. The per-peptide sign-flip was cross-checked against a second, structure-aware model rather than published as-is, and came back null — but the available checkpoint is infectious-disease- not cancer-trained, so that isn't fully decisive either. Low on data quality throughout. |
Foundation models trained directly on raw DNA sequence, and whether what they learn transfers to predicting the real-world consequences of a mutation — without ever looking at a protein sequence, a structure, or an alignment.
A 2026 preprint (Wei, Li & Lehner) measured the fitness effect of essentially every possible mutation across bacteriophage ΦX174's genome — every single-nucleotide substitution and every single-amino-acid substitution, ~44,500 variants in total — then benchmarked roughly 15 protein-sequence variant-effect predictors (the ESM family, EVE, GEMME, VESPA, Progen2, and more) against the real measurements. Every one of those predictors reads protein sequence, or a multiple-sequence alignment built from many related proteins; none of them look at DNA directly. This project fills that specific gap: Evo2, a genome-scale DNA language model pretrained across bacteria, archaea and phage, scored zero-shot on the exact same benchmark.
Method: for each of the 44,478 designed mutations, compare Evo2's own likelihood of the mutant genome against the wild-type genome — a bigger drop in likelihood means the model finds that specific DNA change more disruptive — and correlate that drop against the mutation's real, lab-measured effect on phage fitness, gene by gene, using the paper's own metric (per-gene Spearman correlation).
At the amino-acid level — the paper's own benchmark, identical rows, identical metric — Evo2 ranks 16th of 35 predictors (mean per-gene Spearman ρ=0.257, median 0.329, n=28,380 variants). A model that has never seen a protein sequence or a multiple-sequence alignment beats every single ESM-family model tested (ESM1v, ESM2 up to 15B parameters, ESM3, ESMC up to 6B), plus Tranception and PoET — while trailing the structure-aware and MSA-based specialists (ESCOTT, GEMME) and the best single-sequence protein language models (VESPA, Progen2).

At the nucleotide level — a comparison no prior study could make, since no protein model can be scored on raw DNA — Evo2's own predictions correlate strongly and consistently with measured fitness for every gene with an unambiguous reading frame: Spearman ρ=0.55–0.65 (all p<0.001). Genes B, E and K have no such unambiguous nucleotide positions at all — every base in them also sits inside another gene's overlapping reading frame, a real quirk of ΦX174's densely packed genome, not a gap in the analysis.

Caveat worth keeping attached to any of this: this is zero-shot in the sense that the fitness measurements were never a training target — the same situation every protein model in the comparison faces with ΦX174's own proteins — but ΦX174 (the first DNA genome ever sequenced) was almost certainly present somewhere in Evo2's training data, so it is not a test of generalization to a genuinely unseen genome.
evo2_1b_base) turned out to require FP8 precision and a Hopper-class GPU per Evo2's own requirements — corrected to evo2_7b_base, which runs in plain bf16 on any CUDA GPU with enough memory, before any compute was spent| Data | The Lehner-lab ΦX174 complete-mutagenesis dataset (Wei, Li & Lehner, bioRxiv 2026, CC-BY): 16,098 single-nucleotide and 28,380 single-amino-acid substitutions across the phage's 11 overlapping genes, each with a measured fitness effect from a competitive growth assay, plus the paper's own published scores from ~15 protein variant-effect predictors for direct comparison. Reference genome: NCBI NC_001422, adjusted for the library's own 4-site strain difference. |
|---|---|
| Method | Zero-shot scoring with Evo2-7b-base (Arc Institute/NVIDIA, StripedHyena 2 architecture, pretrained on 8.8 trillion nucleotides across bacteria, archaea and phage): for every variant, the delta between the mutant and wild-type genome's model log-likelihood, correlated (per-gene Spearman) against measured fitness — matching the source paper's own evaluation exactly. |
| Compute & cost | Colab A100 (plus one Colab L4 comparison run) · roughly a day of wall-clock GPU time across ~9 session-preemption/resume cycles. |
| Deliverable | A DNA-model entry in the paper's own protein-VEP ranking table (16th of 35, mean ρ=0.257) plus a first-of-its-kind nucleotide-level result (ρ=0.55–0.65) that no prior study could produce, since no protein model can be scored on raw DNA. |
| Baseline to beat | The paper's own published predictor rankings — best tested is ESCOTT (ρ=0.54), weakest tested here is Tranception-L without retrieval (ρ=0.01). |
| Pitfall watch | ΦX174 was almost certainly present in Evo2's pretraining data, so this is seen-genome/unseen-phenotype zero-shot, not held-out-genome generalization. Amino-acid-level mutant genomes use a reconstructed minimal-nucleotide-change codon where the exact library codon isn't recoverable from the data — same protein effect, negligible DNA-level noise. |
| Risk | Low on the headline numbers — the full designed variant set was scored and evaluated with the paper's own metric on identical rows. The nucleotide-vs-amino-acid gap (nt-level correlations run higher than aa-level for every gene in common) is a genuine open question, not yet explained. |