In Silico Validation: What We Tested in Neoantigen Retriever

A cancer-vaccine candidate starts as a sequence hypothesis: a tumour mutation may change a protein, and part of that altered protein may be displayed to immune cells. Before choosing targets for laboratory testing, we need to know whether our software constructs the right sequences and whether its predictions help us select useful candidates.

We evaluated Neoantigen Retriever using four public resources: canine mammary tumour sequencing, canine DLA-associated peptides, and two human resources with experimental immune-recognition outcomes. Each resource answers a different question. Together they let us examine sequence construction, RNA evidence, prediction and candidate ranking.

The findings are useful and mixed. We obtained reproducible sequence-construction results and executed a selected raw DNA/RNA workflow. An existing predictor enriched observed canine peptides over shuffled controls. Adding expression improved the small TESLA ranking panels, but binding alone performed better in the larger IMPROVE comparison. This article explains what those results mean for the next version of Retriever.

This is an accessible account of the computational research release completed in October 2026. The companion technical report contains the methods, exclusions and run references. It is available for scientific review; independent researcher reproduction remains open.

DNA is the instruction sequence; a gene is a named region within it. A mutation is a change at a particular location. Tumour and matched-normal DNA reads help identify changes associated with the tumour rather than inherited differences. Alignment places reads on a reference genome, and variant calling records candidate changes in a VCF file.

Gene and transcript annotation then tell us how a supported change could alter a protein. We generate short overlapping peptide windows containing the alteration. In our construction experiment, 8–11 means the number of amino acids in each peptide, not the number of candidates per mutation. A single mutation can produce many related windows.

DLA is the dog’s version of the MHC presentation machinery; HLA is the human version. An allele is a particular version of that machinery. Binding prediction needs both the peptide and the relevant allele. Candidate ranking can then combine predictions with other evidence to choose a limited experimental shortlist.

RNA adds two distinct measurements. Gene expression estimates how much RNA comes from a gene, combining normal and mutated versions and potentially several cell types in the tumour biopsy. Mutation-specific RNA support asks whether reads spanning the exact change actually carry the altered sequence. The second is more specific to our candidate.

Project Rosie connected existing components into a prototype: an annotated variant file, pVACseq, an existing prediction model, a weighted scorer, explanatory outputs and an mRNA sequence-design output. A separate VEP wrapper could prepare annotation. Its canine demonstration used synthetic variants at real canine gene coordinates.

Retriever shifted the focus toward inspectable research processing on published specimens. We added explicit sample/reference checks, a custom coding-sequence and peptide-construction implementation, recorded exclusions, RNA evidence and repeatable benchmarks. Selected raw DNA and RNA processing used established genomics tools.

This distinction matters when interpreting the experiments: our evaluated mammary construction route used our Python implementation, not VEP/pVACseq. We independently checked emitted sequences with BCFtools. A matched-input VEP/pVACseq comparison is still a useful next check. We did use pVACbind, another pVACtools component, for a separate peptide-input comparator.

We did not train a new binding model in this release. The current evaluated open-model route used MHCflurry 2.0.6 with its recorded upstream ensemble. Software construction, binding prediction and ranking policy are separate components, and their evidence should be assessed separately.

The 2019 canine mammary study [1] supplied published variants, clinical/sample metadata, expression values and public sequencing resources. It was useful because we could first inspect construction from processed files, then examine the raw-read route in a selected case.

We inventoried 197 cases. Restricting to malignant cases, matched-DNA metadata, tumour-RNA columns and exact paired VCF labels left ten qualifying cases. Eleven exact paired labels existed overall, but ten met the RNA requirement. These were source-label matches, not genotype-confirmed identity. Unresolved legacy aliases were not silently merged.

For these ten cases, our Python code used CanFam3.1 and Ensembl release 91, explicit tumour sample selection, reference-allele checks and declared coding/transcript eligibility rules. It constructed 14,109 peptide windows. All emitted windows agreed with an independent BCFtools consequence/reconstruction check, and candidates, exclusions and RNA-linkage outputs repeated exactly.

That answers a bounded question: the sequences we emitted matched the independent reconstruction. It does not measure complete sensitivity across all transcript or mutation classes. Our implementation translates individual variants and records unsupported contexts; it does not phase several mutations into one altered sequence.

We also processed CMT-002 from actual tumour and matched-normal FASTQ files. BWA-MEM 0.7.17 aligned reads; samtools 1.13 and GATK 4.6.2.0 handled alignment processing, duplicate marking, paired Mutect2 calling and FilterMutectCalls. Calling used annotation-derived coding intervals with 100-base padding, rather than the study’s original capture BED.

The selected raw case produced 70 PASS variants and 605 peptide windows, all independently checked. Raw processing completed once; fresh downstream processing repeated the construction results. This demonstrated execution from reads to checked sequences, while variant-calling accuracy against an independent biological truth set remained unmeasured.

Published gene-expression values let us link candidate genes to RNA abundance in the ten-case analysis. They could not tell us on their own whether the specific mutated version was transcribed. For that question, we processed matched tumour RNA from CMT-002.

HISAT2 2.2.1 aligned the RNA reads to the compatible reference, followed by samtools processing and our quality-controlled fragment counter. We required suitable mapping/base quality, unique alignment and local allele context, and counted overlapping mates once. The recorded support rule required at least three alternate-supporting fragments and ten usable fragments.

Of the 70 DNA PASS variants, seven met that RNA-support rule, 57 lacked sufficient usable coverage, and six had adequate coverage but fell below the alternate-support threshold. Only one of the seven supported variants generated candidate windows in the evaluated construction output, giving 38 overlapping windows.

The practical lesson is that “not confirmed in RNA” has more than one meaning. Insufficient coverage is different from observing adequate coverage without enough mutant evidence. Those distinctions should survive into a candidate report. The counts repeated from the same alignment; the later mutation-specific support was not yet integrated as a tested feature in the binding/expression ranking policy.

The canine immunopeptidomics study [2] supplied peptides experimentally recovered with DLA presentation machinery and reported donor genotypes. We curated 16,359 observations across four canine donors from the study’s supplementary tables. We did not reprocess its raw mass-spectrometry data.

Our question was whether an existing model would rank observed peptides above a controlled artificial background. MHCflurry 2.0.6 [5] scored observed peptides and composition-preserving shuffled alternatives. For each donor, K was the number of unique eligible observed peptides. In a roughly balanced observed/decoy pool, random top-K selection would recover approximately half the observed peptides.

Three dogs had complete supported genotype coverage. Observed-peptide recovery was 69.0% for Lola, 72.6% for 163828A and 60.0% for Bogey. Lily was excluded because the reported genotype included unsupported DLA-88*501:02. Three engineered human-host DLA contexts were analysed separately and were not counted as additional dogs.

These results show enrichment against the chosen shuffled background, with repeatable ranks and metrics. They are not calibrated canine binding measurements. Exact-peptide training exposure was material: the audit found matches for 191 Lola peptides, 432 for 163828A and 1,733 for Bogey at some allele in the supplied training table. Shuffles are also artificial comparisons rather than laboratory-confirmed negatives.

This experiment therefore gives us a transparent baseline and coverage audit to improve upon. It also shows why a future canine benchmark needs careful donor/allele splits and model-training exposure checks.

We used human assay resources for questions that our canine sequencing cohort could not answer. TESLA [3] includes peptide/HLA candidates with experimental recognition outcomes, source predictions and expression. These supplied measured labels for evaluating shortlists; we did not reprocess its raw DNA/RNA.

In TESLA S4, 608 source rows from six patients included 37 positive recognition labels; 404 rows entered the common-feature ranking comparison. Binding alone recovered eight positives in the top-ten lists. Our fixed policy combining within-patient binding and expression ranks with 80/20 weights recovered 12.

The separate S7 panel contained 310 rows from three patients and four positives. With the policy fixed before row-level outcome evaluation, top-ten recovery was one positive for binding alone and three for 80/20. Published aggregate information was already known, and four positives provide limited evidence for broad performance claims.

We also ran pVACbind 7.1.5 with MHCflurry as an established peptide-workflow comparator. It recovered nine top-ten positives in S4 and three in S7 on the reported panels. Exact peptide/HLA matching mattered: a shorter predicted peptide did not automatically inherit a longer assayed peptide’s recognition label. This compared configured prediction/filtering workflows, not full genomic pVACseq construction.

A separate TESLA analysis asked whether predicted binding order agreed with measured competition IC50. It included 496 eligible rows, representing 493 unique pairs across 11 HLA alleles, after excluding missing measurements and ambiguous zeros. Source NetMHCpan predictions had higher within-allele correlation in eight alleles; MHCflurry did in three. Agreement varied by allele, and correlation did not establish numerical calibration.

IMPROVE [4] supplied the additional expression and DNA variant-allele-frequency fields needed to compare our policies with Rosie’s original scorer. Its source table contained 17,520 rows from 70 patients and 81 samples, with 467 positive labels. Removing rows missing required expression or DNA VAF left 17,412 common-feature rows.

We executed fresh MHCflurry predictions and imported the original Rosie scoring module unchanged. Its configured inputs included those affinities, source DNA VAF and the source NetMHCpan binding-percentile fallback. Rosie retained its native filters and 50/30/20 scoring weights. Our 80/20 binding/expression rule was a fixed heuristic, not a trained model.

At ten candidates per patient, binding alone recovered 42 positives, the 80/20 policy recovered 39, and configured Rosie recovered 34 across 700 shortlist positions. Rosie’s filters also changed eligibility, so common input rows did not make all policy choices identical. Its native top-20 result was reported separately.

The answer was useful: adding expression did not consistently improve ranking, and more scoring terms did not automatically produce a better shortlist. We retained this outcome rather than retuning the policy to improve the reported result. IMPROVE was an exploratory comparison selected after earlier results; training exposure and overlap with TESLA were recorded, so these panels should not be pooled as wholly independent evidence.

The strongest engineering result is scoped sequence-construction agreement and repeatability. The selected raw DNA/RNA work establishes an executable upstream path and separate RNA evidence. The model and scoring comparisons give us baselines, coverage exclusions and mixed performance outcomes to carry into the next iteration.

No single dataset connected every stage. The mammary cases lacked required patient DLA typing, so they did not complete an evaluated patient-specific prediction/ranking route. The canine presentation experiment started from measured ligands, while recognition-based ranking used human HLA panels. Keeping those boundaries visible is what makes the overall evidence useful.

These results define concrete next experiments. First, compare Retriever’s construction with a compatible VEP/pVACseq route on the same variants, reference and declared transcript policies. Second, connect a genuinely typed canine case to molecular and RNA evidence so the integrated prediction/ranking path can be evaluated.

Third, curate a canine evaluation resource with distinct presentation, measured-binding and recognition tracks. A new predictor should face the same eligible inputs and an appropriately held-out comparison. A learned ranking model should be assessed separately from predictor changes, using a shortlist budget that reflects laboratory capacity.

Finally, work with experimental collaborators to define the sample requirements and the measured immune-response endpoint before freezing a peptide panel. The next evidence will come from linking candidate selection to suitable canine measurements. We can then preserve the baseline, learn from positive, negative and inconclusive outcomes, and test whether the next iteration improves selection.

The technical report, versioned supplement and review register below make this work inspectable. We welcome scoped scientific critique and independent reproduction of the recorded analyses.