Reproducing the homology-aware train/val/test split generation workflow

[Liujh0223]

Hi NTv3 team,

Thank you for making the benchmark datasets and split files publicly available.

I have been studying the homology-aware splitting strategy described in the NTv3 paper (Section 4.6.3, p.41) and trying to understand how the published splits.bed files (e.g. rice) were generated.

From the manuscript, my understanding is that the workflow consists of:

1. generating 1 Mb gene-aware genomic windows,
2. retrieving homologous gene pairs from Ensembl BioMart,
3. constructing a weighted window-to-window connectivity graph based on shared homology,
4. applying MCL clustering,
5. assigning clusters into train/validation/test splits,
6. performing additional edge pruning for some species such as rice.

In a previous discussion, I learned that the objective was to minimize rather than completely eliminate homology leakage, which makes perfect sense in practice. github issue

I am now interested in reproducing the split-generation procedure itself, either to recreate the published rice split or to apply the same protocol to additional plant genomes.

In many previous genomic foundation models, train/validation/test sets were typically separated by chromosome-level partitioning. While simple and practical, such strategies can still leave substantial sequence or gene-level homology shared across splits, potentially leading to information leakage and over-optimistic evaluation.

The NTv3 homology-aware splitting strategy is therefore particularly interesting, and I would like to better understand the implementation details behind the published benchmark splits.

Would it be possible to provide more implementation details about this workflow?

In particular, I am interested in:

1. how the 1 Mb gene-aware windows were generated;
2. the exact definition of edge weights between windows;
3. the Ensembl/BioMart release version used;
4. the MCL implementation and selected parameters;
5. how modularity was computed when selecting the inflation value;
6. the precise definition of the 10% homology pruning criterion used for rice;
7. how clusters were assigned to train/validation/test partitions;
8. whether the split generation was deterministic or involved a fixed random seed.

If there is already code or a script used to generate the benchmark splits, a pointer to it would be greatly appreciated.

Thanks for your time and for developing NTv3.

Best regards,
Jianhong.

[Liujh0223]

I have attempted to reproduce the described splitting procedure by using continuous 1 Mb gene-aware windows, Ensembl BioMart homology annotations, constructing a window–window weighted graph based on the total bp length of homologous genes, applying a symmetric connectivity matrix, running MCL clustering with inflation values from 1.5 to 5.0 in steps of 0.5, selecting the optimal inflation based on modularity, applying 10% shared-homology edge pruning, and finally assigning train/validation/test sets at the cluster level.

However, the resulting splits differ substantially from the rice split dataset provided in the paper. I suspect this discrepancy may be due to differences in the BioMart data version, the exact definition of the 10% window-size threshold, or differences in the MCL implementation and parameter settings, which could ultimately lead to different clusters.

This suggests that, for downstream post-fine-tuning of NTv3 on previously seen species, it may be necessary to use the genome splitting strategy and implementation provided by your team, rather than independently generating genome partitions.

I would therefore sincerely appreciate your support on this issue. For some species, multiple reference genomes or genome assemblies are available. Rice, for example, has several genomes representing different varieties. If NTv3 is later used for post-fine-tuning to learn across different rice varieties, determining how to properly split these additional genomes becomes a challenging and important problem.