HLTK Exploration

This repository contains the code used to explore the Guaraní subset of the HLTK dataset.
The goal is to inspect, filter, and quantify Guaraní content across multiple .jsonl.zst files, producing verified and language-filtered versions of the dataset.

---

Download

The original HLTK JSONL-ZST files were downloaded from the official HLPT Project:

• https://data.hlpt-project.org/three/sorted/gug_Latn.map

This index file provides the direct download links for the Guaraní subset (gug_Latn):

• https://data.hlpt-project.org/three/sorted/gug_Latn/5_1.jsonl.zst
• https://data.hlpt-project.org/three/sorted/gug_Latn/6_1.jsonl.zst
• https://data.hlpt-project.org/three/sorted/gug_Latn/7_1.jsonl.zst
• https://data.hlpt-project.org/three/sorted/gug_Latn/8_1.jsonl.zst
• https://data.hlpt-project.org/three/sorted/gug_Latn/9_1.jsonl.zst
• https://data.hlpt-project.org/three/sorted/gug_Latn/10_1.jsonl.zst

These files were placed in the data/ directory for processing.

---

Exploration

The dataset was explored by inspecting the number of documents, words, and characters per file.
Word counts were computed using three methods:

• Whitespace tokenization (split method)
• spaCy’s multilingual tokenizer (including punctuation)
• spaCy’s multilingual tokenizer (excluding punctuation)

---

Processing and Filtering

Each document’s language was identified using the LanguageIdentifier module
(from the GuaranIA pipeline).
This ensemble uses:

• FastText
• GlotLID
• OpenLID

Documents were labeled as Guaraní only if all three models agreed (strict 3-model voting):
Accepted language codes: grn, gug, gug_Latn, gn.

Two final datasets were produced:

• verified_hltk_standard_all.jsonl — all documents
• verified_hltk_standard_grn.jsonl — only documents identified as Guaraní

---

Characteristics

The combined dataset processed in this exploration yields the following statistics:

• Documents in Guaraní: 9,757
• Total words (split method): 6,069,689
• Total words including punctuation (spaCy): 7,660,138
• Total words excluding punctuation (spaCy): 6,360,188
• Total characters: 41,684,424
• Average words per document: 622.09
• Average characters per document: 4,272.26
• Average proportion of Guaraní text (strict ensemble): 0.66

---