Ore Mba'e Exploration 📖🔍

This repository contains the code used to explore the text extracted from the book 📕Che Ñe'e, Che Purahéi from the 🔗Ore Mba'e Foundation. The book compiles a series of 222 poems and songs in both Spanish and Guarani. The goal is to extract all Guarani texts from that list.

Input Data 📥

The input data consists on three json files 🗂️data/raw that were created by processing three splits of the original book in pdf format 🗂️data/raw/pdf. The files contain lists of ordered chunks of text that were extracted from the pdf files.

Data Processing ⚙️

The pipeline 📄orembae_extraction_pipeline.py is compressed by the following steps:

1. Extract and compile all chunks from the different files.
2. Process chunks and extract any usefult text from them.
3. Iterate trhough the chunks to separate them into groups according to the list of titles and authors 📝poems_authors.jsonl.
4. Filter out any rogue (non-useful) chunks and join the chunks for each poem or song in a single text.
5. Put all useful content in the extracted format, as specified in the 🗃️Existing Guarani Corpora repository:
   • Identify the source file(s) of the document.
   • Identify the language of the document using the 🗃️Language Identifier Tool.
   • Count the number of words on each document using split(), spacy with punctuation, and spacy without punctuation.
   • Count the number of characters on each document.
   • Separate the Guarani documents from the ones in Spanish.
6. Save the documents to the output directory 🗂️data/processed.

Output Data 📤

The output data 📝guarani_documents_orembae.jsonl consists on 134 Guarani documents. Aditionally, the 88 discarded documents 📝discarded_documents_orembae.jsonl are also stored.

Characteristics 📊

The compiled Guarani documents have the following characteristics:

• Total number of documents: 134
• Average language score: 0.99
• Average number of words using split(): 123.58
• Average number of words using spacy with punctuation: 138.80
• Average number of words using spacy without punctuation: 123.42
• Total number of words using split(): 16560
• Total number of words using spacy with punctuation: 18599
• Total number of words using spacy without punctuation: 16538
• Average number of characters: 874.06
• Total number of characters: 117124

Setup and Usage 🛠️

To use the code in this repository, you need permission to clone the 🗃️Language Identifier Tool