opencc-jieba-rs

High-performance Rust-based Chinese text converter using Jieba segmentation and OpenCC dictionaries.

A Rust-based Chinese text converter powered by OpenCC lexicons, using Jieba for word segmentation to improve phrase-level accuracy. This project aims to provide high-performance and accurate Simplified ↔ Traditional Chinese (zh-Hans ↔ zh-Hant) conversion.

Features

• 📦 Simple CLI tool for converting between Simplified and Traditional Chinese.
• 🔍 Lexicon-driven phrase conversion using OpenCC dictionaries.
• ⚡ Accurate segmentation powered by Jieba with a combined Hans + Hant dictionary.
• 🔠 Works with both Simplified (zh-Hans) and Traditional (zh-Hant) Chinese text.
• 🛠️ Designed to be embedded as a Rust library or used standalone.

🔽 Downloads

• Windows (arm64/x64/x86)
• macOS (arm64/x64)
• Linux (arm64/x64)

---

Installation

git clone https://github.com/laisuk/opencc-jieba-rs
cd opencc-jieba-rs
cargo build --release --workspace

The CLI tool will be located at:

target/release/opencc-jieba

Usage: opencc-jieba convert

opencc-jieba convert: Convert Chinese Traditional/Simplified text using OpenCC

(Windows)
Usage: opencc-jieba.exe convert [OPTIONS] --config <conversion>
(Linux / macOS)
Usage: opencc-jieba convert [OPTIONS] --config <conversion>

Options:
  -i, --input <file>         Read original text from <file>.
      --in-enc <encoding>    Encoding for input: UTF-8|GB2312|GBK|gb18030|BIG5 [default: UTF-8]
  -o, --output <file>        Write converted text to <file>.
      --out-enc <encoding>   Encoding for output: UTF-8|GB2312|GBK|gb18030|BIG5 [default: UTF-8]
  -c, --config <conversion>  Conversion configuration: [s2t|s2tw|s2twp|s2hk|s2hkp|t2s|t2tw|t2twp|t2hk|t2hkp|tw2s|tw2sp|tw2t|tw2tp|hk2s|hk2sp|hk2t|hk2tp|jp2t|t2jp]
  -p, --punct <boolean>      Punctuation conversion: [true|false] [default: false]
  -h, --help                 Print help

Usage: opencc-jieba segment

opencc-jieba segment: Segment Chinese input text into words

Usage: opencc-jieba.exe segment [OPTIONS]

Options:
  -i, --input <file>           Input file to segment
  -o, --output <file>          Write segmented result to file
  -d, --delim <character>      Delimiter character for segmented text (use " " for space) [default: /]
  -s, --separator <character>  Separator character for segmented mode=tag (use " " for space) [default: /]
  -m, --mode <mode>            Segmentation mode: cut | search | all | tag [default: cut] [possible values: cut, search, all, tag]
      --no-hmm                 Disable HMM for segmentation and tagging
      --in-enc <encoding>      Encoding for input: UTF-8|GB2312|GBK|gb18030|BIG5 [default: UTF-8]
      --out-enc <encoding>     Encoding for output: UTF-8|GB2312|GBK|gb18030|BIG5 [default: UTF-8]
  -h, --help                   Print help

Usage: opencc-jieba office

Supported Office formats: .docx, .xlsx, .pptx, .odt, .ods, .odp, .epub

opencc-jieba office: Convert Office or EPUB documents using OpenCC

Usage: opencc-jieba.exe office [OPTIONS] --config <config>

Options:
  -i, --input <file>     Input <file> (use stdin if omitted for non-office documents)
  -o, --output <file>    Output <file> (use stdout if omitted for non-office documents)
  -c, --config <config>  Conversion configuration <config> [possible values: s2t, s2tw, s2twp, s2hk, s2hkp, t2s, t2tw, t2twp, t2hk, t2hkp, tw2s, tw2sp, tw2t, tw2tp, hk2s, hk2sp, hk2t, hk2tp, jp2t, t2jp]
  -p, --punct            Enable punctuation conversion
  -f, --format <ext>     Force office document format <ext>: docx, xlsx, pptx odt, ods, odp, epub
      --keep-font        Preserve original font styles
      --auto-ext         Infer format from file extension
  -h, --help             Print help

Example

# Convert Simplified Chinese to Traditional Chinese
opencc-jieba convert -i input.txt -o output.txt --config s2t

# Convert Traditional Chinese (Taiwan Standard) to Simplified Chinese
opencc-jieba convert -i input.txt -o output.txt --config tw2s

# Convert Traditional Chinese (Taiwan Standard) to Simplified Chinese with idioms
opencc-jieba office -i input.docx -o output.docx --config tw2sp --punct --format docx --keep-font

# Segment text file contents then output to new file
opencc-jieba segment -i input.txt -o output.txt --delim ","

# Segment with POS tagging (format: word:tag)
opencc-jieba segment -i input.txt -o output.txt --mode tag --delim " " --separator ":"

# Segment from console input with POS tagging
opencc-jieba segment --mode tag --delim " " --separator ":"

• Supported conversions:
  • s2t – Simplified to Traditional
  • s2tw – Simplified to Traditional Taiwan
  • s2twp – Simplified to Traditional Taiwan with idioms
  • s2hk – Simplified to Hong Kong Traditional
  • s2hkp – Simplified to Hong Kong Traditional with phrase preferences
  • t2s – Traditional to Simplified
  • t2tw – Traditional to Taiwan variants
  • t2twp – Traditional to Taiwan variants with phrase preferences
  • t2hk – Traditional to Hong Kong variants
  • t2hkp – Traditional to Hong Kong variants with phrase preferences
  • tw2s – Traditional Taiwan to Simplified
  • tw2sp – Traditional Taiwan to Simplified with idioms
  • tw2t – Taiwan variants to Traditional
  • tw2tp – Taiwan variants to Traditional with phrase normalization
  • hk2s – Hong Kong variants to Simplified
  • hk2sp – Hong Kong variants to Simplified with phrase normalization
  • hk2t – Hong Kong variants to Traditional
  • hk2tp – Hong Kong variants to Traditional with phrase normalization
  • jp2t – Japanese Shinjitai to Traditional
  • t2jp – Traditional to Japanese Shinjitai

Lexicons

By default, it uses OpenCC's built-in lexicon paths.

---

Library Usage

To add this crate to your project:

cargo add opencc-jieba-rs

Or add the following line to your Cargo.toml:

opencc-jieba-rs = "0.8.0"

Use opencc-jieba-rs as a library:

use opencc_jieba_rs::{OpenCC, OpenccConfig};

fn main() {
    let opencc = OpenCC::new();

    assert_eq!(opencc.convert("这是一个测试", "s2t", false), "這是一個測試");

    // Direct one-pass Hong Kong phrase APIs.
    assert_eq!(opencc.t2hkp("鼠標"), "滑鼠");
    assert_eq!(opencc.hk2tp("滑鼠"), "鼠標");

    // The same conversion through the strongly typed public API.
    assert_eq!(
        opencc.convert_with_config("鼠標", OpenccConfig::T2hkp, false),
        "滑鼠"
    );
}

📦 Crate: opencc-jieba-rs on crates.io
📄 Docs: docs.rs/opencc-jieba-rs

---

C API Usage (opencc_jieba_capi)

You can also use opencc-jieba-rs via a C API for integration with C/C++ projects. The maintained C and C++ headers live in capi/include; add that directory to your compiler's include path.

Example

#include <stdio.h>
#include "opencc_jieba_capi.h"

int main(int argc, char **argv) {
    void *opencc = opencc_jieba_new();
    const char *config = u8"s2twp";
    const char *text = u8"意大利邻国法兰西罗浮宫里收藏的“蒙娜丽莎的微笑”画像是旷世之作。";
    printf("Text: %s\n", text);
    int code = opencc_jieba_zho_check(opencc, text);
    printf("Text Code: %d\n", code);
    char *result = opencc_jieba_convert(opencc, text, config, true);
    code = opencc_jieba_zho_check(opencc, result);
    printf("Converted: %s\n", result);
    printf("Converted Code: %d\n", code);
    if (result != NULL) {
        opencc_jieba_free_string(result);
    }
    if (opencc != NULL) {
        opencc_jieba_delete(opencc);
    }

    return 0;
}

Output

Text: 意大利邻国法兰西罗浮宫里收藏的“蒙娜丽莎的微笑”画像是旷世之作。
Text Code: 2
Converted: 義大利鄰國法蘭西羅浮宮裡收藏的「蒙娜麗莎的微笑」畫像是曠世之作。
Converted Code: 1

Notes

• opencc_jieba_new() initializes the engine.
• opencc_jieba_convert(...) performs the conversion with the specified config (e.g., s2t, t2hkp, hk2tp).
• opencc_jieba_free_string(...) must be called to free the returned string.
• opencc_jieba_delete(...) must be called to free OpenCC instance.
• opencc_jieba_zho_check(...) to detect zh-Hant (1), zh-Hans (2), others (0).

---

Project Structure

• src/lib.rs – Main library with segmentation logic.
• capi/opencc_jieba_capi – C ABI implementation crate.
• capi/include – Canonical public C and C++ headers.
• tools/opencc-jieba/src/main.rs – CLI tool (opencc-cs) implementation.
• dicts/ – OpenCC text lexicons which converted into JSON format.

---

Dictionary compression (Zstd)

Zstandard - zstd: A fast lossless compression algorithm, targeting real-time compression scenarios at zlib-level and better compression ratios.

zstd -19 src/dictionary_lib/dicts/dictionary.json -o src/dictionary_lib/dicts/dictionary.json.zst
zstd -19 src/dictionary_lib/dicts/dict_hans_hant.txt -o src/dictionary_lib/dict_hans_hant.txt.zst

These .txt files are used for development only.
The runtime uses .zst files generated with zstd.
These are included in the crate, but the .txt source files are not.

Generate and load a custom conversion dictionary

Power users can edit the OpenCC source files under dicts/ and generate a runtime conversion pack with the workspace tool:

cargo run -p dict-generate -- --format zstd --output dictionary.json.zst

Load that pack without enabling any Cargo feature:

use opencc_jieba_rs::OpenCC;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let cc = OpenCC::try_new_with_dictionary_zstd("dictionary.json.zst")?;
    println!("{}", cc.s2t("汉字", false));
    Ok(())
}

Custom OpenCC conversion dictionaries and Jieba user dictionaries are independent and can be combined in either order:

use opencc_jieba_rs::OpenCC;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Start with the custom OpenCC conversion pack, then add Jieba terms.
    let mut cc = OpenCC::try_new_with_dictionary_zstd("dictionary.json.zst")?;
    cc.load_user_dict("dicts/user_dict.txt")?;

    // Or start with a Jieba user dictionary and replace the conversion pack:
    let mut cc = OpenCC::try_new_with_user_dict_path("dicts/user_dict.txt")?;
    cc.load_dictionary_zstd("dictionary.json.zst")?;

    Ok(())
}

The loader validates Zstd data, JSON structure, and the dictionary schema version before replacing the active conversion mappings.

Schema 3 uses the upstream-aligned JP/HK slots:

Runtime slot	Source dictionary
hk_phrases	HKPhrases.txt
hk_phrases_rev	HKPhrasesRev.txt
hk_variants_phrases	HKVariantsPhrases.txt
hk_variants	HKVariants.txt
hk_variants_rev_phrases	HKVariantsRevPhrases.txt
hk_variants_rev	HKVariantsRev.txt
jps_phrases	JPShinjitaiPhrases.txt
jps_characters	JPShinjitaiCharacters.txt
jps_characters_rev	JPShinjitaiCharactersRev.txt

The legacy JPVariants.txt and JPVariantsRev.txt slots are not emitted in schema-3 packs. Schema-2 custom packs remain loadable through compatibility fallbacks.

---

User dictionary

opencc-jieba-rs supports loading Jieba user dictionaries without directly using the lower-level jieba-rs API.

Default user dictionary path

use opencc_jieba_rs::OpenCC;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Loads dicts/user_dict.txt
    let opencc = OpenCC::new_with_user_dict()?;
    Ok(())
}

Custom user dictionary path

use opencc_jieba_rs::OpenCC;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let opencc = OpenCC::try_new_with_user_dict_path("dicts/user_dict.txt")?;
    Ok(())
}

Load multiple user dictionaries

use opencc_jieba_rs::OpenCC;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut opencc = OpenCC::new();

    opencc.load_user_dict("dicts/user_dict.txt")?;
    opencc.load_user_dict("dicts/user_cantonese_dict.txt")?;

    Ok(())
}

User dictionary format

The user dictionary must follow the jieba-rs format:

word freq [tag]

Example:

云计算 100000 n
人工智能 100000 n
区块链 10 nz
Palantir 100000 nz
帕兰提尔 100000 nz
OpenAI 100000
ChatGPT 100000

Note:

• freq is required
• freq must be a valid integer
• tag is optional
• lines containing only word are not supported

For tagged entries, use:

帕兰提尔 100000 nz

For untagged entries, use:

OpenAI 100000

Do not omit the frequency or put the tag in the frequency field.

User dictionaries are loaded into the current tokenizer in order.
Conflict handling follows jieba-rs behavior.

---

📦 Dependency Notes

• Core dependencies (jieba-rs, rayon) are pinned for stability.
• Other dependencies are allowed to float to benefit from upstream fixes.

⚠️ MSRV note: This crate is developed with Rust 1.75.0 in mind. Most users on modern Rust do not need special setup.

For older toolchains, see: MSRV-1.75.0-GUIDE.md

---

Credits

• OpenCC – Lexicon source.
• jieba-rs - Jieba tokenization.

License

• This project is licensed under the MIT License. See the LICENSE file for details.
• See THIRD_PARTY_NOTICES.md for bundled OpenCC lexicons (Apache License 2.0).

Contributing

Contributions are welcome! Please open issues or submit pull requests for improvements or bug fixes.