how to use icu::collator for gb18030 encoding?

[qwm-db]

as rust only support utf-8 encoding string, and the collator api use str slice as parameter, so seems it's not feasible to pass gb18030 encoding bytes?

[sffc]

ICU4X supports Han text in UTF-8 and UTF-16. The following Han orderings are available:

• big5han
• gb2312han
• pinyin
• stroke
• unihan
• zhuyin

Note that big5han and gb2312han need to be explicitly enabled in datagen. When used, they collate the text in the order that would result if the text were in either of those two encodings. However, they are not recommended for modern use; one of the other orderings, such as "pinyin" or "stroke", should produce the best quality collation results.

[qwm-db]

thanks @sffc for shedding light here.

sorry, two follow up questions, many thanks in advance!

1. double confirm here: so the input string is in either utf-8 or utf-16 encoding? not in gb18030 encoding?
2. if I wana to use icu_collator crate, where can I get gb18030 data to setup data_provider?

pub fn try_new_unstable(
data_provider: &D,
locale: &DataLocale,
options: CollatorOptions
) -> Result<Self, CollatorError>

[markusicu]

Correct -- modern i18n libraries necessarily operate on Unicode text.
When you have text in a legacy codepage, you need to convert to/from Unicode.
You can use ICU4C/ICU4J or likely one of several Rust libraries to convert from GB18030 to Unicode.

[sffc]

You can use encoding_rs, built by ICU4X contributor @hsivonen, to convert between GB-18030 and UTF-8.

[qwm-db]

again many thanks @markusicu and @sffc for your kind help.

btw where can I find the data of gb18030 datakey to setup the DataProvider?
https://docs.rs/icu_provider/1.2.0/icu_provider/trait.DataProvider.html

[hsivonen]

There is no gb18030 key for DataProvider. The encoding_rs crate contains the mapping tables from gb18030 to UTF-8 independently of ICU4X data. pinyin is the default collation order when requesting zh as the locale.