You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
mindspore/tests/ut/data/dataset/testTokenizerData/basic_tokenizer.txt

7 lines
502 B

This file contains ambiguous Unicode characters!

This file contains ambiguous Unicode characters that may be confused with others in your current locale. If your use case is intentional and legitimate, you can safely ignore this warning. Use the Escape button to highlight these characters.

Welcome to Beijing北京欢迎您
長風破浪會有時,直掛雲帆濟滄海
😀嘿嘿😃哈哈😄大笑😁嘻嘻
明朝1368—1644年和清朝1644—1911年是中国封建王朝史上最后两个朝代
明代1368-1644と清代1644-1911は、中国の封建王朝の歴史における最後の2つの王朝でした
명나라 (1368-1644)와 청나라 (1644-1911)는 중국 봉건 왕조의 역사에서 마지막 두 왕조였다
Tĥïŝ ĩš â fůňķŷ Šťŕĭńġ