Hugging Face's logo Hugging Face
  • Models
  • Datasets
  • Spaces
  • Buckets
  • Docs
  • Enterprise
  • Pricing
    • Website
      • Tasks
      • HuggingChat
      • Collections
      • Languages
      • Organizations
    • Community
      • Blog
      • Posts
      • Daily Papers
      • Hardware
      • Learn
      • Discord
      • Forum
      • GitHub
    • Solutions
      • Team & Enterprise
      • Hugging Face PRO
      • Enterprise Support
      • Inference Providers
      • Inference Endpoints
      • Storage Buckets

  • Log In
  • Sign Up

MarcoDotIO
/
kotodama-lm-ja-tokenizer

Japanese
sentencepiece
japanese
tokenizer
unigram
llm
kotodamalm
jpn-bench
Model card Files Files and versions
xet
Community
kotodama-lm-ja-tokenizer
1.98 MB
Ctrl+K
Ctrl+K
  • 1 contributor
History: 3 commits
MarcoDotIO's picture
MarcoDotIO
Clarify tokenizer evaluation metadata
0e15076 verified 7 months ago
  • evaluation
    Publish KotodamaLM Japanese tokenizer 7 months ago
  • training
    Publish KotodamaLM Japanese tokenizer 7 months ago
  • .gitattributes
    136 Bytes
    Publish KotodamaLM Japanese tokenizer 7 months ago
  • README.md
    3.47 kB
    Clarify tokenizer evaluation metadata 7 months ago
  • SHA256SUMS
    1.36 kB
    Clarify tokenizer evaluation metadata 7 months ago
  • sentencepiece_config.json
    466 Bytes
    Publish KotodamaLM Japanese tokenizer 7 months ago
  • special_tokens_map.json
    136 Bytes
    Publish KotodamaLM Japanese tokenizer 7 months ago
  • tokenizer.model
    934 kB
    xet
    Publish KotodamaLM Japanese tokenizer 7 months ago
  • tokenizer.vocab
    721 kB
    Publish KotodamaLM Japanese tokenizer 7 months ago
  • tokenizer_config.json
    322 Bytes
    Publish KotodamaLM Japanese tokenizer 7 months ago