Code

Supporting more BERT-like models#89

Code

Closed

ashvardanian wants to merge 10 commits intohuggingface:mainhuggingface/swift-transformers:mainfrom ashvardanian:mainashvardanian/swift-transformers:mainCopy head branch name to clipboard

+163-47

Commits

Commits on Apr 9, 2024

• Add BertPreTokenizer

  ashvardanian

  committed
  9b67fe1

• Drop PreTokenizer suffix

  ashvardanian

  committed
  ec905f9

• Drop Normalizer suffix

  ashvardanian

  committed
  5d36013

• Add BertProcessing

  ashvardanian

  committed
  0a818ac

• Order-agnostic tokenValue

  Show description for c14fa13
  ashvardanian

  committed
  c14fa13

• Handle arrays

  ashvardanian

  committed
  a723369

Commits on Apr 10, 2024

• Add WordPieceDecoder

  ashvardanian

  committed
  1251447

• Improve TokenizerModel mapping

  ashvardanian

  committed
  4060e8f

Commits on Apr 11, 2024

• Capitalization-agnostic tokenizer lookup

  ashvardanian

  committed
  9ef46a5

Commits on Apr 21, 2024

• Fix: UTF8 support in Unigram tokenizer

  ashvardanian

  committed
  89fb5d9