eduagarcia/multilingual-tokenizer-leaderboard at main

multilingual-tokenizer-leaderboard

3.12 MB

Ctrl+K

1 contributor

History: 113 commits

eduagarcia

feat: add total bytes, total chars, and edit distance metrics to API and frontend components

a2366b9 5 months ago

dataset
not commited changes, notebook 6 months ago
docs
refactor: update EBPB metrics to use rate-distortion formula and add full vocabulary variant 5 months ago
frontend
feat: add total bytes, total chars, and edit distance metrics to API and frontend components 5 months ago
.dockerignore

3.87 kB
react v1 6 months ago
.gitattributes

1.55 kB
not commited changed... 6 months ago
.gitignore

3.83 kB
react v1 6 months ago
Dockerfile

790 Bytes
react v1 6 months ago
README.md

261 kB
update README 6 months ago
api.py

67.3 kB
feat: add total bytes, total chars, and edit distance metrics to API and frontend components 5 months ago
battle.py

11.3 kB
fix tokenizer battle 6 months ago
client.py

8.38 kB
api separation from gradio v1 6 months ago
compact_results.py

4.7 kB
batch files in zip 6 months ago
dataset_meta.yaml

38 kB
domain filtering and selection for multilingual tab about 1 year ago
display.py

9.37 kB
update explanation of parity 6 months ago
env.py

1.34 kB
feat: update LeaderboardTable to sort by EBPB and add EBPB to display columns 5 months ago
fix_ebpb_metrics.py

19.6 kB
refactor: update EBPB metrics to use rate-distortion formula and add full vocabulary variant 5 months ago
fix_tokenizer_type.py

10.8 kB
fix eval 6 months ago
init_lb.py

123 kB
initlb update 6 months ago
pyproject.toml

808 Bytes
fix requirements 2 5 months ago
recalculate_derived_metrics.py

13.8 kB
refactor: update EBPB metrics to use rate-distortion formula and add full vocabulary variant 5 months ago
reevaluate_lb.py

11.9 kB
cleanup cache of already ran models 6 months ago
requirements.txt

450 Bytes
fix requirements 2 5 months ago
results_aggregator.py

35.8 kB
refactor: update EBPB metrics to use rate-distortion formula and add full vocabulary variant 5 months ago
search_filter.py

15.6 kB
feat: add total bytes, total chars, and edit distance metrics to API and frontend components 5 months ago
search_for_unique_tokenizers.ipynb

1.07 MB
fix eval 6 months ago
start.py

1.57 kB
react v1 6 months ago
submit.py

11.4 kB
Add field to select subfolders and target_mode of tokenizers about 1 year ago
tokenizer_evaluate.py

149 kB
refactor: update EBPB metrics to use rate-distortion formula and add full vocabulary variant 5 months ago
update_likes.py

10.1 kB
get author info and hide duplicates by author verified and followers 6 months ago
update_readme.py

1.83 kB
Vectorized calculation for aggregated results, better code separation about 1 year ago
upload_repacked_results.py

4.87 kB
refactor: update EBPB metrics to use rate-distortion formula and add full vocabulary variant 5 months ago
util.py

9.48 kB
refactor: update EBPB metrics to use rate-distortion formula and add full vocabulary variant 5 months ago
uv.lock

689 kB
fix requirements 2 5 months ago