admin-valentin/openai-cookbook
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Update How_to_count_tokens_with_tiktoken.ipynb (#368)

Browse source 
* Update How_to_count_tokens_with_tiktoken.ipynb

add c# tiktoken lib.

* Update How_to_count_tokens_with_tiktoken.ipynb

fix format.

* Update How_to_count_tokens_with_tiktoken.ipynb

fix format.
This commit is contained in:
aiqinxuancai 2023-05-06 04:09:05 +08:00 committed by GitHub
parent 5008f3a3d3
commit 23bd266fef
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 2 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
examples/How_to_count_tokens_with_tiktoken.ipynb
View file

@ -1,6 +1,7 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
@ -36,7 +37,7 @@
"\n",
"For `cl100k_base` and `p50k_base` encodings:\n",
"- Python: [tiktoken](https://github.com/openai/tiktoken/blob/main/README.md)\n",
"- .NET / C#: [SharpToken](https://github.com/dmitry-brazhenko/SharpToken)\n",
"- .NET / C#: [SharpToken](https://github.com/dmitry-brazhenko/SharpToken), [TiktokenSharp](https://github.com/aiqinxuancai/TiktokenSharp)\n",
"- Java: [jtokkit](https://github.com/knuddelsgmbh/jtokkit)\n",
"\n",
"For `r50k_base` (`gpt2`) encodings, tokenizers are available in many languages.\n",