admin-valentin/openai-cookbook
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix iteration error (#208)

Browse source 
* fix iteration error

array iteration error: sentences up till max_len of token exceeds should be returned

* split list once instead of twice

split list once instead of twice and take parsed string up till max token length is exceeded
This commit is contained in:
Swastik Banerjee 2023-03-22 03:15:56 +05:30 committed by GitHub
parent feb53449e0
commit 3dde564948
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
examples/fine-tuned_qa/olympics-1-collect-data.ipynb
View file

@ -154,7 +154,7 @@
" for i, sentence in enumerate(sentences):\n",
" ntokens += 1 + count_tokens(sentence)\n",
" if ntokens > max_len:\n",
" return \". \".join(sentences[:i][:-1]) + \".\"\n",
" return \". \".join(sentences[:i]) + \".\"\n",
"\n",
" return long_text\n",
"\n",