admin-valentin/openai-cookbook
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge pull request #146 from hunkim/patch-1

Browse source 
Add handling for last chunk in split_into_sentences function
This commit is contained in:
Ted Sanders 2023-03-16 16:11:05 -07:00 committed by GitHub
commit 8f06acdd82
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 4 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
apps/web-crawl-q-and-a/web-qa.py
View file

@ -248,6 +248,10 @@ def split_into_many(text, max_tokens = max_tokens):
# Otherwise, add the sentence to the chunk and add the number of tokens to the total # Otherwise, add the sentence to the chunk and add the number of tokens to the total
chunk.append(sentence) chunk.append(sentence)
tokens_so_far += token + 1 tokens_so_far += token + 1
# Add the last chunk to the list of chunks
if chunk:
chunks.append(". ".join(chunk) + ".")
return chunks return chunks