How to stream Text Completion requests¶

By default, when you request a completion, the entire completion is generated before being sent back in a single response.

If you're generating long completions, waiting for the response can take many seconds.

To get responses sooner, you can 'stream' the completion as it's being generated. This allows you to start printing or processing the beginning of the completion before the full completion is finished.

To stream completions, set stream=True when calling the chat completions or completions endpoints. This will return an object that streams back the response as data-only server-sent events. Extract chunks from the delta field rather than the message field.

Downsides¶

Note that using stream=True in a production application makes it more difficult to moderate the content of the completions, as partial completions may be more difficult to evaluate.

Example code¶

Below, this notebook shows:

1. What a streaming text completion response looks like

In [1]:

# imports
import easyllm  # for API calls
import time  # for measuring time duration of API calls

# imports import easyllm # for API calls import time # for measuring time duration of API calls

2. How to stream a text completion¶

With a streaming API call, the response is sent back incrementally in chunks via an event stream. In Python, you can iterate over these events with a for loop.

Let's see what it looks like:

In [1]:

from easyllm.clients import huggingface

huggingface.prompt_builder = "llama2"

# a ChatCompletion request
response = huggingface.Completion.create(
    model="meta-llama/Llama-2-70b-chat-hf",
    prompt="Count to 50, with a comma between each number and no newlines. E.g., 1, 2, 3, ...",
    stream=True  # this time, we set stream=True
)

for chunk in response:
    print(chunk)

from easyllm.clients import huggingface huggingface.prompt_builder = "llama2" # a ChatCompletion request response = huggingface.Completion.create( model="meta-llama/Llama-2-70b-chat-hf", prompt="Count to 50, with a comma between each number and no newlines. E.g., 1, 2, 3, ...", stream=True # this time, we set stream=True ) for chunk in response: print(chunk)

{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' Sure', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '!', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' Here', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' it', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' is', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ':', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '\n', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '\n', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '1', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '2', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '3', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '4', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '5', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '6', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '7', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '8', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '9', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '1', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '0', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '1', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '1', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '1', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '2', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '1', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '3', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '1', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '4', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '1', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '5', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '1', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '6', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '1', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '7', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '1', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '8', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '1', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '9', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '2', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '0', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '2', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '1', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '2', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '2', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '2', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '3', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '2', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '4', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '2', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '5', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '2', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '6', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '2', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '7', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '2', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '8', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '2', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '9', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '3', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '0', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '3', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '1', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '3', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '2', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '3', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '3', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '3', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '4', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '3', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '5', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '3', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '6', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '3', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '7', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '3', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '8', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '3', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '9', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '4', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '0', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '4', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '1', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '4', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '2', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '4', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '3', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '4', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '4', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '4', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '5', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '4', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '6', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '4', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '7', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '4', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '8', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '4', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '9', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ',', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': ' ', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '5', 'logprobs': 0.0}]}
{'id': 'hf-PokypeK2an', 'object': 'text.completion', 'created': 1691129904, 'model': 'meta-llama/Llama-2-70b-chat-hf', 'choices': [{'index': 0, 'text': '0', 'logprobs': 0.0}]}

As you can see above, streaming responses have a text field which holds the generated tokens.

Below is an example where we print the generated text as it comes in, and stop when we see a </s> token.

In [2]:

from easyllm.clients import huggingface

huggingface.prompt_builder = "llama2"

# a ChatCompletion request
response = huggingface.Completion.create(
    model="meta-llama/Llama-2-70b-chat-hf",
    prompt="Count to 10, with a comma between each number and no newlines. E.g., 1, 2, 3, ...",
    stream=True  # this time, we set stream=True
)

for chunk in response:
    print(chunk["choices"][0]["text"],end="")

from easyllm.clients import huggingface huggingface.prompt_builder = "llama2" # a ChatCompletion request response = huggingface.Completion.create( model="meta-llama/Llama-2-70b-chat-hf", prompt="Count to 10, with a comma between each number and no newlines. E.g., 1, 2, 3, ...", stream=True # this time, we set stream=True ) for chunk in response: print(chunk["choices"][0]["text"],end="")

  Sure! Here it is: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10.

In [ ]: