How to use Text (Instruction) Completion clients¶

EasyLLM can be used as an abstract layer to replace text-davinci-003 with open source models.

You can change your own applications from the OpenAI API, by simply changing the client.

Chat models take a series of messages as input, and return an AI-written message as output.

This guide illustrates the chat format with a few example API calls.

0. Setup¶

Before you can use easyllm with Amazon SageMaker you need to deploy the model to a SageMaker endpoint. You can do this by following one of the bloh posts below:

• Deploy Llama 2 7B/13B/70B on Amazon SageMaker
• Deploy Falcon 7B & 40B on Amazon SageMaker
• Introducing the Hugging Face LLM Inference Container for Amazon SageMaker

Once you have your endpoint deploy copy the endpoint name of it. The endpoint name will be our model paramter. You can get the endpoint name in the AWS management console for Amazon SageMaker under "Inference" -> "Endpoints" -> "Name" or when you deployed your model using the sagemaker sdk you can get it from the predictor.endpoint_name attribute.

1. Import the easyllm library¶

In [ ]:

# if needed, install and/or upgrade to the latest version of the OpenAI Python library
%pip install --upgrade easyllm

# if needed, install and/or upgrade to the latest version of the OpenAI Python library %pip install --upgrade easyllm

In [6]:

# import the EasyLLM Python library for calling the EasyLLM API
import easyllm

# import the EasyLLM Python library for calling the EasyLLM API import easyllm

2. An example chat API call¶

A text API call has two required inputs:

• model: the name of the model you want to use (e.g., meta-llama/Llama-2-70b-chat-hf) or leave it empty to just call the api
• prompt: a text prompt that is sent to the model to generate the text

Compared to OpenAI api is the huggingface module also exposing a prompt_builder and stop_sequences parameter you can use to customize the prompt and stop sequences. The EasyLLM package comes with build in popular methods for both of these parameters, e.g. llama2_prompt_builder and llama2_stop_sequences.

Let's look at an example chat API calls to see how the chat format works in practice.

In [1]:

import os 
# set env for prompt builder
os.environ["HUGGINGFACE_PROMPT"] = "llama2" # vicuna, wizardlm, stablebeluga, open_assistant
os.environ["AWS_REGION"] = "us-east-1"  # change to your region
# os.environ["AWS_ACCESS_KEY_ID"] = "XXX" # needed if not using boto3 session
# os.environ["AWS_SECRET_ACCESS_KEY"] = "XXX" # needed if not using boto3 session

from easyllm.clients import sagemaker

# Changing configuration without using environment variables
# sagemaker.prompt_builder = "llama2"
# sagemaker.api_aws_access_key="xxx"
# sagemaker.api_aws_secret_key="xxx"

# SageMaker endpoint name
MODEL="huggingface-pytorch-tgi-inference-2023-08-08-14-15-52-703"

response = sagemaker.Completion.create(
    model=MODEL,
    prompt="What is the meaning of life?",
    temperature=0.9,
    top_p=0.6,
    max_tokens=256,
)
response

import os # set env for prompt builder os.environ["HUGGINGFACE_PROMPT"] = "llama2" # vicuna, wizardlm, stablebeluga, open_assistant os.environ["AWS_REGION"] = "us-east-1" # change to your region # os.environ["AWS_ACCESS_KEY_ID"] = "XXX" # needed if not using boto3 session # os.environ["AWS_SECRET_ACCESS_KEY"] = "XXX" # needed if not using boto3 session from easyllm.clients import sagemaker # Changing configuration without using environment variables # sagemaker.prompt_builder = "llama2" # sagemaker.api_aws_access_key="xxx" # sagemaker.api_aws_secret_key="xxx" # SageMaker endpoint name MODEL="huggingface-pytorch-tgi-inference-2023-08-08-14-15-52-703" response = sagemaker.Completion.create( model=MODEL, prompt="What is the meaning of life?", temperature=0.9, top_p=0.6, max_tokens=256, ) response

Out[1]:

{'id': 'hf-dEMeXTUk3Y',
 'object': 'text.completion',
 'created': 1691508711,
 'model': 'huggingface-pytorch-tgi-inference-2023-08-08-14-15-52-703',
 'choices': [{'index': 0,
   'text': " The meaning of life is a question that has puzzled philosophers, theologians, scientists, and many other thinkers throughout history. Here are some possible answers:\n1. Religious or spiritual beliefs: Many people believe that the meaning of life is to fulfill a divine or spiritual purpose, whether that be to follow a set of moral commandments, to achieve spiritual enlightenment, or to fulfill a specific mission or calling.\n2. Personal fulfillment: Some people believe that the meaning of life is to find personal fulfillment and happiness. This can be achieved through pursuing one's passions, building meaningful relationships, and cultivating a sense of purpose and meaning in one's life.\n3. Contribution to society: Many people believe that the meaning of life is to make a positive impact on the world and to contribute to the greater good. This can be achieved through various means, such as working to make the world a better place, helping others, or creating something of value.\n4. Learning and growth: Some people believe that the meaning of life is to learn and grow as individuals, to expand one's knowledge and understanding of the world, and to develop one's skills",
   'finish_reason': 'length'}],
 'usage': {'prompt_tokens': 11, 'completion_tokens': 256, 'total_tokens': 267}}

As you can see, the response object has a few fields:

• id: the ID of the request
• object: the type of object returned (e.g., text.completion)
• created: the timestamp of the request
• model: the full name of the model used to generate the response
• usage: the number of tokens used to generate the replies, counting prompt, completion, and total
• choices: a list of completion objects (only one, unless you set n greater than 1)
  • text: the generated text
  • finish_reason: the reason the model stopped generating text (either stop, eos_token, or length if max_tokens limit was reached)
  • logprobs: optional the log probs of each generated token.

Extract just the reply with:

In [2]:

print(response['choices'][0]['text'])

print(response['choices'][0]['text'])

 The meaning of life is a question that has puzzled philosophers, theologians, scientists, and many other thinkers throughout history. Here are some possible answers:
1. Religious or spiritual beliefs: Many people believe that the meaning of life is to fulfill a divine or spiritual purpose, whether that be to follow a set of moral commandments, to achieve spiritual enlightenment, or to fulfill a specific mission or calling.
2. Personal fulfillment: Some people believe that the meaning of life is to find personal fulfillment and happiness. This can be achieved through pursuing one's passions, building meaningful relationships, and cultivating a sense of purpose and meaning in one's life.
3. Contribution to society: Many people believe that the meaning of life is to make a positive impact on the world and to contribute to the greater good. This can be achieved through various means, such as working to make the world a better place, helping others, or creating something of value.
4. Learning and growth: Some people believe that the meaning of life is to learn and grow as individuals, to expand one's knowledge and understanding of the world, and to develop one's skills

In [ ]: