Llama 2 70B Agent/Tool use example¶
This Jupyter notebook provides examples of how to use Tools for Agents with the Llama 2 70B model in EasyLLM. This includes an example on how to use tools with an LLM, including output parsing, execution of the tools and parsing of the results. It is a very simplified example. If you are interested in Agents you should checkout langchain or the ReAct pattern.
Why do LLMs need to use Tools?¶
One of the most common challenges with LLMs is overcoming the lack of recency and specificity in their training data - answers can be out of date, and they are prone to hallucinations given the huge variety in their knowledge base. Tools are a great method of allowing an LLM to answer within a controlled context that draws on your existing knowledge bases and internal APIs - instead of trying to prompt engineer the LLM all the way to your intended answer, you allow it access to tools that it calls on dynamically for info, parses, and serves to customer.
Providing LLMs access to tools can enable them to answer questions with context directly from search engines, APIs or your own databases. Instead of answering directly, an LLM with access to tools can perform intermediate steps to gather relevant information. Tools can also be used in combination. For example, a language model can be made to use a search tool to lookup quantitative information and a calculator to execute calculations.
# if needed, install and/or upgrade to the latest version of the EasyLLM Python library
%pip install --upgrade easyllm
Getting an open LLM to act like an agent or use tools is incredibly hard. However, with Llama 2 70B it is now possible. Let's see how we can get it running!
Basic example of using a tool with Llama 2 70B¶
In the basic we are are going to only use one abstract tool, a calculator. Our model can use the calculator run mathmatical operations. To make it easy we provide some few-shot example for the model to better understand what it needs to do.
Note: This is adapted from the example by pinecone.
system_message = """Assistant is a expert JSON builder designed to assist with a wide range of tasks.
Assistant is able to respond to the User and use tools using JSON strings that contain "action" and "action_input" parameters.
All of Assistant's communication is performed using this JSON format.
Assistant can also use tools by responding to the user with tool use instructions in the same "action" and "action_input" JSON format. Tools available to Assistant are:
- "Calculator": Useful for when you need to answer questions about math.
- To use the calculator tool, Assistant should write like so:
```json
{{"action": "Calculator",
"action_input": "4+4"}}
```
Here are some previous conversations between the Assistant and User:
User: Hey how are you today?
Assistant: ```json
{{"action": "Final Answer",
"action_input": "I'm good thanks, how are you?"}}
```
User: I'm great, what is the square root of 4?
Assistant: ```json
{{"action": "Calculator",
"action_input": "sqrt(4)"}}
```
Result: 2.0
Assistant: ```json
{{"action": "Final Answer",
"action_input": "It looks like the answer is 2!"}}
```
User: Thanks could you tell me what 4 to the power of 2 is?
Assistant: ```json
{{"action": "Calculator",
"action_input": "4**2"}}
```
Result: 16.0
Assistant: ```json
{{"action": "Final Answer",
"action_input": "It looks like the answer is 16!"}}
```
Here is the latest conversation between Assistant and User."""
In addition to our system message which holds the information for our tools we need to create a user template, which includes the input from the user and tells the model to use tools or not.
prompt = f"{system_message}\n\nUse your existing tools and respond with a JSON object with with 'action' and 'action_input' values \nUser: {{user_input}}"
Now lets combine both and create a request using easyllm.
from easyllm.clients import huggingface
# Changing configuration without using environment variables
huggingface.prompt_builder = "llama2"
# huggingface.api_key="hf_xxx"
def agent(prompt):
response = huggingface.Completion.create(
model="meta-llama/Llama-2-70b-chat-hf",
prompt=prompt,
temperature=0.1,
max_tokens=128,
stop=["```\n","Result: "],
debug=False,
)
return response["choices"][0]["text"]
Now we can begin asking questions
output = agent(prompt.format(user_input="hey how are you today?"))
output
' Assistant: ```json\n{"action": "Final Answer",\n "action_input": "I\'m good thanks, how are you?"}\n```'
What happens if we ask a math question?
output = agent(prompt.format(user_input="What is 4 multiplied by 2?"))
output
' Assistant: ```json\n{"action": "Calculator",\n "action_input": "4*2"}\n```\n'
Great! It works! It correctly selects the tool. Okay now to make it work we need to parse the output and execute it in the case for the calculator
import json
import re
def parser(input):
pattern = r'```json\n(.*?)```'
match = re.search(pattern, input, re.DOTALL)
if not match:
raise ValueError("Couldn't parse the output.")
parsed_data = json.loads(match.group(1))
return parsed_data
output = parser(output)
output
{'action': 'Calculator', 'action_input': '4*2'}
Okay, Now lets execute it using the eval function from python
def use_tool(tool,tool_input):
if tool == "Calculator":
return eval(tool_input)
else:
raise Exception("Unknown tool: " + tool)
Okay, now lets combine everyting and the cacluator result to our agent again.
def use_calculator(input, first_call=True):
if first_call:
input_prompt = prompt.format(user_input=input)
else:
input_prompt = input
# make the agent call
response = agent(input_prompt)
# parse the output if possible
parsed = parser(response)
# check if the output is our final answer or if it is a tool
if parsed["action"] == "Final Answer":
return parsed["action_input"]
# if not try to use the tool
tool_output = use_tool(parsed["action"], parsed["action_input"])
# add message to the agent
next_prompt = f"{input_prompt}\n{response}\nResponse: {tool_output}"
# recursively call the agent with the output of the tool
return use_calculator(next_prompt, False)
use_calculator("What is 19 * 11?")
'It looks like the answer is 209!'