Creating MacOS-Agent Part 2: Applied a Few Shot Prompts to LLama2
Use a few shot prompts to improve and guarantee how LLama2 7B performs with the help of Claude
In this article, you will learn
- How to do a few shot prompts with LLAMA2.
- How to use bigger LLM to help craft prompts for smaller ones.
Background of Experiment
I am working on LLM as an automation tool and I used Claude via AWS Bedrock to produce an Applescript content string and inject it directly into osascript the command. You can check out the article “Creating MacOS-Agent Part 1: Use LLM to Automate Your MacOS”.
Then I moved away from paid LLM to SLM(Small Language Model) using the ollama to run llama2 7B on my local to save some cost and have fun with a different model. The initial result is not nearly good enough.
I push LLama2 a little bit more with this prompt, and seed and temperature
"""
Instruction: You are a Human Wish to MacOS Applescript translator
Human Wish: Open Notes
Applescript:
"""Still not enough.
I almost give up….
The first thing I notice is, of course how the smartness is different.
Now I have to level up my prompt engineering game to improve LLM
I jump to Claude immediately.
Prompt Exchange
User: “OK help me build a prompt that translates human wish into an AppleScript file with a few shot prompt techniques please!”
Claude: “Here is an attempt to create a few-shot prompt to translate human wishes into AppleScript code using a large language model:
Prompt:
Title: Translate natural language wishes into AppleScript code
Instructions: Translate the following natural language wishes into valid AppleScript code to accomplish the wish.
Examples:
Wish: Open Safari Code: tell application “Safari” to activate
Wish: Set volume to 50% Code: set volume output volume 50
Wish: Take a screenshot and save to desktop Code: tell application “Finder” to set theClipboard to (open location “desktop” as alias)’s POSIX path tell application “System Events” to keystroke “3” using command down
Wish: Turn on Do Not Disturb mode Code: tell application “System Preferences” reveal anchor “doNotDisturb” of pane id “com.apple.preference.notifications” end tell
Wish: Create a new folder on desktop called Test Code: tell application “Finder” to make new folder at desktop with properties {name:”Test”}
Wish: {Your wish here} Code:
So this provides a few examples for the model to learn from in a natural language to AppleScript code translation task, with a space for the human to provide a new wish and have the model generate the code. The model can then leverage its language capabilities to generate AppleScript code matching the wish.”
I of course extracted the answer above and applied it to ollama run llama2 right away but it still doesn’t reply just the Applescript content. There were still “Hi there” “Sure thing” etc etc. You get the point. Now I have to research some more and dig deeper.
After Research,
I found out LLama2 use a specific template which is the [INST][/INST] tag
User: “Wrap all the Wish: …. with [INST] [/INST] tag please revise”
Claude: “Here is the prompt revised to wrap the wish examples in [INST] [/INST] tags:
Prompt:
Title: Translate natural language wishes into AppleScript code
Instructions: Translate the following natural language wishes into valid AppleScript code to accomplish the wish.
Examples:
[INST]
Wish: Open Safari
[/INST]
Code: tell application “Safari” to activate[INST]
Wish: Set volume to 50%
[/INST]
Code: set volume output volume 50[INST]
Wish: Take a screenshot and save to desktop
[/INST]
Code: tell application “Finder” to set theClipboard to (open location “desktop” as alias)’s POSIX path
tell application “System Events” to keystroke “3” using command down[INST]
Wish: Turn on Do Not Disturb mode
[/INST]
Code: tell application “System Preferences”
reveal anchor “doNotDisturb” of pane id “com.apple.preference.notifications”
end tell[INST]
Wish: Create a new folder on desktop called Test
[/INST]
Code: tell application “Finder” to make new folder at desktop with properties {name:”Test”}[INST]
{Your wish here}
[/INST]
Code:The [INST] [/INST] tags allow me to clearly identify and separate the wish examples from the instructions and rest of the prompt. Let me know if you would like me to modify or refine this further.”
Result
This time the result is exactly what I want. The LLAMA2 understands the instructions and the output format!
If you shrink down the few shots to one shot, the prompt should look something like this
"""
Instructions: Translate the following natural language wishes into valid AppleScript code to accomplish the wish.
Examples:
[INST]
Wish: Open Safari
[/INST]
Code: tell application "Safari" to activate
[INST]
Wish: Open Notes
[/INST]
Code:
"""
Conclusion
The hard lesson learned. Each LLM has its different format to follow and one must read its document and get to know the model first before start using it.
Also with the Few Shot Prompt technique, I can get consistent results with smaller models. This leads to extra cost savings.
Next, I will show you how I moved from Bash Script to Typescript Project for better software development. Follow for more here
