There are many resources out there on how to engineer prompts, and probably on how to navigate the quirks of the various Large Language Model (LLM) clients available. This is just a quick and dirty hashing out of things I've had to think about repeatedly. Hopefully, they'll help you bypass some of my frustrations.
But first, please note that the results I get are with ChatGPT v4. If you are still on v3.5 you will not achieve the same quality of output
Understand the memory limitations of ChatGPT
ChatGPT and other LLMs (like Claude.ai) have different capabilities when it comes to memory and performance. There are pros and cons for each platform. Many of the details are hidden even from the "bot" itself so it's hard to get specifics.
Here's what I've been able to find.
- Both showcase impressive conversational abilities, but Claude appears more adept at hypotheticals, reasoning, and admitting knowledge gaps.
- ChatGPT falters after 5-6 exchanges, while Claude maintains context across dozens of exchanges.
- ChatGPT was trained via scraping internet data; Claude used high-quality, ethically-sourced datasets.
- ChatGPT optimizes for engagement; Claude focused on safety, truthfulness, and human values.
- ChatGPT lacks memory; Claude has contextual memory and consistency.
- Claude prioritizes safety and ethics; ChatGPT risks potential harms.
- Claude reasons and exercises discretion; ChatGPT aims to satisfy prompts.
- ChatGPT uses GPT-3.5; Claude uses Constitutional AI for stability.
- Claude seems more advanced in coherence, reasoning, safety, and responsible refusal.
- ChatGPT remains impressive but has limitations in memory and stability.
- Claude shows progress mitigating risks through technical innovations.
- Claude balances creativity with wisdom and openness with integrity.
- Claude's Constitutional AI acts as a "Superego" to balance unrestrained models.
- Claude points to AI that enhances humanity rather than undermining it.
- Both offer advanced natural language capabilities.
- Claude claims larger, more up-to-date knowledge base.
- Claude supports English initially but adds languages soon.
- Claude has 100K token limit for larger memory window.
- Claude designed for faster performance around 5 seconds.
These points are clearly biased toward Claude. However, I've found that ChatGPT - when used wisely - produces better and more consistent results. Feel free to try them both out for comparison. They are both continually improving.
If you'd like to read further, here are a couple of sources:
https://medium.com/@amer_as/chatgpt-vs-claude-how-the-leading-ai-assistants-stack-up-22d15d8d41c2
I have one prompt that is nearly 13k characters long. I have had to refactor and trim this beast down a couple of times when ChatGPT choked on it. While I tried migrating to Claude.ai because of its incredibly large memory, it only occasionally gave me the results I was looking for because I've tweaked my prompts for a completely different LLM.
When working with Claude, it indicated that each time I updated a session it had to re-read the entire session, which would slow things down. I'm not certain if ChatGPT operates that way. It seems to hold everything in memory and then bleed off content as it runs out of room. . 🤷🏻♂️
How to Combat Memory Limitations
I do a lot of hierarchical construction. Once I build a Job Map, for example, I have to generate customer success statements for each step. If I have 18 steps things can start to get a little squirrely as I approach the end, e.g., it stops including examples, breaks rules, it only gives me one example, etc. This is indicative of ChatGPT's need for a nap. Put that session to sleep and start another one. Trust me, you'll be get better results with a refresh.
How to Use Variables to Make Life Simple
I'm a recovering developer who was used to plugging in variables or parameters to make my code more flexible. I don't like to hardcode things that will be different each time I use the code / prompt. Therefore, my prompts usually end with a variable section. I go into this more in the prompt explanations but here is a simple scenario to explain it:
Let's say we're studying the job of a 8 to 10 year old child who is setting up a lemonade stand business in their neighborhood. (example I show in a video later). Depending on the prompt type you'll see different variables listed at the end of the prompt. You will need to fill in the values you want to pipe into your prompt as it runs (think "mail merge").
Here's an example (no particular prompt):
end user: 8 to 10 year old (aka the job executor)
job: setting up a lemonade stand (the core job to be done)
context: in their neighborhood (the context in which the job is getting done - this can also be built into the job statement as an alternative)
n: 20 (the number of items you want in your list)
Within the prompt text itself, you will see variables that look like this...{{end user}}. Each time the prompt is run, it will either reuse the input variable from the prior execution (as long as you are in the same chat session) or inject an update if you override it.
If you wish to override a variable the next time it runs, you can simply type the variable, followed by a colon (:) and then the new value. Let's say you want to change 'n' from 20 (prior run) to 25. Simply type the following:
n: 25
And then hit Enter. If the original request was 20, it will rerun the prompt and output 25 while leveraging the pre-existing values for the other variables (which haven't been overridden).
Note: This is also an area to be aware of when it comes to memory. It's ability to remember variables can degrade the longer you remain in the same chat session. This is where you might consider regular comparative testing between platforms.
How to Eliminate Chattiness in Chat Clients
This one annoys the heck out of me. ChatGPT wants to show you how super excited 🤮it is work with you. So, it repeats back what you've asked it to do. Then when it's done, it summarizes it again. I had to stop this!
You will see as you dive into my prompts the various methods I employed to accomplish this. Ultimately, I had to resort to injecting Custom Instructions (a recent feature) to ensure that this never happened. Custom instructions are not yet available in Claude.ai
It still happens occasionally 😢