Mastering advanced prompt engineering: Part 2
In the first part of our blog series, “Mastering advanced prompt engineering: Part-1,” we explored the evolution of prompt engineering, covering foundational techniques like Chain-of-Thought (CoT) prompting and its variations – Zero-Shot Chain of Thought (ZS-CoT), Auto-CoT, Chain-of-Thought with Self-Consistency (CoT-SC), and Tree of Thoughts (ToT).
Before diving into advanced prompt reasoning, let’s have a look at what we have covered so far.
Chain-of-Thought (CoT) prompting encourages language models to break down reasoning into steps, improving complex problem-solving.
Zero-Shot CoT (ZS-CoT) uses reasoning prompts without specific examples.
Auto-CoT automates prompt selection from large datasets.
Self-Consistency CoT (CoT-SC) chooses the most consistent reasoning path from multiple generated solutions.
Tree of Thoughts (ToT) explores multiple reasoning paths in a branching structure.
With this foundation, we now dive deeper into advanced prompt engineering with large language models (LLMs) and cover more advanced approaches that push the boundaries of AI problem-solving capabilities.
In this second part of our series, we’ll focus on two cutting-edge techniques:
Graph of Thoughts (GoT)
Program of Thoughts (PoT)
These advanced methods represent significant leaps forward in our ability to utilize AI for complex reasoning tasks. By enabling more structured, interconnected, and programmatic thinking processes, GoT and PoT open new possibilities for problem-solving, creativity, and decision-making across various domains.
Graph of Thoughts (GoT) prompting#
Key features of GoT#
Non-linear reasoning: GoT allows for complex, interconnected thought patterns that can better represent intricate problem-solving processes.
Iterative refinement: Thoughts can be revisited and refined, enabling the model to improve its reasoning over time.
Multi-directional connections: Unlike a tree structure, a graph allows for connections between any nodes, facilitating more diverse thought patterns.
Scalability: The graph structure can potentially handle more complex problems than linear or tree-based approaches.
Flexible problem decomposition: GoT allows for dynamic problem decomposition, where sub-problems can be identified and solved as the reasoning progresses.
How GoT Works#
To understand GoT, let’s break it down into its key components and processes:
Graph structure
Thought generation and exploration
Scoring and evaluation
Backtracking
Refinement and iteration
Aggregation
Generation
Output synthesis
Graph structure#
In GoT, thoughts are represented as nodes in a graph, with edges representing dependencies or connections between thoughts. This structure allows for more flexible and intricate reasoning patterns than linear (CoT) or hierarchical (ToT) approaches.
Thought generation and exploration#
The model begins with an input and generates several initial approaches. Unlike ToT, which follows a strict tree structure, GoT allows for arbitrary connections between thoughts at any level. The model can explore multiple paths simultaneously, branching out in various directions as needed.
Example: Let’s say we’re trying to solve a complex math problem. In GoT, the model might generate initial thoughts like:
“Let’s break this into smaller sub-problems.”
“We could use integration by parts here.”
“This reminds me of a similar problem we solved earlier.”
These thoughts can be interconnected and explored in parallel rather than following a strict hierarchy.
In ToT, it might have looked something like this:
“This problem seems complex. Let’s break it into steps: first, solve for X, then move to Y.”
“I could use integration here, or maybe substitution will simplify it faster.”
“Using substitution simplifies this expression, so let’s follow this path.”
Scoring and evaluation#
Similar to ToT, thoughts in GoT can be scored as positive (promising) or negative (less promising). This scoring helps guide the exploration process, focusing on more promising paths while allowing diverse thinking.
Example: Continuing with our math problem:
“Let’s break this into smaller sub-problems.” (Positive score)
“We could use integration by parts here.” (Positive score)
“This reminds me of a similar problem we solved earlier.” (Neutral score)
“Maybe we should give up and try a different approach.” (Negative score)
Backtracking#
GoT incorporates a backtracking mechanism, allowing the model to revisit and refine earlier thoughts. This is particularly useful when a chosen path proves unfruitful, enabling the model to explore alternative routes.
Example: If the “integration by parts” approach leads to a dead end, the model can backtrack to the “break into sub-problems” thought and explore that path further.
Refinement and iteration#
A key feature of GoT is the ability to refine thoughts through iteration. The model can loop back to earlier nodes, updating and improving them based on new insights gained during the exploration.
Example: After exploring the “break into sub-problems” approach, the model might refine the original thought: “Let’s break this into three specific sub-problems: solving the integral, finding the limits, and simplifying the result.”
Aggregation#
GoT introduces two types of thought transformations: aggregation and generation. In aggregation, multiple thoughts are combined into a new, more comprehensive thought. This process can involve merging information from different branches of reasoning.
Example: Consider a writing task where we’re summarizing multiple articles:
Thought 1: “Article A discusses climate change impacts.”
Thought 2: “Article B focuses on mitigation strategies.”
Thought 3: “Article C explores economic implications.”
Aggregated thought: “The articles collectively cover the impacts, mitigation strategies, and economic implications of climate change.”
Generation#
The generation process involves creating new thoughts based on existing ones. This can include breaking down complex ideas into simpler components or expanding on promising concepts.
Example: Starting with the thought “We need to improve customer satisfaction,” the model might generate:
“Let’s analyze current customer feedback.”
“We should review our product features.”
“Training our customer service team could help.”
Output synthesis#
The model synthesizes a final output after exploring the graph, refining thoughts, and aggregating key insights. This output benefits from the comprehensive exploration and refinement process, often resulting in more nuanced and well-considered solutions.
Implementing GoT#
While the concept of GoT is powerful, implementing it effectively requires careful consideration and strategic prompting. Here’s a table summarizing the use case (Designing a marketing strategy for a new product) and general prompts for each stage of implementing the GoT concept:
Stage | Use Case Prompt | General Prompt |
1. Initial prompt design | You’re tasked with creating a marketing strategy for a new product. Consider this problem from multiple angles: market segmentation, pricing, advertising channels, and customer engagement. Generate at least three initial approaches for each area, and then explore how these approaches might be connected or influence each other. Feel free to branch out into sub-ideas as you go. | Consider this problem from multiple angles. Generate at least three initial approaches, and then explore how these approaches might connect or influence each other. Feel free to branch out into sub-ideas as you go. |
2. Guiding exploration and refinement | Now that you’ve generated thoughts on market segmentation, pricing, and advertising channels, how might they influence each other? Are there any potential synergies or conflicts between these strategies? For example, how could the pricing strategy impact the choice of advertising channels? | Now that you’ve generated these initial thoughts, how might they influence each other? Are there any potential synergies or conflicts between them? Let’s revisit the thought about [specific idea]. Based on what we’ve explored so far, how might we refine or expand this concept? |
3. Encouraging backtracking and iteration | Looking at our current graph of thoughts on the marketing strategy, are there any earlier ideas we should revisit or reconsider? For instance, does the customer engagement strategy need adjustments in light of our recent explorations on advertising channels? | Looking at our current graph of thoughts, are there any earlier ideas we should revisit or reconsider in light of our recent explorations? |
4. Facilitating aggregation and synthesis | Now that we’ve explored various aspects of the marketing strategy, let’s synthesize our key insights. How do the different branches of our thought graph—market segmentation, pricing, advertising, and engagement—come together to form a comprehensive marketing plan? | Now that we’ve explored various aspects of this problem, let’s try to synthesize our key insights. How do the different branches of our thought graph come together to form a comprehensive solution? |
Practical applications of GoT#
GoT can be applied to various complex problem-solving scenarios. Here’s a summary of its practical applications:
Practical Applications of GoT | Example |
Strategic planning: GoT can be particularly effective in business strategy development, where multiple factors (market trends, competitive landscape, internal capabilities) must be considered simultaneously. | Input: “Develop a 5-year growth strategy for our tech startup.” GoT process: The model would create nodes for market analysis, product development, funding strategies, and team growth. It would then explore connections between these areas, refine ideas based on potential outcomes, and aggregate insights into a comprehensive strategy. |
Scientific research: In fields requiring the integration of diverse data and theories, GoT can help researchers explore complex hypotheses and experimental designs. | Input: “Propose a research design to study the impact of microplastics on marine ecosystems.” GoT process: The model would generate thoughts on sampling methods, chemical analysis techniques, ecological impact assessment, and data analysis approaches. It would then refine these ideas, exploring their interconnections and potential challenges, before synthesizing a comprehensive research proposal. |
Creative writing: GoT can assist in developing complex narratives with intertwining plotlines and character development. | Input: “Create an outline for a mystery novel set in a futuristic city.” GoT process: The model would generate nodes for character backgrounds, plot twists, futuristic technology elements, and mystery clues. It then explores how these elements connect, refine character motivations based on plot developments, and aggregate ideas into a coherent storyline. |
Policy analysis: For analyzing the potential impacts of new policies or legislation, GoT can help consider various stakeholders, economic factors, and long-term consequences. | Input: “Analyze the potential impacts of implementing a universal basic income.” GoT process: The model would create nodes for economic effects, social implications, implementation challenges, and political considerations. It would then explore the interconnections between these factors, refine predictions based on potential scenarios, and synthesize a comprehensive analysis. |
Advantages and challenges of GoT#
Let’s now have a look at some of the advantages and challenges of GoT:
Advantages#
Holistic problem solving: GoT excels in scenarios where multiple interconnected factors must be considered simultaneously.
Creative ideation: The non-linear nature of GoT can lead to more creative and unexpected solutions.
Adaptability: The graph structure can adapt to problems of varying complexity and nature.
Iterative improvement: The ability to refine thoughts and backtrack leads to more polished and well-reasoned output.
Challenges#
Computational complexity: Managing large graphs can be computationally intensive.
Prompt engineering: Crafting effective prompts for GoT requires skill to guide the model in building and utilizing the graph structure effectively.
Interpretability: The complex graph structure might make it harder for humans to follow the model’s thought process than simple linear approaches.
Consistency maintenance: Ensuring consistency across all nodes and connections can become challenging as the graph grows.
GoT represents a significant advancement in prompt engineering and AI reasoning capabilities. By allowing for more flexible, interconnected, and iterative thought processes, GoT enables language models to tackle increasingly complex and nuanced problems. As we keep improving these techniques, we’re getting closer to AI systems that can solve challenging problems with reasoning similar to humans.
Program of Thoughts (PoT) prompting#
Now that we have explored CoT, which helps organize reasoning processes in a linear or hierarchical manner, we can examine an even more refined approach to
Unlike CoT, which is effective at structuring thoughts but still tends to intertwine reasoning and computation, PoT separates these two processes to enhance logical accuracy in tasks that require step-by-step reasoning.
Reasoning vs. computation: The core distinction in PoT#
To understand why PoT is crucial, it’s essential to first differentiate reasoning from computation.
Reasoning refers to the process of logically analyzing and structuring a problem to find potential solutions. It’s about understanding the connections between different parts of a problem and determining how to approach it.
Computation, however, involves executing specific steps, such as performing calculations or applying a formula, to reach a final result.
PoT separates these two functions, enabling the model to focus on reasoning first, before jumping into computation. This distinction is particularly important for problems that involve algorithmic steps or numerical calculations, where precise execution is key to obtaining correct results.
The illustration below shows the difference between CoT and PoT:
The difference between CoT and PoT is shown in the above figure. For CoT, the iteration runs for 50 times, which leads to extremely low accuracy. In contrast, PoT can express the iteration process with a few lines of code, which can be executed on a Python interpreter to derive an accurate answer.
Key features of PoT#
Separation of reasoning and computation: PoT distinguishes between the logical reasoning required to solve a problem and the actual numerical computations.
Algorithmic thinking: It prompts the model to generate a step-by-step program or algorithm to solve the given problem.
Language-agnostic approach: The generated program can be implemented in various programming languages, making it flexible and adaptable.
Enhanced accuracy: PoT can lead to more accurate results by separating reasoning from computation, especially for complex numerical problems.
How PoT works#
Let’s break down the PoT process step-by-step, using an example to illustrate each stage:
Example problem: Calculate the compound interest on a $10,000 investment over 5 years at an annual interest rate of 5%, compounded quarterly.
Problem analysis#
The LLM analyzes the problem, identifying key components: principal, rate, time, and compounding frequency.
Prompt: Analyze the given problem and explain what we need to calculate.
Variable identification#
The LLM identifies the specific variables needed for computation, ensuring all necessary data points are accounted for.
Prompt: Identify the variables needed to solve this problem.
Formula recognition#
The model selects the appropriate mathematical formula or logic to address the problem efficiently.
Prompt: What formula should we use to solve this problem?
Program structure generation#
The structure of the program is outlined to define its flow and key components before implementation begins.
Prompt: Outline the structure of a program to solve this problem.
Equation implementation#
The LLM implements the selected formula into executable equations, forming the computational core of the solution.
Equation implementation prompt: Implement the equations needed to solve this problem.
Main program and output formatting#
This step integrates equations into a main program with user-friendly output formatting for clear results.
Prompt: Write a main program that formats the output using these equations.
Full program assembly#
The complete program, including any necessary imports or auxiliary functions, is assembled for execution.
Prompt: Assemble the full program, including any necessary imports or additional functions.
Execution and verification#
The final step runs the program and verifies the accuracy of its output, ensuring reliability.
Prompt: Execute the program and verify the results. Do they make sense?
Advantages and challenges of PoT#
Let’s now have a look at some of the advantages and challenges of PoT:
Advantages#
Improved accuracy: By focusing on the structure of the solution rather than the computation, PoT often leads to more accurate results for complex problems.
Transparency: The generated program provides a clear and step-by-step explanation of the problem-solving process.
Scalability: PoT can handle problems of varying complexity by generating appropriate programs.
Language model agnostic: This approach can be used with different LLMs, as it relies on the model’s ability to generate logical steps rather than perform calculations.
Challenges#
Programming knowledge: The effectiveness of PoT may depend on the model’s understanding of programming concepts and syntax.
Error-handling: The approach needs robust error-handling mechanisms for cases where the generated program is incorrect or incomplete.
Computational resources: Executing the generated programs may require additional computational resources.
Combining GoT and PoT for advanced problem-solving#
While both GoT and PoT are powerful techniques on their own, combining them can lead to even more sophisticated problem-solving capabilities. Here’s how they can work together:
Problem decomposition with GoT: Use GoT to break down complex problems into interconnected sub-problems.
Algorithmic solution with PoT: For each sub-problem that requires computational steps, employ PoT to generate a programmatic solution.
Integration of results: Use the graph structure of GoT to integrate the results from various PoT-generated programs.
Iterative refinement: Leverage GoT’s ability for iterative thinking to refine the overall solution based on the outcomes of PoT computations.
Example: Economic impact analysis#
Let’s look at an example:
Problem: Analyze the potential economic impact of implementing a new environmental policy across different industries.
GoT decomposition#
First, we break down the problem into interconnected sub-problems, each represented as a node in the graph. GoT would decompose the problem to something like this:
Node 1: Environmental policy details
Node 2: Affected industries
Node 3: Implementation costs
Node 4: Long-term benefits
Node 5: Job market impact
PoT integration#
PoT is utilized to handle computation-heavy tasks within specific nodes, ensuring accurate and detailed numerical or logical outputs for complex sub-problems.
For Node 3 (implementation costs), use PoT to create a program that calculates costs for each industry based on their size and current practices.
For Node 4 (long-term benefits), develop a PoT program to project environmental improvements and associated economic benefits over time.
For Node 5 (job market impact), create a PoT program to model job losses and creations across different sectors.
GoT result integration#
Synthesize outputs from PoT into the GoT framework to analyze the broader interconnections and relationships between nodes.
Connect the outputs of the PoT programs back into the GoT structure.
Use GoT to analyze the relationships between implementation costs, long-term benefits, and job market impacts.
Iterative refinement#
Finally, use GoT to revisit and refine nodes based on the integrated results, ensuring a robust and nuanced solution.
Based on the integrated results, use GoT to identify areas that need further analysis or refinement.
Employ PoT again to adjust calculations or add new computational models as needed.
This combined approach utilizes the strengths of both techniques: GoT’s ability to handle complex, interconnected problems and PoT’s precision in computational tasks.
Conclusion#
As AI evolves to tackle more complex, nuanced problems, techniques like GoT and PoT bridge the gap between data processing and human-like reasoning. This journey through advanced prompt engineering illustrates how AI can tackle complex problems with depth and insight, closely resembling human thought. Whether in research, strategic planning, or creative projects, GoT and PoT open new doors for AI-enhanced problem-solving. Stay tuned as we continue exploring the future of AI reasoning.
Next steps#
To deepen your knowledge of LLMs, prompt engineering, and retrieval-augmented generation (RAG), we recommend our Generative AI courses.
Free Resources
