Building an LLM-based assistant for the game of Diplomacy

An outline of the ReAct framework vs traditional LLM usage from the original parer

Lost and found in the kitchen

Our goal with the rest of the paper is to guide you through our problem - making a Diplomacy AI assistant - and how we used the ReAct framework to solve it.

2. Diplomacy + AI

Negotiation + Board Strategy = Not Yet

• Diplomacy has an enormous state space - 7 players, 49 board locations with a high degree of connectivity. Comprehensive, multi-step, tree-search-based tools like other state of the art zero-sum game playing agents use are simply unfeasable.
• The game is played with a high-degree of "convention". Agents trained primarially with self-play don't play well against humans, because they learn/play a different "style" of game.
• The problem of connecting natural-language negotiations with actions in the world remains unsolved. This is connected to the relatively understudied area of task-oriented dialogues in NLP, especially regarding learning to deceive or coerce, which is crucial for Diplomacy. GPT is good at talking, but it is a people pleaser ... so struggles to drive a hard bargain in a negotiation.
• Human gameplay data is scarce. Diplomacy is a relatively niche game, in large part because it is convoluted and takes a very long time to play (> 8 hours per game). Learning to play takes time, and being good requires a high degree of strategic analysis as well as negotiation acumen. These qualities aren't always found in the same person.

... but some progress has been made

Design of integrated agents that learn to negotiate effectively atop a strategically robust system remains an open challenge.

Render unto caesar...

Considering these limitations, the idea of "cyborg Diplomacy" emerged. Computers excel at tasks like evaluating possible moves, assessing the gameplay aspects of treaties, and keeping track of multiple opponents' intentions. However, interpersonal aspects like convincing another player to form an alliance, making final decisions, detecting deception, and making final decisions about moves still might be better intermediated by a human player.

Furthermore, one of the most interesting aspects of Diplomacy is the human-to-human negotiation tactics. Could an amateur Diplomacy player with strong negotiation skills beat an experienced Diplomacy player when armed with a tool to help resolve some of the strategic aspects?

3. Building a Diplomacy ReAct Agent

• An LLM that interprets the query, reasons, invokes tools, interprets the results of the tools, and answers the query.
• The LLM gets a prompt that describes the task, the tools, and how to use them.
• A set of tools that are made available to the LLM to accomplish the task.

The goal for our Diplomacy work was to produce a chatbot that could - given some information about the current board state and access to the tooling we had built - answer arbitrary questions from a player. These questions could range from simple facts the board state like "who is strong right now" or "what orders should I do on the next turn" that require basic tool invocation, to complex questions about strategy, deception, agreements, and opponents' intentions, which might require the ReAct LLM to invoke multiple tools joined together in intricate ways.

The overall schema of the agent is below:

schema of our diplomacy / react chatbot

The Prompt

The prompt is a a plain-text or Markdown document that describes to the LLM the overall task, exactly how to respond, and an explanation of the available tools, data structures, and formatting. It is something between software documentation and a very specific job description. Prompt 'engineering' is more an art than a science at this stage. Unlike writing code, the effect of additions, subtractions, and changes of any kind are not obvious.

Powerful LLMs are very flexible. They can correctly interpret misspellings, infer information that isn't clearly stated, and extrapolate very well. Therefore the upshot is that everything kind of works. On the other hand, its very hard to know how close to optimal the prompt is for a given task. What's certain is that writing a decent prompt isn't very hard, and writing a very good prompt is probably very hard. Further research on this, or developing learned / LLM augmented and updated prompts is probably needed.

Introduction, core tools, and formatting

The introduction is simple, it provides context and the general goal of the agent:

    You are a bot that assists people playing the board game Diplomacy. 
    Your job is to answer the user's questions as best you can. 
    You respond only in json that can be parsed by Python's json.loads function.

We only actually specify two generic 'tools' the LLM can use directly. The first is clarifying questions, which is self-evident. The second is the Python **REPL (Read-Eval-Print Loop)** (terminal shell), which is the 'gateway' through which our LLM can interact with the specialized Diplomacy-specific tooling we have built.

    You have access to the following tools, which can help you in your task:
    
    - `PythonREPL`
        - This tool is a python shell. Use it to execute python commands. 
        - If you expect output, it MUST be printed to stdout with the `print` function.
        - You should use multi-line formatting to avoid syntax errors.
        - Note that you have access to anything you've computed in previous calls to `PythonREPL`.
        - _Input_: a valid python program.
        - _Output_: sys.stdout for the program.
    
    - `ClarifyingQuestions`
        - This is a tool that you can use to ask the user clarifying questions.  
        - You can use this tool if you don't think you have enough information to answer the question.  
        - You can also use this tool if you think you're stuck and you want the user to help un-stick you.
        - _Input_: question (in English)
        - _Output_: answer to the question (in English)
    

We also specify the 'ReAct' chain-of-thought prompting, and guide the interaction with the software via the formatting.

Every response you generate should be in the following JSON format:
```
    {
    "thought"    : # you should always think about what you need to do
    "tool"       : # the name of the tool.  This must be one of: [PythonREPL]
    "tool_input" : # the input to the tool
    }
```

We finally specify the way the outputs will be returned to the user:

## Tool Evaluation + Iteration
    After you respond in the above format, we will run `tool(tool_input)` and return the output to you in the following format:
```
    { "observation" : # the output of tool(tool_input) }
```

    You can then use the formulate another `{"thought" : ..., "tool" : ..., "tool_input" : ...}` as needed, until you're convinced you have the answer to the question.

    ## Final Answer
    When you are confident you have the correct answer, output the answer in the following format:
```
    { "answer" : # the answer to the question }
```

The Diplomacy specific tooling / python functions

Examples

The last part of the Markdown document consists of providing various examples of questions, the use of tools, and the expected inputs and outputs. As previously mentioned, its difficult to know how and what to specify here, but it seems that more and more varied examples improved the performance of the bot.

    - User: "Question: What are the possible orders for the army in Moscow?"
    - You:
```
    {
        "thought" : "I need to get all of the possible orders for the army in Moscow.",
        "tool"  : "PythonREPL",
        "tool_input" : "
    poss = get_possible_orders()
    print(poss['MOS'])
    "
    }
```
    - User:
```
    { "observation" : "[{'datc': 'A MOS S A STP - LVN', 'type': 'support', 'node': 'MOS', 'source': 'STP', 'target': 'LVN', 'node_unit_type': 'A', 'source_unit_type': 'A', 'node_power': 'GERMANY', 'source_power': 'GERMANY', 'target_power': None}, {'datc': 'A MOS S A UKR', 'type': 'support_hold', 'node': 'MOS', 'source': 'UKR', 'target': 'UKR', 'node_unit_type': 'A', 'source_unit_type': 'A', 'node_power': 'GERMANY', 'source_power': 'TURKEY', 'target_power': 'TURKEY'}]"}
```
    - You:
```
    { "answer" : "
    The possible orders for the army in Moscow are 
    - supporting A STP - LVN 
    - supporting A UKR
    "}
```

LLM

The LLM is easy, we use GPT-4. Its language and code generation / understanding capabilities are definitely the best available at the moment, and its ability to do complex reasoning is far better than other available LLMs. We implement a simple wrapper to keep the chat history and do some simple error handling.

import openai
class ChatGPT:
    def __init__(self, model='gpt-4', system=""):
        self.model  = model
        self.system = system
        self.system_message = {"role": "system", "content": system}
        self.messages = []

    def __call__(self, message):
        self.messages.append({"role": "user", "content": message})
        result = self.execute()
        self.messages.append({"role": "assistant", "content": result})
        if 'ERROR: You sent the following' in message:
            # Drop error + correction from chat history
            self.messages = self.messages[:-3] + self.messages[-1:]
        return result

    @retry_decorator
    def execute(self):
        completion = openai.ChatCompletion.create(
            model=self.model,
            messages=[self.system_message] + self.messages,
            temperature=0,
        )
    def pprint(self):
        rprint(self.messages)

4. The system in action

Suppose you're playing as Italy (green in the diagram below). The game is in its 7th year and things are going decently, you have taken some of central Europe and Africa, and you're trying to understand where you stand, and what you might want to do.

Board State

Lets try some simple things like trying to understand the board:

In this section we see the chatbot:

• Respond to straightforward queries regarding the state of the game
• Write simple, correct python scripts - and even correct them
• Consistently follow thought / tool / tool-input / observation / answer framework
• Interact with the 'stateful' tooling provided through PythonREPL, allowing for a structured, accurate and grounded interaction.
• Interpret and gloss the output of python functions to return correct answers

Orders

In this section we demonstrate some capaiblities about thinking through a good set of orders:

In this section we see the chatbot:

• Know what it doesn't know (ask clarifying questions) see: "which player are you playing in the game"
• Observe, interpret, and remember extremely long sequences of orders / board state information (certainly far faster and more comprehensively than any person could). It can even deal with its own limitations in this regard (too long function output).
• Write relatively more complex python scripts (with interpretable naming conventions!)
• Reason, together with the human player about a complex problem that involves multiple complex steps.
• Failure w/ Serbia's unit at Turkey... not provide wrong information (but perhaps missing one step of understanding why it isn't working).

Treaties

In this section we try to reason about possible agreements between powers by considering Italy's relatively disandvantageous position:

This section demonstrates the chatbot's ability to:

• Error/delay handling.
• Use our custom tools for implementing and valuting treaties.
• Use our custom tools for choosing counterfactual moves (positive / negative constraints).
• Writing much more complex python scripts.
• Thinking through a pretty complex problem.
• Use more classic LLM tools like think of funny names for a new alliance type and find a way to sell something to someone.

5. The upshot

We hope this specific tool will make the game of Diplomacy more accessible to novice players, and allow them to focus more on the interesting bit... lying and decieving, coercing and backstabbing. We asked our agent what it would be called to play against human players using our diplomacy assistant... "Digital Doping", it said. Good luck playing against us at your local tournament.

But beyond the immediate Diplomacy specific use-case, we think our system is a specific example of a powerful general idea: using LLMs as nocode HMI / chatbot interfaces for complex tooling. Building HMIs is usually a big lift. A GUI, hardcoded functions for "typical" queries, interpreters for inputing those queries (buttons? dropdowns?), hardcoded workflows, glue code, methods / language / visualizers to share results, all to make an extremely brittle tool that can only do what the designer thought of ahead of time.

Compare that to our LLM+ReAct+Tooling Diplomacy assistant: it can be queried and respond to in natural language (or code, or a DSL). In the background (or foreground) it determines and invokes the right "toolkit" calls, glue code, and visualization scripts neccessary to solve the query. And it returns results in natural language. Basically, it's a super-fast personal assistant with deep knowledge of a topic area that can implement a tool to solve your domain specific query, available at scale. This stands somewhat in contrast with the typical useage of LLMs which are totally reliant on the information stored in the weights of the model (knoweldge, reasoning traces) with the occasional extension of a database store.

We hope this provides a good overview of our diplomacy work - and on the wider idea of LLMs as generic HMIs for domain-specific tooling. Please leave a comment or reach out with any questions!

zeek, ben @ jataware dot com