Enterprise Ontology — Part VII — Natural Language Interfaces for Policies — Amazon Bedrock Agent

Generative AI interfaces such as ChatGPT revolutionize user interfaces and allow non-technical people to interact with complex data in natural language. In this part, we will add a natural language interface to the ontology, allowing internal business people in the enterprise and external customers to ask questions about the enterprise policies organized in the ontology and get trustful answers.

Natural Langauge Interface Example

Why connect generative AI models to Ontology?

Generative AI models can easily make mistakes when answering natural language questions (a.k.a. “hallucinations”). There are many reasons for these mistakes, especially when the data needed to answer the questions is not part of their training data and differs from the publicly available data. The ontology data is an excellent example of that missing data. Even if the organization published its policies on public websites, and the generative AI models (a.k.a. LLM) were trained on that data, the organization has no control over updates to these policies in the AI models, which are based on the frequency of the data crawling and model retraining.

On the other hand, showing the AI model the relevant data from the internal ontology before it answers the question will anchor the answer in the “source of truth,” which can also be updated constantly as needed.

Let’s take the car rental company's example above of answering the question, “What is the minimum driver age in NYC?” If you send that question to ChatGPT, it might answer outside the context of car rental, as the question doesn’t specify it.

Example of ChatGPT answer to a policy question

However, if the answer covers car rental policies, it will cover any internal policy of a specific company (Avis or Hertz, for example) that might decide to set a higher minimum age to prevent young drivers or a lower minimal age to attract a younger audience. In any case, the AI model will most likely not be aware of the change and reply incorrectly.

Combining an AI model that can understand natural language questions and operate complex technical data APIs to retrieve relevant data, a strict schema, and updated ontology data can solve these trust problems. The logical flow of the AI agent is as follows:

• The AI agent understands from the natural language input that the question concerns the "minimal driver age” policy.
• It understands that the question is specific to NYC, which is in the state of New York (NY).
• It knows how to find the relevant policies of a given state (such as NY) and executes the API calls to the ontology holding the policies.
• It adds the policy JSON data with the policies' text, including information on the priority of different levels of the ontology (organization policy overrides state policy or the other way around), to the step generating the answer.
• Maintaining context to allow follow-up questions, such as “And in New Jersey?” or requesting clarifications from the user, such as “Which Paris, Paris in Texas, or Paris in France?”.

Policy Question Solution Architecture

Policy Question Flow and Solution Architecture

In previous parts, we discussed creating the ontology using Cloud Directory and implementing the GraphQL interface to the ontology using AppSync and its Lambda resolver. Now, we can connect it to the generative AI flow using AWS Bedrock Agent and a simple chat UI using App Runner.

In the diagram above, we can see the flow:

• The user (internal or external) asks the policy question using a simple chat interface. The chat interface can be hosted on AWS App Runner or any other secure hosting option.
• The question is sent to a Bedrock agent that is connected to a Large Language Foundation Model (LLM) to (1) understand the natural language question of the user, (2) generate the queries to the internal GraphQL Policy Lookup API, and (3) generate the natural language answer that is sent back to the user.
• The queries to the internal GraphQL API are sent through an action Lambda function that exposes an OpenAPI schema for its capabilities and interacts with the GraphQL API.
• The API calls to the GraphQL API hosted on AppSync are translated to the API calls to the Cloud Directory hierarchical data using the Lambda resolver.

Why Bedrock Agent?

There are many new and emerging tools for creating generative AI applications and many LLM options to choose from. Since most of the applications (Cloud Directory, AppSync, etc.) are already in the AWS cloud, it makes sense to choose an integrative service to make the security, deployment, and other integration requirements easier in the AWS cloud. Bedrock has a few more benefits, such as easy model selection, automatic scaling, agent testing, etc.

Let’s start by understanding the customization options of generative AI models and why we choose to use an Agent. There are three main customization options for LLMs:

• “Black Box”—customizing the generation of the LLM using the input to the model, a.k.a. prompt engineering. This is the easiest option compared to the following ones; however, it is also the riskiest. It is hard to test the system, and in most cases, the prompts are getting longer and more complex to handle issues, such as “hallucinations” of the models.
• “Gray box”—Customizing the generation process flow includes calls to tools or actions to look up information in API calls or knowledge bases. This is the Agent concept we are using here, and it balances the ease of building it and the safety and trust it provides.
• “White Box”—Finetuning the LLM foundational models (FM) to learn the context of the internal organization data. This provides, ideally, the safest option, as the model learns a lot about the organization's context and is less likely to make mistakes from the general public data it was originally trained on. However, it is also the hardest option as it requires a lot of data and experienced data engineers and scientists are needed to fine-tune the models based on that data.

The balance between complexity and safety that the “Gray Box” provides is a good place to start. We know that in time, we will gather more data and technical expertise alongside the development of generative AI tools, and we can shift to the “White Box,” improving even further the performance of our AI systems.

What is Bedrock Agent?

AWS Bedrock is a set of generative AI services in the AWS cloud, and the Bedrock Agent service is part of Bedrock's orchestration section.

Orchestrate and execute multistep tasks

Agents orchestrate and analyze the task and break it down into the correct logical sequence using the FM’s reasoning abilities. Agents automatically call the necessary APIs to transact with the company systems and processes to fulfill the request, determining along the way if they can proceed or if they need to gather more information.

When building an Agent with Bedrock, you can configure it to fit many different use cases with the following settings:

• Model Selection — Bedrock wraps various models and provides a unified interface that makes switching models easier over time. For example, you can choose between three versions of Claude 3 models based on their performance, latency, and cost.

Claude 3 version comparison from Anthropic launch blog post

• Action Groups — the tools the Agent can use are defined and executed using Lambda functions. The Lambda function can call different APIs, such as the GraphQL API, as in the ontology use case.
• Advanced Prompts — Bedrock provides templates for the agent prompt, which you can override as needed. For example, the template includes a pre-processing prompt that filters the user input before processing it in the orchestration phase. The template is based on Amazon's experience in Trustful AI, which covers most of the use cases. However, if the use case is more or less sensitive, it is possible to override it. The prompt for the orchestration phase describes the action groups that are available for the Agent and can be tuned to provide further information that is missing from the schema of the action groups (see example below).
• Knowledge Base—Bedrock allows building a knowledge base using a vector database from a set of internal documents, websites, or internal data systems. Although there is room for using knowledge bases in the ontology, please see the following section to understand the limitations of knowledge bases with RAG technology compared to the ontology's hierarchical structure and API.

Why Not Using RAG?

Retrieval Augmented Generation (RAG) is a popular method to achieve similar functionality and provide the LLM with a reliable context for the text generation task. Many services, such as Pinecone, Open Search, Redis, MongoDB, and others, offer the option to take internal documents, calculate embedding vectors to these documents or their chunks, store them in a vector database, and allow semantic search based on vector similarity to users’ queries.

It might seem that we can simply take the policy documents, index them in a vector database, and use RAG to retrieve the relevant documents based on the semantic similarity and get a similar accuracy. However, the ontology's hierarchical structure and the tables' separation into different nodes and leaves of the hierarchical trees give a more accurate and complete policy retrieval mechanism.

For example, a policy document can contain the following sentence: “The minimum driver age in the US is 20, except for New York and Texas with 18, and Michigan with 19.” Using RAG, the LLM will get the whole sentence and might confuse the states and numbers. While using the ontology, the US node will have a value of 20, the NY leaf and the TX leaf will have a value of 18, and the MI leaf will have a value of 19. When a user asks about the minimum age in CA, it will only get the US value of 20, or when asked about the age in Houston, it will get the TX value of 19 and the US value of 20. It will know to override the general value with the more specific one and answer with 19. The LLM is less likely to make mistakes with fewer conflicting values in the same sentence.

Comparison of RAG and Ontology Retrieval

Bedrock Agent Action Group

The action group that we add to the Agent is a set of tools that we give to the LLM to generate the correct answer to the user’s questions. The action group is defined as a Lambda function and an OpenAPI schema file that describes its capabilities. For example, the Lambda function can access the GraphQL API of the ontology and extract the relevant policies for a given question.

For example, the OpenAPI schema below for the Lambda function with the get_policies tool. The schema includes the method (GET), and the parameters of the functions, such as state_id. Since an LLM reads the schema, it can include a natural language description, which makes it easy for the LLM to use the tool correctly. For example, we can instruct the model to use shortcodes (CA for California, for example), and it will be able to do it as these codes are used in the documents that it was trained on, and it is not local to the organization that defined the ontology.

openapi: 3.0.0
info:
  title: Ontology API for policies
  version: "1.0"
paths:
  /get_policies:
    get:
      summary: "Gets the policies of the requested state by its short code ID"
      operationId: get_policies
      description: "Gets the policies of the requested state, country or province by its short code ID (for example, 'CA' for California or 'IT' for Italy)"
      parameters:
        - name: state_id
          in: query
          required: true
          schema:
            type: string
          description: "The state's short code ID"
      responses:
        '200':
          description: "Successful Response"
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/Policy'
components:
  schemas:
    Policy:
      type: object
      properties:
        policy_id:
          type: string
        node_id:
          type: string
        policy_type:
          type: string
        policy_document:
          type: string
        policy_holder:
          $ref: '#/components/schemas/DirectoryNode'
    DirectoryNode:
      type: object
      properties:
        node_id:
          type: string

The function tools are also defined in the orchestration prompt. For example, the rental car organization can define the following prompt:

Human:
You are a research assistant AI that has been equipped with one or more functions to help you answer a <question>. Your goal is to answer the user's question to the best of your ability, using the function(s) to gather more information if necessary to better answer the question. If you choose to call a function, the result of the function call will be added to the conversation history in <function_results> tags (if the call succeeded) or <error> tags (if the function failed). $ask_user_missing_parameters$
You were created with these instructions to consider as well:
<auxiliary_instructions>$instruction$</auxiliary_instructions>

Here are some examples of correct action by other, different agents with access to functions that may or may not be similar to ones you are provided.

<examples>
    <example_docstring> Here is an example of how you would correctly answer a question using a <function_call> and the corresponding <function_result>. Notice that you are free to think before deciding to make a <function_call> in the <scratchpad>.</example_docstring>
    <example>
        <functions>
            <function>
                <function_name>get::OrganizationPolicies::get_policies</function_name>
                <function_description>Gets the rental policies of the requested state or country by its short code ID.</function_description>
                <required_argument>state_id (string): The state, country, or province short code (CA for California or IT for Italy, for example)</required_argument>
                <returns>array: Successful response</returns>
                <raises>object: Invalid request</raises>
            </function>
            $ask_user_function$
        </functions>

        <question>what is the minimum age to rent a car in New York?.</question>
        <scratchpad>
            To answer this question, I will need to:
            1. I do not have knowledge to policy management system, so I should see if I can use any of the available functions to help. I have been equipped with get::OrganizationPolicies::get_policies that gets the policies for a given destination state. I will use this function to gather more information.
        </scratchpad>
        <function_call>get::OrganizationPolicies::get_policies(state_id="NY")</function_call>
        <function_result>{response: [{"policy_type": "young_driver","policy_document": "{\"min_age\":20}, "policy_holder": {"__typename": "Country"}},{ "policy_type": "young_driver", "policy_document": "{\"min_age \":18}", "policy_holder": {"__typename": "State"}}]}</function_result>
        <scratchpad>
            To understand the policies, I will need to:
            2. choose the most relevant policies based on their policy_type value.
            3. prioritize the policies based on their policy_holder, where more specific nodes such as state, override the more general nodes such as organization or region.
            4. choose the most relevant section in the policies that are relevant to the question of the user.
        </scratchpad>
        <answer>18, The New York state policy for minimum driver age overrides the country policy.</answer>
    </example>
...

The prompt's structure might look technical with the XML tags. However, the prompt's customization between use cases is mainly in the natural language parts. It can be done by a domain expert from the business side and doesn’t require a technical person or a software developer.

AI Agent Logic Flow

The power of AI agents comes from their ability to:

• Understand a natural language question from a human user,
• Translate it to a set of API calls to external data sources such as the ontology, a SQL database, or a vector database (RAG),
• Understand the data in the replies to the API calls and
• Generate a natural language answer to the user based on that data.

For example, when the user asks: “What is the minimum age in Houston?” the AI agent will:

• Understand that the question is related to a rental car policy and that it should query the ontology with the rental car policies to answer it.
• Check the schema of the tools it has as part of the action group and find the relevant API get_policies .
• Translate the input of the user for Houston to the expected parameter by the API to the shortcode of Texas, TX .
• Call the API and understand the reply in the form {"minimum_age":18} .
• Generate the answer “The minimum driver age in Houston, TX is 18”.

Please note that I simplified the flow as the reply from the API will include both the policies for Texas and the USA, and the AI agent will use the instruction to override the policies to the more specific ones in the states.

If you want to learn more about building AI with Bedrock Agent, you can check out my online course on O’Reilly and EdX.

This is part VII of the series on “Building Enterprise Ontology,” and you are welcome to read the other parts here:

• Part I — Overview of Ontology Solution on AWS.
• Part II — Business Overview — What is an Enterprise Ontology?
• Part III — Ontology Data Store — AWS Cloud Directory.
• Part IV — Ontology Access API — GraphQL on AppSync and Lambda Resolver.
• Part V — Ontology Setup and Data Loading — AWS Step Functions Automation (coming soon)
• Part VI — Reliable Authorization with Ontology — Amazon Verified Permission (coming soon).
• Part VII — Natural Language Interfaces for Policies — Amazon Bedrock Agents (this post).

Summary

The enterprise ontology we built for policy management provides multiple types of interfaces for consuming policy documents across the organization hierarchy. The most useful interface for business people across the organization and the company's external customers is the natural language interface we built using the AI agent in this post.

The new generative AI technology that burst into our lives in the last couple of years, since the release of ChatGPT, raised many conversations about trust and risks from incorrect but convincing information (a.k.a. hallucinations) AI can generate in replies to human questions. The framework of the strict schema ontology, in combination with the trustful AI agent we can build using AWS Bedrock, allows us to benefit from this exciting technology while minimizing the risks of wrong answers.