Skip to content

Commit

Permalink
Merge branch 'develop' v0.7.5
Browse files Browse the repository at this point in the history
  • Loading branch information
@rstrahan
rstrahan committed Jan 18, 2024
2 parents bd028df + 57d54d3 commit aa0dcc0
Show file tree
Hide file tree
Showing 18 changed files with 415 additions and 175 deletions.
16 changes: 13 additions & 3 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [0.7.5] - 2024-01-17
### Added
- Support for larger prompts by storing LLMPromptSummaryTemplate in S3 rather than SSM. By default, the CF templtae will migrate existing SSM prompts to DynamoDB.

### Fixed
- #125 Updated the pca-aws-sf-bulk-queue-space.py function to correctly count jobs based on IN_PROGRESS as well as QUEUED
- #224 Updated the pca-aws-sf-bulk-queue-space.py function to correctly count both Transcribe and Transcribe Call Analytics (vs just Transcribe).

## [0.7.4] - 2023-12-15
### Added
- Drag/drop upload from call list page
- Refresh call summary from call details page
- Drag/drop upload from call list page.
- Refresh call summary from call details page.

### Fixed
- Accessibility improvements
Expand Down Expand Up @@ -152,7 +160,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- Initial release

[Unreleased]: https://github.com/aws-samples/amazon-transcribe-post-call-analytics/compare/v0.7.3...develop
[Unreleased]: https://github.com/aws-samples/amazon-transcribe-post-call-analytics/compare/v0.7.5...develop
[0.7.5]: https://github.com/aws-samples/amazon-transcribe-post-call-analytics/releases/tag/v0.7.5
[0.7.4]: https://github.com/aws-samples/amazon-transcribe-post-call-analytics/releases/tag/v0.7.4
[0.7.3]: https://github.com/aws-samples/amazon-transcribe-post-call-analytics/releases/tag/v0.7.3
[0.7.2]: https://github.com/aws-samples/amazon-transcribe-post-call-analytics/releases/tag/v0.7.2
[0.7.1]: https://github.com/aws-samples/amazon-transcribe-post-call-analytics/releases/tag/v0.7.1
Expand Down
2 changes: 1 addition & 1 deletion VERSION
View file
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0.7.4
0.7.5
95 changes: 27 additions & 68 deletions docs/generative_ai.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,58 +13,22 @@ PCA also supports 'Generative AI Queries' - which simply means you can ask quest

## Generative AI Insights

When enabled, PCA can run one or more FM inferences against Amazon Bedrock or Anthropic APIs. The prompt used to generate the insights is configured in a [AWS Systems Manager Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html). The name of the parameter is `LLMPromptSummaryTemplate`.
When enabled, PCA can run one or more FM inferences against Amazon Bedrock or Anthropic APIs. The prompt used to generate the insights is stored in DynamoDB. The name of the table contains the string `LLMPromptConfigure`, and the table partition key is `LLMPromptTemplateId`. There are two items in the table, one with the partition key value of `LLMPromptSummaryTemplate` and the other with the partition key value of `LLMPromptQueryTemplate`.

### Multiple inferences per call
### Generative AI interactive queries

The default value for `LLMPromptSummaryTemplate` is a JSON object with key/value pairs, each pair representing the label (key) and prompt (value). During the `Summarize` step, PCA will iterate the keys and run each prompt. PCA will replace `<br>` tags with newlines, and `{transcript}` is replaced with the call transcript. The key will be used as a header for the value in the "generated insights" section in the PCA UI.
The item in Dynamo with the key `LLMPromptQueryTemplate` allows you to customize the interactive query prompt as seen in the call details page. You can use this to provide model specific prompts. The default valu is in [Anthropic's prompt format](https://docs.anthropic.com/claude/docs/constructing-a-prompt).

Below is the default value of `LLMpromptSummaryTemplate`.

```
{
"Summary":"<br><br>Human: Answer the questions below, defined in <question></question> based on the transcript defined in <transcript></transcript>. If you cannot answer the question, reply with 'n/a'. Use gender neutral pronouns. When you reply, only respond with the answer.<br><br><question>What is a summary of the transcript?</question><br><br><transcript><br>{transcript}<br></transcript><br><br>Assistant:",
"Topic":"<br><br>Human: Answer the questions below, defined in <question></question> based on the transcript defined in <transcript></transcript>. If you cannot answer the question, reply with 'n/a'. Use gender neutral pronouns. When you reply, only respond with the answer.<br><br><question>What is the topic of the call? For example, iphone issue, billing issue, cancellation. Only reply with the topic, nothing more.</question><br><br><transcript><br>{transcript}<br></transcript><br><br>Assistant:",
"Product":"<br><br>Human: Answer the questions below, defined in <question></question> based on the transcript defined in <transcript></transcript>. If you cannot answer the question, reply with 'n/a'. Use gender neutral pronouns. When you reply, only respond with the answer.<br><br><question>What product did the customer call about? For example, internet, broadband, mobile phone, mobile plans. Only reply with the product, nothing more.</question><br><br><transcript><br>{transcript}<br></transcript><br><br>Assistant:",
"Resolved":"<br><br>Human: Answer the questions below, defined in <question></question> based on the transcript defined in <transcript></transcript>. If you cannot answer the question, reply with 'n/a'. Use gender neutral pronouns. When you reply, only respond with the answer.<br><br><question>Did the agent resolve the customer's questions? Only reply with yes or no, nothing more. </question><br><br><transcript><br>{transcript}<br></transcript><br><br>Assistant:",
"Callback":"<br><br>Human: Answer the questions below, defined in <question></question> based on the transcript defined in <transcript></transcript>. If you cannot answer the question, reply with 'n/a'. Use gender neutral pronouns. When you reply, only respond with the answer.<br><br><question>Was this a callback? (yes or no) Only reply with yes or no, nothing more.</question><br><br><transcript><br>{transcript}<br></transcript><br><br>Assistant:",
"Politeness":"<br><br>Human: Answer the question below, defined in <question></question> based on the transcript defined in <transcript></transcript>. If you cannot answer the question, reply with 'n/a'. Use gender neutral pronouns. When you reply, only respond with the answer.<br><br><question>Was the agent polite and professional? (yes or no) Only reply with yes or no, nothing more.</question><br><br><transcript><br>{transcript}<br></transcript><br><br>Assistant:",
"Actions":"<br><br>Human: Answer the question below, defined in <question></question> based on the transcript defined in <transcript></transcript>. If you cannot answer the question, reply with 'n/a'. Use gender neutral pronouns. When you reply, only respond with the answer.<br><br><question>What actions did the Agent take? </question><br><br><transcript><br>{transcript}<br></transcript><br><br>Assistant:"
}
```

The expected output after the summarize step is a single json object, as a string, that contains all the key/value pairs. For example:

```
{
"Summary": "...",
"Topic": "...",
"Product": "...",
"Resolved": "...",
"Callback": "...",
"Politeness": "...",
"Actions": "...",
}
```


### Single FM Inference

Some LLMs may be able to generate the JSON with one inference, rather than several. Below is an example that we've seen work, but with mixed results.
The default value is:

```
<br>
<br>Human: Answer all the questions below, based on the contents of <transcript></transcript>, as a json object with key value pairs. Use the text before the colon as the key, and the answer as the value. If you cannot answer the question, reply with 'n/a'. Only return json. Use gender neutral pronouns. Skip the preamble; go straight into the json.
<br>Human: You are an AI chatbot. Carefully read the following transcript within <transcript></transcript>
and then provide a short answer to the question. If the answer cannot be determined from the transcript or
the context, then reply saying Sorry, I don't know. Use gender neutral pronouns. Skip the preamble; when you reply, only
respond with the answer.
<br>
<br><questions>
<br>Summary: Summarize the transcript in no more than 5 sentences. Were the caller's needs met during the call?
<br>Topic: Topic of the call. Choose from one of these or make one up (iphone issue, billing issue, cancellation)
<br>Product: What product did the customer call about? (internet, broadband, mobile phone, mobile plans)
<br>Resolved: Did the agent resolve the customer's questions? (yes or no)
<br>Callback: Was this a callback? (yes or no)
<br>Politeness: Was the agent polite and professional? (yes or no)
<br>Actions: What actions did the Agent take?
<br></questions>
<br><question>{question}</question>
<br>
<br><transcript>
<br>{transcript}
Expand All @@ -75,35 +39,30 @@ Some LLMs may be able to generate the JSON with one inference, rather than sever

The `<br>` tags are replaced with newlines, and `{transcript}` is replaced with the call transcript.

**Note:** This prompt generates 7 insights in a single inference - summary, topic, product, resolved, callback, agent politeness, and actions.

The expected output of the inference should be a single JSON object with key-value pairs, similar to above.
### Generative AI insights

### Call list default columns
The item in Dynamo with the key `LLMPromptSummaryTemplate` contains 1 or more attributes. Each attribute is a single prompt that will be invoked for each call analyzed. Each attribute contains an attribute name and value. The attribute name is an integer, followed by a `#`, followed by the name of the insight. The number signifies the order of the insight. For example, `1#Summary` will show up first.

The call list main screen contains additional pre-defined columns. If the output of the inference contains JSON with the column names (or the names are keys in the multiple inferences per call), the values will propogate to the main call list. The names columns are: `Summary`, `Topic`, `Product`, `Resolved`, `Callback`, `Politeness`, `Actions`. They are also in the default prompt.
Default attributes:

## Generative AI Queries
| Key | Description | Prompt |
| ----- | -------- | ---------- |
| `1#Summary` | What is a summary of the transcript? | `<br><br>Human: Answer the questions below, defined in <question></question> based on the transcript defined in <transcript></transcript>. If you cannot answer the question, reply with 'n/a'. Use gender neutral pronouns. When you reply, only respond with the answer.<br><br><question>What is a summary of the transcript?</question><br><br><transcript><br>{transcript}<br></transcript><br><br>Assistant:` |
| `2#Topic` | What is the topic of the call? | `<br><br>Human: Answer the questions below, defined in <question></question> based on the transcript defined in <transcript></transcript>. If you cannot answer the question, reply with 'n/a'. Use gender neutral pronouns. When you reply, only respond with the answer.<br><br><question>What is the topic of the call? For example, iphone issue, billing issue, cancellation. Only reply with the topic, nothing more.</question><br><br><transcript><br>{transcript}<br></transcript><br><br>Assistant:` |
| `3#Product` | What product did the customer call about? | `<br><br>Human: Answer the questions below, defined in <question></question> based on the transcript defined in <transcript></transcript>. If you cannot answer the question, reply with 'n/a'. Use gender neutral pronouns. When you reply, only respond with the answer.<br><br><question>What product did the customer call about? For example, internet, broadband, mobile phone, mobile plans. Only reply with the product, nothing more.</question><br><br><transcript><br>{transcript}<br></transcript><br><br>Assistant:` |
| `4#Resolved` | Did the agent resolve the customer's questions? Only reply with yes or no. | `<br><br>Human: Answer the questions below, defined in <question></question> based on the transcript defined in <transcript></transcript>. If you cannot answer the question, reply with 'n/a'. Use gender neutral pronouns. When you reply, only respond with the answer.<br><br><question>Did the agent resolve the customer's questions? Only reply with yes or no, nothing more. </question><br><br><transcript><br>{transcript}<br></transcript><br><br>Assistant:` |
| `5#Callback` | Was this a callback? | `<br><br>Human: Answer the questions below, defined in <question></question> based on the transcript defined in <transcript></transcript>. If you cannot answer the question, reply with 'n/a'. Use gender neutral pronouns. When you reply, only respond with the answer.<br><br><question>Was this a callback? (yes or no) Only reply with yes or no, nothing more.</question><br><br><transcript><br>{transcript}<br></transcript><br><br>Assistant:` |
| `6#Politeness` | Was the agent polite and professional? | `<br><br>Human: Answer the question below, defined in <question></question> based on the transcript defined in <transcript></transcript>. If you cannot answer the question, reply with 'n/a'. Use gender neutral pronouns. When you reply, only respond with the answer.<br><br><question>Was the agent polite and professional? (yes or no) Only reply with yes or no, nothing more.</question><br><br><transcript><br>{transcript}<br></transcript><br><br>Assistant:` |
| `7#Actions` | What actions did the Agent take? | `<br><br>Human: Answer the question below, defined in <question></question> based on the transcript defined in <transcript></transcript>. If you cannot answer the question, reply with 'n/a'. Use gender neutral pronouns. When you reply, only respond with the answer.<br><br><question>What actions did the Agent take? </question><br><br><transcript><br>{transcript}<br></transcript><br><br>Assistant:` |

For interactive queries from within PCA, it uses a different parameter, named `LLMPromptQueryTemplate`. This will only run a single inference per question.
The `<br>` tags are replaced with newlines, and `{transcript}` is replaced with the call transcript. Some Bedrock models such as Claude require newlines in specific spots.

The default value is:
#### Customizing

```
<br>
<br>Human: You are an AI chatbot. Carefully read the following transcript within <transcript></transcript>
and then provide a short answer to the question. If the answer cannot be determined from the transcript or
the context, then reply saying Sorry, I don't know. Use gender neutral pronouns. Skip the preamble; when you reply, only
respond with the answer.
<br>
<br><question>{question}</question>
<br>
<br><transcript>
<br>{transcript}
<br></transcript>
<br>
<br>Assistant:
```
You can add your own additional attributes and prompts by editing this item in DynamoDB. Make sure you include an order number and insight name in the attribute name. For example `9#NPS Score`. You can use any of the above prompts as a starting point for crafting a prompt. Do not forget to include `{transcript}` as a placeholder, otherwise your transcript will not be included in the LLM inference!

The `<br>` tags are replaced with newlines, and `{transcript}` is replaced with the call transcript.
### Call list default columns

The call list main screen contains additional pre-defined columns. If the output of the inference contains the column names, the values will propogate to the main call list. The names columns are: `Summary`, `Topic`, `Product`, `Resolved`, `Callback`, `Politeness`, `Actions`. They are also in the default prompt.

24 changes: 16 additions & 8 deletions pca-main-nokendra.template
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
AWSTemplateFormatVersion: "2010-09-09"

Description: Amazon Transcribe Post Call Analytics - PCA (v0.7.4) (uksb-1sn29lk73)
Description: Amazon Transcribe Post Call Analytics - PCA (v0.7.5) (uksb-1sn29lk73)

Parameters:

Expand Down Expand Up @@ -758,11 +758,17 @@ Resources:
ServiceToken: !GetAtt TestBedrockModelFunction.Arn
LLMModelId: !Ref SummarizationBedrockModelId

LLMPromptConfigure:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: pca-server/cfn/lib/llm.template

########################################################
# SSM Stack
########################################################
SSM:
Type: AWS::CloudFormation::Stack
DependsOn: LLMPromptConfigure
Properties:
TemplateURL: pca-ssm/cfn/ssm.template
Parameters:
Expand Down Expand Up @@ -876,6 +882,7 @@ Resources:
- ShouldDeployBedrockBoto3Layer
- !GetAtt BedrockBoto3Layer.Outputs.Boto3Layer
- ''
LLMTableName: !GetAtt LLMPromptConfigure.Outputs.LLMTableName

PCAUI:
Type: AWS::CloudFormation::Stack
Expand Down Expand Up @@ -911,6 +918,7 @@ Resources:
- ShouldDeployBedrockBoto3Layer
- !GetAtt BedrockBoto3Layer.Outputs.Boto3Layer
- ''
LLMTableName: !GetAtt LLMPromptConfigure.Outputs.LLMTableName

PcaDashboards:
Type: AWS::CloudFormation::Stack
Expand Down Expand Up @@ -1086,10 +1094,10 @@ Outputs:
Description: Lambda function arn that will generate a string of the entire transcript for custom Lambda functions to use.
Value: !GetAtt PCAServer.Outputs.FetchTranscriptArn

LLMPromptSummaryTemplateParameter:
Description: The LLM summary prompt template in SSM Parameter Store - open to customise call summary prompts.
Value: !Sub "https://${AWS::Region}.console.aws.amazon.com/systems-manager/parameters/${SSM.Outputs.LLMPromptSummaryTemplateParameter}"
LLMPromptQueryTemplateParameter:
Description: The LLM query prompt template in SSM Parameter Store - open to customise query prompts.
Value: !Sub "https://${AWS::Region}.console.aws.amazon.com/systems-manager/parameters/${SSM.Outputs.LLMPromptQueryTemplateParameter}"
LLMPromptSummaryTemplate:
Description: The LLM summary prompt template in DynamoDB Table - open to customise summary prompts.
Value: !Sub "https://${AWS::Region}.console.aws.amazon.com/dynamodbv2/home?region=${AWS::Region}#edit-item?itemMode=2&pk=LLMPromptSummaryTemplate&route=ROUTE_ITEM_EXPLORER&sk=&table=${LLMPromptConfigure.Outputs.LLMTableName}"

LLMPromptQueryTemplate:
Description: The LLM query prompt template in DynamoDB Table - open to customise query prompts.
Value: !Sub "https://${AWS::Region}.console.aws.amazon.com/dynamodbv2/home?region=${AWS::Region}#edit-item?itemMode=2&pk=LLMPromptQueryTemplate&route=ROUTE_ITEM_EXPLORER&sk=&table=${LLMPromptConfigure.Outputs.LLMTableName}"
Loading

0 comments on commit aa0dcc0

Please sign in to comment.