Develop and Troubleshoot Nuance Mix AI Powered Chat Bots locally and on Microsoft Azure.
- Overview
- About the Client
- Setup
- Functionality
- Pre-Requisites
- Quick Start 🚀
- Getting Started
- Conventions for Rich UI
- Conventions for Link Handling
- Data Access Integrations
- Themes/Branding
- Publishing to Azure
- License
This demo client offers a sample integration to Nuance Mix's Conversational AI Runtime Services, specifically
- Dialog (DLGaaS)
- Speech Recognition (ASRaaS)
- Natural Language Understanding (NLUaaS)
- Text to Speech (TTSaaS) and the
- Runtime Event Log API
The client uses the HTTP/1.1 transcoded, or WebSocket transport version of the APIs (rather than the native HTTP/2 gRPC versions).
The prime purpose of this tool is to assist in Bot development and troubleshooting, including offering controlled hosted availability using Azure StaticWebApps.
Azure Functions with Core Tools are used throughout the development and deployment lifecycle.
For more information on how to leverage the client across various environments, see networking options, and be sure to secure your functions. Monitoring is made available through Application Insights.
Disclaimer: This sample client is not intended to illustrate a production-ready implementation; it is intended to aid development, demonstration, troubleshooting, and other such use cases.
- Gatsby, React and React-Bootstrap for the Client/Frontend
- Azure Functions Python for the API
- Acts as a Proxy for Nuance Services, with Data Integrations for Mix.dialog/DLGaaS
Please see the Exchanging data section in the Mix documentation. It is crucial to understand the concepts as you design your Mix projects and leverage this client -- it acts more like a gateway.
- Bot engagements using Nuance Mix's DLGaaS HTTP/1.1 or WebSockets API (web-gRPC not HTTP/2 gRPC)
- Audio or Text Input, Audio or Text Output
- PRESENTATION LAYER: Rich UI through conventions
- DATA: Functions for development through Client Fetch, and when deployed, enables External Fetch usage
- DATA: Client Fetch handler for Location Data supplied in userData
- LOGS: Log Events, Filtering and Timeline Visualization
- CHANNEL SIMULATION: Visual VA, IVR, SmartSpeaker, SMS (audio in/out, text in/out, or any permutation)
- CHAT: Standalone mode for launching without logs and for data collection (
/chat/
path) - THEMES: Supports branding with use of logo, assets like fonts and stylesheets
- Speech Recognition using ASRaaS WebSockets API (web-gRPC not HTTP/2 gRPC)
- Natural Language Understanding using NLUaaS HTTP/1.1 API (not HTTP/2 gRPC)
- Text to Speech Synthesis using TTSaaS HTTP/1.1 API (not HTTP/2 gRPC)
- Event data using the Log Events HTTP/1.1 API
Use Data Access nodes with the client_fetch
configuration and write the integration layer locally with the intention of separate hosted Functions (through external_fetch
). These Functions would be referenced and configured within Mix.dialog and Mix.dashboard respectively.
The following illustrates a scenario where the client is deployed to Azure, and Data Access nodes have been configured to use external_fetch
within Mix.
This simplifies the client handling, deferring to the Functions themselves, and offers lifecycle controls within Mix.
client_fetch
is appropriate; this has been set up such that dlgaas.js
will invoke local ClientFetchHandlers
.
- Git
- Brew (MacOS/Linux) or Chocolatey (Windows)
- OpenSSL and Mkcert
- If using Docker..
- Docker Engine 20.10.0+
- If using a Native Host..
- Node.js 18+ (Client Toolchain)
- Python 3.9+ (Functions)
- Azure Functions Core Tools
git clone git@github.com:nuance-communications/mix-demo-client-azstaticwebapps.git
Pre-requisite for either Docker or Native Host based:
make certs-setup
Assumes running the application using Docker:
make launch
If using Native Host, run two processes:
make native-run-api-secure
make native-run-app-secure
Various opertions are described in a Makefile.
make help
CERTS: certs-(prep|setup)
*LAUNCH: launch
DOCKER: containers-(build|run|restart|stop|status|logs|clean)
NATIVE.BUILD: native-build-(app|api)
NATIVE.RUN: native-run-(app|api)-(secure|insecure)
NATIVE.CLEANUP: native-clean
DATA.ACCESS: new-data-access-endpoint
There are two main modes of operation intended: docker and native host. Launch defaults to docker.
Installs the pre-requisities for certificate creation, namely openssl
and mkcert
. A package manager is used here: brew or choco depending on your platform.
Creates certificates and stores password as needed for local use.
Primary command, sets up certificates and runs with docker compose.
Builds the containers needed to run the app. Reflects the full static web app, including the APIs.
Runs the containers for the app. Reflects the full static web app, including the APIs.
Restarts the containers of the app.
Stops the containers running.
Provides status for the containers.
Follow the logs for the running containers.
Removes all the images for the app.
Builds the client application package and brings in associated dependencies.
Builds the functions package and sets up a virtual environment.
Runs the native application securely (leveraging certs).
Runs the native functions securely (leveraging certs).
Runs the native application insecurely. Must update local.settings.json
accordingly. Not default mode.
Runs the native functions insecurely. Must update local.settings.json
accordingly. Not default mode.
Cleans all application and functions related resources.
Helper to bootstrap a data access endpoint, leveraging a template.
If you do not have mkcert
or openssl
installed:
make certs-prep
To set up the necessary certificates:
make certs-setup
make containers-run
Run the two processes.
App:
make native-run-api-secure
API:
make native-run-api-secure
make new-data-access-endpoint
See the official Azure Functions Reference - Python for more information.
If using Docker, update the values in .env
.
If running Natively, update the following environment variables to override the default values:
export oauth_server_url="https://auth.crt.nuance.com/oauth2"
export base_url_dlgaas="https://dlg.api.nuance.com/dlg/v1"
export base_url_nluaas="https://nlu.api.nuance.com/nlu/v1"
export base_url_ttsaas="https://tts.api.nuance.com/api/v1"
export base_url_logapi="https://log.api.nuance.com"
export oauth_scope="dlg nlu tts log"
When deployed, these can be set per deployment environment in the Azure Portal for the Static Web App.
See resources/local.settings.json
for the base file.
The default setting enable CORS for the Functions, and sets a worker count as desired. Be mindful of the http
vs https
distinction. By default, the application is set up to use https
.
This sample offers a stub integration with SendGrid for email capabilities.
This can be used if creating a Data Access node with the name Server_Send_Email
, which in turn will call the email-api-send
locally.
To configure this integration, provide the following environment variables:
export sendgrid_api_key="<REPLACE_ME>"
export sendgrid_from_email="<REPLACE_ME>"
export sendgrid_custom_token="<REPLACE_ME>"
Note: Override values as needed in .env
when using docker-compose
SENDGRID_TOKEN
. This is to thwart unintentional usage.
This client employs certain conventions within Mix.dialog to offer special types of rendering in the presentation layer.
To take advantage of these, navigate to:
- QA node > Node Properties
- User Input > Optional > Interactivity
Provide the following strings in the "type" field.
Special Input Types for QA Nodes:
email
phone
currency
date
To provide hints (aka placeholders), the sendData must include a variable with the type followed by 'Hint', ie. 'emailHint', 'phoneHint' and so on.
For Interactivity (aka selectables):
carousel
buttons
colorpicker
(description should contain the HEX value)
Note: when using Rich Text markup in messages, if elements contains classes from Bootstrap they will be rendered accordingly in this client. See chat.js
to see the allowed DOM elements and attributes.
Images can be rendered when using selectables and the carousel
type. Edit STUB_SELECTABLE_IMAGES
in shared.js
to experiment with resources.
This client employes certain conventions within Mix.dialog to offer special types of event handling in the presentation layer.
To take advantage of these, simply mark up your Rich Text messages appropriately with the data attributes, depending on the intended interaction.
To trigger an intent or entity selection, leverage the following attributes:
data-mix-action="selectable"
data-mix-selectable-id
- can beINTENT
or the entity namedata-mix-selectable-value
<a href="#"
data-mix-action="selectable"
data-mix-selectable-id="INTENT"
data-mix-selectable-value="iBuyPhone">
Buy a Phone
</a>
<a href="#"
data-mix-action="selectable"
data-mix-selectable-id="ePhoneCapacity"
data-mix-selectable-value="128GB">
128GB
</a>
To trigger a simulated user input, leverage the following attributes:
data-mix-action="input"
data-mix-input
<a href="#"
data-mix-action="input"
data-mix-input="$500">
$500
</a>
This client offers developers the ability to use the client_fetch
mode of the Data Accss node in Mix.dialog and integrate with an Azure Function.
This pattern would apply in a gateway-style integration, however the intention of this set up, is to eventually have the integrations use external_fetch
pointing to deployed Functions.
Create a new Function specifying the endpoint URL and the data access node name to use if client-driven:
./scripts/create-da-handler.sh "weather-api-city-conditions" "Server_Weather_CityConditions"
Navigate to api/weather-api-city-conditions/__init__.py
and start integrating.
By default, POST requests are expected with the body containing the sendData payload. Update api/weather-api-city-conditions/function.json
if other methods are desired.
ExternalFetchHandlers
in dlgaas.js. Create a function with the Data Access node's name, pointing to the newly defined endpoint.
There are cases where the use of Mix.dialog Data Access node's client fetch is intended for the end client (vs. the gateway pattern employed here).
Add a handler to ClientFetchHandlers
in dlgaas.js named with the Data Access node's name.
At this time, geolocation (lat/lng) is supported through the HTML5 API, when the client is running securely.
To request the client to provide it's location, set the Data Access node's ID to Client_Location_LatLng
, and this will return a location
object.
The data integration layer in this client is handled through Functions, and complimented by sample integrations in api/providers.py
.
Each data access access is represented by it's own Function, in preparation for use when deployed and hosted.
This client illustrates integrations with: Yahoo Finance, AccuWeather, SendGrid and a Mock Phone Store.
This API does not require authentication, in this case: the requests are made with no further checks.
POST /api/finance-api-asset-price
Third party service authentication is required and done through passing the token in the payload of the request. In this case: apiKeyAccuweather
is used with the service and must be saved in the session (stored in a variable).
POST /api/weather-api-city-search
POST /api/weather-api-city-conditions
Third party service authentication is configured as part of the Function, but a SENDGRID_TOKEN
is required to execute calls from known sources.
In this case: the token must been assigned in Mix at the PROJECT level.
POST /api/email-api-send
This integration illustrates the use of Dynamic Wordsets, consistently filtering options based on the user's selection, derived from static data.
POST /api/store-api-filter-phone-wordset
POST /api/store-api-request-purchase
POST /api/store-api-purchase
The chat skin can be branded.
A typical brand definition includes:
- Logo (image)
- Fonts
- Stylesheet
Assets are located in /app/src/static/brands/
- Fonts can be stored here as well
The query parameter brand
can be used to preload this configuration. Assign the value to the name of the subfolder.
Here is a sample CSS, assuming that the font files are also located in the directory.
/* Brand Assets */
@font-face {
font-family: 'Font-Bold';
src: url('./Font-Bold.woff2') format('woff');
font-weight: bold;
font-style: normal;
}
@font-face {
font-family: 'Font';
src: url('./Font-Regular.woff2') format('woff');
font-weight: normal;
font-style: normal;
}
:root {
/* MyBrand*/
--skin-logo: url("/brands/mybrand/logo.svg");
--skin-logo-header: url("/brands/mybrand/logo-header.svg");
--skin-logo-header-width: 30px;
--skin-logo-header-height: 30px;
--skin-logo-main-width: 25%;
--skin-logo-main-height: auto;
--skin-color: #336699;
--skin-font: 'Font', 'Helvetica';
--skin-font-size: 14px;
--skin-chat-bg-color: #fff;
--skin-header-text-color: #fff;
--skin-header-bg-color: #27251F;
--skin-footer-bg-color: #fff;
--skin-message-text-color: #27251F;
}
To deploy this client, follow the Azure StaticWebApps deployment guide to publish. Update app/static/staticwebapp.config.json
as needed.
Essentially:
- Create a GitHub Repo
- Create an Azure StaticWebApp pointing to the GitHub Repo (use: gatsby, point to
app/
, andapi/
) - Configure accordingly in Azure
- Create a Application Insights resource and link to StaticWebApp (
APPINSIGHTS_INSTRUMENTATIONKEY
) - Add environment variables (
sengrid_api_key
,sendgrid_from_email
)
- Create a Application Insights resource and link to StaticWebApp (
- update content-security-policy for client -
staticwebapp.config.json
- update client host (ASR_SERVICE_URL, DLG_SERVICE_URL) -
shared.js
- update environment variables for functions -
.env
This source code is licensed under the Apache-2.0 license found in the LICENSE.md file in the root directory of this source tree.