How to get notified if your ChatGPT bot could not answer a question.
Once you have created your knowledge store and deployed a ChatGPT powered bot using ThinkAutomation, there will sometimes be cases where the bot is unable to answer a question. This could be due to missing knowledge, or specific information missing or not written correctly in the knowledge store.
You can setup your Automation so that you can get notified whenever the bot is unable to help. You can then make any updates to the knowledge store.
This is done by telling ChatGPT to respond in a certain way if it is unable to help. In the Automation we then check for that response and send an email.
For example, below is a basic ChatGPT powered bot automation:
ChatMessage = HTML Entity Decode ( %Msg_Body% )ChatMessage = Mask Profanities ( %ChatMessage% )ChatMessage = Trim ( %ChatMessage% )
If %ChatMessage% Equal To [webchatstart] Then // Respond to chat starting Return Welcome %Msg_FromName%! I can answer questions about Widgets.End If
// Add default contextChatGPT Add Context "You are an enthusiastic representative working at Widgets Company. Given the provided sections from our documentation, answer the user's question using only that information, outputted in markdown format. If you are unsure and the answer is not explicitly written in the documentation, say 'Sorry, I don't know how to help with that.'" (Required) To Conversation %Msg_ConversationId%
// Add context from the knowledge storeChatGPT Add Context From Knowledge Store Widgets Search %ChatMessage% Return 4 Most Relevant To Conversation %Msg_ConversationId%
// Send to ChatGPT to get the responseChatGPTResponse =ChatGPTResponse = ChatGPT Say "Question: """ %ChatMessage% """ Answer: " Using gpt-3.5-turbo-16k Conversation %Msg_ConversationId%
// Check if response is 'Sorry, I don't know how to help with that.'If %ChatGPTResponse% Starts With "Sorry, I don't know how to help with that." Then // Let support team know Send Email To support@widgets.com "Bot could not help: %Msg_From%"End If
// Return response to the chatReturn %ChatGPTResponse%In the default context include "If you are unsure and the answer is not explicitly written in the documentation, say 'Sorry, I don't know how to help with that.'" - this tells ChatGPT to respond with specific text if it is unable to answer.
After the ChatGPT action we check if the response is equal to "Sorry, I don't know how to help with that." - if it is we send an email, this contains the %ChatMessage% variable so we can see what the user asked.
The "Sorry, I don't know" message can be anything - provided we explicitly tell ChatGPT to respond with some text if it is unable to help - and then check for that text after asking ChatGPT for a response.