How to Audit a Marketing Technology - An example of a Chatbot Audit

It’s that time of the year again, or should I say that time of the quarter? It’s time to inspect your marketing activities, analyse what worked well for you and what didn’t. But there’s an extra something you need to monitor this time- your new marketing technology. But since it’s your first, you’re at a standoff. How do you analyse a tool that you’ve never used before? 

Digital marketing solutions are all the rage but reviewing their performance is crucial to check for gaps and overlaps. According to Gartner, 29% of CMOs spent their budget on marketing technology, but only 33% of organisations saw an actual benefit from it. It’s either because CMOs overestimate the technological capabilities of the tool, or they fail to evaluate their KPIs. Regardless of the reason, assessing them is essential to get the expected ROI. 

Your tech audit will differ according to the type of tool you integrate and your goals. To keep it simple for you, we’ve given a comprehensive example of a chatbot audit. 

Example: Marketing Audit of a chatbot

Pre-integration Audit Process

Step 1:Auditing the Goals - What are your KPIs?

Setting your priorities helps you identify what you need to measure in the first place. Any audit you conduct starts with defining the goals. In this case, define your reasoning backed by strong pain points. 59% of businesses use chatbots to improve customer support, and 36% use them to generate leads. Although bots are used for various purposes, these two are the primary reasons businesses turn towards chatbots. What are you trying to achieve using the technology? Do you want to increase conversion on websites? Do you want to automate lead qualification? Do you want to scale your customer support? List down the primary areas you want to measure and then decide your Key Performance Indicators. 

As for chatbots, KPIs can range from:

• Conversion Rate: What is your website traffic, and how many anonymous users are you converting into leads? The conversion rate will show you how much your bot is actually helping your website engagement activities. 

• The number of Leads Generated: If you need your bot to generate business value, you first need to know how many leads it generated. Have a CRM integration with the bot to record bot leads. 

• Retention rate: You want to know if your users are returning and having more than one interaction with the bot. This helps in understanding customer satisfaction and knowing if the flow is prepared in the right direction. 

• Total Number of Conversations: Tracking the number of conversations can help you know the engagement level on your website and other platforms. 

• The number of support queries handled: You want your bot to handle chat conversations from start to finish and address customer problems. If the questions are complex, also measure the human takeover rate. It doesn’t mean your bot failed. It means it understood its technical limitations and moved aside for a human to take over. 

Your martech can have many use-cases, but it is your job to identify which use-case you want to adopt it for. You cannot set the goal of generating leads and measure the number of support queries handled. 

Step 2:Defining the Persona: Audience audit 

To deliver an authentic user experience, you need to know who you’re targeting. Define your audience demographics and find out who they are and what they want from you. This is even more important when you’re talking about virtual assistants, as you need to know the openness of customers to talk to machines and not humans. Various studies have shown that chatbots are very popular amongst the younger generation due to their willingness to help themselves without speaking to a human. 75% of customers above the age of 45 have never interacted with a bot. 

Age is just one of the parameters that help you define your audience persona. You also need to determine where are your visitors from and what language do they speak. Conversational marketing tools like bots have the potential to tap vernaculars and dialects to target a particular region or country-specific. For example, a leading Indian hospital wanted to enhance website engagement and generate online appointments. But since India is a linguistically rich nation, it didn’t want to limit its bot language to English. It also deployed its bot in Hindi and Gujarati so more people would be comfortable with using it.

Source: WotNot

Including other languages helped the hospital give a personalised experience to the visitors and have more than 88k+ conversations on the bot!

Step 3: Competitor Audit 

Just like how you conduct a competitor analysis for any marketing campaign, you need to do the same while deploying a marketing technology. Take your top 5 competitors and check if they have a bot on their platforms. See the vendors they’ve chosen, identify what the bot does, how their conversation flow is, what tone it uses, and where it has deployed the bot. This will help you know your market better and how your marketing technology fits your industry. It will help you prepare your bot strategy and give a direction in deciding and reviewing the KPIs. 

Step 4: Branding Audit 

After analysing everything else, you need to ask yourself the most crucial question- How will my bot be different from those of the competitors? Your chatbot is a mirror to your brand. You need to monitor if the bot is rightly representing your brand and urging a brand recall. Is the flow using your content strategy? Does the bot have your name and logo? Does the font match your other content on-site? Some of these things are very minute, but they can have a long-term effect on brand recall. Some of the top considerations you need to consider while branding your bot is:

Giving it a personality: Think about your bot as an employee. You would train your employees to aid the customer and speak to them within the company guidelines. Be it offering product recommendations, solving customer problems, or providing quick answers; your bot should be your de-facto brand ambassador. You need to determine what is the overall sentiment associated with the brand. Add that in your chatbot, and ultimately it will decide your brand tone and emotional quotient. 

Embedding Branding Elements: Adding your branding elements in your bot can act as a significant differentiating factor for your brand. Name your bot that resonates with your brand name, retain your logo, colour, and font in the bot. This will also help you provide a consistent customer experience across platforms.

Source: WotNot

In the following example, notice that the bot colour is aligned with the brand tone as well and it also named its bot “Accuris Buddy”. It ensured that the bot is rightly representing the brand that encourages positive brand reinforcement and goodwill. 

Post-integration Audit process

Step 5: Hygiene Audit

While your performance audit covers attributes that require monitoring every once in a while, your hygiene audit covers attributes that require regular inspection for your technology to perform to its fullest potential. As for a chatbot, you need to look at the aspects that would impact its performance. 

Conversational Flow: I’m again directing this to the KPI. If your goal is to generate leads, your bot needs to get the lead information in the conversation. This type of flow would drastically differ from a bot whose main motive is to offer support to site visitors. Even in lead generation, you need to experiment. That’s the only way to know which flow is the most appealing to the site visitors. Are you asking for lead information at the beginning of the conversation itself, or are you doing it while having the conversation? You can only know what’s best for you if you keep experimenting with the flow. For example, a leading Indian university knew it had great website traffic, but its primary motive was to generate leads from the bot.

To do so, it asked for lead information at the beginning itself before taking the conversation forward. Within a few months, it had 1100+ leads and a conversion rate of 3.31%. Would they have got the same response with a different conversation flow where they ask the lead information at the end? Less likely, since visitors wouldn’t provide their information if they already have a browsing experience prior. The flow would be very different if the university were deploying bot for other reasons, such as assisting current students or providing university FAQs. To know more about the conversational flow in a bot, I’d suggest you check this out: How to design an amazing chatbot.

Bot Script: While this needs to be a part of inspecting your conversational flow, I kept it a separate point to empathise the importance of tone and grammar and your script. It may seem like a trivial detail, but it's too important to miss it. Even the slightest grammatical error can damage a user’s experience. Hence, you need to look for errors in the script and rectify them. Another crucial point is to have an empathetic tone. According to a recent study, it was concluded that people prefer talking to virtual assistants that are happy. Regardless of the approach you take, entirely sounding robotic won’t work for you. You can have a formal conversation and still sound humane. 

Channel: Since bots can be deployed on multiple channels; you need to monitor if the bot is deployed on the right medium. Where do you get the highest number of visitors? Is it your website? Is it your Facebook page? What medium do your prospects use to communicate on a regular basis? For example, an Indian chemical company deployed a bot on WhatsApp to schedule safe appointments during Covid-19. It realised that with more than 340 million active users of WhatsApp in the country, it was the best bet to converse with visitors. As a result, it scheduled more than 200 appointments and had more than 800 conversations on the bot! 

Had the company deployed a bot elsewhere, it wouldn’t have fetched the results it did. 

Analytics: Chatbot analytics gives you a window to monitor your chatbot activity. You need to inspect the increase/decrease in the number of visitors, conversations, average session length, total messages, etc. Check if your tool has a dashboard that summarises your activity instead of calculating every single KPI. For example, Maruti Techlabs built a bot on WotNot, and its dashboard gave a quick snapshot of the bot activity within a given time frame. While choosing your vendor, you need to ensure that you have a dashboard that provides you with an overview of Martech’s performance. Since you need to analyse the activities regularly, a dashboard will save time and shed light on areas that need work.

Step 6: Performance Audit 

Of course, the most obvious one would be to audit the bot’s performance. This is the process that will show you if your tech is going according to your plan. To audit this part, you need a checklist. List down all the attributes that you want to measure and score them. In the case of a chatbot, list down your goals and rate them from 1-10 in terms of their fulfilment, with one being the lowest and ten highest. Rate the importance of the attributes, so you know how crucial they are to your goals. Find out the chatbot checklist below. For the sake of the example, I have given ratings, included their importance level and calculated the total.

More than 50: Excellent

30-50: Average

Below 30: Poor 

You can also add other parameters and KPIs that you want to measure and give them scores and determine their importance level. Your final score will show you whether your technology is performing exceptionally well, average, or below expectations. 

Just how you had a performance review, you can have a product review too to know its user-friendliness.

You need to create a framework for assigning attributes to your performance areas. You can involve other professionals in this task to get a varied opinion. Let everyone rate their experience of incorporating the tool and get the average of the total. 

Step 7: Prepare a Future course of Action.

All your auditing activities need to bear fruit. Based on your conclusions, you need to develop a future course of action. If the bot is not fulfilling its purpose, think about the changes you can make. You can change the conversational flow, or change your channel, have new integrations, etc. If the bot is bringing results, look at what you did right and improve your current strategy. 

Similarly, check if your bot is giving you a good ROI and providing business value. This isn’t something you’ll see in just weeks of deployment. It might take a few months for your bot to actually bring results. So you need to have patience. Your Martech audit requires significant time and resources from your end, so you need to make decisions based on your auditing process and leverage the scope of improvements.