The question.
In the middle of last year we changed the terminology on one of the main navigation sections on the website.  The Help and Support  section of the website contains a large amount of the content of the website.  The reasons behind the change were several.  The tab was getting quite low levels of usage and informal feedback seemed to be showing that users were a little confused about the purpose of the section.  So we changed it to Help which benchmarking against other similar sites seemed to be the most common term that was used.  Ideally we would have A/B tested the two versions, but we tend to stay away from having different versions as they can cause their own support headaches.   So our question is – What difference has changing the terminology made to usage of this section?

Analytics approach
We settled on using four sets of data from Google Analytics and looked at the last six months of 2010 compared with the equivalent period of 2009.  The four pieces of data we decided to use were:

  • the percentage of clicks on the Help and Support/Help tab on the home page – using the beta In-Page Analytics tool against the home page; 
  • the percentage of total site page views represented by the page views of the Help/Help and Support home page – by comparing the site page views with a filtered version of Top Content to look at the Help home page;
  • the page views of the whole Help/Help and Support section as a percentage of the whole site page views – using a similar filter that includes all the help content; and finally
  • the percentage of users of the help home page that come from the home page – using the Navigation Summary in the Content section

Looking at thHelp and Support graphose four pieces of data gave us some results that showed that across all those measures use was down by small amounts.    The percentage change varied a bit between each of the measures but there was a distinct reduction in the users accessing that section. 

Oddly there doesn’t seem to be any evidence that people are finding it harder to find the help they need – we don’t seem to have more telephone enquiries or people saying they can’t find the help they need on the website.  So it’s some more evidence to fit into our redevelopment of the website.

One of the problems with the data is that we can’t be sure that there has been a change in behaviour that would have happened anyway and isn’t to do with the website terminology change.  On reflection we should have A/B tested the change so we would know that the data was being collected at the same time.   And we need to think some more about how to apply what analytics is telling us.

Advertisements