Using Analytics to Track Changes at Your Website

We recently did a live content update for DNNMasters, a DotNetNuke development company that has created some of the must-have modules for DNN websites. DNNMasters also does custom modules and development for DNN users.

DNNMasters had been seeing a slight but steady fall in traffic for some time. They have a lively forum and popular products, but they had very little content on their homepage, and much of what they had written was extremely technical. We added a lot of content and made what they had more accessible.

They were also using terms for keywords which, while common among DNN developers, weren’t popular searches. In fact, the name of their company had more search volume than some of the terms they were focusing on. We included the popular keywords they were missing in the new content.

Finally, we organized the content a bit better, made it more consistent across pages, and suggested a few visual changes to communicate more clearly.

DotNetNuke is a Microsoft content management system and the guys at DNNMasters of course take care of their own website, so I was able to go in and make content changes and give them suggestions for some changes in information architecture, and all the changes were made over a period of three days.

A couple of days later the Masters saw some things at Analytics that showed they needed some further changes in their meta tags. As we’ve seen before, there are some things you can see in Analytics immediately after making changes in a site.

It has been a week since we did the work, so I had a look at Analytics to check the results. At first glance, things look good. Comparing the week since completion with the week before we began the work, we see some changes:

• Traffic is up by 12.14%.
• Absolute visitors are up 25.94%.
• Page views are up 33%.
• The bounce rate is down slightly, by 1.74%.

We’ve also seen significant increases in traffic reaching the site by some of the specific keywords we wanted to target: “DNN modules” is up by 800%, “DotNetNuke custom module” doubled, and we saw far more specific product names showing up as keywords.

Can we say, “Our work here is done!” and take off? Not yet. There are quite a few factors that keep this from being proof of success:

• While we filtered out the workers from the analytics, there’s nothing to prove that the DNNMasters didn’t go out and tell their friends about the new update at their website. We usually wait two weeks, by which time you can generally isolate any social peaks of that type and leave them out of your calculations, but one week isn’t long enough to be sure of accomplishing that. If you’re looking at your analytics after site changes and see a spike in traffic on the day of launch, you should set the dates in the dashboard to exclude that day.
• We compared the most recent week with a random control week, but if you’re looking at only a couple of weeks you can’t be certain that there aren’t other variables besides the site changes. The one we know about — we filtered workers out — would tend to cause the numbers to be lower, not higher. We were able to use geographic data, too, to make certain that we weren’t seeing abnormalities. However, this is a company with global traffic which may be affected by factors we didn’t consider.
• Google Intelligence, the page at Google Analytics where Google tells you when it feels surprised, showed only one alert: people generally were spending more time at the site since the changes than before the changes. The increases in traffic weren’t enough to surprise Google.  By contrast, Trout Fishing in America’s new website has had 31 alerts in the past four days, providing stronger evidence of changes in their traffic patterns.

While we’re happy to see some evidence of a positive trend for the DNNMasters, we can’t say definitively that our changes have turned things around. How, then, can you tell whether your changes are really making a difference or not?

• Compare in more ways and across a broader spectrum. For example, we can see that our post-change week’s overall traffic is up by 65.6% over last year at the same time. We can see that direct traffic hasn’t been as high since May. We can see that search traffic is up only slightly, and is still within the normal range for the past several months — no surprise, since it often takes search engines a while to notice changes. Still, more different comparisons give us a broader picture.
• Look for surprises. Google’s Intelligence is a help with this, but you may see more surprises because the algorithm is surprised by changes while you, if you have made changes to your website, are expecting changes. If you don’t see the changes you expect, or if you see different changes from the ones you expect, dig deeper.
• Wait. While we like to watch closely in order to respond swiftly to things that show up early, we don’t do client reports till two weeks have passed, and you can tell much more after two months. In the early stages, watch for overall improvements: increasing traffic, falling bounce rate, better focus in the keywords. Watch also for the specific goals you had for your changes. In this example, we’re working for increased traffic; in other cases, we’re working to change the source of traffic, to increase conversions, or to change the way visitors use the site, so we’d look at other metrics. Some of those change more quickly and some respond more slowly to changes.

In our example, we’ve seen only positives thus far. If we had seen negative trends, we’d have reevaluated the changes we made. As we continue to watch, we’ll expect to see the same trends show up more strongly. If they don’t, we’ll need to reevaluate and tweak the changes we made.

I used to work in retail. We’d make changes and wait for sales figures or comments from customers to determine whether to continue in a new direction or to shift to another approach or perhaps even backpedal. Analytics allows us to watch much more closely and respond more quickly than if we had to rely entirely on sales figures. It also allows us to operate with more accuracy than if we relied on feelings, whether our own or those of the small set of customers who spoke up.