Learning/teaching analytics - digging deeper

Following on from my last post, why isn’t the news on my Research skills required by PhD students cloudscape as good as it looks at first sight?

Here are the viewing figures. 856 page views of my Cloudscape. Lovely, regular peaks in the viewing figures every Monday. What happens every Monday? I go and check out how much people are using my clouds – that’s what. So at least 400 of those page views are me collecting data. That, in turn, affects my bounce rate – which is 0% on all those data collecting days. In this case, the site’s own analytics are more helpful – the cloudscape has had 323 views (it doesn’t count the logged-in author).

Something else I get from the viewing figures is a spike on 28 June. What happened there? It seems that someone from The Open University arrived three times from Google with related research queries and spent 2 hours 50 minutes on the site. That’s great, they were obviously finding it useful – but that cuts everyone else’s average time on the site considerably.

More depressingly, here are 20 of the 32 searches that brought people to this cloud – where they then spent no time and bounced away from the site again. Some of them are probably in the wrong place and need to leave to look for IT skills or UK skills. The majority, though, should be staying – I’m losing at least 12% of visitors who should have found this a useful resource.

What do visitors see in that fraction of a second they spend on my Cloudscape before bouncing away? Possibly the first line of text ‘Research Skills required by PhD students, as defined by the UK Research Councils’. And, yes, half of those bouncing visitors are outside the UK – although only an eighth of views originate outside the UK. Time for a rewrite – these are generic research skills, relevant worldwide.

More broadly, what have I learned about implementing analytics on a learning site?

• Search for a way to set aside visits by the site owner and the content author – it looks as if this needs to be done on the site itself, rather than through Google Analytics.
• Set aside outliers for separate consideration.
• Give content authors access to the keywords that are working for that content, and the keywords that are not.
• In fact, if the aim is to improve the learning/teaching effectiveness of resources, it would be good if authors could access short analytics reports without having to filter requests through the site owner.

Google Analytics and learning/teaching potential

What else can Google Analytics tell me about the potential of my resources on Cloudworks as teaching materials? Well, I’d expect my Cloudscapes, which are essentially index pages, to lead people to the associated clouds (content pages).

I’ll focus here on Research Skills required by PhD Students. I created this back in February – and it involved a fairly straight transfer of Web materials to Web 2.0. Pretty much all of this material was already available on the Open University Intranet, I moved it over to Cloudworks, sorted out the dead links, added some new links and it’s now open for people to add to, comment on and discuss.

The original, Intranet, version of the web page didn’t work too well.The page stats over a 14-month period showed that people were following the links based on where they fell on the page. Links at the top left did very well, followed by the first link after each heading. Links on the bottom right were only followed once or twice a month. This looks like a classic browsing pattern – people arrive, click around to see what is on offer, but don’t make any serious use of the page or its linked resources.

When the material was added to Cloudworks, the pattern of usage became more even. The previously dominant A1: Recognising research problems was replaced by B2: Compliance with ethical requirements which, more recently, has been overtaken by B6: Justifying research methods. I like to think that that reflects seasonal variation in the concerns of PhD students at the OU (the target audience for the clouds and for the original site) but that’s currently just a guess.

If the Cloudscape is doing its job as a learning/teaching resource I’d expect to see a low level of bounces from that page, a high level of people moving through to the linked clouds, people moving through to the pages related to their original search terms, and those people who do move on to linked clouds spending at least a few minutes on the site.

I seem to have a fair-sized readership – 848 page views since February. Two-thirds of those arriving on the page are bouncing directly off the site but those arrive from elsewhere on the site don't tend to leave when they get to my cloudscape. I’m glad to see that they are following the links, and that those who stay on the site move to linked pages.

The unique visitors who landed directly on the page spent nearly 8 hours there in total, which averages out at around four minutes each. What’s more, people with relevant, detailed searches seem to be spending time on the site.

According to my original hypothesis, Google Analytics is showing this to be a page that looks likely to be supporting teaching and learning. People want to find out more about skills required for research, this page provides links to relevant material, they follow these links and spend some time looking at them.

All good news? Not entirely. In my next post I’ll look at some of the things that are going wrong, and the analytics that help to identify these.

Using analytics to improve practice

I’m looking at a Cloudworks page I set up earlier this year, when I was creating resources for Open University research students in connection with a postgrad conference here at the university. Each of the presentations at the conference had its own cloud, and they have all received a fairly steady amount of visits each week since the conference. The most popular cloud, and the one I’m looking at here, is Skills Audit (currently with 199 views, according to the Cloudscape figures).

Students at the OU have to complete a Skills Audit at the end of the first year of their PhD – considering their progress in relation to a set of research skills identified and agreed by the UK research councils. Having done it myself, I’d say it was a valuable exercise, but I remember resenting having this chore imposed upon me.

Before I looked at Google Analytics, I was fairly pleased with this cloud and its performance. It’s aimed at first-year PhD students at the OU and it’s picking up around ten views a week at the moment. It links to complete lists of research skills and resources for developing and assessing them, it links to the actual skills adit that OU students have to fill in, and it links to the rest of the postgrad conference, and it contains a liveblog of a detailed conference session on the Skills Audit, as well as a biography of the speaker. Not only that, but the SocialLearn gadgets on the page point visitors to other related resources. All the links and info you need in one handy cloud.

I’m not so happy, though, when I get to the analytics. Visitors are spending an average of 43 seconds on the page, but only one or two have spent much longer there. More seriously, I have an 88.6% bounce rate, so visitors aren’t using this as a gateway to other resources on Cloudworks. This gets even worse (90.65%) when I look at the key words people used to get to the page.

Let’s take a closer look at a representative sample. That long line of zeroes represents time spent on the page, the 100%s represent the bounce rate and the exit rate. So people are arriving who should be looking for exactly this resource and, in less than a second, are consistently deciding that what I have provided is no good for them.

And, when I spend half a second on the cloud, I can see exactly why. The cloud begins with a long biography of the seminar presenter – that’s not what people are looking for. I know why it’s at the top – it was the only content I had available before the conference took place – but it shouldn’t be there now.

So the analytics have helped me to improve my own practice – now I need to think through how I can extend this finding in order to have a wider impact.