Learning/teaching analytics - digging deeper

Following on from my last post, why isn’t the news on my Research skills required by PhD students cloudscape as good as it looks at first sight?

Here are the viewing figures. 856 page views of my Cloudscape. Lovely, regular peaks in the viewing figures every Monday. What happens every Monday? I go and check out how much people are using my clouds – that’s what. So at least 400 of those page views are me collecting data. That, in turn, affects my bounce rate – which is 0% on all those data collecting days. In this case, the site’s own analytics are more helpful – the cloudscape has had 323 views (it doesn’t count the logged-in author).

Something else I get from the viewing figures is a spike on 28 June. What happened there? It seems that someone from The Open University arrived three times from Google with related research queries and spent 2 hours 50 minutes on the site. That’s great, they were obviously finding it useful – but that cuts everyone else’s average time on the site considerably.

More depressingly, here are 20 of the 32 searches that brought people to this cloud – where they then spent no time and bounced away from the site again. Some of them are probably in the wrong place and need to leave to look for IT skills or UK skills. The majority, though, should be staying – I’m losing at least 12% of visitors who should have found this a useful resource.

What do visitors see in that fraction of a second they spend on my Cloudscape before bouncing away? Possibly the first line of text ‘Research Skills required by PhD students, as defined by the UK Research Councils’. And, yes, half of those bouncing visitors are outside the UK – although only an eighth of views originate outside the UK. Time for a rewrite – these are generic research skills, relevant worldwide.

More broadly, what have I learned about implementing analytics on a learning site?

• Search for a way to set aside visits by the site owner and the content author – it looks as if this needs to be done on the site itself, rather than through Google Analytics.
• Set aside outliers for separate consideration.
• Give content authors access to the keywords that are working for that content, and the keywords that are not.
• In fact, if the aim is to improve the learning/teaching effectiveness of resources, it would be good if authors could access short analytics reports without having to filter requests through the site owner.

Google Analytics and learning/teaching potential

What else can Google Analytics tell me about the potential of my resources on Cloudworks as teaching materials? Well, I’d expect my Cloudscapes, which are essentially index pages, to lead people to the associated clouds (content pages).

I’ll focus here on Research Skills required by PhD Students. I created this back in February – and it involved a fairly straight transfer of Web materials to Web 2.0. Pretty much all of this material was already available on the Open University Intranet, I moved it over to Cloudworks, sorted out the dead links, added some new links and it’s now open for people to add to, comment on and discuss.

The original, Intranet, version of the web page didn’t work too well.The page stats over a 14-month period showed that people were following the links based on where they fell on the page. Links at the top left did very well, followed by the first link after each heading. Links on the bottom right were only followed once or twice a month. This looks like a classic browsing pattern – people arrive, click around to see what is on offer, but don’t make any serious use of the page or its linked resources.

When the material was added to Cloudworks, the pattern of usage became more even. The previously dominant A1: Recognising research problems was replaced by B2: Compliance with ethical requirements which, more recently, has been overtaken by B6: Justifying research methods. I like to think that that reflects seasonal variation in the concerns of PhD students at the OU (the target audience for the clouds and for the original site) but that’s currently just a guess.

If the Cloudscape is doing its job as a learning/teaching resource I’d expect to see a low level of bounces from that page, a high level of people moving through to the linked clouds, people moving through to the pages related to their original search terms, and those people who do move on to linked clouds spending at least a few minutes on the site.

I seem to have a fair-sized readership – 848 page views since February. Two-thirds of those arriving on the page are bouncing directly off the site but those arrive from elsewhere on the site don't tend to leave when they get to my cloudscape. I’m glad to see that they are following the links, and that those who stay on the site move to linked pages.

The unique visitors who landed directly on the page spent nearly 8 hours there in total, which averages out at around four minutes each. What’s more, people with relevant, detailed searches seem to be spending time on the site.

According to my original hypothesis, Google Analytics is showing this to be a page that looks likely to be supporting teaching and learning. People want to find out more about skills required for research, this page provides links to relevant material, they follow these links and spend some time looking at them.

All good news? Not entirely. In my next post I’ll look at some of the things that are going wrong, and the analytics that help to identify these.

Learning analytics / teaching analytics

My current focus is on learning analytics. How can we tell from site analytics whether someone is learning, engaging in activities that have been shown to support learning, or exhibiting behaviours that are associated with learning? And, rather than develop Wheel 2.0, I'm looking at available analytics and whether they can be harnessed to do this. Hence the current focus on Google Analytics.

A problem is that identifying learning means that I need to be able to associate activity with specific individuals or groups of individuals. As I discussed below, I can do that to some limited extent with Google Analytics but it's not really set up for me to do that and, more to the point, focusing on individuals in this way feels intrusive and, I think, would need informed consent from those concerned if I pursued it to any extent.

So Google Analytics can give me some pointers as to whether learning (activities/behaviours) are taking place, but to link this to individual learners or groups of learners would involve another set of analytics, and those learners would have to be aware of what was taking place.

Coming at this from another perspective - how about teaching analytics? I'm not thinking here of time/motion studies about level of activity and output - I'm more interested in helping teachers / educators judge the value their output has for others. Google Analytics are potentially more helpful here, because the authors of online resources, and the creators of online discussions have publicly identified themselves, and so resources can be tied to individuals.

So, if I examine an online resource, I could look at how many visits it receives, how long those visits last and whether people move on to use linked resources. I can examine the effects of sharing a link to that resource on Twitter. How effective are my Twitter links compared with those of people widely known for their expertise in the field?

More broadly, if I look at an educational resource (I'm currently looking at Cloudworks) I can begin to identify the most effective resources and behaviours. In my next post, I'll describe some preliminary work I have done on this.

Academic Analytics: A New Tool for a New Era

I've followed a link in OUseful Info to Educause, where I'm looking at
Academic Analytics: A New Tool for a New Era
John P. Campbell, Peter B. DeBlois, and Diana G. Oblinger
EDUCAUSE Review, vol. 42, no. 4 (July/August 2007): 40–57

This identifies several uses for analytics in education:

• To manage enrolment, using standardised exam scores, high school coursework, and other information to determine which applicants will be admitted.
• To inform fund-raising. By building a data warehouse containing information about alumni and friends, institutions can use predictive models to identify those donors who are most likely to give.
• To aid retention by identifying students most at risk of dropping out
• To assess which proactive interventions have the best influence on academic success and retention. 
• To predict student success within a course

They also highlight three characteristics of successful academic analytics-based projects (link to referenced PDF):

1. Leaders who are committed to evidence-based decision-making
2. Administrative staff who are skilled at data analysis
3. A flexible technology platform that is available to collect, mine, and analyze data

Within the OU, the IET Student Statistics department takes a leading role in analytics projects like these. Other departments, such as Communications, also make use of analytics data.

My focus is on learning analytics - how we can use online analytics to identify learning, conditions that support learning and behaviours that support learning.