Google Analytics and learning/teaching potential

What else can Google Analytics tell me about the potential of my resources on Cloudworks as teaching materials? Well, I’d expect my Cloudscapes, which are essentially index pages, to lead people to the associated clouds (content pages).

I’ll focus here on Research Skills required by PhD Students. I created this back in February – and it involved a fairly straight transfer of Web materials to Web 2.0. Pretty much all of this material was already available on the Open University Intranet, I moved it over to Cloudworks, sorted out the dead links, added some new links and it’s now open for people to add to, comment on and discuss.

The original, Intranet, version of the web page didn’t work too well.The page stats over a 14-month period showed that people were following the links based on where they fell on the page. Links at the top left did very well, followed by the first link after each heading. Links on the bottom right were only followed once or twice a month. This looks like a classic browsing pattern – people arrive, click around to see what is on offer, but don’t make any serious use of the page or its linked resources.

When the material was added to Cloudworks, the pattern of usage became more even. The previously dominant A1: Recognising research problems was replaced by B2: Compliance with ethical requirements which, more recently, has been overtaken by B6: Justifying research methods. I like to think that that reflects seasonal variation in the concerns of PhD students at the OU (the target audience for the clouds and for the original site) but that’s currently just a guess.

If the Cloudscape is doing its job as a learning/teaching resource I’d expect to see a low level of bounces from that page, a high level of people moving through to the linked clouds, people moving through to the pages related to their original search terms, and those people who do move on to linked clouds spending at least a few minutes on the site.

I seem to have a fair-sized readership – 848 page views since February. Two-thirds of those arriving on the page are bouncing directly off the site but those arrive from elsewhere on the site don't tend to leave when they get to my cloudscape. I’m glad to see that they are following the links, and that those who stay on the site move to linked pages.

The unique visitors who landed directly on the page spent nearly 8 hours there in total, which averages out at around four minutes each. What’s more, people with relevant, detailed searches seem to be spending time on the site.

According to my original hypothesis, Google Analytics is showing this to be a page that looks likely to be supporting teaching and learning. People want to find out more about skills required for research, this page provides links to relevant material, they follow these links and spend some time looking at them.

All good news? Not entirely. In my next post I’ll look at some of the things that are going wrong, and the analytics that help to identify these.

Learning analytics / teaching analytics

My current focus is on learning analytics. How can we tell from site analytics whether someone is learning, engaging in activities that have been shown to support learning, or exhibiting behaviours that are associated with learning? And, rather than develop Wheel 2.0, I'm looking at available analytics and whether they can be harnessed to do this. Hence the current focus on Google Analytics.

A problem is that identifying learning means that I need to be able to associate activity with specific individuals or groups of individuals. As I discussed below, I can do that to some limited extent with Google Analytics but it's not really set up for me to do that and, more to the point, focusing on individuals in this way feels intrusive and, I think, would need informed consent from those concerned if I pursued it to any extent.

So Google Analytics can give me some pointers as to whether learning (activities/behaviours) are taking place, but to link this to individual learners or groups of learners would involve another set of analytics, and those learners would have to be aware of what was taking place.

Coming at this from another perspective - how about teaching analytics? I'm not thinking here of time/motion studies about level of activity and output - I'm more interested in helping teachers / educators judge the value their output has for others. Google Analytics are potentially more helpful here, because the authors of online resources, and the creators of online discussions have publicly identified themselves, and so resources can be tied to individuals.

So, if I examine an online resource, I could look at how many visits it receives, how long those visits last and whether people move on to use linked resources. I can examine the effects of sharing a link to that resource on Twitter. How effective are my Twitter links compared with those of people widely known for their expertise in the field?

More broadly, if I look at an educational resource (I'm currently looking at Cloudworks) I can begin to identify the most effective resources and behaviours. In my next post, I'll describe some preliminary work I have done on this.

Hourly reports on Google Analytics

Google Analytics allows you to break down activity on your site by hour, but this function isn’t easy to find in the current version.

I have set up a custom report to do this (custom reporting is available on the left-hand side of your Google Analytics screen).

Custom reports are set up by dragging metrics (blue) from the left of the screen and adding dimensions (green).

I have dragged over Entrances – which counts how many people arrive on the site – and split it down by hours of the day. So, for example, I can see that between 9am and 10am there have been 1015 site entrances during the last month – and there have only been 100 between 1am and 2am.

As usual, I can filter the report by selecting a particular date, or range of dates, at the top of the screen. On 21 June most people arrived in the hour before midday, while only two people arrived 1am-2pm.

My hourly report is set to drill down to ‘Page Title’ (that’s the second green dimension that I dragged on to my custom report. This means I can click on any hour of the day and see where those entrances took place during that hour.

As my focus is on the two-day OU online conference (21-22 June 2010) I can now focus right in and see where people arrived on the site during specific sessions.
What is more, I can then subdivide that information, so I know how many people arriving on a certain page during a certain session are new visitors, or which country they come from.

By the time I have narrowed it to time, date and city where the visitor is based, though, the level of granularity is such that it has ethical implications because I now have a fairly good idea who some of those individual visitors are. I can click through and see who their service provider is (for example ‘open university’, ‘university of leeds’) and what their connection speed is (an online conference on dial-up? – ouch)

Using Google Analytics at this level of granularity seems/is intrusive. That’s a tension for my work on learning analytics – because how can learning analytics work if they don’t split down to individual level? I guess the distinction has to be that I should be in the user’s hands to turn them on and off, and to decide their own privacy levels in different situations.

Building on previous experience

Tony Hirst helpfully Twittered a Google search term for locating his posts on Google Analytics and his uncourse
allintitle: "course analytics" site:ouseful.open.ac.uk

He began (24 October 2007) with a focus on:
"four distribution (rather than average) measures that are useful for analysing user behaviour on non-ecommerce websites:
* Visitor loyalty - how often has each user visited the site over a given period;
* Visitor recency - of all the people who have visited the site, how many have visited in the last N days;
* Length of visit - how long do visitors stay on site;
* Depth of visit - how many pages on the site are seen on each visit"

His focus (25 Oct) was on "website analytics can be used applied to online course websites in order to gain a better understanding of online study habits and the bahaviour of students taking an online course." The length of visit figure gave an idea of how long students were long to spend online studying course material (approx 30 mins in this case).

On 26 Oct he focused on timing of visits. Students on the course appeared to be less likely to visit on a Saturday, and seemed to be online more at lunchtimes and in the early to mid-evening. Apart from these daily and weekly patterns, there were also spikes associated with deadlines.

This is a small dataset (around 100 students) in the context of the OU, and they were studying an online computer course which makes them likely to be atypical in terms of computer use. Still, it gives some broad hypotheses - students will prefer online material provided in up to 30-minute chunks and they are more likely to be available for collaborative activities in lunchtimes and evenings.