Friday 16 July 2010

Learning/teaching analytics - digging deeper

Following on from my last post, why isn’t the news on my Research skills required by PhD students cloudscape as good as it looks at first sight?

Here are the viewing figures. 856 page views of my Cloudscape. Lovely, regular peaks in the viewing figures every Monday. What happens every Monday? I go and check out how much people are using my clouds – that’s what. So at least 400 of those page views are me collecting data. That, in turn, affects my bounce rate – which is 0% on all those data collecting days. In this case, the site’s own analytics are more helpful – the cloudscape has had 323 views (it doesn’t count the logged-in author).

Something else I get from the viewing figures is a spike on 28 June. What happened there? It seems that someone from The Open University arrived three times from Google with related research queries and spent 2 hours 50 minutes on the site. That’s great, they were obviously finding it useful – but that cuts everyone else’s average time on the site considerably.

More depressingly, here are 20 of the 32 searches that brought people to this cloud – where they then spent no time and bounced away from the site again. Some of them are probably in the wrong place and need to leave to look for IT skills or UK skills. The majority, though, should be staying – I’m losing at least 12% of visitors who should have found this a useful resource.

What do visitors see in that fraction of a second they spend on my Cloudscape before bouncing away? Possibly the first line of text ‘Research Skills required by PhD students, as defined by the UK Research Councils’. And, yes, half of those bouncing visitors are outside the UK – although only an eighth of views originate outside the UK. Time for a rewrite – these are generic research skills, relevant worldwide.

More broadly, what have I learned about implementing analytics on a learning site?
  • Search for a way to set aside visits by the site owner and the content author – it looks as if this needs to be done on the site itself, rather than through Google Analytics.
  • Set aside outliers for separate consideration.
  • Give content authors access to the keywords that are working for that content, and the keywords that are not.
  • In fact, if the aim is to improve the learning/teaching effectiveness of resources, it would be good if authors could access short analytics reports without having to filter requests through the site owner.

Google Analytics and learning/teaching potential

What else can Google Analytics tell me about the potential of my resources on Cloudworks as teaching materials? Well, I’d expect my Cloudscapes, which are essentially index pages, to lead people to the associated clouds (content pages).

I’ll focus here on Research Skills required by PhD Students. I created this back in February – and it involved a fairly straight transfer of Web materials to Web 2.0. Pretty much all of this material was already available on the Open University Intranet, I moved it over to Cloudworks, sorted out the dead links, added some new links and it’s now open for people to add to, comment on and discuss.

The original, Intranet, version of the web page didn’t work too well.The page stats over a 14-month period showed that people were following the links based on where they fell on the page. Links at the top left did very well, followed by the first link after each heading. Links on the bottom right were only followed once or twice a month. This looks like a classic browsing pattern – people arrive, click around to see what is on offer, but don’t make any serious use of the page or its linked resources.

When the material was added to Cloudworks, the pattern of usage became more even. The previously dominant A1: Recognising research problems was replaced by B2: Compliance with ethical requirements which, more recently, has been overtaken by B6: Justifying research methods. I like to think that that reflects seasonal variation in the concerns of PhD students at the OU (the target audience for the clouds and for the original site) but that’s currently just a guess.

If the Cloudscape is doing its job as a learning/teaching resource I’d expect to see a low level of bounces from that page, a high level of people moving through to the linked clouds, people moving through to the pages related to their original search terms, and those people who do move on to linked clouds spending at least a few minutes on the site.

I seem to have a fair-sized readership – 848 page views since February. Two-thirds of those arriving on the page are bouncing directly off the site but those arrive from elsewhere on the site don't tend to leave when they get to my cloudscape. I’m glad to see that they are following the links, and that those who stay on the site move to linked pages.

The unique visitors who landed directly on the page spent nearly 8 hours there in total, which averages out at around four minutes each. What’s more, people with relevant, detailed searches seem to be spending time on the site.

According to my original hypothesis, Google Analytics is showing this to be a page that looks likely to be supporting teaching and learning. People want to find out more about skills required for research, this page provides links to relevant material, they follow these links and spend some time looking at them.

All good news? Not entirely. In my next post I’ll look at some of the things that are going wrong, and the analytics that help to identify these.

Thursday 15 July 2010

Using analytics to improve practice

I’m looking at a Cloudworks page I set up earlier this year, when I was creating resources for Open University research students in connection with a postgrad conference here at the university. Each of the presentations at the conference had its own cloud, and they have all received a fairly steady amount of visits each week since the conference. The most popular cloud, and the one I’m looking at here, is Skills Audit (currently with 199 views, according to the Cloudscape figures).

Students at the OU have to complete a Skills Audit at the end of the first year of their PhD – considering their progress in relation to a set of research skills identified and agreed by the UK research councils. Having done it myself, I’d say it was a valuable exercise, but I remember resenting having this chore imposed upon me.

Before I looked at Google Analytics, I was fairly pleased with this cloud and its performance. It’s aimed at first-year PhD students at the OU and it’s picking up around ten views a week at the moment. It links to complete lists of research skills and resources for developing and assessing them, it links to the actual skills adit that OU students have to fill in, and it links to the rest of the postgrad conference, and it contains a liveblog of a detailed conference session on the Skills Audit, as well as a biography of the speaker. Not only that, but the SocialLearn gadgets on the page point visitors to other related resources. All the links and info you need in one handy cloud.

I’m not so happy, though, when I get to the analytics. Visitors are spending an average of 43 seconds on the page, but only one or two have spent much longer there. More seriously, I have an 88.6% bounce rate, so visitors aren’t using this as a gateway to other resources on Cloudworks. This gets even worse (90.65%) when I look at the key words people used to get to the page.

Let’s take a closer look at a representative sample. That long line of zeroes represents time spent on the page, the 100%s represent the bounce rate and the exit rate. So people are arriving who should be looking for exactly this resource and, in less than a second, are consistently deciding that what I have provided is no good for them.

And, when I spend half a second on the cloud, I can see exactly why. The cloud begins with a long biography of the seminar presenter – that’s not what people are looking for. I know why it’s at the top – it was the only content I had available before the conference took place – but it shouldn’t be there now.

So the analytics have helped me to improve my own practice – now I need to think through how I can extend this finding in order to have a wider impact.

Learning analytics / teaching analytics

My current focus is on learning analytics. How can we tell from site analytics whether someone is learning, engaging in activities that have been shown to support learning, or exhibiting behaviours that are associated with learning? And, rather than develop Wheel 2.0, I'm looking at available analytics and whether they can be harnessed to do this. Hence the current focus on Google Analytics.

A problem is that identifying learning means that I need to be able to associate activity with specific individuals or groups of individuals. As I discussed below, I can do that to some limited extent with Google Analytics but it's not really set up for me to do that and, more to the point, focusing on individuals in this way feels intrusive and, I think, would need informed consent from those concerned if I pursued it to any extent.

So Google Analytics can give me some pointers as to whether learning (activities/behaviours) are taking place, but to link this to individual learners or groups of learners would involve another set of analytics, and those learners would have to be aware of what was taking place.

Coming at this from another perspective - how about teaching analytics? I'm not thinking here of time/motion studies about level of activity and output - I'm more interested in helping teachers / educators judge the value their output has for others. Google Analytics are potentially more helpful here, because the authors of online resources, and the creators of online discussions have publicly identified themselves, and so resources can be tied to individuals.

So, if I examine an online resource, I could look at how many visits it receives, how long those visits last and whether people move on to use linked resources. I can examine the effects of sharing a link to that resource on Twitter. How effective are my Twitter links compared with those of people widely known for their expertise in the field?

More broadly, if I look at an educational resource (I'm currently looking at Cloudworks) I can begin to identify the most effective resources and behaviours. In my next post, I'll describe some preliminary work I have done on this.

Wednesday 14 July 2010

Hourly reports on Google Analytics

Google Analytics allows you to break down activity on your site by hour, but this function isn’t easy to find in the current version.

I have set up a custom report to do this (custom reporting is available on the left-hand side of your Google Analytics screen).

Custom reports are set up by dragging metrics (blue) from the left of the screen and adding dimensions (green).

I have dragged over Entrances – which counts how many people arrive on the site – and split it down by hours of the day. So, for example, I can see that between 9am and 10am there have been 1015 site entrances during the last month – and there have only been 100 between 1am and 2am.

As usual, I can filter the report by selecting a particular date, or range of dates, at the top of the screen. On 21 June most people arrived in the hour before midday, while only two people arrived 1am-2pm.

My hourly report is set to drill down to ‘Page Title’ (that’s the second green dimension that I dragged on to my custom report. This means I can click on any hour of the day and see where those entrances took place during that hour.

As my focus is on the two-day OU online conference (21-22 June 2010) I can now focus right in and see where people arrived on the site during specific sessions.
What is more, I can then subdivide that information, so I know how many people arriving on a certain page during a certain session are new visitors, or which country they come from.

By the time I have narrowed it to time, date and city where the visitor is based, though, the level of granularity is such that it has ethical implications because I now have a fairly good idea who some of those individual visitors are. I can click through and see who their service provider is (for example ‘open university’, ‘university of leeds’) and what their connection speed is (an online conference on dial-up? – ouch)

Using Google Analytics at this level of granularity seems/is intrusive. That’s a tension for my work on learning analytics – because how can learning analytics work if they don’t split down to individual level? I guess the distinction has to be that I should be in the user’s hands to turn them on and off, and to decide their own privacy levels in different situations.

Academic Analytics: A New Tool for a New Era

I've followed a link in OUseful Info to Educause, where I'm looking at
Academic Analytics: A New Tool for a New Era
John P. Campbell, Peter B. DeBlois, and Diana G. Oblinger
EDUCAUSE Review, vol. 42, no. 4 (July/August 2007): 40–57

This identifies several uses for analytics in education:
  • To manage enrolment, using standardised exam scores, high school coursework, and other information to determine which applicants will be admitted.
  • To inform fund-raising. By building a data warehouse containing information about alumni and friends, institutions can use predictive models to identify those donors who are most likely to give.
  • To aid retention by identifying students most at risk of dropping out
  • To assess which proactive interventions have the best influence on academic success and retention. 
  • To predict student success within a course

They also highlight three characteristics of successful academic analytics-based projects (link to referenced PDF):
  1. Leaders who are committed to evidence-based decision-making
  2. Administrative staff who are skilled at data analysis
  3. A flexible technology platform that is available to collect, mine, and analyze data

Within the OU, the IET Student Statistics department takes a leading role in analytics projects like these. Other departments, such as Communications, also make use of analytics data.

My focus is on learning analytics - how we can use online analytics to identify learning, conditions that support learning and behaviours that support learning.

Building on previous experience

Tony Hirst helpfully Twittered a Google search term for locating his posts on Google Analytics and his uncourse
allintitle: "course analytics" site:ouseful.open.ac.uk

He began (24 October 2007) with a focus on:
"four distribution (rather than average) measures that are useful for analysing user behaviour on non-ecommerce websites:
* Visitor loyalty - how often has each user visited the site over a given period;
* Visitor recency - of all the people who have visited the site, how many have visited in the last N days;
* Length of visit - how long do visitors stay on site;
* Depth of visit - how many pages on the site are seen on each visit"

His focus (25 Oct) was on "website analytics can be used applied to online course websites in order to gain a better understanding of online study habits and the bahaviour of students taking an online course." The length of visit figure gave an idea of how long students were long to spend online studying course material (approx 30 mins in this case).

On 26 Oct he focused on timing of visits. Students on the course appeared to be less likely to visit on a Saturday, and seemed to be online more at lunchtimes and in the early to mid-evening. Apart from these daily and weekly patterns, there were also spikes associated with deadlines.

This is a small dataset (around 100 students) in the context of the OU, and they were studying an online computer course which makes them likely to be atypical in terms of computer use. Still, it gives some broad hypotheses - students will prefer online material provided in up to 30-minute chunks and they are more likely to be available for collaborative activities in lunchtimes and evenings.

Monday 12 July 2010

Learning analytics

In this blog I will be thinking about learning analytics. How can we identify learning when it takes place online without formal assessment?