Analytics and Institutional Capabilities at JISC CETIS Conference March 13 2013

It’s Cheltenham Races as well as Open Educational Resource week and the launch of phase 2 of our Jisc CETIS Analytics Series with the first case study (Derby University) available now.

Cheltenham 2013

I’m celebrating the two by attending Jisc CETIS Conference 2013. Today I’m blogging Martin Hawksey (prolific Tweeter, @mhawksey, and data guru), my compadre Sheila McNeil @sheilmcn (the drive behind much of the JISC CETIS Analytics Series) and a bunch of delegates for a long session on the aforementioned at Aston Lakeside. Full session details here.

Aston Lakeside

Martin proposes to run through the dreams, realities and nightmares of analytics. Ranjit Sidhu (Sid) (@rssidhu) is here. Sid is behind ‘Sector Statistics’, something that came to my attention a year or so ago being a shared service joining up cros sinstitutional data sets and providing analytics for those paying to take part. Sid gave a lightning talk yesterday warning about the perils of Big Data = Big Projects = Big Wastes of Money.

Martin ran a pre session questionnaire and displayed a visualisation about delegate backgrounds and interests. This brought up the issue of data literacies – can we all interpret the graph? Mixed responses so he explained for us.

Hawksey Graph 1

The questionnaire asked about competency with data manipulation and found that there were no gurus coming forward.

Some great slides here, particularly liked the nightmare analytics scenarios of students being thrown off courses before they fail based on a failure prediction form the analytics.

Nice quote from Tony Hirst (@psychemedia) (Martin and Tony tweet about data analysis into the small hours) – all charts are lies.

Another quote from Ben Fry (Ben presented at that dashboarding event I was at a couple of weeks ago)
Graphs can be a powerful way to represent relationships between data but they are also a very abstract concept, which means that they run teh danger of meaning something only to the creator of the graph. Everything looks like a graph, but almost nothing should ever be drawn as one.

Dashboarding – an illegal torture method whereby senior managers are gagged and Big Data poured onto their face until they pass out (see also waterboarding). Danson 2013.

Mark Stiles brought up the issue of defining the questions before wasting resource analysing the data. A couple of years ago I attended a Gartner Big Data and Analytics conference (before my blogging days) where someone suggested spending 80% of analytics project budget defining the questions. Then do it again.

Next up we have a series of lightning talks.

1. Wilbert Kraan
Course data project at Bolton University using new XCRI feeds considering student records, module descriptions, timetable data and external data sets; HESA, Student Satisfaction Survey etc.
Now Bolton University developed a Jisc funded workload model tool and have been uusing it to help out with a financial crisis over student numbers. They’re making posts redundant at present but I see they are hiring a BI expert.

2. Mark Stiles
Common Core IMS Standard and publishers clamouring to crank out content, so a heads up presumably

3. Martin Hamilton
Analytics for green energy usage

4. Brian Kelly
Tools for data collection and inference to be drawn outliers are interesting, comparisons might identify good practices and bad etc

5. Shane of PebbelPad
Gets asked by customers for more and more data about how people use their ePortfolio system but has concerns about the validity of what is being attempted and anxieties that maybe inferred by some people who may vote with their feet. The data might be valuable, but by mining it we may undermine its value

6. Martin Hawksey
Quick mention of John Campbell and Purdue Signals project. This comes up time and time again. Martin advocates keeping analysis simple. He consumes into google spreadsheets and bolts on widgits, gadgets and gisoms for visualisations as needed. That the tools has macro capability and scripting.

7. Ranjit Sidhu (Sid) (Statistics into Decisions)
It’s a new world of data but should we bother?

Brave New World

Tell me what you want me to say and I’ll make you a graph.
Sid says there is no truth in data. Decide what you want to communicate and identify the data to demonstrate it is the premise. this avoids becoming lost in a sea of data.
Are we becoming more intelligent (in our data use) or just institutionally covering the tracks of our ignorance? Be brave enough to say when the data is of no use.
Big Data is not new. IBM were doing it 20 years ago, It is just dimensions and measures. So Sid is no Big Data promoter. He thinks it’s a fad and argues for more precision when considering goals and ROI. Like Hawksey he is a proponent of small data. It’s cheaper and quicker. If you don;t do anything meaningful with small data, why would you do anything with big data? Is he a cynic with a smile, or a realist?
Sid showed the Unistats web site. It was expected that students would use it to select universities based on performance criteria. It runs at only 7 visits a day. Well, 8 today,

The key to making data useful is in the human interpretation to first identify then answer the important questions.

Sid works with local data. He starts reading VC statements. PG and International Dashboards are usually the first he produces. Web site customer journey is key in terms of navigation. Undergraduate path takes them to UG data then UCAS to apply. Postgraduate has its own path. On 16/8/12 Clearing changed everything in terms of a change in undergraduate activity and behaviour. Recruitment to meet targets became a purely competitive market. The fees effect was seen across the sector. The prospective students became very discernable clients.

The two worst web pages are a 404 (not surprisingly) and a duff search page that fails to find what the student wants and offers poor alternatives. Sid notes this results in an 80% drop off effect therefore universities m=should concentrate on getting the course information visible and making it look appealing so it sells. Clearing dashboards are a low hanging fruit. Note it’s local data again.

Google advertisements by Universities are hugely important and hugely expensive. Sid states that many Universities are being killed in this competitive environment and reports panic in the sector.

How can you compare Robert Gordon with LSE? What questions can you ask? It’s absolutely moronic. Well said Sid.

Exit Sid, enter Mark Stubbs and his circle of analytics

Case study one had a student spending 2006 – 2009 evaluating the impact of the institutional VLE. In particular does VLE usage affect student success? What sorts of interaction yield the best outcomes?

Manchester Met Analytics VLE Cycle

Trends identified covered a breadth of activities including number of content styles students had visited, discussion tool interaction, assessment tool interaction, number of log ons, percentage of 9 – 5 hits and more. The project managed to match around 18K students through interactions and behaviours to degree classification. As one might imagine this yielded a Big Data sample (albeit a local one) requiring analytical techniques and tools. Significance was explored using random forest analysis.

Here are some examples of visualisations produced

Forest Graphs

At risk failure alarm bells included high usage outside 9 – 5, early finish or late start in the year VLE access. Categorised for interaction type, with more analysis to be undertaken. Hmmm. So is this really an example of a time consuming and resource intensive big data project to determine the obvious?

Of course not. This is @thestubbs in action. The team went on to include NSS scores for their courses. The strongest predictor was the question about whether the course was well organised and running smoothly. It was NOT the question about assessment which is widely held as the main predictor for success. The team undertook more analysis against other NSS questions with equally unexpected results. Of course NSS has its critics so one must bear that in mind. For MMU course organisation was the most important aspect to address for improvement. Other analyses are bing undertaken. Meanwhile course organisation falls in terms of statistical relevance as others increase. The next steps are illustrated below

Next Steps For Manchester Met Analytics

Next we have Jean Muton (@myderbi). I’ve worked a good deal with Jean on our Relationship Management programmes. No need to get bogged in that though as we just launched an InfoKit which includes a case study on this talk, video and advice. Basically Jean looked at identification of students not engaging early on with the University, hence either at risk of drop out or not achieving their full potential. Jean is also big on Service Design and I’d recommend this to you. Here’s a presentation I ran with Jean and Simon yesterday on that InfoKit.

The team came up with the concept of Engagement Analytics. As Jean was the subject of a case study today I’ll send you there. Helpful to know how to get to these things as more are due between now and July 13. Open the Observatory, Choose the Analytics Tab, Scroll down to the blogs, choose the appropriate one, click through to the the case study. Jean shared an anecdote about the ‘bicycle diary’ at an HEI which was completed by staff who cycled to work. When the book was full it was taken to registry who shelved it. When the shelf was full the bicycle diaries were archived in the library. On inspection, the archives went back to the second world war. Cyclists were given extra food rations. Considering the student lifecycle and touch points demonstrating engagement, what processes are actually valid and why do we do them the way we do?

Next we’re treated to a talk by Simon Buckingham Shum of the OU Knowledge Media Institute. Simon @sbskmi is on the team behind SOLAR and LAK 13.

Simon describes Learning Analytics as Evolutionary Technology where it’s concerned with the usual educational paradigms of engaging, better assessed with better outcomes delivered at scale. It becomes revolutionary technology by offering insight into interpersonal networks, quality of discourse and writing, lifelong learning dispositions, problem solving strategies (process analytics – a new one on me) and life wide learning.

Simon aired a few OU operational issues including data warehouse, data dictionary, data marts and cubes for data sets, exploration using appropriate tool (SAS for example) – the latter is a skill in shortage at present. Simon calls this role a data wrangler. To move to action. Which fits nicely with the Jisc analytics definition. Phew.

Simon went on to talk about predictive analysis;

OU Predicitive Modelling

Quick plug by Simon for the Jisc funded RETAIN project (one of mine) which looked at VLE data as prediction indicators for student success and failure not disimilar to the material Mark Stubbs aired earlier. That one came from our Business Intelligence Programme and we’re launching a new InfoKit on that in April with case studies, video and plenty of peer advice. It’ll appear here (the old one is there).

Simon went on to show us some really complex tag clouds and visualisations for social analytics. This was first published at LAK 2012 in Vancouver buy Rebecca ferguson @R3beccaF and will be updated at LAK in Leuven 2013.

Discurse analytics next – it has a way to go but seems a great topic for a researcher like Simon and his colleague Rebecca.

We’re just missing a mention to @dougclow learning analytics researcher, lecturer and fellow live blogger (he’s much better that me).

What’s next for Learning Analytics? Well, not surprisingly Simon aired FutureLearn, MOOCS and analytics. He also aired Staff and Skills issues and what it means to be an educational data scientist. I’m really keen on this as, through my involvement in the Jisc digital literacy programme, I can forecast the need for a high level of data literacies across the workforce in the not too distant.

Anyway, Simon always pops his slides up on his blog so look there if this is your thing.

And so I must leave CETIS 2013. It’s been a real pleasure catching up with folks. Here’s a shot of some excellent colleagues enjoying academic discourse in the Sack of Spuds last night.

Sack of Spuds

@rsherratt @sclater @hughdavis and Rikki Prince

1 thought on “Analytics and Institutional Capabilities at JISC CETIS Conference March 13 2013

  1. Pingback: Day 1 Early PM; Learning Analytics and Knowledge Conference April 2013 « Myles Danson

Leave a Reply

Your email address will not be published. Required fields are marked *