Single search box

Reading through Lown, Sierra and Boyer’s article from ACRL on ‘How Users Search the Library from a Single Search Box’ based on their work at NCSU, started me thinking about looking at some data around how people are using the single search box that we have been testing at http://www.open.ac.uk/libraryservices/beta/search/.

About three months or so ago we created a prototype tool that pulls together results from the Discovery product we use (EBSCO Discovery) alongside results from the resources database that we use to feed the Library Resources pages on the library website, and including pages from the library website.  Each result is shown in a box (ala ‘bento box’) and they are just listed down the screen, with Exact Title Matches and Title Matches being shown at the top, followed by a list of Databases, Library Pages, Ebooks, Ejournals and then articles from EBSCO Discovery.  It was done in a deliberately simple way without lots of extra options to manipulate or refine the lists so we could get some very early views about how useful it was as an approach.

Looking at the data from Google Analytics, we’ve had just over 2,000 page views over the three months.  There’s a spread of more than 800 different searches with the majority (less than 10%) being repeated fewer than 6 times.  I’d suspect that most of those repeated terms are ones where people have been testing the tool.

The data also allows us to pick up when people are doing a search and then choosing to look at more data from one of the ‘bento boxes’, effectively they do this by applying a filter to the search string, e.g. (&Filter=EBOOK) takes you to all the Ebook resources that match your original search term.  So 160 of the 2,000 page views were for Ebooks (8%) and 113 f0r Ejournals (6%) for example.

When it comes to looking at the actual search terms then they are overwhelmingly ‘subject’ type searches, with very few journal articles or author names in the search string.  There are a few more journal or database names such as Medline or Web of Science  But otherwise there is a very wide variety of search terms being employed and it very quickly gets down to single figure frequency.  The wordle word cloud at the top of the page shows the range of search terms used in the last three months.

We’ve more work to do to look in more detail about what people want to do but being able to look at the search terms that people use and see how they filter their results is quite useful.  Next steps are to do a bit more digging into Google Analytics to see what other useful data can be gleaned about what users are doing in the prototype.

Library resources and recommendations

I noticed an interesting Jisc-funded project at Liverpool today that I hadn’t previously heard about (blogged by Jisc today) that talked about a method of sharing resources amongst students using a crowdsourcing approach.  The service is called Kritikos and takes several quite interesting approaches.  At the heart of the system is some work that has been done with students to identify resources relevant to their subjects (in this case Engineering) and also to identify results that weren’t relevant (often because some engineering terms have different meanings elsewhere – e.g. stress). That’s an interesting approach as one of the criticisms I’ve heard about discovery systems is that they struggle to distinguish between terms that are used across different disciplines (differentiation for example having separate meanings in mathematics and biology).

The search system uses a Google Custom Search Engine but then presents the results as images which is a fascinating way of approaching this aspect.  Kritikos also makes use of the Learning Registry to store data about students interactions with the resource and whether they found them relevant or not.  It seems to be a really novel approach to providing a search system that could go some way to address one of the common comments that we’ve been seeing in some work we’ve been doing with students. They feel that they are being deluged with too much material and struggle to find the gold nuggets that give them everything they want.

Kritikos looks to be particularly useful for students in the later stages of their degrees, where they are more likely to be doing some research or independent study.  One of the things that we are finding from our work is that students at earlier stages are less interested in what other students are doing or what they might recommend.  But possibly if they were presented with something like Kritikos they might be more inclined to see the value of other students’ recommendations.

Visual.ly Google Analytics infographic

Infographics and data visualisations seem to be very popular at the moment and for a while I’ve been keeping an eye on visual.ly as they have some great infographics and data visualisations.  One of the good things about the visual.ly infographics is that there is some scope to customise them.  So for example there is one about the ‘Life of a hashtag’ that you can customise and several others around facebook and twitter that you can use.

I picked up on twitter the other week that they had just brought out a Google Analytics infographic.  That immediately got my interest as we make a lot of use of GA.  You just point it to your site through your Google Analytics account and then get a weekly email ‘Your weekly insights’ created dynamically from your Google Analytics data.

It’s a very neat idea and quite a useful promotional tool to give people a quick snapshot of what is going on.  So you get Pageviews over the past three weeks, what the trends are for New and Returning Visitors and reports on Pages per visit and Time on site and how that has changed in the past week.

It’s quite useful for social media traffic showing how facebook and twitter traffic has changed over the past week and as these types of media are things that you often want quite quick feedback on it is a nice visual way of being able to show what difference a particular activity might have had.

Obviously as a free tool, there’s a limit to the customisation you can do.  So it might be nice to have visits or unique visitors to measure change in use of the site, or your top referrals, or particular pages that have been used most frequently. The time period is something that possibly makes it less useful for me in that I’m more likely to be want to compare against the previous month (or even this month last year).  But no doubt visual.ly would build a custom version for you if you wanted something particular.

But as a freely available tool it’s a useful thing to have.  The infographic is nicely presented and gives a visually appealing presentation of analytics data that can often be difficult to present to audiences who don’t necessarily understand the intricacies of web analytics.

The Google Analytics Visual.ly infographic is at https://create.visual.ly/graphic/google-analytics/

Discovery is miscellaneous

Everything is miscellaneous
I’ve finally got around to reading “Everything is Miscellaneous” by David Weinberger, (yes I know that is about five years after everyone else, and no real reason not to read it, other than a sense of not wanting to follow everyone else.)    I’m reading it in paperback form as we don’t seem to have it on ebook which gives it a slight sense of being older than it actually is.  Particularly with the pages in the paperback being slightly yellowing.  It is also interesting to me to pick up a library book added to stock in 2009 that has two date stamps on the date label. Two loans in four years brings home what a different world academic libraries are from public libraries.

While there’s a slight sense of things having moved on in the post – twitter world in terms of some of the technologies, it is a really interesting read with lots of things to think about and it is really making me think about the approach we take to providing access to library materials.  I am particularly thinking about how we present material through our library website, either with search tabs for articles, books etc, or by categorising library resources into journals, databases or ebooks, or even by us using different systems to manage different types of material.  As David Weinberger points out that is just a carry over from the old analogue and physical world that makes no real sense to users in a digital world.  And that is something that needs reinforcing regularly as it is easy to lose sight of that.

Tagging, sharing and perspective
One of the things that is starting to come out of our personalisation surveying and focus groups is that users want what is relevant to them.   Well, not a great surprise, but then that isn’t something that our systems really faciliate do they?  Where we are at the moment is to still think in terms of how you get something depends on what type of thing it is.  For a physical library that’s relevant in that the leaf is only on the tree in one place, to use Weinberger’s analogy.  But in the digital world, all the stuff is website content, and all the constraints are artificially created (that doesn’t mean that they are not necessary in some cases).   So you access ebooks through the catalogue because that is where we put them, often for our administrative convenience.  But users might want them in different places at different times.  But in a world where users expect to be able to shape their view of the world by customising the ‘library channel’ as you can do with Spotify or any number of web-scale services, the single ‘take-it or leave-it’ library approach seems curiously archaic. 

Discovery
So what does that mean for discovery and especially for discovery systems?  Are discovery systems the right solution?  Discovery systems and the Google-like search box are an attempt to pull stuff together into one place.  So upload your catalogue into your discovery platform and you can lose the OPAC – maybe.   It seems to me to start to pick up on relevancy ranking becoming a much more important area.  But it still doesn’t really start to approach anything that is particularly ‘socially’ or ‘user-aware’.

As a user you probably want to decide what is relevant to you, you might want to tag that content and probably share it too.  And you’d probably expect to be able to see other users tag and use them to find material relevant for you too.   But with library systems we take the view that we have to have special people who we trust to add accurate metadata.  I hate to say this, but I think that’s another legacy of the physical age and not really viable for the explosion in digital content that is upon us.

So you start to have a model where users expect the system to know something about them (what course they are on for example – does your discovery platform know that?), and to filter based on their likely interests, but then to learn from what they search for (and what others search for, or tag) to find other things they might be interested in.  I start to think that this is at the heart of user disatisfaction with library systems, there is a great disconnection with their experience of the rest of the web.

Is it feasible, could we experiment, what might that space look like?  Discovery is miscellaneous now…

New library tools

New tools concept
Earlier in the week we soft-launched a new section on our library website.  The New Tools section is a space where we can put out new ideas with the aim of trying to get some feedback about whether users will find them useful.  This parallels the work we’re also doing with a group of students from our Student Panel to work with them to design some new features (blogged about earlier in the week).

http://www.open.ac.uk/library/new-tools

Our idea is that we’d use the New Tools section to put up beta tools based on ideas that have come up through a number of ways.  So the ideas that come through the personalisation study work with students will go through a private ‘alpha’ stage where they help with defining the ideas and feeding back on paper prototypes and ‘proof-of-concept’  tools.  Once the tools have been refined the best ones get released as ‘beta’ versions through the New Tools section.  We’d also look at releasing as beta tools some of the ideas that come from other work we’ve done in the past such as in the RISE recommender project and other ideas we’ve come up with.

The idea with the New tools section is that the tools aren’t fully supported but are there for people to try and let us know what they think about them.  If they work then we can refine them and take them into service.  If they aren’t useful then we’ll have a better idea of what people want and what they don’t.

First new tools – single search box
The first two tools that we’ve made available in beta are both around library resources.  The first one is a single search box (I”ve written before about the library quest for the google-like search box – and I’m starting to get more interested in the Google-like search box actually being Google and that libraries might be better concentrating on helping users in Google find library resources that they are entitled to access – but Google’s decision to ‘retire’ Google Reader certainly gives me pause in relying too much on something from Google).   Behind the search box is a search that passes your search string to our version of EBSCO Discovery (using their API) and also to the library resources database that powers the resource lists that are fed into the library website.  The idea behind this is that it will bring together results from our various systems into one place and particularly that it will be better at finding Journal titles that are direct matches.

This single search box is designed to also test the feasibility of bringing together different search results into a single interface.  It’s a bit federated-search-like in that the results are presented in separate boxes (sort of like a stacked bento-box approach inspired by Stanford and others – it’s interesting also to see the approach that Princeton have taken with their beta version of their library website).  We also haven’t strayed too much into the area of adding some of the surrounding functionality (saving citations, sharing etc features) that a fully-fledged system would need.  This is just about testing whether pulling together these results is a workable and useful thing to do.

First new tools – My recent resources
The second tool is about trying to see if giving users access to a list of library resources they have recently accessed is useful to them.   If you’re not an OU user (or aren’t signed-in) you’ll just see a demonstration list of resources.  But if you are signed-in you should see a list of the resources you’ve used, with the most recent ones first.  These resources will include ones you’ve looked at directly from the library website, or ones articles that you’ve viewed through our One Stop search discovery system.  For this prototype we have offered RSS and RIS formats to export your records so you can put them into your favourite reference management tool. We’ve also included a box on the right to list your most used resources, with the number of times in brackets.

The format and description of the entries just picks up the standard format we already use on the library website and we’ve started to add in book covers for ebooks (although that gets me thinking that I’ve never really worked out what the point is of a book cover for an ebook anyway – Kindles seem to take you to the start of the book, not to the cover, so maybe ebook covers aren’t that relevant anymore – but in any case it breaks up the blocks of text neatly).

Next steps
The plan is to develop more prototypes and build up a pool of tools in this space that we can get people to look at and comment on.  Hopefully it will be useful to people,

Whose data is it anyway?

http://www.flickr.com/photos/blenderizer/6062810940/sizes/m/in/photostream/

One of the bits of work that we’re doing at the moment is to talk to students about their thoughts about personalised library services. The aim of the work is to help us to understand what students might want (or not want) and to then use that information to build some tools that we can test with them.  In part it is being driven by a realisation that library websites and systems are competing against expectations that are shaped by sites such as Google and Amazon.  Traditional library websites such as OPACs seem to be a world away from a modern web experience (see Aaron Schmidt’s blogpost on Library Journal for example).

One of the interesting things that is coming out of the work for me is around attitudes and expectations for the personal use data that is being collected as part of user engagement with our systems.   I’d expected that students would be quite guarded about what they would expect a library system to know about them, because generally speaking, libraries rarely seem to use data to provide much in the way of a personalised service.  But expectations seem to be that once a student has logged in then library systems should know their name and the course they are studying, at least.  But that maybe the library systems should also know what previous courses they’d studied or their contact preferences.   And that is really interesting to know as it’s difficult to think of many (any? other than some experimental work) examples of library systems that do track what courses a student is studying and actively use that data to provide a tailored service.

When we asked some specific questions about whether students would object to us using certain data to tailor services, over 90% of respondents didn’t object to us using their course or previous material they’ve accessed as a means of providing personalised services.  More than 80% had no objection to using their previous courses or search terms for personalisation. I think I would have expected a larger number of respondents who objected to the use of their data.

What I think is that there’s a trade-off between privacy and service (highlighted in this article by Li and Unger ‘Willing to pay for quality personalization?‘ (link is to abstract) from European Journal of Information Systems (2012) 21, 621–642. doi:10.1057/ejis.2012.13).  So there is a conscious calculation being made in terms of being able to see that you, the user, are getting a direct benefit from allowing the system to know something about you.  As a user you make that calculation and judge whether it seems reasonable to you or not.  ‘Does the benefit outweigh the loss of privacy?’  It strikes me that there seems to be an element here where users might be ascribing a ‘value’ to their data and they’d ‘trade’ that value for a benefit.  That makes me wonder whether the likes of Google (that essentially make their business model in part at least out of the value they can leverage from user data) have had the effect of making users realise that their ‘data’ also has a value that they can swop for a service?

One Page Project Management

It’s always good to find out about new project management tools and tips, so it was good to spend a few hours the other week at a training session introducing the One Page Project Management (OPPM) approach.    OPPM was something that I was vaguely aware of but not something that I knew too much about, and I probably started from the view of being slightly sceptical that it was possible to encapsulate everything you needed for reporting on your project onto a single page.  Well, not at A4 size anyway.

The training, run by David Sommer, was based around the One Page Project Management idea from Clark Campbell (you can see more information about OPPM and download a free version at their website https://www.oppmi.com/).  I understand David runs the UKSG Practical Project Management courses.

The training covered the ideas behind the concept and then concentrated on working through the template establishing your project and showing how you use it on a day to day basis. It was good to then be able to run through a real project and try to fit it into the template.

The template includes a header with things like the project goal and completion date, then a set of 5 or so objectives, a number of tasks and a list of people.  The template forms a matrix a bit like a gantt chart that shows progress with your project.  So you start with open circles in each time period on the timeline for an activity and then fill in the circles when that activity for that time period has been completed.  The template also lets you assign people and denote their roles and we had a bit of a debate about how we used the notation in that area.

Also in the template was space for some more subjective measures that essentially capture ‘confidence’ using a Red/Amber/Green traffic light system to denote how confident you are that the project will deliver on time for example.  There’s also a section for a couple of measures that might be cost or staff resource or % of content ingested for example.  And finally there’s a small box for a commentary.

A few things struck me during the session.  It’s interesting that it makes you think really carefully about your objectives for your project, and there was a tip about thinking about your objectives in the past tense, e.g. website usability testing completed.  It looks quite good at being able to get people to focus on the key information and where there are key decisions to be made.  Often there’s a tendancy in projects to focus on what’s been done rather than focusing on what is still to do.  I’m always in favour of ‘exception’ reporting where you get attention to focus on the things that are going off track or have changed or particularly where they need some action to get things back on track.  It looks like OPPM might be helpful for that.

Using OPPM as a focus of project meetings is also an interesting idea.  It was said that what tends to happen is that people become more focused on making sure they have done their tasks by the time of the meeting.  That’s a perpetual bane of a project manager’s existence so I liked that idea. I also liked the suggestion that you should print out your OPPM plan out in A3 size and stick it on the wall.  People passing could then see how your project was going at a glance.  And maybe steer clear if there were too many uncompleted tasks.

Overall it was a good day’s training session, quite practical and concentrating on working through the template and seeing how it works in practice.  We’re planning on adopting it for our project reporting proceses so it will be good to see how we get on with it.  Across our teams we’ve several projects running so it will be interesting to see how it copes with the different types of projects that we have.  But first impressions are that it does pretty much what it says in giving you a method of encompassing the key messages about the progress of your project in a single page.

Library data for library users

Encouraged by some thinking about what sort of prototype resource usage tools we want to build to test with users in a forthcoming ‘New tools’ section I’ve been starting to think about what sort of features you could offer to library users to let them take advantage of library data.

Early steps
For a few months we’ve been offering users of our mobile search interface (which just does a search of our EBSCO discovery system) a list of their recently viewed items and their recent searches. The idea behind testing it on a mobile device was that giving people a link to their recent searches or items viewed would make it easier for people to get back to things that they had accessed on their mobile device by just clicking single links rather than having to bookmark them or type in fiddly links. At the moment the tool just lists the resources and searches you’ve done through the mobile interface.

But our next step is to make a similar tool available through our main library website as a prototype of the ‘articles I’ve viewed’. And that’s where we start to wonder about whether the mobile version of the searches/results should be kept separate from the rest of your activities, or whether user expectations would be that, like a Kindle ebook that you can sync across multiple devices, your searches and activity should be consistent across all platforms?

At the moment our desktop version has all your viewed articles, regardless of the platform you used. But users might want to know in future which device they used to access the material maybe? Perhaps because some material isn’t easily accessible through a mobile device. But that opens up another question, in that the mobile version and the desktop version may be different URLs so you might want them to be pulled together as one resource with automatic detection of your device when you go to access the resource.

Next steps
With the data about what resources are being accessed and what library web pages are being accessed it starts to open up the possibility of some more user-centred use of library activity and analytics data.

So you could conceive of being able to match that there is a spike of users accessing the Athens problems FAQ page and be able to tie that to users trying to access Athens-authenticated resources. Being able to match activity with students being on a particular module could allow you to push automatically some more targeted help material, maybe into the VLE website for relevant modules, as well as flag up an indication of a potential issue to the technical and helpdesk teams.

You could also contemplate mining reading lists and course schedules to predict when there are particular activities that are scheduled and automatically schedule pushing relevant help and support or online tutorials to students. Some of the most interesting areas seem to me to be around building skills and using activity (or lack of activity) to trigger promotion of targeted skills building activities. So knowing that students on module X should be doing an activity that involves looking at this set of resources, and being able to detect the students that haven’t accessed those resources, offering them some specific help material, or even contact from a librarian. Realistically those sorts of interventions simply couldn’t be managed manually and would have to rely on some form of learning analytics-type trigger system.

One of the areas that would be useful to look at would be some form of student dashboard for library engagement. So this could give students some data about what engagement they have had with the library, e.g. resources accessed, library skills completed, library badges gained, library visits, books/ebooks borrowed etc. Maybe set against averages for their course, and perhaps with some metrics about what high-achieving students on their course last time did. Add to that a bookmarking feature, lists of recent searches and resources used, with lists of loans/holds. Finished off with useful library contacts and some suggested activities that might help them with their course based on what is know about the level of library skills needed in the course.

Before you can do some of the more sophisticated learning analytics-type activities I suspect it would be necessary is to have a better understanding of the impact that library activities/skills/resources have on student retention and achievement. And that seems to me to argue for some really detailed work to understand library impact at a ‘pedagogic’ level.

Library Analytics

I’d been thinking early this morning about writing up a blog post around some thoughts about ‘Library Analytics’ and thinking that it was interesting how ‘Library Analytics’ had been used by Harvard for their ‘Library analytics toolkit’ and by others as a way of talking about web analytics, but that neither really seemed to me to quite be analagous to the way that the Learning Analytics community, such as Solar,  view analytics.  There are several definitions about Learning Analytics.  This one from Educause’s 7 things you should know about first-generation learning analytics:

Learning analytics (LA) applies the model of analytics to the specific goal of improving learning outcomes. LA collects and analyzes the “digital breadcrumbs” that students leave as they interact with various computer systems to look for correlations between those activities and learning outcomes. The type of data gathered varies by institution and by application, but in general it includes information about the frequency with which students access online materials or the results of assessments from student exercises and activities conducted online. Learning analytics tools can track far more data than an instructor can alone, and at their best, LA applications can identify factors that are unexpectedly associated with student learning and course completion.

Much of the library interest in analytics seems to me to have mainly been about using activity data to understand user behaviour and make service improvements, but I’m increasingly of the view that whilst that is important, it is only half the story.  One of the areas that interests me about both learning analytics and activity data, is the empowering potential of that data as a tool for the user, rather than the lecturer or librarian, to find out interesting things about their behaviour, or get suggested actions or activities, and essentially to be able to make better choices.  And that seems to be the key – just as reviews and ratings are helping people being informed consumers, with sites like Trip Advisor then we should be building library systems that help our users to be informed library consumers.

So it was great to see the announcement of the JiscLAMP project this morning http://infteam.jiscinvolve.org/wp/2013/02/01/jisc-lamp-shedding-light-on-library-data-and-metrics/ announcing the Library Analytics and Metrics project and talking about delivering a prototype shared library analytics service for UK academic libraries.  I was particularly interested to see that the plan is to develop some use-cases for the data and great that Ben Showers shared some of the vision behind the idea.   It’s a great first step to put data on a solid, consistent and sustainable basis, and should build a good platform to be able to exploit that vast reservoir of library data.

Making the library’s search box work like Google

I’ve been reading a great blog post by Peter Morville on Semantic Studios ‘Inspiration Architecture: the Future of Libraries‘ and it includes a great description that really resonated

There was even a big move towards the vision of “library as platform.” Noble geeks developed elaborate schemata for open source, open API, open access environments with linked data and semantic markup to unleash innovation and integration through transparency, crowdsourcing, and mashups. They waxed poetic about the potential of web analytics and cloud computing to uncover implicit relationships and emerging patterns, identify scholarly pathways and lines of inquiry, and connect and contextualize artifacts with adaptive algorithms. They promised ecosystems of participation and infrastructures for the creation and sharing of knowledge and culture.

Unfortunately, the folks controlling the purse strings had absolutely no idea what these geeks were talking about, and they certainly weren’t about to entrust the future of their libraries (and their own careers) to the same bunch of incompetent techies who had systematically failed, for more than ten years, to simply make the library’s search box work like Google.

I’ve highlighted the last bit as it really struck home.  The great search for a library equivalent of the Google Search box is something that is familar to anyone working in trying to build better ways of helping users get to library content.  It has pretty much been a mantra over the past few years.  (for a great summary of how library search systems differ from Google look at Aaron Tay’s blogpost from last May and his blogpost on web scale discovery from December last year.)   So it’s easy to find examples of where libraries and other organisations have tried to put in place a google-like search, from the Biodiversity Heritage Library,  from the American University and from others such as the National Archives and Records Administration (reported in Information Management Journal, March 2011) and Oregon State University (paper by Stefanie Buck and Jane Nichols ‘Beyond the search box’ in Reference & User Services Quarterly March 2012.

The current generation of discovery systems (Summon, EDS, Primo etc) are largely built around the concept of a ‘google-like’ search.  As reported here for McGill University by OCLC for WorldCat Local. In some ways it seems to me that we’ve been concentrating too much on the simplicity of the original Google interface and as Lorcan Dempsey pointed out in his ‘Thirteen Ways of Looking at Libraries, Discovery and the Catalog’

‘a simple search box has only been one part of the Google formula. Pagerank has been very important in providing a good user experience, and Google has progressively added functionality as it included more resources in the search results. It also pulls targeted results to the top of the page (sports results, weather or movie times, for example), and it interjects specific categories of results in the general stream (news, images).’

So although we’ve implemented a ‘google-like’ search box it becomes apparent that it doesn’t entirely solve the problem. It’s a bit like a false summit or false peak.  You think you’ve reached the top but realise that you still have some way to go. Relevancy ranking becomes vitally important and with the Discovery service generation you’ve essentially handed that over to a vendor to control the relevancy algorithm.  You can add your local content into the system and have some control but it is limited.  And you are constrained in what you can add into the discovery platform.  Your catalogue, link resolver/knowledge base generally yes, your institutional repository yes, but your other lists of resources in simple databases, not so easily unless they happen to be OAI-PMH or MARC.

So you look at bringing together content from different systems, probably using the Bento Box approach (as used by Stanford and discussed by them here) where you search across your different systems using APIs etc and return a series of results from each of those systems.  You then get a series of results that come from each of the different systems and incorporate the relevancy ranking of discovery systems, rather than ranking the relevancy of the results in total.   So is that going to be any better for users?  Is it going to be better to sort the results by system, as Stanford have done? or should we be trying to pull results together, as Google do?  That’s something we need to test.

But there’s a nagging feeling that this still all relies on users having to come to ‘the library’ rather than being where the users are.  So OK, we can put library search boxes into the Virtual Learning Environment, so we’ve an article search tool that searches our Discovery system, but if your users start their search activity with Google, then the challenge is about going beyond Google Scholar to get library discovery up to the network level.