Responding to the results of user testing

Did you notice that we launched our new AtoM catalogue last week? I hope so!

In the month whilst preparing for launch we wanted to take the time to find out what a sample of users thought about our new catalogue and here I will summarise some of the findings and the steps that we have taken to react to this feedback.

We had 14 people test the catalogue for us off-site and fill out an online questionnaire which was put together using Google forms. Testing was carried out on AtoM version 2.2.0. The volunteers for user testing were found by putting out a call on Twitter and the results were helpful and constructive (though one user could not access the site so was not able to answer the questions in any meaningful way). Despite the small sample size there were several themes that were mentioned more than once. Interestingly these weren't necessarily the themes that we thought would be mentioned more than once!

Let's start with the positives....

The good things

It's always nice to receive positive feedback and we were encouraged to see that there was plenty of this to come out of the user testing - things that were praised fell into the following categories:


Look and feel - The vast majority of users found the catalogue visually appealing. A couple of people mentioned that they liked the colour scheme and one appreciated the fact that it flowed nicely from our website. The image on the home page was also praised. Others commented on the fact that it was well set out with a clean and clear appearance. One respondent compared it very favourably with other leading archival catalogues.

Our home page image

Functionality - The search functionality of the catalogue was praised as was the faceted classification that allows you to filter your search results. The browse by subject feature had several positive mentions and one person liked the ability to download XML files. Navigation within the catalogue was praised, including a specific comment about the tree-view feature on the left side of the interface.

The data - We were pleased to hear people saying good things about the quality of the data that we have in the catalogue. The information was described as being 'full' and 'comprehensive'. The level of detail held in the Conditions of Access and Use field was mentioned specifically and the fact that you could see when each description was last updated. One respondent stated that they liked the fact the catalogue conformed to recognised archival standards and that it was clear from the interface which rules had been used to create the data.

Digital objects - Several of the testers mentioned specifically that they liked the inclusion of digital objects within the catalogue. We have not utilised this feature to full effect just yet, but for some of our descriptions a finding aid or an image is available. Users liked the way that AtoM displays the thumbnails in the results list. An archival catalogue can be quite text-heavy so using digital objects to break the text up was seen as a good thing.

The help pages - Our glossary page had a positive mention. We put this together as we recognised that archival terminology can be a bit of a mystery to non-archivists (myself included) so being able to define some of the key terms we use was a priority for us.

My favourite comment under the question "What did you like about the catalogue?" was "Almost everything". This highlights to me that we have pretty much got it right but of course we shouldn't put our feet up - there is always room for improvement!

The not-so-good things

We also received comments about the things which weren't working so well in our new catalogue:

Look and feel - Of the users who did not think the catalogue was visually appealing, one comment was that it was 'bland' and that too much space on the front page was taken up by the image. The same person didn't like the fact that all the navigation was on the left and they couldn't find the search box. Another respondent thought that the links on the left hand side were too small and their eye wasn't drawn to them because of the large image on the front page. It was thought by one person that the location of the main image on the front page looked odd because it wasn't central.

Our response: We wondered about trying to increase the size of the text in the left hand navigation bar in order to make these links stand out a bit more but concluded that this may well upset the balance of the current design. Being that the majority of respondents were very happy with the visual appearance of the site, we decided that no changes were needed at this point in time.

Search box - The visibility of the search box was an issue that was raised a couple of times. We are using a slightly customised version of the default Dominion theme within AtoM and this puts the search box at the top of the screen. One person didn't find the search box at all whilst testing the catalogue. Another found it but wasn't immediately sure of its purpose as its location and proximity to the University of York logo suggested it would search our website rather than our catalogue. This may have been a direct result of our decision to style the catalogue to mirror the look and feel of our website as we do have a similar sized website search box in the top bar of our website.

Our response: We have given some serious thought to how to make the search box more prominent within AtoM but I'm not convinced there is an obvious solution to this. Prior to the user testing we had already changed the colour of the search box from dark grey to white to make it more visible. We have since made another minor tweak to the default theme to turn the 'Search' text within the search box from grey to black to make it stand out more. We considered making the search box bigger (longer) but our top bar is already getting quite crowded and filling it up any more than necessary does have knock on effects to the responsive design when viewed on smaller screens. 

While I can see a benefit to having the main search box taking centre stage on the catalogue front page, I also see it is useful having it up in the top bar so it is always accessible where ever a user is within the catalogue. We don't intend to make any further changes for the time being.

Search results - Several people mentioned that there were simply too many results when you carry out a search ...and the results that come up are not always relevant. We had already been discussing this very issue on the AtoM mailing list and were not surprised that our users were struggling with this. 

Our response: We are hoping that this is something that will be resolved in future versions of AtoM, but in the meantime we are focusing on educating our users by giving them the information they need in order to run more effective and precise searches (even just using the powerful functionality that is available within the basic search box). 

We think that a change to AtoM's default behaviour which currently searches for multiple words by default with an 'OR' operator rather than an 'AND' would produce search results that were more in line with what our users were expecting. Also, although users of Google will happily run a search that produces many thousands of results and feel comfortable not moving beyond the first ten 'hits', users of archival catalogues do not necessarily take the same approach. There seems to be more of an assumption that the list of results will be relevant and each should be worked through in turn. This is something we are definitely hoping for a solution to in the future.

Filtering the search results - One person expressed a desire to be able to filter a search by date

Our response: We agree that this would be a really useful feature and we were pleased to hear from Artefactual Systems that this will be possible within the next version of AtoM (2.3) which is due out soon. This will also introduce the ability to search within the date field in the advanced search and order results by start date in the results list. I think these features are going to be really valuable to our users.

Navigation - One person reported that the catalogue was hard to navigate but didn't give further details. Another struggled with navigation and described a scenario in which they had got lost within the catalogue. 

Our response: I can easily understand how someone could get lost within our catalogue - it has happened to me too! In some respects this problem is directly related to the powerful functionality of the AtoM interface and relational nature of the underlying data structure. Searching and browsing AtoM isn't a linear journey but rather an opportunity to follow links between one record and another based on shared subject terms or creators. Getting lost is a fairly inevitable consequence of this functionality and I struggle to think of an effective solution (apart from encouraging repeated use of the browser 'back' button to get back to where you started!)

The data - One user reported that there is "not much material yet" and another asked for more digitised documents. It was also mentioned that there were "not enough categories for searching" (we speculate that this might relate to the subject terms we have entered). Another comment received was about the term 'accrual' which is used as a field name within AtoM and also within the data that we enter in that field. It was suggested that this word might be a bit off-putting for some users. It was also mentioned that the lists within the Scope and Content field were"pretty hard reading" and a suggestion was made that this would be more user-friendly if presented as a bulleted list rather than a paragraph of text.

Our response: We did expect to get comments about our data. Just because we have launched our catalogue we do not consider it to be a finished piece of work. Further work on populating the catalogue and a fuller exploration of the functionality around digital objects will follow over the next couple of years. It was interesting to get the feedback about the word 'accrual' - we had actually anticipated much more feedback about the terminology that we use but hadn't considered this word in particular. I do agree that this word is a tricky one for non-archivists and I'm pretty sure I had not encountered it before I came to work at the Borthwick Institute. We don't want to change it on the basis of one comment but did decide to add the term to our glossary (one of the help pages we have created within AtoM) and hope that this helps our users.

The help pages - In our questionnaire we asked people specifically whether they used the catalogue help pages. The majority of users surveyed didn't use the help pages and this was not a surprising result. One person's reason for not using the help was because they "should not need to in a well designed information system". Another person stated that they preferred to "just see if I could use the catalogue instinctively". A couple of people mentioned that the page was too text heavy and someone else reported that they didn't know there were any help pages. Someone also suggested that the help pages should open in a new window.

Our response: As a result of the user testing we have made several changes to our help pages. We have updated the text (specifically to explain how to reduce the number of search results) and added a number of screenshots to help convey the information in a more visual way. 

Our help pages are now more visual and include screenshots - the first graphic simply shows how to access the search box. We have also created some printed and laminated copies of these for use in the searchroom.

Of course we can put a lot of effort into putting the right level of information into our help pages but we can not force people to use them! So, over the last couple of weeks we have been ensuring that our searchroom assistants (the people who will be providing front line support to our users as they grapple with our new catalogue) are aware of the different search options within AtoM and understand how they can be used to best affect.

There are also things we can do to make it clearer to users where the help pages are so that they can easily find them if they want to. By default the help pages in AtoM appear under an 'i' icon alongside other static pages. Replacing this 'i' icon with a '?' seemed to be a sensible step to take in order to make it clearer where help could be found. Artefactual Systems were able to point us to the relevant icon in Font Awesome which was just what we needed to implement this little change. 

We agreed that it may be useful for the help pages to open in a new tab so that someone could access them without losing their place within the catalogue (particularly being that 'getting lost' was also an issue that had been reported). Our help pages now open within a separate tab. We will monitor how users respond to this and whether the potential proliferation of tabs becomes a problem.

It has been a useful exercise reviewing this initial sample of responses and giving some thought to how AtoM and our own implementation of it can be improved. We will be continuing to gather user feedback through further more detailed testing with a smaller sample of users and by pulling together the ad hoc comments we are likely to receive now our catalogue is live. 

Jenny Mitcham, Digital Archivist