Responding to the results of user testing

Did you notice that we launched our new AtoM catalogue last week? I hope so!

In the month whilst preparing for launch we wanted to take the time to find out what a sample of users thought about our new catalogue and here I will summarise some of the findings and the steps that we have taken to react to this feedback.

We had 14 people test the catalogue for us off-site and fill out an online questionnaire which was put together using Google forms. Testing was carried out on AtoM version 2.2.0. The volunteers for user testing were found by putting out a call on Twitter and the results were helpful and constructive (though one user could not access the site so was not able to answer the questions in any meaningful way). Despite the small sample size there were several themes that were mentioned more than once. Interestingly these weren't necessarily the themes that we thought would be mentioned more than once!

Let's start with the positives....

The good things

It's always nice to receive positive feedback and we were encouraged to see that there was plenty of this to come out of the user testing - things that were praised fell into the following categories:


Look and feel - The vast majority of users found the catalogue visually appealing. A couple of people mentioned that they liked the colour scheme and one appreciated the fact that it flowed nicely from our website. The image on the home page was also praised. Others commented on the fact that it was well set out with a clean and clear appearance. One respondent compared it very favourably with other leading archival catalogues.

Our home page image

Functionality - The search functionality of the catalogue was praised as was the faceted classification that allows you to filter your search results. The browse by subject feature had several positive mentions and one person liked the ability to download XML files. Navigation within the catalogue was praised, including a specific comment about the tree-view feature on the left side of the interface.

The data - We were pleased to hear people saying good things about the quality of the data that we have in the catalogue. The information was described as being 'full' and 'comprehensive'. The level of detail held in the Conditions of Access and Use field was mentioned specifically and the fact that you could see when each description was last updated. One respondent stated that they liked the fact the catalogue conformed to recognised archival standards and that it was clear from the interface which rules had been used to create the data.

Digital objects - Several of the testers mentioned specifically that they liked the inclusion of digital objects within the catalogue. We have not utilised this feature to full effect just yet, but for some of our descriptions a finding aid or an image is available. Users liked the way that AtoM displays the thumbnails in the results list. An archival catalogue can be quite text-heavy so using digital objects to break the text up was seen as a good thing.

The help pages - Our glossary page had a positive mention. We put this together as we recognised that archival terminology can be a bit of a mystery to non-archivists (myself included) so being able to define some of the key terms we use was a priority for us.

My favourite comment under the question "What did you like about the catalogue?" was "Almost everything". This highlights to me that we have pretty much got it right but of course we shouldn't put our feet up - there is always room for improvement!

The not-so-good things

We also received comments about the things which weren't working so well in our new catalogue:

Look and feel - Of the users who did not think the catalogue was visually appealing, one comment was that it was 'bland' and that too much space on the front page was taken up by the image. The same person didn't like the fact that all the navigation was on the left and they couldn't find the search box. Another respondent thought that the links on the left hand side were too small and their eye wasn't drawn to them because of the large image on the front page. It was thought by one person that the location of the main image on the front page looked odd because it wasn't central.

Our response: We wondered about trying to increase the size of the text in the left hand navigation bar in order to make these links stand out a bit more but concluded that this may well upset the balance of the current design. Being that the majority of respondents were very happy with the visual appearance of the site, we decided that no changes were needed at this point in time.

Search box - The visibility of the search box was an issue that was raised a couple of times. We are using a slightly customised version of the default Dominion theme within AtoM and this puts the search box at the top of the screen. One person didn't find the search box at all whilst testing the catalogue. Another found it but wasn't immediately sure of its purpose as its location and proximity to the University of York logo suggested it would search our website rather than our catalogue. This may have been a direct result of our decision to style the catalogue to mirror the look and feel of our website as we do have a similar sized website search box in the top bar of our website.

Our response: We have given some serious thought to how to make the search box more prominent within AtoM but I'm not convinced there is an obvious solution to this. Prior to the user testing we had already changed the colour of the search box from dark grey to white to make it more visible. We have since made another minor tweak to the default theme to turn the 'Search' text within the search box from grey to black to make it stand out more. We considered making the search box bigger (longer) but our top bar is already getting quite crowded and filling it up any more than necessary does have knock on effects to the responsive design when viewed on smaller screens. 

While I can see a benefit to having the main search box taking centre stage on the catalogue front page, I also see it is useful having it up in the top bar so it is always accessible where ever a user is within the catalogue. We don't intend to make any further changes for the time being.

Search results - Several people mentioned that there were simply too many results when you carry out a search ...and the results that come up are not always relevant. We had already been discussing this very issue on the AtoM mailing list and were not surprised that our users were struggling with this. 

Our response: We are hoping that this is something that will be resolved in future versions of AtoM, but in the meantime we are focusing on educating our users by giving them the information they need in order to run more effective and precise searches (even just using the powerful functionality that is available within the basic search box). 

We think that a change to AtoM's default behaviour which currently searches for multiple words by default with an 'OR' operator rather than an 'AND' would produce search results that were more in line with what our users were expecting. Also, although users of Google will happily run a search that produces many thousands of results and feel comfortable not moving beyond the first ten 'hits', users of archival catalogues do not necessarily take the same approach. There seems to be more of an assumption that the list of results will be relevant and each should be worked through in turn. This is something we are definitely hoping for a solution to in the future.

Filtering the search results - One person expressed a desire to be able to filter a search by date

Our response: We agree that this would be a really useful feature and we were pleased to hear from Artefactual Systems that this will be possible within the next version of AtoM (2.3) which is due out soon. This will also introduce the ability to search within the date field in the advanced search and order results by start date in the results list. I think these features are going to be really valuable to our users.

Navigation - One person reported that the catalogue was hard to navigate but didn't give further details. Another struggled with navigation and described a scenario in which they had got lost within the catalogue. 

Our response: I can easily understand how someone could get lost within our catalogue - it has happened to me too! In some respects this problem is directly related to the powerful functionality of the AtoM interface and relational nature of the underlying data structure. Searching and browsing AtoM isn't a linear journey but rather an opportunity to follow links between one record and another based on shared subject terms or creators. Getting lost is a fairly inevitable consequence of this functionality and I struggle to think of an effective solution (apart from encouraging repeated use of the browser 'back' button to get back to where you started!)

The data - One user reported that there is "not much material yet" and another asked for more digitised documents. It was also mentioned that there were "not enough categories for searching" (we speculate that this might relate to the subject terms we have entered). Another comment received was about the term 'accrual' which is used as a field name within AtoM and also within the data that we enter in that field. It was suggested that this word might be a bit off-putting for some users. It was also mentioned that the lists within the Scope and Content field were"pretty hard reading" and a suggestion was made that this would be more user-friendly if presented as a bulleted list rather than a paragraph of text.

Our response: We did expect to get comments about our data. Just because we have launched our catalogue we do not consider it to be a finished piece of work. Further work on populating the catalogue and a fuller exploration of the functionality around digital objects will follow over the next couple of years. It was interesting to get the feedback about the word 'accrual' - we had actually anticipated much more feedback about the terminology that we use but hadn't considered this word in particular. I do agree that this word is a tricky one for non-archivists and I'm pretty sure I had not encountered it before I came to work at the Borthwick Institute. We don't want to change it on the basis of one comment but did decide to add the term to our glossary (one of the help pages we have created within AtoM) and hope that this helps our users.

The help pages - In our questionnaire we asked people specifically whether they used the catalogue help pages. The majority of users surveyed didn't use the help pages and this was not a surprising result. One person's reason for not using the help was because they "should not need to in a well designed information system". Another person stated that they preferred to "just see if I could use the catalogue instinctively". A couple of people mentioned that the page was too text heavy and someone else reported that they didn't know there were any help pages. Someone also suggested that the help pages should open in a new window.

Our response: As a result of the user testing we have made several changes to our help pages. We have updated the text (specifically to explain how to reduce the number of search results) and added a number of screenshots to help convey the information in a more visual way. 

Our help pages are now more visual and include screenshots - the first graphic simply shows how to access the search box. We have also created some printed and laminated copies of these for use in the searchroom.

Of course we can put a lot of effort into putting the right level of information into our help pages but we can not force people to use them! So, over the last couple of weeks we have been ensuring that our searchroom assistants (the people who will be providing front line support to our users as they grapple with our new catalogue) are aware of the different search options within AtoM and understand how they can be used to best affect.

There are also things we can do to make it clearer to users where the help pages are so that they can easily find them if they want to. By default the help pages in AtoM appear under an 'i' icon alongside other static pages. Replacing this 'i' icon with a '?' seemed to be a sensible step to take in order to make it clearer where help could be found. Artefactual Systems were able to point us to the relevant icon in Font Awesome which was just what we needed to implement this little change. 

We agreed that it may be useful for the help pages to open in a new tab so that someone could access them without losing their place within the catalogue (particularly being that 'getting lost' was also an issue that had been reported). Our help pages now open within a separate tab. We will monitor how users respond to this and whether the potential proliferation of tabs becomes a problem.

It has been a useful exercise reviewing this initial sample of responses and giving some thought to how AtoM and our own implementation of it can be improved. We will be continuing to gather user feedback through further more detailed testing with a smaller sample of users and by pulling together the ad hoc comments we are likely to receive now our catalogue is live. 

Jenny Mitcham, Digital Archivist

Our catalogue is now live!

Was it really 3.5 years ago when I first blogged about requirements for a new archival management system?

My main aim in getting involved in this project was to create a stable base to build a digital archive on.

If you build a digital archive on wobbly foundations there is a strong chance that it will fall over.

Much safer to build it on top of a system established as the single point of truth for all accessions information your organisation holds. A system which will become the means by which you disseminate information about your digital holdings (alongside the physical ones) and enable users to access copies of born digital and digitised material.

Finally we have such a solution in place!

We chose Access to Memory (AtoM) as our new archival management system, and over the last few years there has been a huge amount of work going on behind the scenes getting it up and running. I'm so pleased that today we are in a position to unveil the results of all of that hard work.

Our new catalogue can be viewed at https://borthcat.york.ac.uk/

In a previous blog post "A is for AtoM" I talked about some of the tasks that have been going on and decisions that have been made to get us up and running, so I won't repeat all of that here.

Suffice to say that a considerable amount of work has gone in to getting AtoM installed, configured and styled. While this has been going on, Project Genesis has been key to getting the catalogue populated with archival descriptions. The task of populating our catalogue will continue via project Genesis until April 2017 and by other channels beyond that.

While our initial focus has been to get a collection level description for each of our archives into the catalogue, further work is required on the wider task of retroconversion - getting a variety of finding aids in a range of different formats into the system. We have managed to tackle some of this in an ad hoc way but there is still much to do.

Our AtoM catalogue is live, but our work is not yet done. I need to start thinking about how we can build digital preservation functionality on top of this (via Archivematica) and of course how we can start to provide more access to our digital holdings through the catalogue interface. Watch this space!

In the meantime, we'd be happy to hear any feedback about our catalogue so do get in touch.




Jenny Mitcham, Digital Archivist

Kicking off phase 3 of "Filling the Digital Preservation Gap"

I realise I've gone a bit quiet on "Filling the Digital Preservation Gap" since the release of our phase 2 project report. I am pleased to pass on the news that we have been funded by Jisc to continue some of our work into Phase 3.

Our Research Data Spring phase 3 kick off meeting was held yesterday at the Hull History Centre and we celebrated with a suitably spring-themed cake!

Our Research Data Spring chicken cake

So here is a run down of what we are planning to do in phase 3:

The big one at the top of the list is Archivematica implementation. Both York and Hull are going to be working on their own proof of concept implementations of Archivematica integrated with their existing repositories (and potentially other systems within the RDM workflow). We may not be able to follow the implementation plans from our phase 2 report in full (as we have not been funded in full) but both institutions plan to get an implementation up and running with a focus on a single use case.

I for one am very excited about this implementation phase. This is what our work over the previous two phases has been leading up to. The ground work laid in phases 1 and 2 has been incredibly valuable, but it will be great to move from talking about Archivematica to actually working with it!

We are also going to continue to look at the issue of unidentified file formats. This has been a recurring theme during phases 1 and 2 and is particularly pertinent for research data which comes in such a huge variety of formats. We are going to work with The National Archives to ensure a few more research data file formats are represented in PRONOM. We will also give further thought to our workflows for handling unidentified files and how tools such as Archivematica can help.

We will of course be continuing our dissemination and outreach work. Some of this has already happened over the last couple of months.

• I gave a presentation at the IDCC16 conference in Amsterdam in February and discussed why active digital preservation is often left out of RDM workflows - the slides can be viewed here
• Julie Allinson presented a case study about our project at a workshop entitled 'Digital Preservation: Strategic Issues' at the National Library of Wales in February
• Myself and Simon Wilson from Hull produced a poster for the UK Archives Discovery Forum last month to promote some of the themes of our project so far and make sure the wider archives community is aware of our work

Our UKAD 2016 poster

• At the UK Archivematica meeting last month I gave a presentation which summarised the outcomes of the development work we funded in phase 2. This can be found here

Watch out for us at 'Research Data, Records and Archives: Breaking the Boundaries' in Edinburgh later this month and Open Repositories in Dublin in June.

Of course we will also be keeping you posted on this blog as phase 3 of our project progresses, so watch this space

Jenny Mitcham, Digital Archivist