Showing posts with label EAD. Show all posts
Showing posts with label EAD. Show all posts

Monday, 18 September 2017

Harvesting EAD from AtoM: we need your help!

Back in February I published a blog post about a project to develop AtoM to allow EAD (Encoded Archival Description) to be harvested via OAI-PMH (Open Archives Initiative Protocol for Metadata Harvesting): “Harvesting EAD from AtoM: a collaborative approach”

Now that AtoM version 2.4 is released (hooray!), containing the functionality we have sponsored, I thought it was high time I updated you on what has been achieved by this project, where more work is needed and how the wider AtoM community can help.


What was our aim?


Our development work had a few key aims:

  • To enable finding aids from AtoM to be exposed as EAD 2002 XML for others to harvest. The partners who sponsored this project were particularly keen to enable the Archives Hub to harvest their EAD.
  • To change the way that EAD was generated by AtoM in order to make it more scalable. Moving EAD generation from the web browser to the job scheduler was considered to be the best approach here.
  • To make changes to the existing DC (Dublin Core) metadata generation feature so that it also works through the job scheduler - making this existing feature more scalable and able to handle larger quantities of data

A screen shot of the job scheduler in AtoM - showing the EAD and
DC creation jobs that have been completed

What have we achieved?

The good

We believe that the EAD harvesting feature as released in AtoM version 2.4 will enable a harvester such as the Archives Hub to harvest our catalogue metadata from AtoM as EAD. As we add new top level archival descriptions to our catalogue, subsequent harvests should pick up and display these additional records. 

This is a considerable achievement and something that has been on our wishlist for some time. This will allow our finding aids to be more widely signposted. Having our data aggregated and exposed by others is key to ensuring that potential users of our archives can find the information that they need.

Changes have also been made to the way metadata (both EAD and Dublin Core) are generated in AtoM. This means that the solution going forward is more scalable for those AtoM instances that have very large numbers of records or large descriptive hierarchies.

The new functionality in AtoM around OAI-PMH harvesting of EAD and settings for moving XML creation to the job scheduler is described in the AtoM documentation.

The not-so-good

Unfortunately the EAD harvesting functionality within AtoM 2.4 will not do everything we would like it to do. 

It does not at this point include the ability for the harvester to know when metadata records have been updated or deleted. It also does not pick up new child records that are added into an existing descriptive hierarchy. 

We want to be able to edit our records once within AtoM and have any changes reflected in the harvested versions of the data. 

We don’t want our data to become out of sync. 

So clearly this isn't ideal.

The task of enabling full harvesting functionality for EAD was found to be considerably more complex than first anticipated. This has no doubt been confounded by the hierarchical nature of the EAD which differs from the simplicity of the traditional Dublin Core approach.

The problems encountered are certainly not insurmountable, but lack of additional resources and timelines for the release of AtoM 2.4 stopped us from being able to finish off this work in full.

A note on scalability


Although the development work deliberately set out to consider issues of scalability, it turns out that scalability is actually on a sliding scale!

The National Library of Wales had the forethought to include one of their largest archival descriptions as sample data for inclusion in the version of AtoM 2.4 that Artefactual deployed for testing. Their finding aid for St David’s Diocesan Records is a very large descriptive hierarchy consisting of 33,961 individual entries. This pushed the capabilities of EAD creation (even when done via the job scheduler) and also led to discussions with The Archives Hub about exactly how they would process and display such a large description at their end even if EAD generation within AtoM were successful.

Some more thought and more manual workarounds will need to be put in place to manage the harvesting and subsequent display of large descriptions such as these.

So what next?


We are keen to get AtoM 2.4 installed at the Borthwick Institute for Archives over the next couple of months. We are currently on version 2.2 and would like to start benefiting from all the new features that have been introduced available... and of course to test in earnest the EAD harvesting feature that we have jointly sponsored.

We already know that this feature will not fully meet our needs in its current form, but would like to set up an initial harvest with the Archives Hub and further test some of our assumptions about how this will work.

We may need to put some workarounds in place to ensure that we have a way of reflecting updates and deletions in the harvested data – either with manual deletes or updates or a full delete and re-harvest periodically.

Harvesting in AtoM 2.4 - some things that need to change


So we have a list of priority things that need to be improved in order to get EAD harvesting working more smoothly in the future:


In line with the OAI-PMH specification: 

  • AtoM needs to expose updates to the metadata to the harvester
  • AtoM needs to expose new records (at any level of description) to the harvester
  • AtoM needs to expose information about deletions to the harvester
  • AtoM also needs to expose information about deletions to DC metadata to the harvester (it has come to my attention during the course of this project that this isn’t happening at the moment) 

Some other areas of potential work


I also wanted to bring together and highlight some other areas of potential work for the future. These are all things that were discussed during the course of the project but were not within the scope of our original development goals.

  • Harvesting of EAC (Encoded Archival Context) - this is the metadata standard for authority records. Is this something people would like to see enabled in the future? Of course this is only useful if you have someone who actually wants to harvest this information!
  • On the subject of authority records, it would be useful to change the current AtoM EAD template to use @authfilenumber and @source - so that an EAD record can link back to the relevant authority record in the local AtoM site. The ability to create rich authority records is such a key strength of AtoM, allowing an institution to weave rich interconnecting stories about their holdings. If harvesting doesn’t preserve this inter-connectivity then I think we are missing a trick!
  • EAD3 - this development work has deliberately not touched on the new EAD standard. Firstly, this would have been a much bigger job and secondly, we are looking to have our EAD harvested by The Archives Hub and they are not currently working with EAD3. This may be a priority area of work for the future.
  • Subject source - the subject source (for example "Library of Congress Subject Headings") doesn't appear in AtoM generated EAD at the moment even though it can be entered into AtoM - this would be a really useful addition to the EAD.
  • Visible elements - AtoM allows you to decide which elements you wish to display/hide in your local AtoM interface. With the exception of information relating to physical storage, the XML generation tasks currently do not take account of visible elements and will carry out an export of all fields. Further investigation of this should be carried out in the future. If an institution is using the visible elements feature to hide certain bits of information that should not be more widely distributed, they would be concerned if this information was being harvested and displayed elsewhere. As certain elements will be required in order to create valid EAD, this may get complicated!
  • ‘Manual’ EAD generation - the project team discussed the possibility of adding a button to the AtoM user interface so that staff users can manually kick-off EAD regeneration for a single descriptive hierarchy. Artefactual suggested this as a method of managing the process of EAD generation for large descriptive hierarchies. You would not want the EAD to regenerate with each minor tweak if a large archival description was undergoing several updates, however, you need to be able to trigger this task when you are ready to do so. It should be possible to switch off the automatic EAD re-generation (which normally triggers when a record is edited and saved) but have a button on the interface that staff can click when they want to initiate the process - for example when all edits are complete. 
  • As part of their work on this project, Artefactual created a simple script to help with the process of generating EAD for large descriptive hierarchies - it basically provides a way of finding out which XML files relate to a specific archival description so that EAD can be manually enhanced and updated if it is too large for AtoM to generate via the job scheduler. It would be useful to turn this script into a command-line task that is maintained as part of the AtoM codebase.

We need your help!


Although we believe we have something we can work with here and now, we are not under any illusions that this feature does all that it needs to in order to meet our requirements in the longer term. 

I would love to find out what other AtoM users (and harvesters) think of the feature. Is it useful to you? Are there other things we should put on the wishlist? 

There is a lot of additional work described in this post which the original group of project partners are unlikely to be able to fund on their own. If EAD harvesting is a priority to you and your organisation and you think you can contribute to further work in this area either on your own or as part of a collaborative project please do get in touch.


Thanks


I’d like to finish with a huge thanks to those organisations who have helped make this project happen, either through sponsorship, development or testing and feedback.




Jenny Mitcham, Digital Archivist

Friday, 16 June 2017

A typical week as a digital archivist?

Sometimes (admittedly not very often) I'm asked what I actually do all day. So at the end of a busy week being a digital archivist I've decided to blog about what I've been up to.

Monday

Today I had a couple of meetings. One specifically to talk about digital preservation of electronic theses submissions. I've also had a work experience placement in this week so have set up a metadata creation task which he has been busy working on.

When I had a spare moment I did a little more testing work on the EAD harvesting feature the University of York is jointly sponsoring Artefactual Systems to develop in AtoM. Testing this feature from my perspective involves logging into the test site that Artefactual has created for us and tweaking some of the archival descriptions. Once those descriptions are saved, I can take a peek at the job scheduler and make sure that new EAD files are being created behind the scenes for the Archives Hub to attempt to harvest at a later date.

This piece of development work has been going on for a few months now and communications have been technically quite complex so I'm also trying to ensure all the organisations involved are happy with what has been achieved and will be arranging a virtual meeting so we can all get together and talk through any remaining issues.

I was slightly surprised today to have a couple of requests to talk to the media. This has sprung from the news that the Queen's Speech will be delayed. One of the reasons for the delay relates to the fact that the speech has to be written on goat's skin parchment, which takes a few days to dry. I had previously been interviewed for a article entitled Why is the UK still printing its laws on vellum? and am now mistaken for someone who knows about vellum. I explained to potential interviewers that this is not my specialist subject!

Tuesday

In the morning I went to visit a researcher at the University of York. I wanted to talk to him about how he uses Google Drive in relation to his research. This is a really interesting topic to me right now as I consider how best we might be able to preserve current research datasets. Seeing how exactly Google Drive is used and what features the researcher considers to be significant (and necessary for reuse) is really helpful when thinking about a suitable approach to this problem. I sometimes think I work a little bit too much in my own echo chamber, so getting out and hearing different perspectives is incredibly valuable.

Later that afternoon I had an unexpected meeting with one of our depositors (well, there were two of them actually). I've not met them before but have been working with their data for a little while. In our brief meeting it was really interesting to chat and see the data from a fresh perspective. I was able to reunite them with some digital files that they had created in the mid 1980's, had saved on to floppy disk and had not been able to access for a long time.

Digital preservation can be quite a behind the scenes sort of job - we always give a nod to the reason why we do what we do (ie: we preserve for future reuse), but actually seeing the results of that work unfold in front of your eyes is genuinely rewarding. I had rescued something from the jaws of digital obsolescence so it could now be reused and revitalised!

At the end of the day I presented a joint webinar for the Open Preservation Foundation called 'PRONOM in practice'. Alongside David Clipsham (The National Archives) and Justin Simpson (Artefactual Systems), I talked about my own experiences with PRONOM, particularly relating to file signature creation, and ending with a call to arms "Do try this at home!". It would be great if more of the community could get involved!

I was really pleased that the webinar platform worked OK for me this time round (always a bit stressful when it doesn't) and that I got to use the yellow highlighter pen on my slides.

In my spare moments (which were few and far between), I put together a powerpoint presentation for the following day...

Wednesday

I spent the day at the British Library in Boston Spa. I'd been invited to speak at a training event they regularly hold for members of staff who want to find out a bit more about digital preservation and the work of the team.

I was asked specifically to talk through some of the challenges and issues that I face in my work. I found this pretty easy - there are lots of challenges - and I eventually realised I had too many slides so had to cut it short! I suppose that is better than not having enough to say!

Visiting Boston Spa meant that I could also chat to the team over lunch and visit their lab. They had a very impressive range of old computers and were able to give me a demonstration of Kryoflux (which I've never seen in action before) and talk a little about emulation. This was a good warm up for the DPC event about emulation I'm attending next week: Halcyon On and On: Emulating to Preserve.

Still left on my to do list from my trip is to download Teracopy. I currently use Foldermatch for checking that files I have copied have remained unchanged. From the quick demo I saw at the British Library I think that Teracopy would be a more simple one step solution. I need to have a play with this and then think about incorporating it into the digital ingest workflow.

Sharing information and collaborating with others working in the digital preservation field really is directly beneficial to the day to day work that we do!

Thursday

Back in the office today and a much quieter day.

I extracted some reports from our AtoM catalogue for a colleague and did a bit of work with our test version of Research Data York. I also met with another colleague to talk about storing and providing access to digitised images.

In the afternoon I wrote another powerpoint presentation, this time for a forthcoming DPC event: From Planning to Deployment: Digital Preservation and Organizational Change.

I'm going to be talking about our experiences of moving our Research Data York application from proof of concept to production. We are not yet in production and some of the reasons why will be explored in the presentation! Again I was asked to talk about barriers and challenges and again, this brief is fairly easy to fit! The event itself is over a week away so this is unprecedentedly well organised. Long may it continue!


Friday

On Fridays I try to catch up on the week just gone and plan for the week ahead as well as reading the relevant blogs that have appeared over the week. It is also a good chance to catch up with some admin tasks and emails.

Lunch time reading today was provided by William Kilbride's latest blog post. Some of it went over my head but the final messages around value and reuse and the need to "do more with less" rang very true.

Sometimes I even blog myself - as I am today!




Was this a typical week - perhaps not, but in this job there is probably no such thing! Every week brings new ideas, challenges and surprises!

I would say the only real constant is that I've always got lots of things to keep me busy.

Jenny Mitcham, Digital Archivist

Friday, 10 February 2017

Harvesting EAD from AtoM: a collaborative approach

In a previous blog post AtoM harvesting (part 1) - it works! I described how archival descriptions within AtoM are being harvested as Dublin Core for inclusion within our University Library Catalogue.* I also hinted that this wouldn’t be the last you would hear from me on AtoM harvesting and that plans were afoot to enable much richer metadata in EAD 2002 XML (Encoded Archival Description) format to be harvested via OAI-PMH.

I’m pleased to be able to report that this work is now underway.

The University of York along with five other organisations in the UK have clubbed together to sponsor Artefactual Systems to carry out the necessary development work to make EAD harvesting possible. This work is scheduled for release in AtoM version 2.4 (due out in the Spring).

The work is being jointly sponsored by:



We are also receiving much needed support in this project from The Archives Hub who are providing advice on the AtoM EAD and will be helping us test the EAD harvesting when it is ready. While the sponsoring institutions are all producers of AtoM EAD, The Archives Hub is a consumer of that EAD. We are keen to ensure that the archival descriptions that we enter into AtoM can move smoothly to The Archives Hub (and potentially to other data aggregators in the future), allowing the richness of our collections to be signposted as widely as possible.

Adding this harvesting functionality to AtoM will enable The Archives Hub to gather data direct from us on a regular schedule or as and when updates occur, ensuring that:


  • Our data within the Archives Hub doesn’t stagnate
  • We manage our own master copy of the data and only need to edit this in one place
  • A minimum of human interaction is needed to incorporate our data into the Hub
  • It is easier for researchers to find information about the archives that we hold without having to search all of our individual catalogues


So, what are we doing at the moment?


  • Developers at Artefactual Systems are beavering away working on the initial development and getting the test site ready for us to play with.
  • The sponsoring institutions have been getting samples of their own AtoM data ready for loading up into the test deployment. It is always better when testing something to have some of your own data to mess around with.
  • The Borthwick have been having discussions with The Archives Hub for some time about AtoM EAD (from version 2.2) but we’ve picked up these discussions again and other institutions have joined in by supplying their own EAD samples. This allows staff at the Hub to see how EAD has changed in version 2.3 of AtoM (it hasn’t very much) and also to see how consistent the EAD from AtoM is from different institutions. We have been having some pretty detailed discussions about how we can make the EAD better, cleaner, fuller - either by data entry at the institutions, automated data cleaning at The Hub prior to display online or by further developments in AtoM.


What we are doing at the moment is good and a huge step in the right direction, but perhaps not perfect. As we work together on this project we are coming across areas where future work would be beneficial in order to improve the quality of the EAD that AtoM produces or to expand the scope of what can be harvested from AtoM. I hope to report on this in more detail at the end of the project, but in the meantime, do get in touch if you are interested in finding out more.







* It is great to see that this is working well and our Library Catalogue is now appearing in the referrer reports for the Borthwick Catalogue on Google Analytics. People are clearly following these new signposts to our archives!

Jenny Mitcham, Digital Archivist

Tuesday, 15 November 2016

AtoM harvesting (part 1) - it works!

When we first started using Access to Memory (AtoM) to create the Borthwick Catalogue we were keen to enable our data to be harvested via OAI-PMH (more about this feature of AtoM is available in the documentation). Indeed the ability to do this was one of our requirements when we were looking to select a new Archival Management System (read about our system requirements here).

Look! Archives now available in Library Catalogue search
So it is with great pleasure that I can announce that we are now exposing some of our data from AtoM through our University Library catalogue YorSearch. Dublin Core metadata is automatically harvested nightly from our production AtoM instance - so we don't need to worry about manual updates or old versions of our data hanging around.

Our hope is that doing this will allow users of the Library Catalogue (primarily staff and students at the University of York) to happen upon relevant information about the archives that we hold here at the Borthwick whilst they are carrying out searches for other information resources.

We believe that enabling serendipitous discovery in this way will benefit those users of the Library Catalogue who may have no idea of the extent and breadth of our holdings and who may not know that we hold archives of relevance to their research interests. Increasing the visibility of the archives within the University of York is an useful way of signposting our holdings and we think this should bring benefits both to us and our potential user base.

A fair bit of thought (and a certain amount of tweaking within YorSearch) went into getting this set up. From the archives perspective, the main decision was around exactly what should be harvested. It was agreed that only top level records from the Borthwick Catalogue should be made available in this way. If we had enabled the harvesting of all levels of records, there was a risk that search results would have been swamped by hundreds of lower level records from those archives that have been fully catalogued. This would have made the search results difficult to understand, particularly given the fact that these results could not have been displayed in a hierarchical way so the relationships between the different levels would be unclear. We would still encourage users to go direct to the Borthwick Catalogue itself to search and browse lower levels of description.

It should also be noted that only a subset of the metadata within the Borthwick Catalogue will be available through the Library Catalogue. The metadata we create within AtoM is compliant with ISAD(G): General International Standard Archival Description which contains 26 different data elements. In order to facilitate harvesting using OAI-PMH, data within AtoM is mapped to simple Dublin Core and this information is available for search and retrieval via YorSearch. As you can see from the screen shot below, Dublin Core does allow a useful level of information to be harvested, but it is not as detailed as the original record.

An example of one of our archival descriptions converted to Dublin Core within YorSearch

Further work was necessary to change the default behaviour within Primo (the software that YorSearch runs on) which displayed results from the Borthwick Catalogue with the label Electronic resource. This is what it calls anything that is harvested as Dublin Core. We didn't think this would be helpful to users because even though the finding aid itself (within AtoM) is indeed an electronic resource, the actual archive that it refers to isn't. We were keen that users didn't come to us expecting everything to be digitised! Fortunately it was possible to change this label to Borthwick Finding Aid, a term that we think will be more helpful to users.
Searches within our library catalogue (YorSearch) now surface Borthwick finding aids, harvested from AtoM.
These are clearly labelled as Borthwick Finding Aids.


Click through to a Borthwick Finding Aid and you can see the full archival description in AtoM in an iFrame

Now this development has gone live we will be able to monitor the impact. It will be interesting to see whether traffic to the Borthwick Catalogue increases and whether a greater number of University of York staff and students engage with the archives as a result.

However, note that I called this blog post AtoM harvesting (part 1).

Of course that means we would like to do more.

Specifically we would like to move beyond just harvesting our top level records as Dublin Core and enable harvesting of all of our archival descriptions in full in Encoded Archival Description (EAD) - an XML standard that is closely modelled on ISAD(G).  This is currently not possible within AtoM but we are hoping to change this in the future.

Part 2 of this blog post will follow once we get further along with this aim...






Jenny Mitcham, Digital Archivist

Friday, 18 March 2016

'A' is for AtoM

Over the past couple of years we have been busy working away behind the scenes on our implementation of Access to Memory (AtoM) at the Borthwick Institute for Archives and very soon we will be launching our new catalogue to the public.

I haven’t said much about AtoM on this blog thus far but it has been a huge preoccupation over the last couple of years. Here I attempt to redress that balance.

It turns out that deciding to adopt a system is relatively simple, working out exactly how you are going to use it is far more complex!

What follows is a list of just some of the things we have been thinking about and working on over the last couple of years as we move towards launch. I present you with the A to Z* of implementing AtoM….

A is for Accession Mask

We were very keen to use AtoM for accessioning…in fact the need to urgently find a new system for recording our accessions was the key driver for getting us moving with AtoM in the first place.

As we started using AtoM for recording new accessions we realised we needed to get the accessions mask right. This is just one of the configuration options within AtoM and it enables you to create unique references for your accessions. We wanted to ensure that our new accession numbers were in the same format as our previous ones so with a  little bit of help and advice from the AtoM user forum settled on the mask “%Y/%iii” which creates  numbers in our preferred format of [yyyy]/[no]. Now we just need to remember to reset the accession number to ‘0’ at the start of each new year so that our running number sequence starts again. This is just one of the ways that an institution can configure AtoM to suit local preferences.


B is for Business as Usual

Any organisation when adopting a new and complex system like AtoM needs to think beyond initial implementation and consider how the solution can be embedded into their workflows for the longer term? The ultimate goal for us is getting AtoM seen as 'business as usual' at the Borthwick. We are not there yet (though perhaps we almost are when it comes to working with accessions data). Getting us to the point where AtoM is not a standing item on our meeting agendas is something to aim for in the future!

C is for Customising the look and feel

AtoM gives you some options for customising of the look and feel of the front end. Being that the AtoM interface is going to be the primary means through which our users will browse and view information about our holdings, we want the interface to look consistent with our other communications. It needs to be clear that it belongs to us. Using our brand colours was a quick win and we also put some additional effort into creating an attractive image for the home page to make it look more visually appealing.

Note that there is a limit to the level of customisation that can be done without developer support. Within the admin interface of AtoM some basic changes to theme colours can be made, but I quickly found that changing the background colour to our Borthwick orange did not look pretty! Much better to call in our local technical experts to tweak the CSS behind the scenes.

D is for Drop Down Lists

AtoM comes ready populated with wordlists (called taxonomies) that populate the drop down lists to support data entry, however, institutions can change these to meet their own local needs. We have had to tweak a few of the taxonomies within AtoM, for example the deposit types in the accessions section and the levels of description (after much internal debate!).

E is for Experimenting

In order to understand AtoM we knew we really need to get some of our data into it. We experimented with some structured finding aids that already existed in EAD format and had a go at importing them. We discovered that data may not always import in the way you expect.

One of the key problem areas for us has been the way AtoM handles the <bioghist> element in our EAD files. The issue is documented here. Essentially what it tends to mean for us is that we end up with lots of untitled authority records when we import an EAD finding aid. This has been a bit of a barrier for us in getting more of our existing catalogues into AtoM. Experimenting and carrying out tests to check the behaviour however, does allow us to consider how we can tackle the issue and work towards a solution for future data imports.

F is for Friendly Advice

Though there is much detail in the AtoM documentation, anyone starting to use a new system such as AtoM will inevitably get to the point where they need to speak to someone, or see another implementation. The AtoM mailing list and the staff at Artefactual Systems are friendly and helpful and it is easy to get quick answers to specific questions. It is also incredibly valuable to have a local AtoM user to talk to, to bounce random questions off (particularly ones that may sound too silly or trivial for the mailing list!).

G is for Give it a Name

In the last few weeks before AtoM launch it occurred to us that we needed to decide what to call it. Internally we have simply been calling it ‘AtoM’ but we realised that this label is of little use to our users. As we started to finalise the interface and prepare the publicity for launch date we agreed that we would call it the ‘Borthwick Catalogue’. Perhaps not very imaginative but it is at the very least a concise description of its content and purpose!

H is for Help Pages

An online archival catalogue is quite a complex thing and we are aware that some of our users may be a bit daunted by it. Help pages are therefore really important to describe how to search and filter the results.

AtoM comes with some standard static pages, that can be very easy edited. We've been working on our help pages and expect we will be editing these further once we have completed our user testing. We have also created another static page to act as a glossary of archival terms. Although one of AtoM's big selling points for us was the fact it was aligned with archival standards and terminology, we are concerned that our users may struggle with some of the language used.

I is for ISDIAH

Within AtoM the archival descriptions from an institution all link back to an ISDIAH record that describes the archival institution. This record is useful for users of our data, whether browsing within the AtoM interface directly or through aggregators.

We have had some internal debate on the extent  to which we should replicate information that is on our website, but have decided that providing links to the relevant content would be better in many cases. For information about access and opening hours and the extent of our holdings, we want to ensure that the information is accurate and up to date, and having another place where this information would need to be edited adds an extra overhead.

J is for Just Start!

For a while we were stuck in a chicken and egg situation. Not sure how to use AtoM until it was set up properly and ready to go, and not sure how to set it up until we had started using it and fully understood the issues we would encounter.

Reading the documentation is essential but testing and experimenting with AtoM are really the best ways of working it out. Only by importing different datasets into AtoM or by creating new ones direct into the web form did we really understand how it worked and how this impacted on our own internal workflows. Learn by doing!

K is for Kittens (because they are never really free)

AtoM is open source and freely available for all. However, Artefactual Systems who support it stress it is “free as in free kittens”. In other words, you can have AtoM for free but it isn’t cost neutral - you need someone to install it, manage the server, configure it, and administer it. Populating it is also going to require a huge outlay of staff time.

On top of this, there will undoubtedly be things that you want AtoM to do that it doesn't yet do. If you are implementing AtoM, have a budget for funding further developments. Sponsored developments will then benefit the wider AtoM community and together we can make AtoM better and better. Quite early on in our AtoM implementation project we funded a small piece of work to include covering dates within the accessions module of AtoM as we felt that this was important information to record during the accessioning process and we did not want to lose this data from our existing accessions records when we imported them into the system. Of course we are hoping this feature will also be valuable for other AtoM users. There will undoubtedly be other feature developments we will sponsor in the future.

L is for Local Guidance

One of AtoM’s key selling points to us was the fact that it was created in association with the International Council on Archives (ICA) and is closely aligned with their metadata standards. There is however still a need for local guidance on how we intend to use some of these metadata fields.

In response to this we have created our own AtoM handbook to sit alongside the documentation that Artefactual provides. The handbook doesn't duplicate the official documentation, but describes our local procedures and requirements for data entry. This is all the more necessary given the fact that the majority of the data fields within AtoM are free text fields. With multiple users entering data into AtoM, it is important to have local guidance to ensure we maintain some consistency in the way we describe our archives.

M is for MySQL access

When we initially assessed AtoM against our requirements for an archival management system, it performed well but it didn't do everything we needed it to do.  Searching and reporting functionality within AtoM does not currently meet all of our needs. It was considered essential then that we had another method of querying the data within AtoM and producing reports and statistics. To do this, we need access to the MySQL database that sits behind AtoM.

Access to the the data via a free tool (I use Squirrel but there are other options out there) and a working knowledge of Structured Query Language allows you to do pull out exactly the data you require.

AtoM has quite a complex and involved data structure so getting to grips with this was a bit of a learning curve, but having now got a working query to enable me to extract an annual summary of all accessions we have received over a given year I feel ready for the next challenge that is thrown my way!

N is for Not Perfect

AtoM (like all complex systems) has its limitations. It ticks many boxes for us but it does not tick them all. There are several areas where we think it could improve and we have been discussing these with the user community and developers and hope to influence its roadmap. As with all open source solutions, rather than complaining about what it doesn't do well, the user community should be working together to solve problems and support improvements. AtoM is not perfect but we are confident that it is moving in the right direction and getting better all the time.

O is for Objects (digital ones!)

One of the main reasons I got involved with AtoM implementation was because I wanted a stable base to build a digital archive on – a single point of truth about our holdings and a single system through which our users could access information about our holdings. Being able to expose access copies of our born digital archives and digitised content via AtoM is something we haven’t yet explored in full but this work will become a priority over the next couple of years. Once AtoM is launched I will be turning my attention back to Archivematica in order to help get this moving.

P is for Populating AtoM

This is undoubtedly the biggest challenge we have. Over the course of the 60 years we have been in existence, the Borthwick has created a wealth of catalogues and finding aids. Of course, these are in a range of different formats and states of completion. Some are digital, some are not. Of the digital ones, some are structured data and some are not. Some comply to modern archival standards and some don’t. Some are complete but some do not include information about more recent accruals to the archives. Just working out the current state of play is a challenge in itself.

Being both pragmatic and realistic about what is achievable is a good place to start. Getting all of this information into AtoM is a huge task and not something we can do quickly. While we have managed to enter some full finding aids into AtoM, we have not had the staff time to do as much as we would have liked. What we have prioritised though, is the creation of a collection level description for each archive that we hold and this is being achieved through Project Genesis.

Populating AtoM with our accessions data was also not without its problems but now this has been achieved we are able to browse and search all of our accessions data in one place for the first time - a really important step for us!

Q is for Quality

In an ideal world, all our data within AtoM would be of a high quality.

...but we do not live in an ideal world.

Accepting that legacy data will not always meet current standards or be as accurate as we would like is key to moving forward with a system such as this.

We are striving for a full range of high quality and standards compliant finding aids within AtoM but difficult decisions have to be made. Is it better to expose a small number of perfect catalogues or a larger number of catalogues that don’t contain all the mandatory ISAD(G) fields? The second option gets my vote.

R is for Reference Codes

Quite early on, we had to make a decision about whether or not to inherit reference codes. This is a setting you can change within the admin section of AtoM and a very important one to give some thought to before you go too far down the data entry or import route.

AtoM can either be set up so that you enter the full reference code for each level of the hierarchy of archival description, or it can be set up to inherit previous levels of its reference code depending on its position within the hierarchy.

There is no right or wrong answer here and each institution will need to work out what will suit them best.  It can be hard to make a decision like this at the point where you are just starting out. Until you start to use AtoM in earnest you may not understand the full implications of your decision. Having initially agreed internally that we were going to inherit the reference code to save time with data entry and help guard against human errors, we subsequently changed our minds and decided not to inherit. This decision was influenced heavily by the way AtoM displays the reference numbers to the end user and how the archival hierarchy appears on the left side of the interface. We wanted the full reference to be displayed alongside each element of the hierarchy to help our users interpret the data and more easily see how the different levels relate to each other.

Time will tell whether we've made the right decision or not, but I imagine that once we have a substantial quantity of data within AtoM, this will become a harder decision to change!

S is for Session Timeout

Beware the inactive session timeout! AtoM times out by default after 30 minutes of inactivity. This has caused us problems when creating detailed descriptions within AtoM. If completing the Scope and Content field for a large and complex archive, it is necessary to spend some time consulting the physical archives and composing a description. Colleagues sometimes found that by the time they came to save their record the session had timed out. Naturally this was the source of great frustration.

We experimented with trying to extend the inactive session timeout period but these efforts were not successful. To avoid data loss we do encourage staff to regularly save their work. A text editor can also be used to compose descriptions. With an autosave function and no timeout, data is safer here and can be pasted into AtoM once it is complete.

T is for Training

Artefactual Systems offer introductory training sessions in AtoM and delivered one of these to Borthwick staff via WebEx at the start of our implementation project. This was well worth the expense, ensuring that staff understood the capabilities of the system and had a basic grounding in how to use it. I had my reservations about how well a training session via WebEx would work, but needn't have worried on that score. We heard Sarah Romkey from Artefactual Systems in Canada loud and clear and she was able to maintain a high level of enthusiasm throughout the session despite the fact that we had got her out of bed very early in the morning.

Training is not just a one off exercise. Now we are further along in our AtoM implementation we will be arranging further staff training to focus more on our local use of AtoM and internal processes and workflows.

U is for User Profiles and Roles

We have been giving some thought to who needs to do what within AtoM.

  • Who should have access to the import and export functions?
  • Who will be able to add new users to the system?
  • Who needs the ability to edit the static pages?
  • Who can publish and delete archival descriptions?
  • Who can change the accessions counter?

We are keen that AtoM is widely used by our staff and want to ensure that everyone has the necessary permissions to be able to carry out their work. User roles may evolve over time but some initial decisions do need to be made in the early stages of implementation.

V is for Volunteers

Prior to release of AtoM we have been calling for volunteers from our user base to help us test AtoM and give us their feedback.

We have put a lot of work into getting our AtoM instance ready to release and we have had our users in mind at many stages of the process. We now need to find out whether we have got it right. User testing is ongoing and we envisage we will be making some changes to AtoM once the feedback is collated.

We are really looking forward to seeing what people think.

W is for Web Address

We have made some decisions about the web address we will use for our production version of AtoM. The default url had ‘atom’ in it, but we wanted to change this to something more meaningful. AtoM means something to us and perhaps to other archives professionals but not to our users.

So, we have replaced ‘atom’ with something more descriptive and meaningful to our users – we will be plastering this url over the bookmarks and other publicity we are creating for our scheduled launch date so we want to get it right!

X is for XML

We do not want people to have to come to us to find out what we hold, we want our data to be signposted as widely as possible via other portals and aggregators both nationally and internationally. By doing so we facilitate serendipitous discovery and attract new users.

To this end we have been talking to external aggregators such as the Archives Hub to find out whether our AtoM data can be incorporated into their portal. We have been exporting sample data as EAD XML files so that the Archives Hub can assess it and see if it can be incorporated into their portal. A few initial problems with the EAD that AtoM creates have been ironed out and we are moving closer to being able to make this a reality over the next few months.

Y is for YorSearch

One of the features of AtoM we have been looking at before launch is the OAI-PMH functionality. We have used this to enable our AtoM data to be surfaced as simple Dublin Core metadata via our University Library Catalogue, YorSearch. It will be interesting to see whether students and staff members from the University (who may not have thought to consult our catalogue directly) will be approaching us in the future to consult our archives.




So, these are some of the things we have been thinking about and working on over the last year or so whilst moving our AtoM implementation from idea to reality. Hopefully it is of use to others who are embarking on the same process.

And of course, watch this space for news of launch!



* Actually an A-Y ...did anyone notice that there was no letter 'Z'?


Jenny Mitcham, Digital Archivist

Friday, 28 March 2014

Discovering archives: it's all about the standards

Yesterday at the UK Archives Discovery Forum we mostly talked about standards.*

Specifically metadata standards for resource discovery of archives, both physical and digital.. Standards are key to making archival data discoverable and of course this is our main reason for being - we preserve things so that they can be reused - they can only be reused if they can be discovered.

The day was really relevant to work we are currently doing at the Borthwick Institute, with the installation of a new archival management system (AtoM) underway and scoping work ongoing for a retroconversion project which will help us move our legacy catalogues into this new system - both major initiatives intended to make our catalogue data more widely discoverable.

Nick Poole from the Collections Trust talked about user focused design (both for physical buildings and digital interfaces), how we should avoid putting barriers between our users and the information we need. The gov.uk website is an obvious example of how this approach to design can work in a digital sphere and their design principles are on-line. This is something I think we can all learn from.

He also touched on the Open Data agenda and how the principles of making data ‘open by default’ are sometimes seen as being at odds with traditional models for income generation in the archives sector. Nick argues that by opening up data we are allowing more people to find us and making way for new opportunities and transactions as they engage further with the other services we have to offer.

He also mentioned that we can be ‘digitally promiscuous’ - making our data available in many different ways via many different platforms. We do not need to keep our data close to our chests but should be signposting what we have and drawing people in. We can only really do this if we make use of data standards. Standards help us to exchange and share our data and allow others to find and interpret it.

Jane Stevenson talked about the importance of standards to the Archives Hub.  Aggregating data from multiple sources would be very tricky if no-one used metadata standards. The problem is that the standards that we have are not perfect. Encoded Archival Description (EAD), the XML realisation of ISAD(G), can be too flexible and thus is realised in different ways by different institutions. Even those archives using CALM as their archival cataloguing system may have individual differences in how they use the metadata fields available to them. This does make life as an aggregator more challenging.

Once data is standardised into the Archives Hub flavour of EAD it can be transformed again into other data standards allowing it to be cross searchable beyond the UK archives sector. Jane touched on their work with RDF and linked data and the opportunities this can bring.

We should make use of opportunities to join the European stage. The Archives Hub are 'country manager' for Archives Portal Europe (APE) thus making it a simple matter for Hub contributors to push their data out beyond national borders. For those archival descriptions that link directly to a digital object, the opportunity exists to make this data available through Europeana. This takes our data beyond the archives sector, allowing our collections to be cross-searched alongside other European digital cultural heritage resources. In my mind, this really is the start of ‘digital promiscuity’ and an opportunity I feel we should be embracing (if we can accept the necessity to open up our metadata with a CC0 licence).

Geoff Browell from Kings College London talked about what we as archivists can offer our users over and above what they can get by visiting Google. He highlighted our years of experience at indexing data and pointed out that at approximately half of users of the AIM25 search interface appreciate our efforts in this area and use the index terms provided to browse for data in preference to the google-style free text search. He thinks that we should be talking more closely with both our users and the interface developers to ensure we are giving people what they need. He mentioned that delivery of data to users should be a conversation not a one-sided process.

The National Archives asked us for comment on a beta version of the new Discovery interface which will provide a new portal into selected UK archival holdings. They are encouraging conversation with users by encouraging ‘tagging’ of pages within the search interface.

Malcolm Howitt from Axiell discussed how systems and software can support standards. Standards is a topic that is often raised and they are asked to support many of them. They are keen to help where they can and need to work with the community to ensure that they know what is required of them. The different flavours of EAD was again raised as an issue but Malcolm pointed out that when standards work, the user doesn’t even need to be aware of them.

The National Archives, Kew, London by Jim Linwood on Flickr CC BY 2.0


Reflections

I think we are all in agreement that metadata standards are necessary and we need to work with them in order to make our catalogue data more visible. Some further issues were picked out in the final session of the day where attendees were invited to share their thoughts on the standards they use and the ones they would like to know more about.


  1. Do we need a standard for accessions data? Would this be a specific subset of ISAD(G) or does it need further definition? The next step in our planned implementation of AtoM is to populate it with accessions data from various different sources and I expect there will be some issues to deal with along the way as a result of lack of standards in this area.
  2. How do we describe digital material? Is ISAD(G) fit for this purpose? As born digital material becomes more and more prevalent in our collections this will become more of an issue. The use of PREMIS to hold technical preservation metadata will be essential alongside the resource discovery metadata but is this enough? This is undoubtedly an area for future exploration.
  3. Does the hierarchical nature of ISAD(G) and EAD hold us back? If we can’t create detailed resource discovery metadata for an archive until we know both the hierarchy and its place in the hierarchy does this slow us down in getting the information out there?



*…mostly standards - with the addition of a surprisingly entertaining session on copyright from Roman Deazley – check out the CREATe project for more on this topic


Jenny Mitcham, Digital Archivist

The sustainability of a digital preservation blog...

So this is a topic pretty close to home for me. Oh the irony of spending much of the last couple of months fretting about the future prese...