Moving a proof of concept into production? it's harder than you might think...

Myself and colleagues blogged a lot during the Filling the Digital Preservation Gap Project but I’m aware that I’ve gone a bit quiet on this topic since…

I was going to wait until we had a big success to announce, but follow on work has taken longer than expected. So in the meantime here is an update on where we are and what we are up to.

Background

Just to re-cap, by the end of phase 3 of Filling the Digital Preservation Gap we had created a working proof of concept at the University of York that demonstrated that it is possible create an automated preservation workflow for research data using PURE, Archivematica, Fedora and Samvera (then called Hydra!).

This is described in our phase 3 project report (and a detailed description of the workflow we were trying to implement was included as an appendix in the phase 2 report).

After the project was over, it was agreed that we should go ahead and move this into production.

Progress has been slower than expected. I hadn’t quite appreciated just how different a proof of concept is to a production-ready environment!

Here are some of the obstacles we have encountered (and in some cases overcome):

Error reporting

One of the key things that we have had to build in to the existing code in order to get it ready for production is error handling.

This was not a priority for the proof of concept. A proof of concept is really designed to demonstrate that something is possible, not to be used in earnest.

If errors happen and things stop working (which they sometimes do) you can just kill it and rebuild.

In a production environment we want to be alerted when something goes wrong so we can work out how to fix it. Alerts and errors are crucial to a system like this.

We are sorting this out by enabling Archivematica's own error handling and error catching within Automation Tools.

What happens when something goes wrong?

...and of course once things have gone wrong in Archivematica and you've fixed the underlying technical issue, you then need to deal with any remaining problems with your information packages in Archivematica.

For example, if the problems have resulted in failed transfers in Archivematica then you need to work out what you are going to do with those failed transfers. Although it is (very) tempting to just clear out Archivematica and start again, colleagues have advised me that it is far more useful to actually try and solve the problems and establish how we might handle a multitude of problematic scenarios if we were in a production environment!

So we now have scenarios in which an automated transfer has failed so in order to get things moving again we need to carry out a manual transfer of the dataset into Archivematica. Will the other parts of our workflow still work if we intervene in this way?

One issue we have encountered along the way is that though our automated transfer uses a specific 'datasets' processing configuration that we have set up within Archivematica, when we push things through manually it uses the 'default' processing configuration which is not what we want.

We are now looking at how we can encourage Archivematica to use the specified processing configuration. As described in the Archivematica documentation, you can do this by including an XML file describing your processing configuration within your transfer.

It is useful to learn lessons like this outside of a production environment!

File size/upload

Although our project recognised that there would be limit to the size of dataset that we could accept and process with our application, we didn't really bottom out what size dataset we intended to support.

It has now been agreed that we should reasonably expect the data deposit form to accept datasets of up to 20 GB in size. Anything larger than this would need to be handed in a different way.

Testing the proof of concept in earnest showed that it was not able to handle datasets of over 1 GB in size. Its primary purpose was to demonstrate the necessary integrations and workflow not to handle larger files.

Additional (and ongoing) work was required to enable the web deposit form to work with larger datasets.

Space

In testing the application we of course ended up trying to push some quite substantial datasets through it.

This was fine until everything abrubtly seemed to stop working!

The problem was actually a fairly simple one but because of our own inexperience with Archivematica it took a while to troubleshoot and get things moving in the right direction again.

It turned out that we hadn’t allocated enough space in one of the bits of filestore that Archivematica uses for failed transfers (/var/archivematica/sharedDirectory/failed). This had filled up and was stopping Archivematica from doing anything else.

Once we knew the cause of the problem the available space was increased but then everything ground to a halt again because we had quickly used that up again ….increasing the space had got things moving but of course while we were trying to demonstrate the fact that it wasn't working, we had deposited several further datasets which were waiting in the transfer directory and quickly blocked things up again.

On a related issue, one of the test datasets I had been using to see how well Research Data York could handle larger datasets consisted of c.5 GB consisting of about 2000 JPEG images. Of course one of the default normalisation tasks in Archivematica is to convert all of these JPEGs to TIFF.

Once this collection of JPEGs were converted to TIFF the size of the dataset increased to around 80 GB. Until I witnessed this it hadn't really occurred to me that this could cause problems.

The solution - allocate Archivematica much more space than you think it will need!

We also now have the filestore set up so that it will inform us when the space in these directories gets to 75% full. Hopefully this will allow us to stop the filestore filling up in the future.

Workflow

The proof of concept did not undergo rigorous testing - it was designed for demonstration purposes only.

During the project we thought long and hard about the deposit, request and preservation workflows that we wanted to support, but we were always aware that once we had it in an environment that we could all play with and test, additional requirements would emerge.

As it happens, we have discovered that the workflow implemented is very true to that described in the appendix of our phase 2 report and does meet our needs. However, there are lots of bits of fine tuning required to enhance the functionality and make the interface more user friendly.

The challenge here is to try to carry out the minimum of work required to turn it into an adequate solution to take into production. There are so many enhancements we could make – I have a wish list as long as my arm – but until we better understand whether a local solution or a shared solution (provided by the Jisc Research Data Shared Service) will be adopted in the future it is not worth trying to make this application perfect.

Making it fit for production is the priority. Bells and whistles can be added later as necessary!





My thanks to all those who have worked on creating, developing, troubleshooting and testing this application and workflow. It couldn't have happened without you!

Jenny Mitcham, Digital Archivist

Modelling Research Data with PCDM

This is a guest post by Julie Allinson, Technology Development Manager for Library & Archives at York. Julie has been working on York's implementation for the 'Filling the Digital Preservation Gap' project. This post discusses preliminary work to define a data model for 'datasets'.

For Phase three of our 'Filling the Digital Preservation Gap', I've been working on implementing a prototype to illustrate how PURE and Archivematica can be used as part of a Research Data management lifecycle. Our technology stack at York is Hydra and Fedora 4 but it's an important aspect of the project to ensure that the thinking behind the prototype is applicable to other stacks. Central to adding any kind of data to a research information system, repository or preservation tool is the data model that underpins the metadata. For this I've been making use of the Portland Common Data Model (PCDM) and it's various extensions (particularly Works).

In the past couple of years there has been a lot of work happening around PCDM, described as "a flexible, extensible domain model that is intended to underlie a wide array of repository and DAMS applications". PCDM provides a small Models ontology of classes and properties, with extension ontologies for Works and Use, among others. I like PCDM because it is high level enough to provide a language to talk across different domains and use cases.

Datasets data model version 1

My first attempt at a data model for datasets based on PCDM can be seen below.

Datasets Data Model v1

Starting with the Dataset object above, for York this is equivalent to a dataset record in our PURE research information system. The only metadata expected at this level is about the dataset as a whole, and for us, will largely come from PURE.

Below this, you'll see that I've begun to model in some OAIS constructs: the Archival Information Package (AIP) and Dissemination Information Package (DIP). The AIP is the deposit of data, prepared for preservation, a single package of data "consisting of the Content Information and the associated Preservation Description Information (PDI), which is preserved within an OAIS." (DCC Glossary). The DIP is a representation of the Content Information in the AIP, produced for dissemination. OAIS, by the way, gives no standard approach for structuring the AIP or DIP.

As it stands, this model does not consider how to 'unpack' the AIP at all and for our prototype - we are (or were - see below) intending to simply point to the AIP in Archivematica.

The DIP as illustrated above is based on what Archivematica generates. The diagram includes the processing configuration and METS files that Archivematica produces by default as filesets, for illustration. These aren't part of the dataset as deposited, hence not making them 'Works' in their own right.

GenericWork is intended for each Unit or 'Work' in the dataset as deposited, or representation thereof. GenericWork is intended for use with any kind of data object. it might be independently re-used, eg. as a member of another dataset. In most cases for research data we probably won't know much about what the data is and so GenericWork will be used, but sometimes it may make sense to use existing models. For example, if the dataset is a collection of images then a more tailored Image model could be used for each Image, or if a dataset includes some existing objects that are already in our repository, those might already have different models. They can still be members of our dataset.

The model is intended to allow for a Dataset to include multiple AIPs and thus I have suggested a local predicate for hasDIP / hasAIP to establish the relationship between the AIP and the DIP.

The trouble with DIPs : an alternative model

Discussing this model with Justin Simpson from Artefactual Systems recently, I got to thinking that the DIP is a really artificial construct, effectively a presentation 'view' on the data and not really separate 'Work' in it's own right.  Archivematica's DIP, at present, provides only files, and doesn't reflect the original folder structure, which may well be meaningful and necessary for a dataset. Perhaps what we really need is the AIP, described fully in Fedora, leaving presentation of the DIP to the interface level? A set of rules for producing a DIP to our own local specification might go like this (using the PCDM Use ontology): if there is a Preservation Master file, produce a Service File for user access, otherwise present the Original File to the user.

The new model would look something like this:

Datasets Data Model alternative

The DIP would be a view constructed from elements of the AIP:

This model feels like a better approach than that in version 1 as it facilitates describing the 'whole' dataset. I do have some immediate questions about the model, though:

• Is a dataset really a pcdm:Collection, rather than a Work?
• If the GenericWork is each data file irrespective of whether it can be used/understood on it's own, how useful is that in reality? Is the GenericWork really needed, or are FileSets enough? Is there genuinely value is identifying each individual piece of data as a 'Work'? (re-use outside of the dataset, for example)

And when thinking beyond the model, about how this would actually work for different use cases, implementations questions start to surface. 

Beyond the model

1) Dataset size and structure

Datasets may contain thousands, millions even, of files structured into folders where folders may impart meaning to the data, or be purely arbitrary. Fedora 4 can, by design, handle a folder structure using it's implementation of LDP Basic Containers. As illustrated below, each folder is a 'Basic Container' and each data file is a Work, with FileSet and File objects.

• AIP ldp:contains folder1
• folder1 ldp:contains folder2
• folder2 ldp:contains folder3
• folder3 ldp:contains GenericWork
• GenericWork pcdm:hasMember FileSet 

But if each of those folders are objects in Fedora and each file is in Fedora with a Work, FileSet and several File objects, then the number of Fedora objects begins to rise exponentially

• Would it be better to avoid object-cost to Fedora of creating many many objects? 
• What alternative approaches could we take? Use BagIt and have Fedora reference only the 'bag'? Store only data files and create an 'order' to represent the folder structure (as outlined in PCDM 2.0)?

2) Storing Files

'Where should data actually be stored permanently?' is another practical question I've been thinking about. On the one hand, Archivematica makes the AIP and it's files available via URLs in the Storage Service and stores the file in a location of your choosing. On the other, Fedora can contain data files, or reference them via a URI. This gives gives us the flexibility to do several things:

1. Leave the AIP and DIP in Archivematica's stores and use URIs in Fedora to reference all of the files and build a PCDM-modelled view of the data (Archivematica as preservation and access store).
2. Manage all files in Fedora, treating the Archivematica copy of the data as temporary (Archivematica as sausage factory).
3. Have two copies of AIP data, one in Archivematica and one in Fedora (LOCKSS model).
4. Manage Preservation files in Archivematica and delivery/access files in Fedora (Archivematica as preservation and Fedora as access).

Keeping data in Archivematica makes is easy to do additional preservation actions in future, such as re-ingesting when format policy rules change, whereas managing all files within Fedora unlocks the possibilities of Fedora's audit functionality, fixity checking and versioning. Having two copies is attractive as a preservation strategy, but could be difficult to justify and sustain if data collections grow to a significant size. 

On balance I think option (4) is best for the short-term with other options worth re-considering as both Archivematica and Fedora mature. But I'd be really keen to hear different views.

Conclusions

Hopefully this post illustrates that creating an outline data model is pretty easy, but when it comes to thinking about it in terms of implementation decisions, all kinds of ifs and buts start coming up.

In the model above, each data file is a Work. Each work contains one or more FileSets, and each FileSet contains one or more different representations of the file. 

Is it really possible to define a general datasets model that could encompass data from across disciplines, of various sizes, structures and created for a variety of purposes? Data that might be independently re-usable (a series of oral history interviews) or might only be understandable in combination with other files (a database schema document, for example)?

This is very much a work in progress, and I'd really welcome feedback from others who have done allied work or anyone who has suggestions and comments on the approaches and issues outlined above.

Jenny Mitcham, Digital Archivist

Spreading the word on the "other side of the pond"

A guest post by Richard Green who has been leading the University of Hull's technical investigations for "Filling the Preservation Gap".

Jenny is away from her desk at the moment so I've been deputised to provide a blog post around the work we've been doing at the University of Hull as part of the Jisc-funded "Filling the Preservation Gap" (FPG) project.  In particular we (the FPG team) want to mention a poster that we prepared for a recent conference in the US.

Hull has had a digital repository in place for a considerable number of years.  It has always had the Fedora (now Fedora Commons) repository software at its heart and for several years now has deployed Hydra over the top of that - indeed, Hull was a founder member of the Hydra Project.  With the established repository goes an established three-stage workflow for adding content.  Content is initially created in a “proto-queue” by a user who, when (s)he is happy with it, transfers it to the ownership of the Library who take it through a quality assurance process.  When the team in the Library is happy with it the content is transferred to the repository "proper" with appropriate access permissions.  The repository contains a wide range of materials and over time we are developing variants of this basic workflow suited to each content type but this activity is constrained by limited resources and we know there are other variations we would like to, and can, develop when circumstances permit. The lack of a specific workflow for research data management (RDM), encompassing the possible need for long-term preservation, was one of the reasons for getting involved in the FPG project.

Whilst the focus of the FPG project is clearly research data it became apparent during our initial work with Archivematica that its preservation needs were not so far removed from the preservation needs that we have for some of our other content.  That being the case we have kept our eye on the bigger picture whilst concentrating on RDM for the purposes of our Jisc project.  We have spent some time putting together an initial all-encompassing design through which an RDM workflow for the FPG project would be but one possible path. It is that overall picture that became our poster.

The Hydra Community holds one major get-together each year, the "Hydra Connect" conference.  The last full week in September saw 200 people from 55 institutions gather in Minneapolis for Connect 2015.  A regular feature of the conferences, much appreciated by the audience, is an afternoon given over to a poster session during which attendees can talk about the work they are doing with Hydra.  Each institution is strongly encouraged to contribute and so Hull took along its grand design as its offering.

Poster for Hydra Connect 2015 in Minneapolis, MN, September 2015

So that’s the poster and here’s a somewhat simplified explanation!

Essentially, content comes in at the left-hand side.  The upper entry point corresponds to a human workflow of the type we already have.  The diagram proposes that the workflow gain the option of sending the digital content of an object through Archivematica in order to create an archival information package (AIP) for preservation and also to take advantage of such things as the software’s capability to generate technical metadata.  The dissemination information package (DIP) that Archivematica produces is then “mined” for content and metadata that will be inserted into the repository object already containing the creator’s descriptive metadata record.  One of the items mined is a UUID that provides a tie-up between the record in the repository and the AIP which goes to a separate preservation store.

The lower entry point corresponds to an automated (well, maybe semi-automated) batch ingest process.  In this case, the DIP processor creates a repository object from scratch and, in addition to possible dissemination files and technical metadata, provides the descriptive metadata too.  There are a number of scenarios for generating the descriptive metadata; at one extreme it might be detailed fields extracted from an accompanying file, at the other it might be minimal metadata derived from the context (the particular ingest folder and the title of the master file, for instance).  There will be circumstances when we create a metadata-only record for the repository and do not include dissemination files in the repository object; under these circumstances the UUID in the metadata would allow us to retrieve the AIP from store and create a new DIP should anyone ever request the data itself.

Finally, we have content already in the repository where it is being “kept safe” but which really justifies a proper preservation copy.  We shall create a workflow that allows this to be passed to Archivematica so that an AIP can be created and stored.  It is probable that this route would use the persistent identifier (PID) of the Fedora object as the link to the AIP.

Suffice it to say that the poster was well received.  It generated quite a lot of interest and, some of it, from surprising quarters.  In conversation with one well-established practitioner in the repository field, from a major US university, I was told “I’ve never thought of things quite like that – and you’re probably right!” It’s sometimes reassuring to know that the work we undertake at the smaller UK universities is nevertheless respected in some of the major US institutions!


If you have any comments, or want further details, about our work then please get in touch via this blog.  We’re interested in your thoughts, perspectives and ideas about the approach.

Jenny Mitcham, Digital Archivist

Hydra: multiple heads are better than one

Trinity College, Dublin

I spent a couple of days this week in Dublin at the Hydra Europe Symposium. Hydra has been on my radar for a little while but these two days really gave me an opportunity to focus on what it is and what it can do for us. This is timely for us as we are currently looking at systems for fulfilling our repository and digital archiving functions. 

At York we currently use Fedora for our digital library so developments within the Hydra community are of particular interest because of its relationship to Fedora.

Chris Awre from the University of Hull stated that the fundamental assumptions on which Hydra was built were that:

1. No single system can provide the full range of repository based solutions for an institutions needs

2. No single institution can resource development of a full range of solutions on its own

This chimes well with our recent work at York trying to propose a technical architecture that could support deposit, storage, curation and access to research data (among other things). There is no one solution for this and building our own bespoke system from scratch or based purely on Fedora would clearly not be the best use of our resources.

The solution that Hydra provides is a technical framework that multiple institutions can work with but that can be built upon with adopting institutions developing custom elements tailored to local workflows. Hydra has one body but many heads supporting many different workflows.

We were told pretty early on within the proceedings that for Hydra, the community is key. Hydra is as much about knowledge sharing as sharing bits of code.

“If you want to go fast go alone, if you want to go far, go together” – This African proverb was used to help explain the Hydra concept of community. In working together you can achieve more and go further. However, some of the case studies that were presented during the Symposium clearly showed that for some, it is possible to go both far and fast using Hydra and with very little development required. Both Trinity College Dublin and the Royal Library of Denmark commented on the speed with which a repository solution based on Hydra could be up and running. Speed is of course largely dependent on the complexity or uniqueness of the workflows you need to put in place. Hydra does not provide a one-size-fits-all solution but should be seen more as a toolkit with building blocks that can be put together in different ways.

Dermot Frost from Trinity College Dublin summed up their reasons for joining the Hydra community, saying that they had had experience with both Fedora and DSpace and neither suited their needs. Fedora is highly configurable and in theory does everything you need to do, but you need a team of rocket scientists to work it out. DSpace is a more out-of-the-box solution but you can not configure it in the way you need to to get it to conform to local needs. Hydra sits between the two providing a solution that is highly configurable, but easier to work with than Fedora.

Anders Conrad from the Royal Library of Denmark told us that for their repository solution, 10-20% of material is deemed worthy of proper long term preservation and is pushed to the national repository. The important thing here is that Hydra can support these different workflows and allows an organisation to put one repository in place that could support different types of material with different values placed on the content and thus different workflows going on within it. The 'one repository - multiple workflows' model is very much the approach that the University of Hull have taken with their Hydra implementation. Richard Green described how data comes in to the repository through different routes and different types of data are treated and displayed in different ways depending on the content type.

And what about digital preservation? This is of course my main interest in all of this. One thing that is worth watching is Archivesphere, a Hydra head that is being created by Penn State designed to "create services for preserving, managing, and providing access to digital objects, in a way that is informed by archival thinking and practices" and including support for both PREMIS and EAD metadata. This is currently being tested by Hydra partners and it will be interesting to see how it develops.

Another thing to think about is how Hydra could meet my digital preservation requirements that I published last year (note they have changed a little bit since then). I think the answer to this is that it probably could meet most of them if we wanted to develop the solutions on top of existing Hydra components. Archivesphere is already starting to introduce some of the required functionality to Hydra, for example file characterisation, normalisation and fixity checking. I guess the bigger question for me is whether this is the best approach for us or whether we would be preferable to make use of existing digital archiving software (Archivematica for example) and ensure the systems can talk to each other effectively.

Jenny Mitcham, Digital Archivist