Advice for our donors and depositors

Anyone who knows anything about digital archiving knows that one of the best ways to ensure the longevity of your digital data is to plan for it at the point of creation.

If data is created with long term archiving in mind and following a few simple and common sense data management rules, then the files that are created are not only much easier for the digital archivist to manage in the future, but also easier for the creator to work with. How much easier is it to locate and retrieve files that are ordered in a sensible and logical hierarchy of folders and named in a way that is helpful? We are producing more and more data over time and as the quantity of data increases, so do the size of our problems in managing it.

We do not have many donors and depositors at the Borthwick who regularly put digital archives into our care but this picture will no doubt change over time. For those who do deposit digital archives, it is important that we encourage them to put good data management into practice and the earlier we speak to them about this the better.

'Bitrot' From the Digital Preservation Business
Case Toolkit http://wiki.dpconline.org/

Last week I was fortunate enough to be invited to speak to a group of people from one of our depositor organisations who are likely to start giving us digital data to archive in the future. They were from a large organisation with no central IT infrastructure and many people working from home on their own computers. Good data management is particularly important in this sort of scenario. This was a great opportunity for me to test out what could become the basis of a standard presentation on digital data management techniques that could be delivered to our donors and depositors.

I started off talking about what digital preservation is and why we really need to do it. It is always handy to throw in a few cautionary tales at this point as to what happens when we don't look after our data. I think these sorts of stories resonate with people more than just hearing the dry facts about obsolescence and corruption. I made a good plug for the 'Sorry for your data loss' cards put together by the Library of Congress earlier this year as this is something that any of us who have experienced data loss can relate to.

I then moved on to my own recent tale of digital rescue, using the 5 1/4 inch floppy disks from the Marks and Gran archive as my example (discussed in a previous blog post). This was partly because this is my current pet project, but also because it is a good way to cement and describe the real issues of hardware and software obsolescence and how we can work around these.

In the last section of the presentation I gave out my top tips on data management. I wanted the audience to go home with a positive sense of what they can start to do immediately in order to help protect their data from corruption, loss or misinterpretation.

Much of what I discussed in this section was common sense stuff really. Topics covered included:

• how to name files sensibly
• how to organise files well within a directory structure
• how to document files
• the importance of back up
• the importance of anti-virus software

The presentation went well and sparked lots of interesting questions and debate and it was encouraging to see just how accessible this topic is to a non-specialist audience. Some of the questions raised related to current 'hot topics' in the digital archiving world which I hadn't had time to mention in any depth in my presentation:

• How do you archive e-mails?
• Is cloud storage safe?
• What is wrong with pdf files?
• What is the life span of a memory stick?

I had an answer to all of these apart from the last one, for which I have since found out the answer is 'it depends'. I have recently been told on Twitter that most digital preservation questions can be answered in this way!

Jenny Mitcham, Digital Archivist

Do sound engineers have more fun?

At the end of last week I was at the British Library on their excellent ‘Understanding and Preserving Audio Collections’ course.

British Library and Newton by Joanna Penn on Flickr

The concept of ‘Preserving audio’ is not a new one to me. Audio needs to be digitised for preservation and access and that pushes it firmly into my domain as digital archivist. I know the very basics such as the recommended file formats for long term preservation, but when faced with a real life physical audio collection on a variety of digital and analogue carriers it is hard to know what the priorities are and where exactly to start. This is where the ‘Understanding audio’ part of the course came to my rescue, filling in some of the gaps in my knowledge.

The course

The first day of the course was fascinating. We were given a run-down of the history of audio media and were introduced to (and in many cases, allowed to handle) many different physical carriers of audio. Hearing a wax cylinder being played on an original phonograph was a highlight for me. Digital archivists don’t normally get to play with the physical artefacts held within archives!  Perhaps most useful in this session was learning how to recognise different types of physical media and spot the signs of physical degradation.

In the following two days the emphasis moved on to digital carriers and digital files. Interestingly digital carriers were seen to be more vulnerable than analogue in many respects. Digitisation workflows were also discussed and we got the chance to see around the digitisation studios with a wide range of equipment demonstrated. This was the point at which I started to wish I was a sound engineer!

Not so different after all…

One of the things that struck me throughout the three days was that this really isn't an alien subject to me at all. Familiar concepts were repeated again and again about obsolescence of technology, lack of standards (particularly when a new type of media takes off), the importance of metadata, the idea that future technologies may be able to do a better job of this than us, and the vain hopes that an ‘everlasting’ media carrier may be made available to us and solve all of our problems. Standard topics in any introductory presentation concerning digital archiving!

What was new and interesting to me though was that for audio media a time limit has been internationally agreed for taking action. We need to plan to digitise our audio and preserve it within a digital archive within the next 15 years because there will come a point at which this strategy will not be possible any more. We have a limited window of opportunity to work in. Digitising obsolete analogue and digital carriers is becoming harder to do (as the media degrades in a variety of different ways) and more expensive (as the necessary hardware and parts becomes harder to source). In fact, whereas digitisation of documents is becoming cheaper over time as new technologies are introduced, the digitisation of audio is becoming more expensive over time as the necessary equipment becomes harder to get hold of.

Has such a time limit has ever been discussed for rescuing data from obsolete digital media such as floppies and zip disks? If so, it is not one that has hit my radar.

Putting the learning into context

The Borthwick Institute and the University of York curate some substantial music collections, but we have also been carrying out an audit of all the other bits and pieces of audio that are buried within some of our other collections. Currently we have a list with basic information about each item including the media type, the location in the strongroom and descriptive information taken from the label or packaging. The next step was to work out a digitisation strategy for these items.

This is all well and good, but work on this stalled as it quickly found its way into the ‘too difficult’ box. Following on from the information absorbed on this course, I now have the ability to start the process of prioritising the audio for digitisation based on variables such as vulnerability of the physical media and the condition of individual items. Also taking into account whether the content is unique or of particular interest.

Another benefit is that I now feel that I could now hold a conversation with a sound engineer! This is key to planning a digitisation project. Every discipline has its own particular language or jargon and happily I now have some understanding of waveforms, equalisation curves and sampling rates. At the very least I know what to ask for if writing a specification for an audio digitisation project and have a wealth of references, resources and contacts at my finger tips if I need to find out more.



Jenny Mitcham, Digital Archivist

A short detective story involving 5 ¼ inch floppy disks

Earlier this year my colleague encountered two small boxes of 5 ¼ inch floppy disks buried within the Marks and Gran archive in the strongrooms of the Borthwick Institute. He had been performing an audit of audio visual material in our care and came across these in an unlisted archive.

This was exciting to me as I had not worked with this media before. As a digital archivist I had often encountered 3 ½ inch floppies but not their larger (and floppier) precursors. The story and detective work that follows, took us firmly into the realm of ‘digital archaeology’.

Digital archaeology: 

“The process of reclaiming digital information that has been damaged or become unusable due to technological obsolescence of formats and/or media” (definition from Glossaurus)

Marks and Gran were a writing duo who wrote the scripts of many TV sitcoms from the late 1970's on-wards. ‘Birds of a Feather’ was the one that I remember watching myself in the 80’s and 90's but their credits include many others such as ‘Goodnight Sweetheart’ and ‘The New Statesman’. Their archive had been entrusted to us and was awaiting cataloguing.

Clues on the labels

There were some clues on the physical disks themselves about what they might contain. All the disks were labelled and many of the labels referred to the TV shows they had written, sometimes with other information such as an episode or series number. Some disks had dates written on them (1986 and 1987). One disk was intriguingly labelled 'IDEAS'. WordStar was also mentioned on several labels 'WordStar 2000' and 'Copy of Master WordStar disk'. WordStar was a popular and quite pioneering word processing package of the early 80’s.

However, clues on labels must always be taken with a pinch of salt. I remember being guilty of not keeping the labels on floppy disks up to date, of replacing or deleting files and not recording the fact. The information on these labels will be stored in some form in the digital archive but the files may have a different story to tell.

Reading the disks

The first challenge was to see if the data could be rescued from the obsolete media they were on. A fortuitous set of circumstances led me to a nice chap in IT who is somewhat of an enthusiast in this area. I was really pleased to learn that he had a working 5 ¼ inch drive on one of his old PCs at home. He very kindly agreed to have a go at copying the data off the disks for me and did so with very few problems. Data from 18 of the 19 disks was recovered. The only one of the disks that was no longer readable appeared from the label to be a backup disk of some accounting software - this is a level of loss I am happy to accept.

Looking at file names

Looking at the files that were recovered is like stepping back in time. Many of us remember the days when file names looked like this - capital letters, very short, rather cryptic, missing file extensions. WordStar like many of the software packages from this era, was not a stickler for enforcing use of file extensions! File extensions were also not always used correctly to define the file type but sometimes were used to hold additional information about the file.

Looking back at files from 30 years ago really does present a challenge. Modern operating systems allow long and descriptive file names to be created. When used well, file names often provide an invaluable source of metadata about the file. 30 years ago computer users had only 8 characters at their disposal. Creating file names that were both unique and descriptive was difficult. The file names in this collection do not always give many clues as to what is contained within them.

Missing clues

For a digital archivist, the file extension of a file is a really valuable piece of information. It gives us an idea of what software the file might have been created in and from this we can start to look for a suitable tool that we could use to work with these files. In a similar way a lack of file extension confuses modern operating systems. Double click on a file and Windows 7 has no idea what it is and what software to fire up to open it. File characterisation tools such as Droid used on a day to day basis by digital archivists also rely heavily on file extensions to help identify the file type. Running Droid on this collection (not surprisingly) produced lots of blank rows and inaccurate results*.

Another observation on initial inspection of this set of files is that the creation dates associated with them are very misleading. It is really useful to know the creation date of a file and this is the sort of information that digital archivists put some effort into recording as accurately as they can. The creation dates on this set of files were rather strange. The vast majority of files appeared to have been created on 1^st January 1980 but there were a handful of files with creation dates between 1984 and 1987. It does seem unlikely that Marks and Gran produced the main body of their work on a bank holiday in 1980, so it would seem that this date is not very accurate. My contact in IT pointed out that on old DOS computers it was up to the user to enter the correct date each time they used the PC. If no date was entered the PC defaulted to 1/1/1980. Not a great system clearly and we should be thankful that technology has moved on in this regard!

So, we are missing important metadata that will help us understand the files, but all is not lost, the next step is to see whether we can read and make sense of them with our modern software. 

Reading the files

I have previously blogged about one of my favourite little software programmes, Quick View Plus – a useful addition to any digital archivist’s toolkit. True to form, Quick View Plus quickly identified the majority of these files as WordStar version 4.0 and was able to display them as well-formatted text documents. The vast majority of files appear to be sections of scripts and cast lists for various sitcoms from the 1980’s but there are other documents of a more administrative nature such as PHONE which looks to be a list of speed dial shortcuts to individuals and organisations that Marks and Gran were working with (including a number of famous names).

Unanswered questions

I have not finished investigating this collection yet and still have many questions in my head:

• How do these digital files relate to what we hold in our physical archive? The majority of the Marks and Gran archive is in paper form. Do we also have this same data as print outs? 
• Do the digital versions of these files give us useful information that the physical do not (or vice versa)? 
• Many of the scripts are split into a number of separate files, some are just small snippets of dialogue. How do all of these relate to each other? 
• What can these files tell us about the creative process and about early word processing practices?

I am hoping that when I have a chance to investigate further I will come up with some answers.

* I will be providing some sample files to the Droid techies at The National Archives to see if they can tackle this issue.

Jenny Mitcham, Digital Archivist

Twelve interesting things I learnt last week in Glasgow

The Cloisters were a good place to shelter from the heat! 

Photo Credit: _skynet via Compfight

Last week I was lucky enough to attend the first iteration of the Digital Preservation Coalition's Advanced Practitioner Course. This was a week long course organised by the APARSEN project and based at the University of Glasgow on the warmest week of 2013 so far. On the first day Ingrid Dillo begun by telling us that 'data is hot' - by the end of the week it was not only data that was hot.

It would be a very long blog post if I was to try and do justice to each and every presentation over the course of the week, (I took 30 pages of notes) so here is the abridged version:

A list of twelve interesting things:

These are my main 'take home' snippets of information. Some things I already knew but were reinforced at some point over the week, and others are things that were totally new to me or provided me with different ways of looking at things. Some of these things are facts, some are tools and some are rather open-ended challenges.

A novel way to present a cake.
Photo credit: Jenny Mitcham

1) We can think about data and interpretations of data using the analogy of a cake (Ingrid Dillo). The raw ingredients of the cake (eggs, flour, sugar etc) represent the raw data. The cake as it comes out of the oven is the information that is held within that raw data. The iced and decorated cake is the presentation of the data - data can be presented in lots of different ways just as the same cake could be decorated in many different ways. The leftover crumbs of the cake on the plate after it is eaten represents the intangible knowledge that we have gained. This really reinforces for me the reason behind our digital preservation actions - curating and preserving the raw data so that others can create alternative interpretations of that data and we all can benefit from the knowledge that that will bring.

2) Quantifying the costs of digital curation is hard (Kirnn Kaur). No surprise really considering there have been so many different projects looking at cost models for digital preservation. If it was easy perhaps it would have been solved by a single project. The main problem seems to be that many of the cost models that are currently out there are quite specific to the organisation or domain that produced them. Some interesting work on comparing and testing cost models has been carried out by the APARSEN project and a report on all of this work is available here.

3) In 20 years time we won't be talking about 'digital preservation' anymore, we will just be talking about 'preservation' (Ruben Riestra). There will be nothing special about preserving digital material, it will just be business as usual. I eagerly await 2033 when all my headaches will be over!

4) I am right in my feeling that uncompressed TIF is the best preservation file format for images. The main contender, JPEG2000 is complicated and not widely supported (Photoshop Elements dropped support for it due to lack of interest from their users) (Tomasz Parkola).

5) There is a tool created by the SCAPE project called Matchbox. I haven't had a chance to try it out yet but it sounds like one of those things that could be worth it's weight in gold. The tool lets you compare scanned pages of text to try and find duplicates and corresponding images. It uses visual keypoints and looks for structural similarity (Rainer Schmidt and Roman Graf). More tools like this that automate some of the more tedious and time consuming jobs in digital curation are welcome!

6) Digital preservation policy needs to be on several levels (Catherine Jones). I was already aware of the concept of high level guidance (we tend to call this 'the policy') and then preservation procedures (which we would call a 'method statement' or 'strategy') but Catherine suggests we go one step further with our digital preservation policy and create a control policy - a set of specific measurable objectives which could also be written in a machine readable form so that they could be understood by preservation monitoring and planning tools and the process of decision making based on these controls could be further automated. I really like this idea but think we may be a long long way off to achieving this particular policy nirvana!

7) The SCAPE project has produced a tool that can help with preservation and technology watch ...that elusive digital preservation function that we all say we do ...but actually do it in such an ad hoc way that it feels like we may have missed something. Scout is designed to automate this process (Luis Faria). Obviously it needs a certain amount of work to give it the information that it needs in order for it to know what exactly it is meant to be monitoring (see comment on machine readable preservation policies above), but this is one of those important things the digital preservation community as a whole should be investing their efforts into and collaborating on.

8) There is a new version of the Open Planets Foundation preservation planning tool Plato that sounds like it is definitely worth a look (Kresimir Duretec). I first came across Plato when it was first released several years ago but like many, decided it was too complex and time consuming to bring into our preservation workflows in earnest. The project team have now developed the tool further and many of the more time consuming elements of the workflow have been automated. I really need to re-visit this.

9) The beautifully named C3PO (Clever, crafty content profiling for objects) is another tool we can use to characterise and identify the digital files that we hold in our archives (Artur Kulmukhametov). We were lucky enough to have a chance to play with this during the training course. C3PO gives a really nice visual and interactive interface for viewing data from the FITS characterisation tool. I must admit I haven't really used FITS in earnest because of the complexity of managing conflicts between different tools, and the fact that the individual tools currently wrapped in FITS are not kept up-to-date. C3PO helps with the first of these problems but not so much with the later. The SCAPE project team suggested that if more people start to use FITS then the issue might be resolved more quickly. It does become one of those difficult situations where people may not use the tool until the elements are updated but the tool developers may not update the elements until they see the tool being more widely used! I think I will wait and see.

Of the handful of example files I threw at FITS, my plain text file was identified as an HTML file by JHove so not ideal. We need greater accuracy in all of these characterisation tools so that we can have greater trust the output they give us. How can we base our preservation planning actions on these tools unless we have this trust?

10) Different persistent identifier schemes offer different levels of persistence (Maurizio Lunghi). For a Digital Object Identifier (DOI) you only need the indentifier and metadata to remain persistent, the NBN namespace scheme requires that the accessibility of the resource itself is also persistent. Another reason why I think DOI's may be the way to go. We do not want a situation where we are not allowed to de-accession or remove access to any of the digital data within our care.

11) I now know the difference between a URN a URL and a URI (Maurizio Lunghi) - this is progress!

12) In the future data is likely to be more active more of the time, making it harder to curate. Data in a repository or archive could be annotated, tagged and linked. This is particularly the case with research data as it is re-used over time. Can our current best practices of archiving be maintained on live datasets? Repositories and archives may need to rethink how we support researchers in this to allow for different and more dynamic research methodologies, particularly with regards to 'big data' which can not simply be downloaded (Adam Carter).

So, there is much food for thought here and a lot more besides.

I also learnt that Glasgow too has heat waves, the Botanic Gardens are beautiful and that there is a very good ice cream parlour on Byres Road!

Jenny Mitcham, Digital Archivist

Assessing the value of digital surrogates

I had an interesting meeting with colleagues yesterday to discuss how we manage digital surrogates - digitised versions of physical items we hold in the archives.

At the Borthwick Institute we do a fair amount of digitisation for a variety of reasons. These range from the large scale digitisation projects such as the York Cause Papers, to digitisation to create images for publications and exhibitions to single pages of Parish Registers for family history researchers.

As I am in the process of setting up a digital archive for the Borthwick Institute, effectively managing these digital surrogates also becomes my concern. The need to preserve these items is not as pressing as for born digital data (because they are only copies, not originals) however, to start to build up the collections that we have in digital form, to allow access to users who can not visit our searchroom, and to avoid having to carry out the same work twice, appropriate creation and management of this data is important. Although there will be a clear distinction in the digital archive between material that has been donated or deposited with us in digital form, and material we have digitised in-house, both these types of data need to be actively managed and migrated over time.

One of the big questions we have been mulling over is how we decide which digitised material we keep and which we discard. There is no pressing need to keep everything we digitise. Much of the reprographics work we carry out for orders would consist of a single page of a larger volume. Creating appropriate metadata to describe exactly which section of the item had been digitised would undoubtedly become an administrative burden and the re-use potential of that individual section would be limited. We therefore need to make some pragmatic decisions on what we keep and what we throw away.

Here are some points to consider:

• What is the condition of the original document? Is it fragile? Do we need a digital surrogate so we can create a copy for access that avoids any further handling of the original?
• How easy is it to digitise the original? If problematic because of it's large size or tight binding then we should ensure we maintain the digital surrogate to avoid having to digitise again in the future.
• Is the section to be digitised suitable for re-use or would it make little sense out of context (for example if it is only one small section of a larger item)?
• Is the archive catalogued to item level? This would certainly make it easier to administer the digital surrogates and ensure they can be related to their physical counterparts
• What are the access conditions for the original document? Is there any value in maintaining a digital copy of a document that we can not make more widely available?

I am interested in how others have addressed this decision making process and how digital surrogates are treated within a digital archive so would welcome your thoughts...

Jenny Mitcham, Digital Archivist

Some thoughts about our preservation policy

We are updating the Borthwick Institute's preservation policy. Originally written in 2007, it is due for review by our preservation archivist and conservation team but my interest in this task is to ensure that digital archives are represented within the policy. In the current policy there is no mention of the preservation of digital archives. There was no need for us to mention them six years ago but times are changing - this has now become a priority for us.

It was interesting having the opportunity to sit down and read our existing preservation policy. The only other preservation policies I had ever read up to this point (not that many of them I might add) related purely to digital material so had quite a different emphasis. As I am still quite new to the world of traditional archives, reading about everything that is in place to protect and preserve the documents in our care is an education for me.

The first question in my head is whether to crack on with this now or wait until such a time that the structure of the digital archive is more firmly in place. By getting the policy for the preservation of digital material written now we are jumping the gun a bit as procedures are not yet established. However by writing a policy at this point we are at least setting out our intentions and providing an overview of how we will approach the preservation of digital material. There are certain things I am confident we will be doing. The finer detail of how this will occur will follow in time and may be incorporated into a more detailed preservation strategy document in the future.

The second decision to make regarding the revised preservation policy was whether we integrate the digital within the current policy or create a separate document. Both approaches are legitimate and widely used. For the Borthwick we have decided that an integrated policy is the way forward. Ultimately, we want our systems for receiving, managing and providing access to archives to be seamless and media blind. It shouldn't matter to our depositors or users whether the media is digital or not, they should be confident in our ability to preserve and provide access to them regardless. Digital preservation should not be a specialist outpost, it should be fully integrated in the psyche of both our staff and our users.

There are differences to the way we preserve physical and digital archives:

• With physical archives (paper and parchment) it is most important for the material to be appropriately packaged and stored in the correct environmental conditions. Once the conditions are right, they can be largely left alone. Intervention should only be required where specific issues occur.
• With digital archives it is all about continuous active management. The digital environment is fast moving and the threat of obsolescence is never far away. Leaving the data alone in a static environment is a very risky approach.

Despite these differences, the basic premis of preservation is the same. What is highlighted in our current preservation policy is the idea of "preservation for access". This is why we are all here after all. Whether the material is physical or digital, we need to ensure that we preserve them so that others may access them in the future.

Jenny Mitcham, Digital Archivist

Preservation metadata

Yesterday I attended the DPC event on preservation metadata which focused on the PREMIS and METS standards. Coming at a time when I am reviewing various systems and software for managing digital archives and research data this was quite timely. The main message I took away from the day is that the key to actually creating and managing this metadata as ever is having the necessary tools!

Jelly Beans by kayaker1204, on Flickr
 - lots of these were consumed yesterday!

Earlier this year with the help of an intern we collected together digital media deposited with the Borthwick Institute and got the data safely stored on our digital archive file store. For the time being, the metadata I am holding about these files is pretty limited (I am currently working to level one of the NDSA Levels of Digital Preservation – metadata comes further down the line so I’m temporarily off the hook!).

I do have some technical metadata for files (MD5 checksums and output from DROID) however, the need to have a system in place to hold information about preservation actions is becoming more pressing. Working on the premis that doing something is better than doing nothing, there have been certain files that I have already been compelled to migrate to file formats more suitable for long term archiving. Documenting these kinds of actions is an essential part of any digital archivists job but my problem at the moment is that I have no proper system in which to hold this information.

The benefits of using PREMIS are clear. As a couple of the speakers at yesterday's event stated, it provides a metadata schema that includes all things “that most working preservation repositories are likely to need to know to support digital preservation”. It is flexible enough that you can pick and choose from the elements, using only those that you require. If this is the case then I do not see a reason to re-invent the wheel and design a new schema to record the technical metadata and preservation actions that I need to record.

So what is stopping me from starting to use this today? Firstly a confession – I like databases. Given the choice I would much rather work with databases than XML. This isn’t a problem in itself. Rob Sharpe from Tessella explained that a system can be externally PREMIS compliant even if PREMIS isn’t used internally. So long as the metadata recorded by an archive can be mapped to PREMIS then PREMIS XML could be exported and made available to others.

Following on from this, it should not really matter whether you prefer databases or XML so long as you have the right tools for the job. A preservation system in which much of the preservation metadata is automatically generated and where a user-friendly interface is provided for the creation and editing of preservation metadata would be the ideal. How the data is stored behind the scenes is not so important as long as you know you are storing the right bits of information.

Fortunately there are several digital preservation tools into which PREMIS is incorporated. Some of these such as Rosetta and Archivematica are already on my list of systems to look at. Armed with the knowledge I gained yesterday (and a pocket full of jelly beans) I am in a good position to push forward with my investigations.

Jenny Mitcham, Digital Archivist