NLN Digital preservation – Blog

PREMIS without a single right answer – our approach to preservation metadata and events

Tue, 24 Mar 2026 00:00:00 +0000

🗂️ Preservation metadata describes the origin of digital material, where it comes from and how it was created. It documents the actions and events (in this blogpost referred to as “events”) that have affected a digital object, such as creation, migration, validation, and transfer. This type of metadata is particularly important for digital materials, as it ensures traceability and provides a record of what has been done to the object throughout its lifecycle.

🕰️ Historically, there has been limited focus on systematic work with preservation metadata for digital content at the National Library. Following the establishment of the Digital Preservation Team, dedicated attention has been given to how both data and metadata will be better managed over time. The PREMIS standard¹, which is designed to support the structured description of preservation metadata, has been discussed on several occasions. When the decision was made to adopt the E-ARK standard for our preservation packages, the choice became clear, as E-ARK also recommends and refers to the use of PREMIS for this type of metadata. In recent years, we have documented certain preservation events in a separate database. As part of the ongoing work to upgrade our DPS solution², we also decided to explore the possibility of implementing PREMIS events.

🧐 As we had little prior knowledge of PREMIS, we had to begin by familiarizing ourselves with what the standard is, what it is intended for, and how it can be applied. We quickly discovered that there is limited guidance on how it should be implemented in practice, and that the standard functions more as a flexible framework that can be interpreted and realized in many different ways.

📊 Next our approach was to begin with a systematic review of the PREMIS Data Dictionary. We examined all entities and their semantic units to map what we already document, what would be relevant to document in the short term (to get started), what we would like to implement over time, and what we consider less relevant for our collection. Throughout, we have had to balance the need for sufficient documentation against the risk of ending up with unnecessary amounts of metadata being stored and managed. It is also important to keep in mind the scale of our holdings, larger than most, about 19 petabytes (19.000 terabytes) with unique data by the end of 2025. Even by documenting only a limited set of events in the first DPS solution, we have already accumulated more than 53 million events at the package level and over 76 million events at the file level.

🔄 We then identified which events were relevant for the team to document internally within the DPS workflow, during ingest, within the preservation process, and at the point of dissemination. As an extension of this work, we saw that it could be possible for depositors to add PREMIS events via our API at the same time as they submit new information packages (SIPs). In this way, we can also enable depositors to include relevant preservation metadata that has been documented before the digital material is received in the DPS. Events submitted via the API are preserved in the same way as those created within the DPS.

📋 To determine which events we consider relevant to document, we based our work on the Library of Congress list of eventTypes and refined it to better fit our material and preservation environment. At present, we allow 11 event types to be submitted via our API, while some additional types are used internally within the DPS. The set of event types currently accepted through the API has initially been developed for the National Library’s internal production workflows, but will be adapted for external depositors as needed.

💾 So far, we have adopted a simple modeling approach for PREMIS events, documenting three core components: object, event, and agent. For the object, we record two levels: the intellectual entity (essentially the information package) and the file level. We have decided to continue using our own event database, rather than writing events to files within the information package. This approach makes it easier to collect information and add new events without having to update the information package itself. Events are stored in the database in JSON format, not in PREMIS.xml. We have planned for the possibility of writing events from the database into information packages (for example, DIPs at dissemination) in PREMIS.xml format. This also applies to events submitted by depositors via the API. For clarity, our databases are maintained according to the same preservation standards as our bit-level storage.

In addition to recommending the submission of preservation metadata as events via the API, it is also possible to include this type of metadata as part of the information package (SIP). More information can be found here: SIP 1.0 (E-ARK).

🚀 We view the work we have done so far as a starting point. We have chosen to document what we consider important for now, to get up and running, with a deliberate approach that allows us to adjust course along the way. For us, it is more important to establish a practice that can be further developed than to aim for a perfect solution from the start, where we risk spending time on endless discussions - and few practical solutions.

Documentation on the use of event elements and event types is published here: Events/preservation metadata.

More technical information is available here: Swagger DPS Submission Service API.

Please feel free to reach out if you have any comments or questions 😊

Preservation Metadata Implementation Strategies (PREMIS) is a metadata standard for recording information required for preservation of digital objects. The standard’s documentation and metadata schema are hosted by the Library of Congress. ↩︎
Digital Preservation Services (DPS) is an umbrella term for services and software used to manage digital preservation at the National Library. The system ingests data for preservation, ensures data integrity, and manages access to preserved data. ↩︎

Backend Developer – Preserving the Nation’s Memory

Fri, 16 Jan 2026 00:00:00 +0000

Backend Developer – Preserving the Nation’s Memory

We have received funding to expand our team with a new IT position.

We are currently a multidisciplinary product team of eight people, half of whom are IT developers. The team is responsible for the development and management of the National Library’s core systems for digital long-term preservation.

By the end of 2025, the National Library’s digital collection amounts to approximately 19 petabytes, consisting of more than 2 billion files. We receive more than 6 terabytes of new data daily, distributed across around 5,000 archival objects.

Please feel free to contact us if you are interested, or if you know someone who might be.

You can read more about the position here.

Mission accomplished!

Wed, 17 Dec 2025 00:00:00 +0000

The National Library of Norway, the Norwegian Directorate for Cultural Heritage, and KulturIT were commissioned by the Norwegian Ministry of Culture and Equality to carry out a pilot project for the long-term preservation of digital museum objects using the National Library’s digital preservation solution (DPS). The assignment was given in December 2024.

Over the past year, the National Library and KulturIT have worked closely together to develop and test a solution for the ingest and delivery of digital museum objects, initially focusing on images. For the Digital Preservation team at the National Library, this work required the establishment of several new functions within the DPS.

An API interface has been developed to enable communication between various data systems and the DPS. The API handles authentication and authorization, and all data exchange is governed by agreements between the National Library and the institutions that chooses to use the service. More information about the API is available here.

For data ingest and delivery, a data exchange platform based on the Amazon S3 protocol was tested. This solution allows users to transfer data without needing to understand how the data is stored within the platform.

Requirements have been established for data delivered to the National Library to comply with the E-ARK standard for the structure and content of information packages. More information about these requirements is available here.

In addition, metadata requirements have been defined for information packages submitted for preservation. These metadata are considered essential for the management and preservation of the material within the preservation environment and are submitted via the API. More information about the metadata requirements is available here.

KulturIT has, in parallel, developed functionality for selecting museum objects for preservation, as well as solutions for for ingesting and retrieving preserved museum objects from the National Library’s digital preservation system (DPS) based on our requirements.

The solution has been tested by three of KulturIT’s owner museums, and the pilot project is considered successful.

A final report has been submitted to the Ministry of Culture and Equality. The report describes how the assignment was carried out and recommends establishing a permanent service for the long-term digital preservation of images for museums. This service should build on the results of the pilot and include an initial phase involving five museums.

In the longer term, the service may be expanded to include additional museums and extended to cover other media types, such as text, audio, and moving images.

The National Library of Norway, the Norwegian Directorate for Cultural Heritage, and KulturIT have collaborated closely throughout the pilot and support the report’s recommendations for further work.

Workshop with KBNL

Thu, 27 Nov 2025 00:00:00 +0000

On 18-19 november, three colleagues from the KB, National Library of the Netherlands (KBNL), visited the Digital Preservation Team at the National Library of Norway (NLN).

Guests from KBNL: Inge Hofsink, Richard Ligtenberg og Lonneke Smit

Workshop with KBNL

Over the course of two days, we had a tightly packed schedule, starting with a guided tour of the National Library’s facilities.

While NLN holds all media types and carries out extensive in-house digitisation, KBNL primarily handles text-based material, and all digitisation is carried out through tender processes in which they receive the digitised result.

After the tour, we organized a joint workshop. The focus here was to share each institution’s experiences working with digital preservation.

NLN presented the architecture of our Digital Preservation System (DPS) and its roadmap. KBNL presented their preservation solution, which is built on an internally developed system called DAPPR and uses ExLibris Rosetta and S3 for object storage (they also use ExLibris Alma). There was also a focus on experiences with metadata standards such as Dublin Core, MODS, and PREMIS.

The final part of the workshop was devoted to discussing organisational challenges related to digital preservation work. How can we ensure that institutions keep a continuous focus on digital preservation? How can predictable framework conditions be established to sustain this work? Can we avoid leaving behind a period of digital dark age? Many interesting and valuable reflections emerged. We will return to this topic in a later blog post on Digital preservation at the National Library of Norway.

The NLN–KBNL workshop

World Digital Preservation Day

Thu, 06 Nov 2025 00:00:00 +0000

World Digital Preservation Day is an annual event held on the first Thursday of November. The event is organized by the Digital Preservation Coalition (DPC), and the day features events, campaigns, and the sharing of experiences worldwide. Its goal is to highlight both the value and the challenges of long-term preservation of digital data. This year’s theme is Why preserve?

Why digital preservation at the National Library of Norway?

The National Library of Norway’s mandate is to collect, preserve, and make accessible all published content across all types of media. This means that everything published in Norway, or published abroad about Norway, must be preserved and made available both now and in the future - whether it is physical or born-digital content. “All types of media” include, among other things, books, journals, e-books, audiobooks, posters, cards, advertisements, theatre programs, all Norwegian websites, radio broadcasts, and television programs - the list is extensive ! For this reason, the National Library is considered the most important source of knowledge about Norway and Norwegian society.

The library’s societal mission is to give the people of Norway access to our shared knowledge and cultural history. This is achieved by preserving, digitizing, and disseminating everything published for the public. Access to our cultural heritage is essential for democracy and public enlightenment, and provides a foundation for identity and historical understanding. ¹

The efforts in digital preservation at the National Library are closely related to:

📚 Ensuring that today’s society, culture, and knowledge remain accessible for future generations - Properly managed digital content lasts longer. Active preservation protects material from technological obsolescence, and with sufficient metadata, it can be understood and playable far into the future.
🌍 Bridging generations and geographies, making knowledge and culture accessible to everyone - Digital content can be accessed anytime, anywhere, making sources for understanding the past and present more readily available to all.
⚖️ Preserving digital collections to ensure compliance, accountability, and transparency across sectors - When public institutions preserve documents, data, and communication digitally in a secure and systematic way, it enables transparency and reuse of information, makes it possible to trace decisions, actions, and resource use, and ensures that laws, regulations, and requirements are followed.
🛡️ Protecting the integrity of our sources, research, and history - In an era of misinformation and manipulation, preserving original documents, images, and audio recordings digitally with secure metadata standards ensures authenticity and reliability. Digital preservation allows research data, methodologies, and results to be stored in ways that can be verified and reused. Storing content in secure environments and multiple independent copies makes deliberate manipulation more difficult.
🌱 Enabling sustainability, adaptability, and long-term thinking - not just storage! - Digital preservation reduces unnecessary duplication and physical wear on archival materials while using resources efficiently over time. Digital collections can be easily integrated into new platforms, used in research, teaching, or public outreach, and adapted to future needs. In short, digital preservation is not just about storing files - it’s about building a system that keeps knowledge accessible and relevant over time, regardless of technological or organizational changes.

Happy World Digital Preservation Day! 🎉

National library of Norway strategy and societal mission, https://www.nb.no/om-nb/mandat-og-strategi/ ↩︎

Specifications for SIP Submission and Metadata Requirements

Tue, 08 Jul 2025 00:00:00 +0000

As part of the pilot project on long-term preservation of museums’ digital cultural heritage, the team has been working on defining a standardized approach for the transfer of digital objects to the National Library of Norway. The work is based on the E-ARK specification, from which it is defined more specific requirements for the delivery of information packages.

In addition, the team have focused on how to ensure that a minimum set of descriptive metadata is submitted alongside the digital objects. The primary purpose of this is to enable basic search and identification within the preservation environment, and it is not intended as a substitute for richer metadata delivered within the package itself. The approach is based on the Dublin Core metadata standard, which is a basic standard, features open fields (no controlled vocabularies), and can be used for a variety of media types.

Documentation for SIP submission is available here: SIP 1.0.

Documentation on metadata submission requirements is available here: Metadata Requirements.

Initially, the published documentation is intended for use within the scope of the pilot project. It primarily focuses on the submission of still image material. However, the intention is to use the same specifications for internal deposit workflows at the National Library as well. Further updates and modifications to the documentation will be introduced as we proceed beyond the pilot.

Do not hesitate to contact us should you have any comments or questions! 👩🏻‍💻

Preserving 1,6 million hours of television

Thu, 12 Jun 2025 08:00:00 +0100

The National Library of Norway has recently completed one of its largest digital preservation projects ever: transferring the entire archive of historical digital television broadcasts to a new preservation system. Over 1.6 million hours of television, spread across as many files, were analyzed, quality assured, and repackaged—amounting to a total of 1,800 terabytes of data.

This is the story of how we did it.

Background: One Hour, One File

Since 2007, Norwegian broadcasters have delivered television broadcasts digitally to the National Library of Norway — one MP4 file per calendar hour of TV transmission. Each file contains everything aired during one hour, regardless of where programs start or end. In practice, this means a single TV program may be spread across multiple files.

The files were stored in an older bit repository (Oracle HSM) in three copies. Now, the entire digital collection were to be rearchived into a modern preservation system — a process that imposed new requirements for quality, metadata, and packaging.

Challenges and Decisions

✅ Checksums for All Files
Some broadcasters, like TV2 and TVNorge, delivered files with checksums¹ — a digital signature confirming that the file hasn’t changed. But files from other broadcasters lacked these, so we had to generate our own checksums.

🔠 Standardizing File Names
The file names contain important information — broadcaster, channel, date, and time — but many did not follow the standard format. Some lacked information about the broadcast time, others had the channel name misplaced. Before automated processing could begin, thousands of file names had to be corrected.

🔍 Validating Technical Properties
Each file was analyzed using the tools MediaInfo² and MediaConch³ to validate the format. We checked that the files contained both audio and video, had the correct duration, and weren’t truncated or empty.

📄 Metadata and MODS
Since none of the files had catalog data, we generated MODS⁴ metadata for each one. Information from the file names was extracted and combined with technical metadata from the analysis tools.

✏️ Expanding Channel Names to Full Form
In the original file names, abbreviated channel names were used, such as “BLI” for “TV2 BLISS.” These were converted into full channel names in the metadata to ensure future clarity and understanding.

📦 Packaging Files in E-ARK Format
All files were packaged according to the E-ARK standard⁵, a European standard for long-term preservation. We used the open-source tool Commons-ip for both packaging and validation.

📝 Documenting Preservation Activities with PREMIS
Changes, anomalies, and technical conditions were documented using PREMIS⁶ metadata, ensuring the entire rearchiving process is traceable and verifiable in the future.

Technical Solution and Progress

Work began in autumn 2024 with mapping and preparation. The actual rearchiving began in November 2024 and was completed on February 4, 2025. The process was automated using Apache NiFi⁷, which managed data processing with high control over flow and capacity.

By combining automation with targeted manual reviews, we significantly increased efficiency. Daily processing capacity rose from 25 to over 40 terabytes.

Overview of data volume rearchived per day. 1 TB = 1 GB

Findings and Anomalies

With a dataset this large, some anomalies were unavoidable. Here are some examples of what we encountered:

Duplicates: Some were caused by seasonal time changes (switching to and from daylight saving time), others were identical broadcasts from different NRK districts. Files were assessed, and the best versions were kept.
Incorrect Checksums: Several files arrived with invalid checksums. After closer inspection, we found that all copies were in fact identical, indicating that the error occurred prior to delivery. We updated the checksums, and the issue was documented.
Empty Files: Around 30 files contained no data. In cases where it was possible, we retrieved the original files. Otherwise, the files were documented as empty with appropriate metadata.
Unknown File Types: Some MP4 files had incorrect signatures and were unrecognized by file identification tools. Some turned out to be QuickTime files in the wrong container. These were also documented.
Truncated Files: Some files were technically cut short, containing only partial content. These were preserved with documented notes on the issue.

What We Learned

This project showed that when preserving large volumes of data over the long term, it is critical to have robust tools, standards, and reliable control routines. It also demonstrated the importance of combining automation with professional judgment.

By using Apache NiFi for data flow and the E-ARK standard for archival packaging, we developed a scalable, reusable solution for future preservation efforts.

In Conclusion

Television is a central part of our shared memory. By securing and structuring this material for the future, we enable tomorrow’s researchers, journalists, and the public to understand how Norway has evolved—hour by hour.

Pilot Project for Long-term Digital Preservation of Museum Collections

Tue, 28 Jan 2025 08:00:00 +0100

The National Library of Norway, in collaboration with KulturIT and commissioned by the Ministry of Culture, has initiated a pilot project focused on ensuring the long-term preservation of museums’ digital cultural heritage. This pilot represents a strategic initiative in preserving Norway’s digital cultural heritage for future generations.

The primary objective is to evaluate whether the National Library’s digital preservation solution can be integrated with KulturIT’s collection management systems currently used by museums. The pilot emphasizes security protocols, user experience optimization, and process automation, with subsequent evaluation based on cost-benefit analysis.

Key Objectives and Technical Components

The pilot project encompasses several critical technical and operational elements:

Technical Integration: Implementation of machine-to-machine communication between KulturIT’s systems and the National Library’s preservation infrastructure, using REST API technology. This integration facilitates efficient ingestion and retrieval of archival files.
Intermediate Storage: Development of an S3-protocol-based intermediate storage platform, ensuring secure and efficient file transfer operations between systems.
Standardized Package Formats: Implementation of E-ARK standards to maintain interoperability and ensure consistency in long-term data preservation.
Authentication and Authorization: Development of robust security mechanisms to ensure that only authorized users and systems can access preserved files and associated functionality.
Museum User Interface: Implementation of an intuitive interface enabling museums to select preservation-worthy data and efficiently retrieve preserved data.

The pilot will be conducted with a minimum of two participating museums, initially focusing on the long-term preservation of digital images. Beyond technical infrastructure testing, the solution will validate that all archival packages received from museums meet specified preservation requirements.

Architectural Overview

The following architectural diagram illustrates the proposed solution framework:

Proposed DPS Architecture

Clients: Entities responsible for submitting or retrieving materials preserved by the National Library.
Intermediate Storage: Platform services implementing shared intermediate storage solutions based on S3 protocol.
Archival Storage: Bit-repository services where preservation data is archived following the 3+2+1 principle. That is three copies, two technologies (disk, tape, tape) and one geographically distributed copy.
DPS - Digital Preservation Services: The National Library’s digital preservation solution, based on the OAIS standard. This encompasses intermediate storage solutions, bit-repository management, API interfaces, and verification mechanisms for materials received and distributed within the National Library’s preservation environment.

Pilot Scope Limitations

The pilot explicitly excludes functionality for access copies or metadata updates. However, the potential for future expansion to include these capabilities will be evaluated during the assessment phase.

Project Management and Reporting Structure

The pilot is managed by the National Library of Norway in partnership with KulturIT and the Directorate of Culture. The project team will maintain regular reporting cycles to the Ministry of Culture. A comprehensive final report, including findings, cost estimates, and recommendations for future development, will be submitted by December 1, 2025.

The National Library of Norway is looking forward to productive collaboration with KulturIT in developing solutions that will enhance museums’ capabilities for preserving and accessing digital cultural heritage materials. This pilot represents a significant step toward ensuring the long-term preservation of Norway’s digital cultural heritage.

Moving 4.3 million digital newspapers

Wed, 06 Nov 2024 12:46:00 +0100

The National Library of Norway is in the process of replacing its bit-repository purchased in 2007 with a more modern preservation solution for digital material. This solution is based on in-house developed software called DPS (Digital Preservation Services) and uses IBM-HPSS as the underlying system for data storage.

Transition to the new preservation solution

In 2023, all archiving of data at the National Library was moved to the new DPS preservation solution. By this time, the old bit-repository contained over 14 Petabytes¹ of digitized and deposited historical material, which must be re-archived to DPS. A key part of this process is analysing and repackaging the historical data to meet the requirements of the new DPS.

It was decided that the newspaper collection would be the first material type to be re-archived. This process was carried out as a collaborative project between the Text team and the Digital Preservation team at the National Library. The Text team is responsible for digitizing all text-based material, and the Preservation Team is responsible for preserving the digital collection.

Brief overview of the National Library’s digital newspaper collection

The collection consists of born-digital and digitized newspapers from 1763 up until today.
The total newspaper collection consists of approximately 4.6 million newspapers across 1,800 newspaper titles. Of these, around 4.3 million newspapers were to be re-archived to DPS.
In total, over 16 million packaged files had to be moved from the old bit-repository.
This amounted to a total of approximately 2.5 Petabytes of data.

The National Library’s digital newspaper collection comes from three sources:

Legally deposited PDF newspapers.
These are the print files of the newspapers, which are downloaded in PDF format. The newspapers are downloaded by the National Library from the newspaper publishers daily, then processed before being prepared for dissemination and digital preservation. As of 2024, a total of 220 published newspapers are received in this way on a daily/regular basis. This material makes up about 6% of the newspaper collection.
Newspapers scanned from microfilm.
These are paper newspapers that were first photographed on microfilm and then the microfilm was digitized. The digitization was carried out by commercial companies. Most of these newspapers are published before the year 2000. This material makes up about 41% of the newspaper collection.
Newspapers scanned from paper.
Original paper newspapers that have been scanned and processed internally by the National Library. This material makes up about 53% of the newspaper collection.

Newspapers can vary greatly in size and number of pages, depending on the publicised era and publication format.

For almost 20 years, the National Library has systematically digitized and received digital newspapers on a large scale. The methods for doing this have evolved over time. This means that newspapers digitized or received 20 years ago were packaged and archived differently than the newspapers being digitized today.

Here is an overview of what is done with digital/digitized newspapers that are archived today:

Each individual page is OCR² processed.
Each individual page is going through a structure recognition process where content is analysed to recognize images, headlines, publisher information, publication date, etc. This information is stored in separate OCR/ALTO³ files.
Separate JP2K (.jpx) files are created for dissemination on nb.no. This also applies to PDF-deposited newspapers.
Separate JP2K (.jp2) files are created for long term preservation.
A METS⁴ structure is created for all newspapers. This links the content of each newspaper so that it is possible to virtually browse through the newspaper, search its content in free text, and get highlighted search results.
As an example: A 20-page newspaper issue can consist of about 160 individual files. These include both metadata files and content files. Typical file types are JP2K-HQ, JP2K-LQ, OCR, ALTO, METS, JHOVE, and other metadata about the newspaper object.
All similar file types are packed together in container files of the tar⁵ type. A newspaper issue consists of a folder with .tar files.

This means that the oldest digitized newspaper material in the collection may not necessarily have been processed in the same way as today’s newspaper material.

Improvements/decisions that impacted the re-archiving process

Before starting the re-archiving of newspapers from the old bit-repository to the new DPS, the content was analysed. Based on the analyses, several decisions were made that influenced the design of the new archival packages in DPS:

All files that are re-archived must have accompanying checksums⁶. Very few of the original newspaper files in SAM-FS had any checksum information.
Internal structure cleanup of existing archival packages was to be performed. Files that were not deemed preservation-worthy would be removed from the archival packages. Examples of such files include temporary status files from the production process.
File identification would be performed on all files. The tool DROID⁷ was used for this. DROID uses the PRONOM⁸ register as authority.
Documentation of what was done with each file during the re-archiving process was required. This was to be done by creating specific “events” in the DPS database, based on the PREMIS⁹ standard.
Data sent to DPS would be deleted in the old bit-repository as soon as DPS confirmed that the data was archived there.

Timeline for the re-archiving process

March-April 2023

The two teams began with preliminary investigations and mapping of the newspaper material in the old bit-repository in the spring of 2023.

May 2023 (migration start)

A test migration to DPS was conducted in May 2023. This revealed the need to clean up the source material in the bit-repository. A production workflow was created specifically for re-archiving newspapers to DPS.

June-July 2023

During the summer months of 2023, the process of generating trustworthy checksums for the newspaper files started. This is described in more detail in a separate blog post.

October 2023

Large scale re-archiving of deposited newspapers and newspapers scanned from microfilm began in October 2023. In total, approximately 1PB of data was re-archived. The process was then halted due to delays in the delivery of new data storage resources to DPS for purchase and tendering reasons. We simply ran out of storage.

January 2024

In January 2024, re-archiving of the newspapers scanned by the National Library began. This amounted to approximately 1.5 PB of data.

June 2024 (migration completed)

Data volume processed over time

The highest volume of data processed and re-archived in one day was: 42.15 TB.

Newspaper volume processed over time

The highest number of individual newspapers migrated in one day was: 97,528.

Results and lessons learned

The archiving speed varied considerably over time, with rates reaching 42 TB per day for simpler newspaper materials. This demonstrated HPSS’s considerable ingestion capacity and helped us establish realistic timelines for future migrations. However, complex materials (like those requiring OCR reprocessing) had significantly lower throughput rates, due to additional pre-processing time.
We experienced an unplanned pause in migration due to lack of storage media. This highlighted that large-scale migrations require comprehensive planning beyond technical considerations. Critical administrative prerequisites include active procurement agreements, completed tender processes, and streamlined purchasing procedures - all of which need significant lead time and budgetary planning.
Old data can be messy, and migration projects present valuable opportunities for cleanup.
- We discovered and removed 4.5 million 0-byte files - remnants of temporary status tracking from historical digitization processes. This cleanup removes uneccessary materials and aligns our preservation packages with our principles.
- We cleaned up material that was incorrectly packaged or had incomplete information, ensuring that the newspaper collection is uniformly archived in DPS.
- Poor OCR quality was identified for two years of newspaper volumes (53,678 issues, 107 TB) from 2010-2011 - the first two years of OCR processing. The files were reprocessed using the current newspaper production workflow, to produce new and improved OCR.
- Some digitized newspapers from microfilm were of very poor quality. These were replaced with digital copies from scans of the original paper newspapers. In total, this amounted to 2,051 newspaper issues. It illustrates how original source material often produce better results than secondary generations of copies.

Svein Erik Molberg and Mona Løkås from the Text Team, Vigdis Sørensen from Digital Preservation.

Some numbers

Overview of re-archived newspapers into DPS

Source	Newspaper editions	Newspaper editions in %	Data volume in TB	Data volume in %
Born digital	247 385	6%	75	3%
Microfilm	1 779 043	41%	784	32%
Paper base	2 250 038	53%	1 580	65%
Re-archived to DPS	4 276 466	100%	2 439	100%

Overview of all newspapers in DPS (re-archived and new acquisitions since 2023)

Source	Newspaper editions	Newspaper editions in %	Data volume in TB	Data volume in %
Born digital	302 372	7%	87	3%
Microfilm	1 899 481	41%	820	31%
Paper base	2 397 660	52%	1 703	66%
Total in DPS today	4 599 513	100%	2 610	100%

Better Late Than Never: Adding Checksums to 16 Million Legacy Files

Wed, 06 Nov 2024 12:45:00 +0100

The National Library of Norway has used SAM-FS¹ as a system for long-term storage and archiving of large amounts of data since 2007. SAM-FS contains 14 Petabytes of data and will soon reach “end of life” status as a product.

In 2022, the National Library decided to replace SAM-FS with a more modern preservation solution for digital material. This new solution is based on in-house developed software called DPS (Digital Preservation Services) and uses IBM-HPSS as the underlying system for data storage.

Over the last 10 years, the National Library has used checksums² as a verification technique for preserved data. In this context, a checksum is a calculated hash string used to verify that a data file has not been subject to any changes. Common checksum calculation algorithms include MD5, SHA-1, SHA-256, or SHA-512. The National Library uses MD5³.

Lack of checksums

Many of the oldest files in SAM-FS lacked checksums when they were stored. As all files in SAM-FS are stored in three copies, you could say that without an accompanying checksum the three copies exist independently of each other. If a discrepancy were to arise between the three copies, we would have no original checksum to use for verification.

Data stored in SAM-FS in 3 instances

As part of the process of migrating data from SAM-FS to the new DPS, it was decided that checksums should be calculated and stored for all files that did not already have one.

Challenge

How could we ensure that files being transferred from SAM-FS to DPS were the same as those originally archived when there were no checksums to verify this? The oldest files were over 20 years old and had been subjected to up to five hardware/platform migrations over time (see Figure 4).

Which of the three file instances stored in SAM-FS should we choose as the starting point for migration? How could we know that this was the “correct file” without having to read and compare all three instances? Reading and comparing all three instances was considered to be difficult, as this involved reading and processing many Petabytes⁴ of data on the same infrastructure that was also used for daily operations.

Solution

The fact that we had the files preserved in multiple instances helped us, as we were about to generate checksums for “old” files for the first time. One copy is stored on disk (disk copy), and two copies are on tape (tape copy 1 and tape copy 2).

Checksum calculation

First (Step 2), we created a program (script) that extracted all files in SAM-FS without checksums from one of the tape copies. A checksum was then calculated for each individual file, which was stored in a database.

This process took just over 2 calendar months to complete for a dataset of 2.5 Petabytes.

Checksum verification and transfer to DPS

The actual task of transferring files from SAM-FS to DPS began after all the files on tape had been read, and checksums had been calculated and stored in a database.

The migration then started by retrieving a file from the disk instance in SAM-FS (step 3). A checksum for this file was then calculated and compared with the checksum stored for the corresponding tape copy file. If these matched, it meant that at least two of the three file copies were identical.

The file was then considered to be OK and was transferred to DPS with the accompanying new checksum (step 4). If the checksum did not match, it meant that either the disk or tape copy of the file was incorrect. This was then reported as an error and had to be followed up manually by checking the third copy to see if it matched one of the two that had already been checked.

Outcome and lessons learned

None of the 16 million files in the dataset of 2.5 Petabytes had checksum discrepancies. This method of ensuring that the files in DPS are authentic after migration was both time- and resource-consuming, but it proved to work well for us.

Another experience we had was that one can trust technical storage systems when it comes to avoiding changes in the bit pattern over time. We have checksum-verified 16 million files that have undergone up to five technological shifts over 20 years, without finding any trace of changes in the bit pattern of any of the files.

Platform/Technology generations in SAM-FS

Overview of technology shifts in the National Library’s SAM-FS long-term storage system. The TB value here is native storage capacity per storage unit, disk/tape:

Time period	Disk copy	Tape copy 1	Tape copy 2
2007-2009	SUN 6140 (1TB)	SUN SL8500 – T10kA (500GB)	SUN SL8500 – T10kA (500GB)
2009-2011	SUN 6180 (2TB)	SUN SL8500 – T10kB (1TB)	SUN SL8500 – T10kB (1TB)
2012-2016	Nexsan (3TB)	SUN SL8500 – T10kC (5TB)	SUN SL8500 – T10kC (5TB)
2016-2019	Nexsan (8TB)	as above	as above
2020-2022	Fujitsu (16TB)	SUN SL8500 – LTO8 (12TB)	SUN SL8500 – LTO8 (12TB)

Hierarchical storage Management System, SAM-FS is also known as Oracle HSM ↩︎
https://www.dpconline.org/handbook/technical-solutions-and-tools/fixity-and-checksums ↩︎
https://en.wikipedia.org/wiki/MD5 ↩︎
1 Petabyte = 1.000 Terabyte ↩︎

Digital preservation poster

Mon, 30 Sep 2024 00:00:00 +0000

In early 2023 the digital preservation team had a poster made to spread awareness of digital preservation in the wider organization. The poster was drawn by our former colleague Vegard Orheim Stoveland in colloboration with the digital preservation team. The poster was based on the first revision of the newly defined principles for digital preservation in the National Library. The poster aimed to ilustrate why digital preservation is important, and how we intend to do it.

We hoped to share a revised version of the poster at iPRES 2024, but sadly it did not get accepted¹. Of course, that means we don’t have to save it for IPres any longer, and can share it freely here.

English version

click the image for full resolution

Full resolution PDF (16 MB) can be found here.

Norwegian version

click the image for full resolution PDF (16 MB)

Full resolution PDF (16 MB) can be found here.

However, our poster showing our Grafana monitoring dashboards was accepted! ↩︎

Rearchiving 2 million hours of digital radio, a comprehensive process

Wed, 28 Aug 2024 00:00:00 +0000

The National Library is in the process of a major overhaul of its 2007 bit-repository, replacing it with a contemporary digital preservation system. This new solution is based on an in-house developed system called DPS (Digital Preservation Services), which uses IBM-HPSS as the underlying bit repository for data storage. This transition, which is expected to span over a couple of years, is necessary to ensure the long-term preservation and accessibility of the National Library’s digital collection.

Transition to a New Preservation Solution

In 2023, all new data archiving was transferred to the new DPS preservation solution. At this time, the old bit repository contained over 14 Petabytes of digitized and legally deposited historical material, which needs to be re-archived into DPS. A key part of this process involves analyzing and repackaging the historical data to meet the new DPS requirements.

Historical Legally Deposited Radio

Among the materials to be re-archived are 2.2 million hours of digital radio, equivalent to 2.5 million files and a total of 1 Petabyte of data. This includes both born-digital and digitized radio programs from the period 1993-2022.

In 1993, there were four radio channels delivering 16,500 hours of radio. By 2022, the number of radio channels had increased to 30, collectively delivering 150,000 hours of radio. With the phasing out of the old bit repository, it became necessary to move this data to the new preservation solution.

DSM to DPS: A Thorough Process

DSM (Digital Longterm Storage) has been the National Library’s internal management system for legally deposited radio for the past 20 years. The data has been stored in an Oracle HSM bit-repository in three instances (disk, tape, tape), and the radio material was fetched daily from various broadcasters. Some radio broadcasts were stored as mp3 and wav files, with accompanying checksum files. Other broadcasts were only stored as mp3.

Before the re-archiving process began, it was decided to generate new MP4 playback files from the wav files to replace the varying qualities of the old mp3 files. The new MP4 files is not to be archived in DPS, as they are secured on the Wowza viewing platform. The new MP4 playback format chosen was 160kbit AAC with the M4A (audio) container. Fraunhofer FDK AAC (libfdk_aac) was used as the codec.

Technical Improvements and Metadata

All objects were packaged according to the E-ARK standard, and metadata was generated in MODS format. File identification was performed using the tool Siegfried, and technical metadata is extracted using Mediainfo. Each file was carefully processed and documented, and “events” were created to record what had been done. These were stored in the DPS database.

Timeline and Experience

Preliminary studies and mapping began in the fall of 2023, and the development of the re-archiving workflow, based on Apache NiFi, started in December 2023. The first version was put into operation on February 27, 2024, and by June 24, 2024, the re-archiving of 2.2 million radio programs was completed.

The generation of new MP4 files was time-consuming, but an infrastructure running 35 parallel threads made it possible to achieve a re-archiving rate of 9 Terabytes per day.

Findings and Lessons Learned

During the process, discrepancies were found in 577 out of 2.2 million objects. None of the errors occurred while the data was stored in the Oracle HSM bit repository. Most errors are related to inadequate control routines when the material was first received and archived.

Checksum mismatch (4 cases)

Cause: Objects failed due to a checksum mismatch, where the recorded checksum and the actual calculated checksum during re-archiving differed.
Action: For each of these objects, all three copies in Oracle HSM (disk, tape, tape) were checked and found to be identical. This means the discrepancy between the actual checksum and the recorded checksum must have occurred before the object was first archived. For these four objects, we chose to include the mp3 file in the preserved object, as it had the correct checksum. Additionally, an “event” was recorded describing why we also preserved the mp3 file.

Null-byte file (1 case)

Cause: An object where the wav file was null byte, causing a checksum mismatch.
Action: We checked all three copies in Oracle HSM and found they were identical. The discrepancy between the actual checksum and the recorded checksum must have occurred before the object was stored in Oracle HSM. For this object, we chose to include the mp3 file in the preserved object, as it had content and the correct checksum. An “event” was also recorded to explain why the mp3 file was preserved.

Corrupt file (1 case)

Cause: A file with the correct checksum that failed because we couldn’t extract “duration” from Mediainfo, revealing that the file was corrupt.
Action: We chose to archive the corrupt file in the absence of alternatives and recorded an “event” describing our findings.

URN validation failed (507 cases)

Cause: Objects with a URN that was not in the expected format. The URN validator extracts data from the URN and uses it to build MODS metadata for the object to be archived. In this case, the validator indicated that the URN content did not match what it expected. The discrepancy appears to be due to file delivery errors related to a broadcaster for a short period in October 2011.
Action: URNs were changed to the correct format.

Duplicate files (48 cases)

Cause: During the re-archiving process, it was discovered that 48 objects from DSM were already archived in DPS. This is related to the transition of radio production workflows on March 21, 2023. At that time, daily legal deposit of radio was moved from DSM to the new DPS. In this process, the first datasets were manually loaded into the new DPS to ensure the transition was correct. By mistake, they were also captured by the old production workflow and archived in DSM. The data from the transition day was therefore stored in both the new DPS and the old DSM. When we started re-archiving the radio, we encountered duplicate errors for the programs from that day since they were already archived in DPS.
Action: No action was needed as the data is correctly archived in DPS.

Duplicate files (16 cases)

Cause: After re-archiving was completed, there were 16 objects that, for various reasons, remained in the queue marked as not archived but were actually archived in DPS. These are mostly related to a cleanup after a backup-related incident on March 1, 2024.
Action: No action was needed as the data is correctly archived in DPS.

Statistics and Results

The re-archiving process resulted in 2.1 million new MP4 files, totalling 143 TB of new playback files on nb.no, replacing 40 TB of the old MP3 files. In total, 2,183,478 archive packages were re-archived in the new DPS preservation environment.

This work represents a significant improvement in the National Library’s ability to preserve and make available digital radio material for future generations.

Some numbers

Statistics from DSM (old preservation environment)

MimeType	Quantity	Bytes	TiB	TB
audio/mpeg	24 997	1 436 936 706 369	1	1
audio/x-wav	2 158 485	1 053 241 022 424 650	958	1 053
text/plain	358 462	12 177 323	0	0

Statistics from DPS (new preservation environment)

Type	Num AIPs	Bytes	TiB	TB
radio-DK	350 929	47 733 509 049 163	43	48
radio	1 832 549	1 007 674 770 422 690	916	1 007
Total	2 183 478	1 055 408 279 471 850	959	1 055

TiB (TebiByte) is bytes/1024 which is the exact unit of measurement for data volume.

TB(TeraByte) is bytes/1000 which is the most commonly used unit of measurement for approximate data volume.

2.1 million new MP4 (M4a) files were produced, which together amounted to 130TiB/143TB with new access files on nb.no. This replaced approx. 40TB with old MP3 access files.

Preferred Digital Formats at the National Library

Fri, 05 Jul 2024 00:00:00 +0000

This is an overview of the digital formats preferred for digital preservation at the National Library.

The Digital Preservation Team has compiled a list of file formats which is preferred by the National Library for digital preservation. The format list is based on recommendations from the respective media departments within the National Library.

The list includes both formats produced by the National Library itself and formats received from others.

The preferred formats is a list of formats we ideally want and are striving for. The acceptable formats will also be accepted and taken care of without conversion or normalization.

All other formats should be discussed with the relevant media departments within the National Library before they are accepted and included in the management system.

The list is published in both Norwegian and English and can be found here:

The Preferred file formats list is available here.

The Digital Preservation Team plans to conduct annual reviews of the list.

Presentation from the Digital libraries: storage for now and forever conference

Wed, 29 May 2024 00:00:00 +0000

We were invited to talk at the Digital libraries: storage for now and forever conference hosted by the IFLA Preservation and Conservation Center at the National Library of Poland. The presentation is a crash course through digital storage and digital preservation practices at the National Library of Norway over the past 20 years. It concludes with our current status and our future plans.

The recording of the event can be found below. Our presentation starts at 4h01m44s. The slides are available as a PDF here.

Ambitions, Goals, and Strategy for Digital Preservation at the National Library

Tue, 20 Feb 2024 00:00:00 +0000

The Digital Preservation Team at the National Library (NLN) has developed its first strategy for digital preservation. This strategy aims to steer, structure, and sharpen the focus of our digital preservation efforts over the coming two years.

Strategy Design

In developing the strategy, we aimed for writing a short and precise document. We wanted to avoid overwhelming the reader with a wall of text, or producing a verbose document that would be of little practical use. The purpose of the strategy would be to serve as a guideline for making key decisions in our digital preservation work.

Our design process was iterative, based on extensive review cycles. Over a span of roughly two months, a dedicated working group of selected team and departmental members met weekly. Each meeting began with a fresh review of the entire document’s text and structure, intentionally allowing time between sessions for individual reflection and idea maturation.

The initial meetings saw significant changes to the document’s structure, but we gradually achieved a stable framework, shifting our focus to refining the wording and content. It was also vital that the strategy remained flexible, avoiding putting constraints or lock-in in relation to external partners or financial considerations. The resulting concise, one-page document provides valuable guidance and a clear direction for NLN’s digital preservation efforts.

This strategy has been reviewed and ratified by the organization, initially by the team’s board of owners and subsequently by the NLN board of directors.

Structure of the Strategy Document

The document is divided into five main sections:

We begin with the rationale for digital preservation at the NLN, grounded in the mandates of Norwegian law.
Next, we outline the NLN’s vision for digital preservation. We use the concept of national digital cultural heritage here to define our scope. This concept highlights that our scope and responsibility extends beyond legally deposited materials, and also includes materials from other cultural institutions.
We then outline four specific goals. The goals aim at effective collection, secure storage, and active management of the digital collection, as well as ensuring current and future access to these materials. Acknowledging that immediate accessibility is a precondition for long-term preservation.
The document also sheds light on some major challenges facing the NLN when it comes to digital preservation, acknowledging the evolving nature of these obstacles.
Finally, we specify three strategic priorities: enhancing knowledge of digital preservation in the organization, adhering to established standards, and employing tools and technology responsibly and sustainably. These are underpinned by the current principles of digital preservation at the NLN.

Utilization of the strategy in the team

To make practical use of this strategy, the team has devised a roadmap outlining key digital preservation initiatives in the coming two years. All activities in the roadmap are tied back to the three strategic priorities.

Activities are prioritized and detailed at the monthly meeting with the board of owners, subsequently organized into “epics”¹ within Atlassian Jira.

For each epic, we outline specific Jira tasks, clarifying objectives and approaches. For activities with unclear scope, the first task will be a “spike”² where we try to break the activity into more manageable tasks, with clear specifications and “definition of done”³. This creates a cohesive thread from the overarching strategy down to the specific tasks to be carried out by the digital preservation team.

Figure of our strategy, roadmap, delivery plan, epics, and tasks

This structured approach has proven effective. It has set a clear direction for NLN’s digital preservation work, constantly reminding the team of our strategic goals through daily Jira tasks and activities.

The strategy is available here.

“Epic.” The Agile Dictionary, agiledictionary.org/309/epic/. Accessed 19 Feb. 2024. ↩︎
“Spike.” The Agile Dictionary, agiledictionary.com/209/spike/. Accessed 19 Feb. 2024. ↩︎
“Definition of done.” The Agile Dictionary, agiledictionary.org/8/definition-of-done/. Accessed 19 Feb. 2024. ↩︎

Team Digital Preservation

Thu, 26 Oct 2023 00:00:00 +0000

In June 2022, a dedicated team was established at The National Library of Norway to manage the preservation of its digital collection. The team is responsible for handling all types of digital material, whether digitized from analog sources or born-digital. This includes media formats such as websites, text documents, images, audio, and moving images.

Areas of responsibility include managing long-term digital preservation solutions and working across the entire process: ingest, quality control, storage, preservation, and access. Data included for long term preservation typically consists of large, high-quality files, as opposed to compressed access copies.

The Digital Preservation Team collaborates closely with several other specialized media teams within the institution. In addition to receiving digital material covered by the Legal Deposit Act, the National Library of Norway also produces large vulumes of data through digitization efforts. This includes both material from its own collections and from institutions across the archive, library, and museum (ALM) sector.

The team is a members of the Digital Preservation Coalition.

Organisation

The Digital Preservation team consist of 8 members:

Trond Teigen

Team lead

Torbjørn Bakken Pedersen

Product lead

Thomas Edvardsen

Tech lead

Vigdis Marie Sørensen

Senior platform developer

Siarhei Kulakou

Application developer

Johannes Karlsen

Application developer

This team reports to a committee of leaders responsible for this area in the National Library. The members are:

IT Director (Product owner)
Director of Digitalizing Cultural Heritage
Head of Metadata Standards Development Section
Head of IT Platform Section

The National Library’s digital collection in numbers

Over 2 billion files
More than 100 different file formats
18 Petabytes of data (that’s 18,000 Terabytes!) stored in 3 copies
The largest single file is 2.5 Terabytes
Daily ingest of new material averages over 6 Terabytes

Data volume by type

Video and television: 22%
Film: 21%
Newspapers: 19%
Web Archive: 16%
Radio and audio: 12%
Books: 8%
Photos: 2%

Technology choices used when working with digital preservation

Apache Kafka for sending messages between systems
Apache NiFi for running the data flows that validate, move, and package data
MariaDB as the database engine
DROID and Siegfried for file format identification
Grafana for statistics and reporting
IBM High Performance Storage System (HPSS) as bit repository
CentOS Linux as server platform
CommonsIP for packaging and validating archival packages using the E-ARK standard