Data Archiving Policy

 

A Summary of Statistics New Zealand's Policy for the Retention and Preservation of Statistical Datasets

 Official statistical datasets are valuable and irreplaceable, with their value increasing through long-term use.
 Principle 1.  Statistical datasets with potential for on-going or long-term use are archived, subject to security, confidentiality, privacy and statutory requirements.
 Principle 2.  Statistics New Zealand provides access to archived datasets, subject to security, confidentiality, privacy and statutory requirements.
 Principle 3.  Statistics New Zealand's datasets are well-managed throughout their lifecycle to ensure successful preservation.
 Principle 4.  Archived statistical datasets are reliable and authentic.
 Principle 5.  Archived statistical datasets are supported by sufficient documentation to enable informed use of the data.

Rationale

This policy statement sets out Statistics New Zealand’s approach to the long term retention and preservation of unit record and aggregate datasets held by Statistics New Zealand and gives guidance on implementing those principles. This statement has been developed from international best practise, in consultation with Statistics New Zealand subject matter areas.

The Statistics Act 1975 provides for the collection of statistical data, balancing the need to provide statistical information to users with the need to protect the confidentiality of respondents and to maintain public trust in the Official Statistics System.

This policy follows from the overarching principle that the statistical datasets created by Statistics New Zealand and other agencies in the Official Statistics System are valuable and irreplaceable, with their utility maximised through on-going use. The Statement of Principles for the Official Statistics System states this under the maximisation principle as ‘Statistical data is treated as an enduring national resource, with their value increasing through widespread and long-term use.” The utility of official statistics lies in their ability to provide snapshots of society, the economy and the environment, and to show patterns and change over time.

However, statistical datasets are fragile and their long-term availability to users is not always maintained through business-as-usual processes. Changes in systems, changes in the focus of a statistical collection and changes in data management can expose datasets to loss or damage or render them inaccessible. Datasets must be properly archived to ensure they will be available for use in the future.

A formal approach to the retention and preservation of datasets will provide for:

  • Future access. Preservation will ensure that datasets are available for use in the long-term.
  • Greater use of data. Future uses of datasets may include answering new research questions, creating new statistical compilations, and the revision or backcasting of time series not currently anticipated. Users may be within Statistics New Zealand, within the Official Statistics System, or may be outside researchers submitting research proposals.
  • Value for money. The ongoing use of datasets ensures maximum value is derived from datasets which may have been expensive or difficult to collect.
  • Reduced burden on operational systems. Datasets no longer required for operational purposes can be stored separately from data still-in-use, relieving the pressure on storage in operational systems and on those responsible for data management.
  • Meeting legal obligations. Retention and preservation of statistical datasets helps Statistics New Zealand to meet its legal obligations, including those under the Statistics Act 1975 and the Public Records Act 2005.

Scope

This policy applies to all statistical datasets created by Statistics New Zealand, or created by another agency but managed by Statistics New Zealand.

The policy covers all data captured in the Process, Analyse and Disseminate phases of the statistical process. It covers unit record and aggregate datasets, both published and unpublished. It covers pilot surveys, sampling frames and customised datasets. Statistical datasets created from administrative data and used to create official statistics, or integrated to create official statistics, are also covered by the policy. The treatment of integrated personal data is consistent with the guidelines in the Data Integration Protocol.

The policy also covers the retention of the metadata that supports statistical datasets, including classifications and standards.

The retention of completed individual statistical schedules or collection instruments is covered by a separate policy.

Although this policy is primarily designed to cover statistical datasets held in digital form, Statistics New Zealand aims to preserve datasets held in non-digital form, and, where possible, to provide access and maintain the supporting metadata for those datasets.