1

The South African Statistical Quality Assessment Framework (SASQAF)

Presentation made at the Conference on Data Quality for International Organisations

Helsinki, Finland, 6–7 May 2010

Seble Worku National Statistics System Division

Statistics South Africa

2

Background• The broader National Statistics System (NSS) is characterized

by three gapsCapacity gap

- Insufficient statistical skills- Inappropriate placement of skilled personnel

Quality gap- Preponderant usage of data of uncertain or

unknown qualityInformation gap

- Insufficient supply of statistical information- Insufficient supply of data on second

economy

• To address the Quality Gap the Statistician General has gazetted through parliament the South African Statistical Quality Assessment Framework (SASQAF) on the 23rd September 2009

• SASQAF quality requirements must be met by all statistics producers in the NSS to qualify as “Official Statistics”.

3

The Statistics Act No. 6 of 1999• Stats SA is governed by the Statistics Act (No. 6 of

1999)• The Stats Act covers “Official” Statistics and “Other”

statistics• Implies, Stats Act covers all statistics produced that

informs policy, planning and monitoring of government performance.

• Section 14(7) of the Act, empowers the Statistician-General to “designate as official statistics any statistics or class of statistics” produced by Stats SA or any organ of state Required that a rational, sustainable and

transparent framework for assessing the quality of those statistics being developed

South African Statistical Quality Assessment Framework (SASQAF) has been developed for this purpose

4

Definition

Official statistics’ definition is statutory – see Statistics Act [No. 6 of 1999]

Practical criteria of official statistics – Must be used in the public domain– Are from organs of state and other agencies that are

partners in the National Statistics System [NSS]– Are sustainable– Have met quality criteria as defined by the Statistician-

General [SASQAF]

National statistics’ definition is implicitly statutory

Official statistics are statistics designated as official statistics by the Statistician-General within the provisions of the Statistics Act

National statistics are statistics not designated as official Statistics by the Statistician-General

5

Structure of the framework

• Provides a structure for the assessment of statistical products based on 1. Dimensions of quality,2. Indicators, 3. Standards,4. Benchmarks

• Each of the 8 quality dimensions consists of number of indicators

• Within the indicators a number of benchmarks are identified relating to a 4-point scale;

1. Quality statistics (4) 2. Acceptable statistics (3) 3. Questionable statistics (2) 4. Poor statistics (1)

6

Relevance

Accuracy

Coherence

TimelinessInterpretability

Accessibility

Meeting realneeds of clients

Correctly describes phenomena it is designed to measure

Info available atdesired referencepoint

Ease of obtaininginfo from agency

Availability of supplementaryinfo and metadata

Harmonisation of different infowithin broadanalyticaland temporalframework

Integrity

Free from political interference:Adherence to objectivity, professionalism, transparency,ethical standards

Methodologicalsoundness

Sound methodologies:•International standards and guidelines - good practice•Agreed practices•Dataset-specific

The Dimensions of Quality

SASQAF

7

Indicators and Standards

1st Edition 2nd Edition

Dimension Indicators Indicators Standards

Prerequisites of Quality 7 8 21

Relevance 7 5 5Accuracy 7 7 36Timeliness 5 4 10

Accessibility 13 12 13

Interpretability 3 3 3

Comparability and Coherence

5 5 12

Methodological soundness

6 6 14

Integrity 6 6 6

Quality 59 56 120

8

Quality dimension Description

0. Prerequisites of quality Refers to the institutional and organisational conditions that have an impact on data quality.

Key components IndicatorAssessment Levels

Quality Statistics Acceptable Statistics

Questionable Statistics

Poor Statistics

Level 4 Level 3 Level 2 Level 1

Legal and institutional environment (including Memoranda of Understanding (MoUs) or Service Level Agreements (SLAs)Privacy and confidentialityResources are commensurate with the needs of statistical programmesQuality is the cornerstone of statistical work

The responsibility for producing statistics is clearly specified.

The responsibility for producing statistics is explicitly specified through a legal framework.

The responsibility for producing statistics is specified through a legal framework.

The responsibility for producing statistics is implied through a legal framework.

The responsibility for producing statistics is not specified.

Standards and policies are in place to promote consistency of methods and results.

All standards and policies are in place to promote consistency of methods and results, and are adhered to.

The majority of standards are in place to promote consistency of methods and results.

Some standards are in place to promote consistency of methods and results.

No standards are in place to promote consistency of methods and results.

Data sharing and coordination among data-producing agencies is clearly specified and adhered to.

Data sharing and coordination among data-producing agencies is explicitly specified through a legal framework.

Data sharing and coordination among data-producing agencies is specified through a legal framework.

Data sharing and coordination among data-producing agencies is implied through a legal framework.

Data sharing and coordination among data-producing agencies is not specified.

Layout of the framework

9

Quality Standards

•Indicator:

The responsibility of producing statistics is clearly specified.

•Standard:A legal arrangement exists that explicitly mandates the production of statistics.

•Benchmarks:Quality statistics: A law or legal arrangement

exists that explicitly provides the mandate for the production of statistics;

Poor statistics: No arrangement exists.

10

Purpose of the framework

SASQAF provides an universal framework for assessing the quality of statistics within Stats SA and the NSS

•Reviews within the NSS: certification of data

• Self-assessment by data-producing agencies: rate own performance

• Assessment by data users based on quality declaration: brings trust in data

• Assessment by international agencies: ease international comparison

11

Purpose of the framework

•SASQAF provides an universal framework for assessing the quality of statistics within Stats SA and the NSS

•Reviews within the NSS: certification of data

• Self-assessment by data-producing agencies: rate own performance

•Assessment by data users based on quality declaration: brings trust in data

•Assessment by international agencies: ease international comparison

12

ConclusionAdherence to SASQAF will

• encourage compliance to the agreed standards, procedures and guidelines resulting in improvement of quality, closing the quality

gap

• ensure that more statistics are certified as official closing the supply gap

• assist the users in assessing the quality of data and products promoting transparency

• ensure that all published products include statements about data quality (quality declaration) informing users about the quality of data and products

• ensure that more statistical products produced in the NSS are declared as “fit for use”

13

Thank you