ASIC 3 Workshop, May 17, 2006 1 System Level Approach to Satellite Instrument Calibration Space Dynamics Laboratory at Utah State University: Joe Tansock,

ASIC3 Workshop, May 17, 2006 1

System Level Approach to Satellite

Instrument Calibration

Space Dynamics Laboratory at Utah State University: Joe Tansock, Alan Thurgood, Gail Bingham, Nikita Pougatchev, Randy Jost

NIST: Raju Datla

Ball Aerospace & Technologies Corp.: Edward Knight


Outline

• Calibration Philosophy • “Specmanship”• Workshop to Improve Calibration• Calibration Planning• Subsystem/Component measurements• Ground Calibration• On-Orbit Calibration

– Internal and external calibration sources

• Satellite Instrument Validation and End-to-end Error Model

• Summary


Calibration Philosophy

• Calibration– Provides a thorough understanding of sensor operation and

performance– Verifies a sensor’s readiness for flight– Verifies requirements and quantifies radiometric and

goniometric performance– Provides the needed tools to convert the sensor output to

engineering units that are compatible with measurement objectives

– Provides traceability to appropriate standards– Estimates measurement uncertainties


Calibration Philosophy – Cal Domains

• A complete calibration will address five responsivity domains

– Radiometric responsivity• Radiance and irradiance traceable to NIST• Response linearity and uniformity corrections• Nominal/outlying pixel identification• Transfer calibration to internal calibration sources

– Spectral responsivity• Sensor-level relative spectral response

– Spatial responsivity• Point response function, effective field of view, optical distortion, and

scatter– Temporal

• Short, medium, and long-term repeatability, frequency response– Polarization

• Polarization sensitivity


Calibration Philosophy – Cal Domains

• The goal of calibration is to characterize each domain independently

– Together, these individually characterized domains comprise a complete calibration of a radiometric sensor

• Domains cannot always be characterized independently

– Complicates and increases calibration effort– Example: Spectral spatial dependence caused by Stierwalt

effect• Calibration parameters are grouped into two

convenient categories– Calibration equation

• Converts sensor output (counts, volts, etc.) to engineering units

– Radiometric model• All parameters not included in calibration equation but required to meet

calibration requirements


Calibration Philosophy – Phases of Cal

• A complete and methodical approach to sensor calibration should address the following phases:

Calibration planning during sensor design

Ground measurements

Subsystem/component measurements

Sensor-level engineering tests and calibration

Sensor-level ground calibration

Integration and test

On-orbit measurements On-orbit calibration


Establishment of Good Specifications Improves Calibration

• Programs often start with a requirement such as

– “The instrument shall be radiometrically calibrated to a 3% absolute error, 1.5% band to band error, and a 0.25% intra-band pixel to pixel error”

• The designers are then asked for cost, schedule, and risk to meet this requirement, which could vary dramatically

– E.g., is “error” a 1-sigma or 3-sigma requirement?

• Furthermore, incomplete, changing, or impossible specifications are often the cause of cost and schedule overruns


So, What Makes a Good Specification?

• A good specification clearly communicates what must be accomplished

– To an audience that is reading (vs. oral communication)• No other “clues” to help understanding

– To an audience that may not be able to ask questions easily

• Example: reading the specification at the end of the program after there’s been personnel turnover

– To an audience that may have a different background, training, or understanding of the problem than the author

“Good” Requirements Tests

(examine every formal requirement with these tests) A. Is the requirement complete (domains, interactions, worst cases)?

B. Is the requirement unambiguous (terminology, grammar)?

C. Is the specification free of errors (for example, typos, math mistakes)?

D. Is there at least one identifiable method to implement this requirement?

E. Is there at least one identifiable method to verify this requirement?

Also see E. Knight, “Lessons

Learned in Calibration

Specsmanship, CALCON 2005

proceedings.


Lessons Learned in Specifications

• Lessons– Cover all domains (spectral, spatial, temporal, radiometric,

polarization)• Including interactions and “worst case” for requirements

– Scrub for ambiguity– Use mathematical equations whenever possible to define

requirements– Have at least one idea for implementation in mind when writing

the specification• Or upon first round of review/questions

– Have at least one idea for verification in mind as well

• Conclusion– The chance of an instrument

• Being “poorly calibrated” • Overrunning cost and schedule targets can be reduced with improved

calibration specsmanship


• EO/IR Calibration & Characterization Workshops held in Feb 2005 and March 2006 at SDL/USU

– Envision self governing community based organization with goal of improving calibration for all participating organizations

• Workshop Objectives– Explore ways to improve the quality of IR/Visible/UV measurements,

community-wide, based on an ISO 17025 standard, as pioneered by the RCS community

• Benefits based on experiences of RCS community– Measurably and quantifiably improve the quality of measurements

made in the community– Facilitate data comparison between sensors, systems, facilities,

programs and customers– Increase in customer confidence in measurement results due to

improved: accuracy, uncertainty, repeatability, comparability, consistent documentation

Workshop to Improve Quality of Calibration


• Universal Agreement– There is an unmet need that can not be addressed by any one

organization

• Intermediate results will continue to be presented at annual CALCON (Calibration Conference)

• For more information– CD available containing the presentations and recommendations

of the 2005 and 2006 workshops.– http://www.sdl.usu.edu/conferences/eo-ir/

• Based on attendee feedback provided at the 2006 workshop, we have started the planning process for the next workshop, to be held Spring 2007, at NIST, in Gaithersburg, MD

Workshop to Improve Quality of Calibration


Calibration Planning

• Calibration planning– Start as soon as possible (I.e. requirements definition,

concept design, sensor design, etc.)– Influence sensor design to allow for efficient and complete

calibration– Encourages optimum sensor design and calibration

approach to achieve performance requirements• Planning phase can help shake out problems

– Schedule and cost risk can be minimized by understanding what is required to perform a successful calibration

– Calibration equipment needs should be identified early to allow time to build and test any required new equipment


Calibration Planning• Identify instrument

requirements that drive calibration

• Identify calibration measurement parameters and group into:

– Calibration equation– Radiometric model

• Flow calibration measurement parameters to trade study

– Schedule – Sensor design feedback– GSE hardware & software– Measurement uncertainty– Risk

• Perform trade study to determine best calibration approach

Mission Requirements

Sensor Design

Cost & Schedule

Risk

GSE Hardware & Software

Calibration Planning

Measurement Uncertainty

Instrument Requirements

Calibration Measurement Parameters• Calibration Equation• Radiometric Model


Subsystem/Component Measurements

• Subsystem and/or component level measurements– Help verify, understand, and predict performance– Collect Parameters for the Radiometric Model that can't be

measured well at the system level – Minimize schedule risk during system assembly

• Identifies problems at lowest level of assembly• Minimizes schedule impact by minimizing disassembly effort to fix a

problem

• System/Sensor level model development and measurements– Allow for the development of Measurement Equation and

Performance prediction– Allow for end-to-end measurements– Account for interactions between subsystems and

components that are difficult to predict


Subsystem/Component Measurements

• Merging component-level measurements to predict sensor level calibration parameters may bring to light systematic system-level uncertainties A,B

– Comparison of System-level estimate using component measurements with end-to-end measurement of SABER relative spectral responsivity (RSR)

• 9 of 10 channels < 5% difference

• 1 channel 24% difference (reason unknown)

• Helps to resolve and correct for component degradation and sensor performance after launch

A.) Component Level Prediction versus System Level Measurement of SABER Relative Spectral response, Scott Hansen, et.al., Conference on Characterization and Radiometric Calibration for Remote Sensing, 1999

B.) System Level Vs. Piece Parts Calibration: NIST Traceability – When Do You Have It and What Does It Mean? Steven Lorentz, L-1 Standards and Technology, Inc, Joseph Rice, NIST, CALCON, 2003


Engineering Ground Calibration

• Engineering calibration – Performed before ground calibration– Perform abbreviated set of all calibration measurements – Verifies GSE operation, test configurations, and test

procedures– Checks out the sensor– Produces preliminary data to evaluate sensor performance– Feedbacks info to flight unit, calibration equipment, procedures,

etc.

• Engineering calibration data analysis– Evaluates sensor performance, test procedures, calibration

hardware performance and test procedures

• Based on results of engineering calibration, appropriate updates can be made to prepare for ground calibration data collection


Ground Calibration

• Provides complete calibration needed to meet related requirements

• Is performed under conditions that simulate operational conditions for intended application/measurement

• Careful in-lab calibration minimizes problems that arise after launch

– Minimizes risk of not discovering a problem prior to launch– Promotes mission success during on-orbit operations

• For many sensor applications– Detailed calibration is most efficiently performed during ground

calibration– On-orbit calibration will not provide sufficient number of sources at

needed flux levels– Operational time required for on-orbit calibration is minimized

• Best to perform ground calibration at highest level of assembly possible

– Sensor-level at a minimum is recommended


• Calibration continues after ground calibration

• Internal Calibration Source Response Trending

– Trend sensor response to quantify relative response changes over time

– Source types• Blackbodies, glow bars, diffusers, lamps, etc.

– Ensure source is stable and repeatable for sensor operational life

Sensor Design/Fabrication

Ground Calibration On-Orbit/Field Operations

Internal Calibration Source Response Trending

On-Orbit/Field Calibration/Verification

Extending Calibration to Operational Environment


Internal Calibration Sources• Challenges

– Ensure calibration source is stable and repeatable for sensor operational life

– Ideally, calibration source should use same optical path as external measurements

• Detailed trade to determine best approach is needed for each specific application

• Considerations => source type, flux level, configuration, power, space, and weight limitations, etc.

– Sources of variability• Temperature stability and/or temperature measurement• Emissivity changes• Thermal variations (external and internal)• Separate drift in observed response between calibration source and sensor

response• Control and/or monitor electronics

• IR internal calibration source developments are required to achieve stringent stability requirements of many climate change measurements


• Track, trend, and update calibration throughout a sensor’s operational life

– In addition to internal calibration sources make use of external calibration sources

• External On-orbit sources– Standard IR stars

• StarsBoo, Lyra, Tau, CMa, Gem, Peg

– Celestial objects• Moon• Planets provide bright variable sources• Asteroids, etc.• Sometimes you have to be creative:

– Off-axis scatter characterization using the moon

– Other techniques• View large area source located on surface of earth (often termed vicarious calibration)• Cross-calibration between sensors• Use of atmospheric lines• Etc.

On-Orbit Calibration Verifies Cal and Quantifies Uncertainty


Satellite Instrument Validation

• The purpose of validation is to assess actual accuracy and precision of Satellite Instruments by comparison with validating measurements

• Apparent differences in results between validating and measurement system

– Satellite and validation data are not co-located in time and space– Satellite and validating system have different vertical and horizontal

resolution– Satellite and validating system have finite accuracy and repeatability– Physical measurement differences (I.e. spectral, sensor, platform,

etc.)

• Validation Assessment Model makes comparisons more accurate by understanding and accounting for theses differences

– Make results comparable

• Validation Assessment Model can be used as a tool to better understand the tradeoff between validation approaches

22ASIC3 Workshop, May 17, 2006

Validation Assessment ModelReconcile differences to make results comparable

Performance Assessment

y

End-to-End Error Model Overall Concept

TrueProfile

xsat

Radianceysat

Parameter Error - δbNoise – ε

Instrument SDR

SmoothingParameter

NoiseRetrieval EDR

TrueProfile

xval

xval

yval

Validation SystemRadiosondes, Aircraft Measurement Systems, Cross-Calibration, etc.

expectedδy expectedδx

ˆ

x

valy valx


Summary

• Calibration Philosophy– What does calibration provide– Calibration domains – Phases of calibration

• Planning through operational environment

• Importance and benefit of good specsmanship– Facilitate clear communication and minimizes risk of failure

• Workshop to improve quality of calibration– Community wide participation working to improve calibration

• Calibration Planning– Address all phases of calibration as early as possible

• Specification and design phases


Summary (cont)

• Calibration Measurements– Subsystem/Component Measurements

• Minimizes schedule risk and facilitates development of instrument model and measurement equation

– Engineering Calibration and Calibration• Methodical and careful approach leads to efficient and thorough

calibration

– Extending Calibration to Operational Environment• Internal calibration sources (I.e. in-flight internal sources)

– Challenges and need for improvement• External on-orbit sources

– External sources and need for improvement

• Satellite Instrument Validation – Overall concept and the need for validation assessment

model to account for differences in space, time, resolution, etc.

Documents

ASIC 3 Workshop, May 17, 2006 1 System Level Approach to Satellite Instrument Calibration Space Dynamics Laboratory at Utah State University: Joe Tansock,