IT SoC LABProf. Young-Chul Kim

ISP System Design for High-Resolution CMOS Image Sen-

sor

Schedule

1. ISP System Summary

2. ISP Software Design and Verification

3. ISP hardware Design and Verification

4. ISP Embedded System Design and Verifica-tion

1. ISP Summary

• ISP System: Background & Objective• ISP System: Environment & Block Dia-

gram• ISP System Design Process• ISP Functional Blocks

ISP System: Background & Objective

ISP(Image Signal Processor) ?╶ Image Processing for compensating picture deteriora-

tion in CIS(CMOS Image Sensor)

CCD image sensor vs CMOS image sensor

CCD(Charge Coupled Device) CMOS(Complementary metal-oxide semiconductor)

Color High color expression Low color expression

Noise Relatively low Relatively high

Power High Low(1/10~1/100 of CCD)

Cost High Low

ISP System: Background & Objective

ISP necessity╶ High user demand for high resolution╶ Increased demand for low-cost, low-power, high-density╶ Characteristics: CMOS Image Sensor

⇒ Low cost⇒ Small size⇒ Low power consumption⇒ Deterioration in image quality ISP process is crucial

ISP Composing blocks╶ Color Interpolation, Color Correction, Auto White Bal-

ance, Gamma Correction, Color Space Conversion, Auto Exposure, Auto Focus etc.

ISP System Design Process

① System spec determination╶ System performance(image quality

& processing speed)╶ Verification Environment╶ Verification platform

② Collection of related information╶ Collection of system and algorithm

data

③ System structure design╶ Functional block selection & entire

architecture design

④ Algorithm design and verification╶ Software based algorithm design and

verification ╶ Use of design tools: Matlab, Visual

C++

System spec determination

Collection of related infor-mation

Software based system simulation

Algorithm design & verifica-tion

Hardware based simulation

System architecture design

Hardware design & verifica-tion

ISP System Design Process

⑤ Software based system simulation╺ Verification platform based system simulator design╺ Eg.) Hyvision Interface board, PC

⑥ Hardware design and verification╺ ModelSim, Matlab based verification

⑦ Hardware based system simulation╺ Verification platform based system simulator design╺ Eg.) Hyvision Interface board, PC, FPGA

ISP Functional Block Diagram

① Input : 8-bit Bayer data

② Output : 4:2:2 16-bit YCbCr

③ Functional blocks : Total 11

④ Serial excution(except AE,

AF)

⑤ Image quality sensitive

blocks

╶ AWB, Color correction

╶ Place upper position

⑥ Grouping according to in-

put method

╶ Processing efficiency en-

hancement

ISP System Functional Diagram

ISP Function block – Color Interpolation

Function or objective╶ Image date obtained from CIS has one color value(Bayer

color pattern) per pixel To reduce transmission bandwidth as well as the no. of CIS’ color lens

number

Need the process to convert Bayer color pattern to full color pattern

Bayer color pattern ver. Full color space structure

ISP Function block – Color Interpolation

Role╶ Conversion Bayer color patterns to RGB full color space

Processing method╶ Obtain target pixel information using surrounding Bayer data╶ Available techniques such as Bilinear, Edge-sensing, Pattern

matching╶ Eg.) 3x3 bilinear color interpolation processing

(b)

(a)

Case (a) Case (b)

R (R1 + R2) / 2 R1

G G3 (G1 + G2 + G3 +G4) / 4

B (B1 + B2) / 2 (B1 + B2 + B3 + B4) / 4

ISP Function Block-Color Correction

Function and Objective╶ Compensation in color distortion due to difference in

transfer characteristic depending on lens wavelength of CIS

Role╶ Perform the color enhancement on output date from CIS╶ Compensation close to pixel value(RGB) of standard

cameras(analog camera, high-end camera like DSLR)

Process① Compensation factor cal-

culation through compari-son of current image and standard image

② Correctness in compensa-tion factor decides per-formance

Compensation fac-tor

- Rc, Gc, Bc : RGB after color correction- Rt, Gt, Bt : RGB before color correction

ISP Functional Block-Color Correction

Implementing Method╶ Use multiple recursive analyzing method when calculat-

ing compensation factor╶ Standard image producing tool

⇒ QpCOLORSOFT 501 : QPCARD 사용⇒ Imatest : Use Macbeth color checker

Result after color correction

Before color correction After color correction

ISP Functional Block-Auto White Bal-ance

Need╶ Distortion phenomenon that different light sources

cause the same object to represent different colors Need compensation to the object to represent its inherent color

Role╶ Compensate the distortion phenomenon by changing

value for color temperature

Process- Obtain the gain for each pixel by deciding color temper-

ature- Available techniques such as Gray World Assumption al-

gorithm, Dynamic Threshold AWB algorithm and etc.

ISP Functional Block-Auto White Bal-ance

Implementing Method╶ Eg.) Use Gray World Assumption algorithm╶ Calculate gain for each channel based on the theory

that color represented by RGB average is same as achromatic color once the color change is enough

Result after auto white balance

Before process After process

ISP Functional Block-Lens Shading Correction

Need╶ Nonlinearity phenomenon that image data become

darker and darker as it is getting distant farther and far-ther from the center. It occurs when image data are col-lected.

Need to adjust brightness around periphery

Role╶ Compensate difference in brightness according to distance from arbi-

trary center circleProcess- Increase Weight value as the distance from center in-

creases

ISP Functional Block-Lens Shading Correction

Implementing Method╶ Eg) Use “Quadratic algorithm”

① Calculate Weight value according to distance from the center② Multiply the weight for each pixel

Result after Lens Shading Correction

Before process After process

ISP Functional Block-Color Space Conversion

Need╶ Need conversion to provide appropriate inputs for the

functional blocks using brightness values(Y)

Role╶ Convert color space from RGB Format to YCbCr Format

Process

① Determine input format and out format╶ Eg) Digital RGB -> ITU-R BT.601 YCbCr format

ISP Functional Block-Color Space Conversion

Result after Color Space Conversion

Process(continued)③ Apply conversion formula : ITU-R BT.601 format

Y = 16 + (65.738*R + 129.057*G + 25.064*B)/256

Cb = 128 + (-37.945*R - 74.494*G + 112.439*B)/256

Cr = 128 + (112.439*R - 94.154*G - 18.285*B)/256

After processBefore process

RGB YCbCr Y Cb Cr

ISP Functional Block-Auto Exposure

Need╶ Required for brightness control according to change of

shooting(or photographing) environment

Role╶ Control integration time and internal analog gain of CIS

camera to acquire appropriate brightness

Type╶ AE with aperture priority

⇒ Control shutter velocity after calculating proper exposure with fixed aperture value

╶ AE with shutter⇒ Automatic aperture value control with fixed shutter speed

ISP Functional Block-Auto Exposure

Result after auto exposure

Before process After process

Process① Generate histogram using change

of input image② Extract Mean Sample Value③ Obtain Integration time 과 analog

gain using MSV

4

0

4

0

)1(

i

i

i

i

x

xi

MSV: Mean Sample Value

ISP Functional Block-Auto Focus

Need╶ Blurring Effect due to different focus on input image╶ Need to find focus

Role╶ Adjust the focus of image by controlling actuator em-

bedded in Image sensor

Process╶ Algorithm finding focus value: Tenengrad method, SML

method, HPACM ex) Tenengrad method : Use Sobel operator to extract horizontal and vertical outlines from brightness value of image

: x sobelO : y sobel O-1 0 1

2 0 2

-1 0 1

-1 0 1

2 0 2

-1 0 1

ISP Functional Block-Auto Focus

① Accumulate inner product of f(x,y), brightness value of im-age

② Algorithm to find correct focus: Hill climbing , global searching

ex) Hill climbing ╶ Search in direction increasing focus from initial lens position P ╶ Find the place just before focus value begins to decrease

|y)}f(x,{y)}f(x, {|: 2y

2 sobel

x y

xsobel OOFVFocusValue

After processBefore process

ISP Functional Block-Auto Focus

Result after Auto Focus

Obtain more clear image by find correct focus compared to one before auto focus is applied

ISP Functional Block-Noise Reduc-tion

Need╶ Need to reduce noise on original image due to image

processing such as image compression

Role╶ Restore original image by removing noise

Process① Analyze the cause and type of noise 잡음의 종류를 분석② Appropriate filter selection/application according to cause and

type

ISP Functional Block-Noise Reduc-tion

Result after Noise Reduction

Process(Continued)③ Ex) Applying Median Filter

Line up surrounding pixel values in ascending or descending order and adjust the center pixel value

Original Image Image with noise After process

Median Filter

ISP Functional Block-Gamma Correction

Need╶ General monitor outputs darker image than original signal Need to compensate brightness

Role╶ Perform brightness compensation in monitor

Process- Improve curved brightness on monitors by modifying

Gamma value

4. ISP Functional Block-Gamma Correction

Result after Gamma Correction

2. ISP Software Design and Verification

• S/W ISP Simulator• S/W ISP Simulator GUI• S/W ISP Design and Verification

S/W ISP Simulator

1. Simulator Environ-ment

- 8M CIS - PC - V2P HyImage board - PCI express card (e-

quipped in PC)2. Tool - Visual C++ 6.0 Based 3. HyImage interface

board - Communication role

bet. CIS and PC - Image data transmis-

sion through PCI communi.

- Transferring control signal through USB communication

HyImage interface board

ISP Simulator

S/W ISP Simulator GUINo. Main function Contents

① Sensor Selection- Sensor selection- Select Y232 in case of 8M CIS

② Display menu

- PLAY :- STOP : - Snapshot : still image shot- Preview Fit : Full screen out-put- 2Play : output in size of ¼

③ Frame Rate - Frame number per second

④ I2C

- Addr : slave address input(HEX) *sensor : 0x1A *actuator : 0x0C- Data : data input- Read : I2C read- Write : I2C write

⑤ Process - Output current progress

⑥ resolution- Full : 3200 x 2408 size- ¼ : 1600 x 1204 size

S/W ISP Simulator GUI

No. Main func-tion Contents

⑦ Pre-process-ing

- color correction control- value : Pop-up for color correction factor adjustment - Lens Shading Correction : LSC control

⑧ 3A- AWB, AE, AF control- Auto White Balance : GWA, SDWGWA, Custom option control

⑨ Auto Expo-sure

- Outdoor - Indoor - Exposure value : Mean Sample value- Luminance gain : CIS luminance gain - Integration time : CIS integration time

⑩ Auto Focus - Global Search, Hill Search selection

⑪ Processing

- Gamma correction- Edge enhancement- Noise reduction, Alpha_trimmed - Control : Contrast, Saturation, Bright-ness adjustment- Histogram Equalization - adjustment

⑫ User Option - Detailed option selection

User Option Pop-up window

Color correction factor setting

S/W ISP Simulator GUIColor correction factor ad-justment

* Rc = 1.772750 x Rt -0.233610 x Gt + 0.024167 x Bt -0.048939

* Gc = -0.214110 x Rt + 1.715830 x Gt – 0.321510 x Bt + 0.027624

* Bc = -0.049861 x Rt -0.486130 x Gt + 1.640180 x Bt + 0.002687

- Factor set when clicked

User option Pop-up Window╶ Custom White Balance

⇒ Light source selection

╶ Correction⇒ Lens shading correction RGB center-

axis selection

╶ Histogram Equalization⇒ alpha, beta value selection

╶ Color⇒ Brightness, Contrast, Saturation, Hue

╶ Enhancement⇒ Gamma, Noise, Edge parameter

S/W ISP Design & Verification-Color interpolation

Applied Algorithm :

╶ 3x3 Bilinear

╶ 5x5 Adaptive Interpolation

3x3 Bilinear interpolation

(b)

(a)

Bayer color pattern

S/W ISP Design & Verification-Color interpolation

5x5 adaptive color interpolation╶ Divide into 8 cases according to pixel placement

⇒ Applying different weight per each case

╶ Add reference pixel applied in case of Bilinear color interpola-tion

Bayer pattern

8-case weight filters% Divide by 8 after applying each weight filter

S/W ISP Design & Verification-Color interpolation

Perfor-mance

result

3x3 Color Interpolation

5x5 adaptive Color Interpola-tion

Reserve edge component

S/W ISP Design & Verification-Color interpolation

Performance Comparison(PSNR)

A : 3x3 color interpolation B : 5x5 adaptive color interpolation Average 1.3dB

S/W ISP Design & Verification-Color interpolation

Program Source Analysis╶ Design S/W simulator using 5x5 adaptive color interpola-

tion╶ Source file : Hyimage.cpp╶ Referecence funtion : color_interpolation()╶ Source analysis : Refer to attached files

S/W ISP Design & Verification-Auto White Bal-ance

Applied algorithm: ╶ GWA(Gray World Assumption) algorithm,╶ SDLWGW(Standard Deviation and Luminance Weighted Gray World Al-

gorithm) algorithm╶ Custom White Balance algorithm

BGR

B

BGRBB

G

BGRGG

R

BGRRR

3/)('

3/)('

3/)('

Gray World Assumption Algorithm ╶ Performance degradation in case that image of single colored

object or background occupies major portion of a full image╶ Assume that ╶ Gain for R, G, B

S/W ISP Design & Verification-Auto White Bal-ance

SDLWGW(Standard Deviation and Luminance Weighted Gray World Algorithm) Algorithm╶ Compensate drawbacks in GWA(Image quality degrada-

tion in case that rate of single color in image is high)╶ Perform based on two-assumptions

⇒ Assumption that color distribution of color is uniform when standard deviation(SD) in an image is high

-> Equation 1 in GWA is justified⇒ Pixels with medium values in brightness play key role in

light source evaluation

S/W ISP Design & Verification-Auto White Balance

1) Divide into N blocks

╶ Setting n=200 for our project

2) Compute SD weighted average for each

blockDivide input image

luminance weighted average computation

SDLWGWGWAOriginal

S/W ISP Design & Verification-Auto White Bal-ance

최종적인 R, G, B 값

3) Compute final R, G, B

╶ In case that SD and luminance of

each block are low

╶ No affection to computation of

gain due to near zero value

BSDLWA

BSDLWAGSDLWARSDLWABB

GSDLWA

BSDLWAGSDLWARSDLWAGG

RSDLWA

BSDLWAGSDLWARSDLWARR

gaingain

gaingain

gaingain

_

3/)___('

_

3/)___('

_

3/)___('

Result after GWA / SDLWGW algorithm ap-plication

S/W ISP Design & Verification-Auto White Balance

Custom White Balance Algorithm╶ Control by user’s decision according to typical light souces

① Tungsten ② Fluorescent③ Sunshine④ Cloud

Tungsten

(3200K)

Fluorescent(4000

K)

Sunshine(500

0K)

Cloud

(6000K)

R-gain 0.8800 0.9496 1.000 1.0233

G-gain 0.9699 0.9847 1.000 1.0078

B-gain 1.8654 1.3288 0.9510 `0.8151

White balance parameter for typical light sources

<Sunshine> <Cloud><Fluores-

cent><Tungsten>

S/W ISP Design & Verification-Auto White Bal-ance

Result after applying Custom White Balance algo-rithm

S/W ISP Design & Verification-Auto White Balance

Program source analysis

╶ S/W simulator designed such that GWA, SDLWGW, Custom White Bal-

ance can be selected in option

╶ Source file : Hyimage.cpp

╶ Related functions : AWB(), SDAWB(), Cunstom_white_valance

╶ Source analysis : Refer to attached files

S/W ISP Design & 및 Verification-Lens Shading Correc-tion

Applied algorithm :╶ Position-Based linear Algorithm

Position-Based linear Algorithm

- Use of 5ⅹ5 Searching window

- Emphasize on the blocks

whose average value are

high

2) rate computation

yYxX

MAXyxmeanif

yyxxIyxmean

centercenter

mean

,

),(5

)2:2,2:2(),(

max

22 )()(

d

yyxxrate

centercenter

1) Search for center point

After processBefore process

S/W ISP Design & Verification-Lens Shading Correction

3) Weight Computation

- Curve control for our project

1,2/1,4/1,8/1,0

11

)1(2

max

controlcurve

controlcurve

ratecontrolcurverateweightweight

Result

Program Source Analysis╶ Use of Position-based linear algorithm╶ Source file : Hyimage.cpp╶ Related function : LSC()╶ Source analysis : Refer to attached files

S/W ISP Design & Verification-Lens Shading Correction

S/W ISP Design & Verification–Gamma Correction

Process

Applied Algorithm╶ Piece-wise linear Gamma correction

Gamma curve for gamma compensation

S/W ISP Design & Verification – Gamma Correc-tion

Process╶ In case of R < 1: Algebraic curve function to increase bright-

ness╶ In case of R = 1: Null function ╶ In case of R > 1: Reverse log function to decrease brightness

R versus Gamma relation

S/W ISP Design & Verification–Gamma Correc-tion

gamma = 1.0 gamma = 2.2

gamma = 0.45 gamma = 0.85

Process

S/W ISP Design & Verification–Gamma Correc-tion

Program Source Analysis╶ Source file : Hyimage.cpp╶ Related fuction : GAMMA()╶ Source analysis : Refer to attached files

S/W ISP Design & Verification–Noise Reduction

Process① Alpha-trimmed filter

╶ Arrange pixel values within Mask in ascending order╶ Remove pixels with higher/ lower alpha value╶ Average the remaining pixels

Applied Algorithm╶ Alpha-trimmed filter╶ MA-MEAN filter╶ L-MMSE filter

Alpha-trimmed filter applying process

S/W ISP Design & Verification–Noise Reduction

Program Source Analysis╶ Use of MA-MEAN, L-MMSE filter╶ Source file : Hyimage.cpp╶ Related functions : recon(), NR(), NR_TEM()╶ Source analysis : Refer to attached files

Applied Algorithm : Unsharp mask Filter

Step1 Mean Filter(3X3)

Step2 ╶ (Original image – Mean filter image) scale

S/W ISP Design & Verification-Edge Enhance-ment

1. Add every pixel values 2. Devide by a size of mask 3. Insert the value into 5th pixel 1. Original image

2. Applying mean filter3. Step 24. Original + Step 3

S/W ISP Design & Verification-Edge Enhancement

Step2 (countinue)╶ Find the edge part through difference bet. Original and

mean filtered mask image╶ Increase the visibility of edges with weighting

Step3 Original image + Step2╶ Increase visibility of resulting image from Step 2

Step4 Clipping ╶ Correct pixels with values over gray-level value

S/W ISP Design & Verification-Edge Enhancement

Performance

Original Edge = 4

Edge = 7 Edge = 9

S/W ISP Design & Verification-Auto Focus

Sharpness Value 측정 알고리즘 : Sum-Modified Laplacian(SML) + Median

초점 위치 탐색 알고리즘 : Global searching , Hill climbing search-ing

Sum-Modified Laplacian(SML) + Median╶ 2 차 도함수 결과의 상쇄를 개선시킨 방법╶ Sharpness Value 값이 큰 영역의 범위가 좁다는 장점╶ 임펄스 잡음을 제거

그림 1

그림 2

본 과제 적용 마스크 : 그림 2 ( 연산 수행 속도 감소 )

소프트웨어 ISP 설계 및 검증 -Auto Focus

초점 위치 탐색 알고리즘 - Global searching ╶ 초점위치의 시작점인 0 값에서부터 일정한 값을 증가시켜

순차적으로 초점 값을 찾는 방식 ╶ 가장 정확한 정 초점 위치를 찾아낼 수 있는 방식╶ 초점 값을 계산하는 영역이 많아지기 때문에 초점위치 탐색 알고리즘

중 가장 수행속도가 느린 방식

본 과제 step_size = 13 current_value = current_step + (48<<10) current_step = 0x50 + step_size

Global 처리 후Global 처리 전

소프트웨어 ISP 설계 및 검증 -Auto Focus

성능

Test time

소프트웨어 ISP 설계 및 검증 -Auto Focus

초점 위치 탐색 알고리즘 - Hill climbing searching╶ 처음 단계에서는 탐색 구간을 크게 증가시켜 탐색한 후 이전의 초점

값이 현재의 초점 값 보다 큰 영역이 있을 시 반대방향으로 이동 시키는 방법

╶ 반대방향으로 이동시 탐색 구간을 이전의 절반으로 감소시켜 좀 더 세밀한 탐색

╶ 임펄스 성 노이즈의 영향을 줄이기 위해 이전의 초점 값이 현재의 초점값 + threshold 이상일 경우 반대방향으로 이동

본 과제 Threshole = 120000 step_size = 90 current_value = current_step + (48<<10) current_step = 0xE0 + step_size

소프트웨어 ISP 설계 및 검증 -Auto Focus

성능

Hill search 처리 전 Hill search 처리 후 , 걸린 시간

소프트웨어 ISP 설계 및 검증 - Color Space Conversion

적용 Y’CbCr Format : ITU-R BT.601 format

소스코드 분석╶ 적용함수 : void color_conversion()

적용결과RGB YCbC

r

적용 전 적용 후

Y =16 + (65.738*R + 129.057*G + 25.064*B)/256

Cb =128 + (-37.945*R - 74.494*G + 112.439*B)/256

Cr =128 + (112.439*R - 94.154*G - 18.285*B)/256

소프트웨어 ISP 설계 및 검증 - Auto Exposure

적용방법 : MSV(Mean Sample Value) 를 이용

Xi - 지역 I 에서의 밝기 합

μ - 0 부터 5 까지의 범위

4

0

4

0

)1(

i

i

i

i

x

xi

╶ 위 식에서 얻어지는 μ 값의 범위에 따라 integration time 과 analog gain 값을 조절

μ integration time Analog gain

>4( 매우 밝음 ) -200 -3

>3.5( 밝음 ) -30 -1

>3.2( 약간 밝음 ) -10 -1

>3.0( 정상 ) 현재 값 유지 현재 값 유지

>1.7( 약간 어두움 ) +20 +1

<1.7( 어두움 ) +100 +5

AE 적용 후

AE 적용 전

소프트웨어 ISP 설계 및 검증 - Auto Exposure

테스트에 따른 상하한 설정

테스트에 따른 상하한 설정 이유① integration time 증가 -> frame rate 감소② integration time 감소 -> 영상이 보이지 않음③ analog gain 무한증가 감소 -> 색감변화

상한 (Hex) 하한 (Hex)

integration

time3000 400

analog gain 50 10Integration time 및 analog gain 의 상하한 경계

소프트웨어 ISP 설계 및 검증 - Auto Exposure

소스코드 분석╶ 사용함수 : void AE_day(double EV,BYTE *pBmpBuffer1) void AE_day(double EV,BYTE *pBmp-Buffer1)╶ 사용이벤트 : IDC_ae_en_day, IDC_ae_en_night

IIC_Write_Proc(BIT16_BIT16, 0x0202, 0x0400);//202 : coarse_integration_time address

IIC_Write_Proc(BIT16_BIT16, 0x0204, 0x00a0); //204 : analogue_gain_code_global

IMX046ES_I2C_Mipi_E__081016.PDF