Click here to load reader

System Architecture for ISP Hardware System Architecture for ISP Hardware FPGA Based ISP Simulator FPGA Based ISP Simulator FPGA Based ISP Simulator GUI

  • View

  • Download

Embed Size (px)

Text of System Architecture for ISP Hardware System Architecture for ISP Hardware FPGA Based ISP Simulator...


3. ISP Hardware Design & VerificationSystem Architecture for ISP HardwareFPGA Based ISP SimulatorFPGA Based ISP Simulator GUI ComponetsDesign & Verification for ISP Functional Blocks System Architecture for ISP Hardware Total 12 functional blocks- Sytem clock : pixel clock Input : 8-bit bayer Output : 16-bit 4:2:2 YCbCr Functional block control through I2C Function selector generates enable singal to each fucnctional block of ISP

Operational frequency for blocks : More than 100 MHz

Hardware architectue for ISP sytemSynthesis Results of ISP Hareware Design

- Op. freq. : 105.85MHz

- Critical path : AWBSetup time of the ISP functional block measured in slow model.Resource usageItemValueTotal logic elements10,338Total registers7,431Total memory bits316,896FPGA Based Real-time ISP SimulatorFPGA Based ISP SimulatorTest Enveronment : 8M CIS PC DE2-70 FPGA board HyImage V4P board

HyImage interface board : Communication bet. PC & FPGA

Operation speed :In case of full resolution : 7~8 Frames per sec.In case of down scaling : 10 Frame per sec.

FPGA Based ISP SimulatorPCB Board Design for FPGA-PC Connection

gpio0, gpio1 FPGA board GPIO

Required of PCB design for FPGA board I/O with a fixed power pins(3.3V)

PCB board for FPGA-PC conncetion

Connection information for PCB mode

FPGA Based ISP Simulator GUIISP Simulator GUI

No.Main functionDetailsDown scaling - Full/down scaling selectionColor interpolation 5x5 adaptive interpolation 3x3 bilinear interpolation Option Selection for Gamma value Setting weight for Edge enhancementAuto focus Selection for operationAE_Option- Setting for in-door/out-doorISP Function Activate each functional blockEdge enhancement weight value setting windowGamma value setting windowFPGA Based ISP Simulator GUIGamma value setting window- possible from 0.1 through 0.9

Edge enhancement setting window- Value setting: 1 ~ 5 - Increase by 0.5ISP Functional Block Design & Verification Color InterpolationApplied Algorithm33 bilinear color interpolationProcessLine buffer : 3 line data store3x3 data generator : 3x3 data matrix generationBilinear operator : bilinear interpolation Applying total 4-level pipeline

ISP Functional Block Design & Verification Color InterpolationProgram Source AnalysisHardware file configuration inter_3_3.v buffer_3_3_8bit.v : 3x3 buffer dpsram3200x8_2.v : 3x3 buffer memory simulation file generation (Modelsim file)inter_3_3.vbuffer_3_3_8bit.vdpsram3200x8_2.vimage_gen.v : 8bits bayer data generation filinter_tb.v : Test-bench top fileSimulation File Generation(Matlab file)rgbtogbrg.m : bayer pattern file generation from image-file inputinterpolation_3_3.m : Result image generation after performing 3x3 interpolationimage_out.m : Converting hardware-based 3x3 interpolation results to a corresponding image file

ISP Functional Block Design & Verification- Color interpolation 5x5 adaptive color interpolation

Hardware file compositioncolor_inter_new.v : color interpolation top filebuffer_5_5.v : 5 line bufferdpsram3200x16.v : memory for 5-line bufferpattern_gen.v : generation pattern value according to bayer patternparam.h : Store 8 masks Simulation file composition4 higher level filesinter_tb.v : Top test-bench fileimage_gen.v : bayer pattern input generationISP Functional Block Design & Verification-Color interpolation Simulation file configurationMatlab filergbtogbrg.m : generate bayer pattern file from image file inputinterpolation_5_5.m : Result image generation after 5x5 interpolation image_out.m : Converting hardware based 5x5 interpolation results to image file

ISP Functional Block Design & Verification Edge enhancementApplied algorithmUnsharp mask filterPerforming processBuffer 3x3 Store 3-line data3x3 data generator Generate 3x3 data matrix Sum 3-line summationDivider Get average dataAdd multiplication result of weight and subtracted value from original pixel data to the original pixelTotal 19-stage pipeline

ISP Functional Block Design & Verification- Auto White BalanceGray World Algorithm

Hardware file compositionAWB.v : AWB top filedivider.vhd : division calculation Simulation file compositionawb_lee3.v : AWB top fileawb_lee3_tb.v : Test bench top filedivider.vhd: 21bit / 11bitdpsram3200x16.v : memory for 5-line bufferim_64x64.txt : Text image fileRGB_image_gen.v : RGB pattern input generationRGB_image_write.v

ISP Functional Block Design & Verification - Auto White BalanceSimulation file compositionMatlab file * im2text.m : convert an image file to a text file * text2im.m : convert a text file to an image file * compare_result.m : result comparison

GWA AWB ISP Functional Block Design & Verification-Auto White Balance Source analysisSee reference files

Applying 1-stage pipeline

Consists of 5 sub-blocks

Signal generator : generate a frame completion signal

FIFO : Perform delay for Divider block

Accumulator : perform total summation R, G, B pixels

Shifter : perform average of R, G ,B

Divider : divide R, G, B average total by each R, G, B average

ISP Functional Block Design & Verification Edge enhancementProgram source analysisHardware file compositionEdge_enhance.v : filedivider.vhd : division calculationdpsram3200x32.v : Line buffer memoryFIFO.v : FIFO processSimulation file composition(Modelsim file)buffer3_3.v : 3 line buffer edge_en_tb.v : edge enhance top fileFIFO.v dpsram3200x32.v edge_enhance.v divider.vhd YCbCr_image_gen.v : YCbCr image generationReg_Nbits.v YCbCr_image_write.vSimulation file composition(Matlab file)im2textYCbCr.m : convert into test file after converting image file to YCbCrtextYCbCr2im.m : convert text file to YCbCr image filecompare_result.m : Result comparison

ISP Functional Block Design & Verification Auto FocusApplied algorithmTenengrad (calculating focusing value)Global search(Movement of focusing position)Performing process3 line Buffer : store 3-line Y data 3x3 data generator : generate 3x3 data matrixFocus value calculating : calculate focus value per frameMain operator : setting the moving range for actuator stepI2C Controller : activate actuator step movement

ISP Funcional Block Design & Verification Auto FocusProgram source analysisHardware file compositionglobal_se.v : filefocus_val.v : calculate focus value I2C_AF.v : Moving Motordpsram3200x16: Line buffer memoryaf_mem.v : memory access

Simulation file composition(Modelsim file)Imposible to derive program by Modelsim (register control)

ISP Functional Block Design & Verification Gamma correctionApplied algorithmPiece-wise linear Gamma-correction

Performing process

Top block for gamma correction

ISP Functional Block Design & Verification Gamma correctionPiece-wise linear gamma correction

4. Embedded System Design & VerificationSoftware ISP SimulatorSoftware ISP Simulator GUIHardware ISP ISP Embedded System

Block diagram of an ISP Embedded System SoCBase 1.0 Based Design Process : ARM926EJS FPGA : Virtex4LV80 Design block : - ISP top I2C controller ISP functional block control ISP function blocks - Dualport SRAM controller - VGA controllerDesign Details

System spec.

Design ToolSoCBase 1.0

SoC base 1.01. Single-processor based SoC development application platform 2. Provided with total 26 IPs AMBA bus Memory controller External unit controllerHardware Design & Verification Dual port SRAM controller

Dual port SRAM controller Stucturedual port SRAM controller design for high-speed store/output of image dataUse dual port FIFO - 8192 x 32 bitsGenerate SRAM control data from Write/read data controller Required in case for output of high-resolution image

Hardware Design & VerificationVGA controller

Clock generation through control of ICS 307 chip input signal Clock generation of 25MHz for 640x480 resolution output VGA controller usage example : Refer to HBE-SoC-IPD Users Guide p.250 ICS 307 chip manual : HBE-SoC-IPD Users Guide p.265

Ics_307blockVga dataCtrollerTop_VGAiCLK_25Ics_sclkics_dataics_strobeoVGA_R[7:0]oVGA_G[7:0]oVGA_B[7:0]oVGA_HSoVGA_VSoVGA_BLANKoVGA_SYNKoVGA_CLOCKiRSTIcs_307 chipClk_108SRAM-VGA Simulation Source file location : sram_vga_test folderInput file : ycbcr_in.txtOutput file : image_out.txtSRAM model generationConfirm image through MatlabRGBText InputImage generatorDual port memory controllerVGA controllerSRAM modelText Output24bits RGB outputYCbCrYCbCrYCbCrYCbCrColor conversionRGBSRAM-VGA SimulationSimulation result

Original image(3200x2408)Resulting image(800x600)Down scaling Pin Configuration


VGAARMFPGAUARTCISISP embedded system verification environmentAccess slave address through Multi-ICEControl ISP functional blocks through embedded software

Future StudyPCB Connection bet. CIS-IPD boardNeed to remove noise of output images

SDRAM memory based dualport memory controller designNeed to increase output resolution

ISP functional block software/hardware partitionSystemC based

RT level simulationUse Seamless CVE based virtual core