Slide 1

P76994369 2010/1/10 Keywords: panorama, SURF, stitching, multi-band blending, LM, bundle adjustment

Slide 2

Introduction Modified SURF Comparison of SIFT and modified SURF Experiments and results Time cost test Conclusion and future works Reference

Slide 3

Stitching multiple images is a popular method of effectively increasing the field of view of a camera, by allowing several views of a scene to be combined into a single view. Stitching includes two main parts: image matching and image blending. find the motion relationship between two images or several images

Slide 4

For matching, there are two ways: direct method and feature detection method Direct method: is inconvenient because it always needs a high quality image. feature detection method: such as SURF and SIFT. For blending: weight averaged linear blending, multi-band blendingetc multi-band blending: good performance between quality of result and time cost

Slide 5

This paper a panorama image stitching process which combines an image matching system: Modified SURF + multi-band blending. 1. find the SURF features from the image(use KNN and RANSAC) 2. estimate the homography matrix according to LM ( Levenberg-Marquardt ) method, 3. adjust the coordinate of images 4. blend the images by multi-blending to remove the stitch seam and illumination discrepancy

Slide 6

SURF method: relying on integral images for image convolutions building on the strengths of the leading existing detectors and descriptors (using a Hessian matrix- based measure for the detector, and a distribution-based descriptor) simplifying these methods to the essential

Slide 7

Integral images allow the computation of rectangular box filters in near constant time This paper used KNN to find the nearest neighbor with setting k to 2 RANSAC is used to estimate a model for finding the minimize error matches set, which can maintain the correct matches by comparing the distance of the closest neighbor to that of second-closest neighbor

Slide 8

After extracting invariant scale features, we got potential feature matches by using k- nearest neighbor method, and then remove the mismatches with RANSAC algorithm. modified SURF Though SIFT get more matches than Modified SURF, Modified SURF is much faster than SIFT

Slide 9

Figure 1 and Fig. 2 give the match results of Modified SURF and SIFT in scale and illumination change respectively: win

Slide 10

modified SURF was not better than SIFT in rotation, Modified SURF is as robust as SIFT in other performance.

Slide 11

Summary: SURF describes image faster than SIFT by 3 times SURF is not as well as SIFT on invariance to illumination change and viewpoint change

Slide 12

bundle adjustment: to transform the images into a same coordinate or computing surface 1. Choose one of the images to be reference surface 2. transform each of other images to the reference surface usual choice for compositing larger panoramas is to use a cylindrical or spherical projection.

Slide 13

Transformation: compute the homography and optimize the parameters of the matrix 1. find out the best neighbor image for each image 2. calculate the distance between the two neighbor images 3. minimize the distance value to adjust the matrix between the neighbor images (use LM)

Slide 14

Levenberg-Marquardt (LM): nonlinear function least square , using nonlinear minimum square evaluation to minimize the transfer error, which is calculated as equation: Is correspondent with points X Homography matrix Euclidian distance

Slide 15

goal : is to produce a resulting image where no transition can be seen between the original source images. Linear method: may results in ghosting artifacts, blurring. But Linear blending method is fast and can be a good compromise between quality and speed if you are not too demanding on quality

Slide 16

multi-band blending (or called by pyramid blending): effective for image stitching without blurring and ghosting artifacts. It will produce much better results than the "Linear" mode. Multiband blending scheme ensures smooth transitions between images despite illumination differences

Slide 17

compare two methods: Refernce: http://www.cs.ubc.ca/~lowe/425/slides/11-PanoramasAR.pdf

Slide 18

Multi-band Blending: compare to linear blending multi-band blending can make image more clear in detail. In the paper, author use 2-band. band weighting( Gaussian function) weighting ( Gaussian function) band blending

Slide 19

The Laplacian pyramid of the final image is formed as equation: Pyramid blending gradually blends the lower frequencies of the images while maintaining a sharper transition for the higher frequencies where X1,k and X2,k are the kth level of Laplacian pyramid decomposition Yk is the kth level of Laplacian pyramid decomposition for the final combination result Mk is the kth level of Gaussian pyramid decomposition of the image mas Mk is the kth level of Gaussian pyramid decomposition of the image mas

Slide 20

Environment: studio 2008 C++ with OpenCV library Flow diagram has tow parts: matching and blending The connection of the two parts is the correspondence pairs.

Slide 21

Flow of matching: the goal of which is to find the largest feature points good to transformation Detect feature points square Euclidean distance ratio between neighbors is calculated estimate a model of consensus set that minimizes matching error Output for blending as itsinput

Slide 22

Flow of blending: H can be estimated based on those correspondence pairs Correct some stitching error of color and illumination. the images has been transformed into the corresponding image in a same coordinate system by the H matrix the images has been transformed into the corresponding image in a same coordinate system by the H matrix

Slide 23

Experiments consist of two parts: panorama quality (stitching) test and time cost test. A good stitching program should make panorama seamless and clear and be fast for using in various application such as real time processing.

Slide 24

Stitching test: there are three seams in Fig.4 Fig. 4. Panorama with obvious seam before blending processing Fig. 5. Panorama with seamless after blending processing of Fig. 4 1 2 3

Slide 25

Next, we did an experiment with large data set. In this experiment, we use 16 images of Camp dataset We will see that the present stitching process can show its good performance for large image dataset

Slide 26

Fig. 6. Panorama stitched 16 images based on SIFT. Fig.7. Panorama stitched 16 images based on modified SURF

Slide 27

The present system is faster than SIFT demo as shown in Fig 8 and Fig 9, due to using fast modified SURF. Fig. 8. Time cost number of images The present system is almost 4 times (3.56~4.46) faster than SIFT demo

Slide 28

Each experiment the time will change a little because of the CPU and memory, the reasonable experiment time is needed Fig 9 shows the time cost when the size of images that used are different

Slide 29

The present panorama image stitching process has a good performance. Due to Modified SURF, high-resolution panorama can be created in case there are some changes of illumination, color, blur and et cetera, and processed fast.

Slide 30

Reasons for good performance: K-NN and RANSAC improves the repeatability of matching. Bundle adjustment and multi-band blending make the panorama seamless. LM is used to estimate the homography, which makes the transformation more accurate

Slide 31

SURF is poor at handling viewpoint change handling illumination change Plus, the present system shows its defects when there are some noise images that are not neighbored removing the noise before stitching processing

Slide 32

Herbert Bay, Andreas Ess, Tinne Tuytelaars, Luc Van Gool. SURF: Speeded Up Robust Features. Computer Vision and Image Understanding (CVIU), Vol. 110, No. 3, pp. 346--359, 2008. Luo Juan and Oubong Gwun. A Comparison of SIFT, PCA-SIFT and SURF. International Journal of Image Processing (IJIP), Volume (3): Issue (4) 2009.