Quasi Multi-Baseline-Stereo Method with Single Camera for Reducing Occlusion

TAKASHI WATANABE, AKIRA KUSANO, TAKAYUKI FUJIWARA, and HIROYASU KOSHIMIZUSIST, Chukyo University, Japan

SUMMARY

A robust 3D image measurement method is proposedin order to suppress occlusion by a quasi multi-baseline-ste-reo method based on a single camera configuration. Thereis no way to suppress completely the “occlusion problem”in the field of stereo geometry. In order to deal with thisproblem, the multi-baseline-stereo configuration is oftenadopted and is basically effective. This paper proposes anew configuration for stereo measurement by means of asingle camera, a mechanical feeder for the object, imageprocessing for the baseline calibration, and in particular,diagonal feeding of the object in front of the camera. Theconcept of this method is to realize a pair of stereo geome-tries in the stereo configuration with a single camera and amechanical feeder. It is technically noted that the baselineof this single camera stereo was precisely calibrated by animage processing procedure in both the horizontal andvertical directions. © 2009 Wiley Periodicals, Inc. ElectronComm Jpn, 93(1): 41–49, 2010; Published online in WileyInterScience (www.interscience.wiley.com). DOI10.1002/ecj.10156

Key words: image processing; computer vision;single camera stereo measurement; occlusion; diagonalparts feeding.

1. Introduction

Stereo measurement methods that use the principleof triangulation are representative of passive measurementmethods, and their use in the measurement of three-dimen-sional shapes in topography is common [1]. In such stereomeasurements, measurement points are extracted frommultiple images captured by two or more cameras, and thedistance from the camera to the target to be measured iscalculated. However, because creating a correspondence

between the extracted measurement points in the images isdifficult due to slow changes in the shading of the images,silhouettes with clear edge information or measurements ofpatterned feature points constitute difficult limitations. Fur-thermore, in images captured with two cameras set up inparallel, occluded areas where shape information cannot beacquired occur, and thus measurements may be limited toan object’s surface where information can be acquired. Athree-lens stereo measurement method using three camerasis known to be effective for resolving these problems [1].Cameras with three lenses in one body are sold commer-cially [2], and are widely used in research on measuring theshape of objects, restoring three-dimensional shapes, andmotion capture [3, 4].

However, if three-dimensional measurements arelimited to when there is clear feature point informationalong the silhouette of an object, then there is no need toconfirm all of the surface information for that object, andthere is no need to resolve the problem of point correspon-dence between difficult images. This approach is requiredfor scanning measurements in manufacturing facilities [5,6], but to satisfy these requirements, there is no need toimprove performance by increasing the number of cameras.As a result, the development of a system that simulates asimple multiple baseline stereo measurement using onecamera instead of three-camera stereo measurements hasthe advantages of reducing costs, saving space, and simpli-fying processing.

This paper describes the performance of quasi multi-ple-baseline-stereo measurements in a single-camera stereosystem with a simple configuration that horizontally movesone fixed camera and the object to be measured. We providepractical examples of measurements using our method andreport on verification of its effectiveness through experi-ments.

2. Single-Camera Stereo Measurement Method

The single-camera stereo measurement method con-sisting of a camera system with one camera involves calcu-lating the distance from the camera by using as the parallax

© 2009 Wiley Periodicals, Inc.

Electronics and Communications in Japan, Vol. 93, No. 1, 2010Translated from Denki Gakkai Ronbunshi, Vol. 127-C, No. 4, April 2007, pp. 652–658

Contract grant sponsors: 2004 IMS Center-HUTOP grant, 2004 grant fromthe Ministry of Education, Sports, Science, and Culture High-TechnologyResearch Center (HRC), and by funding from JST and NEDO.

41

the distance between feature points extracted from twoimages before and after moving the object to be measured.This is a simple system composed of one camera systemand a movement mechanism. However, in combinationwith the problems of stereo measurement described in theprevious section, when there are no limitations on thecamera being moved or on the direction of movement of theobject to be measured, complex processing is required [7].Compared with methods using two or more cameras, de-spite the lack of a need for position calibration between thecameras and the advantages of cost versus performance, notmuch research has been performed.

However, if the distance between the object to betransported on a conveyer belt, the manufactured objects ina self-assembling mechanism, or the individual compo-nents in assembly equipment can be fixed between thecamera and the measurement target, or if the measurementsare limited to an environment in which stable movement ofthe camera or the measurement target can be maintained,then the method above has sufficient value. We have studiedapplications involving electronic scanning of components[8, 9], and have verified their effectiveness. In this paper wedescribe a system based on a single-camera stereo measure-ment method for the quasi multiple-baseline-stereo meas-urement method proposed in this paper.

3. Object of Measurement and Principles ofMeasurement

3.1 Object of measurement

We selected the mechanical parts in Fig. 1 as anexample of a measurement target. As can be seen in Fig. 1,the shape is simple, and the external shape of the part canbe measured adequately using calipers or a micrometer. Ifthe c surface (diagonal part) in Fig. 1 is distorted when it isset under conditions in which it is tightly affixed to otherparts, then the distance a, b (mm) with respect to the tightlyaffixed surface may vary substantially, and the reliability of

the values measured before setup drops substantially. In thispaper, we verify the effectiveness of our system by meas-uring the a, b distance under conditions in which the csurface in Fig. 1 is set onto other mechanical parts.

3.2 Principles of single-camera stereomeasurements

As can be seen in Fig. 2, one camera is set in place,the object to be measured is moved along the horizontalwith respect to the camera, and the distance traveled is takento be the baseline length D. At this point, the base block inFig. 2 which moves the object to be measured and thecamera are set up in parallel. Also, the distance between thecoordinates for the two feature points pn (n = 1, 2: 1: featurepoint extracted before moving the object to be measured;2: feature point extracted after moving the object to bemeasured) before and after movement in the image shownin Fig. 3 is taken to be the parallax d, and the distancebetween the base block (mechanical part) used to move theobject to be measured from the camera, information alreadyknown, is taken to be K. Using Eq. (1), we calculate thedistance H between feature points from the camera shownin Fig. 3.

In order to calculate the parallax d described above,the camera’s field of view must be maintained greater thanthe baseline length D (movement distance of the base block)determined arbitrarily so as to confirm the feature points p1

and p2 in the captured images. As a result, the magnificationof the camera lens is determined while taking into consid-eration the size of the camera elements for capturing im-ages. Furthermore, the distance K between the camera andthe base block can be determined in a relative manner basedon these conditions:

Fig. 1. Basic structure of the mechanical parts. Fig. 2. Single-camera stereo model.

(1)

42

4. The Proposed Method

4.1 Direction of movement of the object to bemeasured to confirm occluded surfaces

Based on the model in Fig. 2, we propose a methodof moving the object to be measured, as shown in Fig. 4,along a diagonal line (γ = ±45°) in the camera’s field ofvision.

This method allows for checking the surfaces Athrough E in Fig. 4. This is a strategy to resolve the occlu-sion of the D and E surfaces in the conventional movementalong the x axis (γ = 0° or 180°), or the B and C surfacesduring movement along the y axis (γ = ±90°).

Figure 5 represents a movement model on the x, yplane for the object to be measured. The movement distance(baseline length) can be extracted as three types of move-ment: movement on the x axis (Dx), movement on the y axis(Dy), and real movement (D), meaning that information on

three distances can be acquired from one action. Moreover,as was the case in Fig. 6 for the distance between featurepoints (parallax) in an image, the three distances (dx, dy, d)can be acquired. The baseline length for the real movementand the movement along the x and y axes, and the distancebetween feature points can be calculated using Eqs. (2) to(5). However, for surfaces present at the same angle as themovement (γ = –45° or y = 45°), recognition is impossible.As a result, in this case the angle of movement must be γ =0° or γ = 180°.

(4)

Fig. 3. Measurement model.

Fig. 4. Quasi multi-baseline-stereo method for singlecamera.

(2)

(3)

(5)

Fig. 5. Feeding distance and baseline length.

Fig. 6. Feature point and disparity in image.

43

4.2 Selection of the parallax and baselinelength

The baseline length and parallax calculated in Eqs.(2) to (5) result in three values for each, which are selectedbased on the presence of feature points and the direction ofthe face. When measuring a feature point that is almostparallel to the x axis of the camera’s field of view, Dy anddy are used; when measuring a feature point that is almostparallel to the y axis of the camera’s field of view, Dx and

dx are used; otherwise, D and d are used. This representsusing the baseline length and parallax that are virtuallyparallel to the slope of the target face (Fig. 7 shows thesituation for selecting the baseline length). Note that theobject being measured here is simple and consists of facesthat are almost parallel to the x axis or y axis. As a result,the distances Dy, dy, Dx, and dx can be used. Moreover, theslope information for the face in the image’s field of viewcan be detected by extracting a straight-line component ofthe face using edge processing.

4.3 Movement state of the object beingmeasured

Figure 8 shows images captured using the movementmethod described in Section 4.1. These images representcaptures of the mechanical part in Fig. 1 moving at an angleof γ = –45°, made with an ordinary CCD camera (1.45megapixel progressive scan) and a ×0.1 macro lens. Athrough D in the three-dimensional linear model in Fig. 8represent the surfaces of the object to be measured. It isclear that the four surfaces A through D were checked in theimages in the process of moving from No. 1 to No. 6.

In order to measure the distance a,b in Fig. 1, we useimages No. 1 and No. 3. In order to measure the dispersedstate of the distance at “b” in Fig. 1, we use the images No.5 and No. 6, which can check the face C, extract the two

Fig. 8. Captured image of mechanical parts.

Fig. 7. Selected baseline length.

44

feature points as shown in Fig. 9, and create a correspon-dence by measuring the distances for b1 and b2.

4.4 Calibration of the baseline length

In Figs. 5 and 7, the movement distance of the objectto be measured is used as the baseline length. However, theuse of a stage with the ability to measure the movementdistance results in increased costs, and is contrary to theintention of developing a simple system. Therefore, theauthors attached a mark, as seen in Fig. 10, on the base block(Fig. 2) set up on the object to be measured, then used imageprocessing to measure the distance between feature pointsfor the shape of the mark extracted using the images beforeand after movement, and used the result as the baselinelength.

The mark has inside it a rectangular or linear compo-nent used to extract the feature points for measurement, andthese lines are shaped so that they can intersect. As can beseen in Fig. 11, after the edge of the mark’s silhouette isformed, a linear approximation is performed using themethod of least squares based on the edge coordinate valuesalong the x axis and the y axis. The intersection of the

approximated lines is taken to be the feature point OPn (n= 1, 2). The distance between these feature points is calcu-lated using Eqs. (2) and (3), and the result is the baselinelength D, Dx, Dy. Note that in this method, tracking thesymbol mark and generating the edge accurately is vital.After determining the direction of movement of the baseblock ahead of time, minimal regions for the range ofmovement of the symbol mark can be segmented, and acorrespondence created by acquiring position informationusing template matching.

4.5 Method to extract feature points

Linear approximation is performed using the methodof least squares for the two edge groups including thefeature points, so as to create a precise correspondence forthe feature points extracted from the images before and aftermovement of the object to be measured. The point wherethe approximated lines intersect is taken to be the point oforigin qn (n = 1: before movement; n = 2: after movement)used for feature point extraction (Fig. 12). An intersection

Fig. 9. Measuring point for verification.

Fig. 11. Setup of measuring point.

Fig. 10. Symbol mark. Fig. 12. Estimated origin point.

45

of an edge and a circle of radius rn (n = 1, 2, . . .) with itscenter at qn is taken to be the feature point pn (n = 1, 2, . . . )used for taking measurements (Fig. 13). Note that the valueof the radius rn and the number of feature points pn are setarbitrarily as required.

In this method for extracting feature points, the abil-ity to create a precise correspondence for extractions fromthe two images before and after movement was verifiedexperimentally.

Similarly, we verified experimentally the effective-ness of a compensation method that tracks changes in theposition of the extracted feature points accompanyingmovement of the object to be measured [8, 9]. Because wepreviously performed an experiment with a fixed stage formovement of an object to be measured in this situation, themethod is omitted here. Processing is required when theobject in question is not fixed to a moving stage.

5. Measurement Experiments and Discussion

5.1 Experimental environment

As shown in Fig. 14, we developed an experimentalsetup, and evaluated the processing time and measuredvalues.

The setup consists of a software environment runningon an ordinary commercial PC.

(1) CPU: Celeron 2.4 GHz(2) OS: Windows 2000(3) Software: HALCON (MVTec, LinX), Visual C++

(Microsoft)(4) Memory used: 100 to 150 MB(5) Camera: 1.45 megapixel black and white progres-

sive scan camera(6) Lens: ×0.1 macro lens

5.2 Evaluation of the validity of the measuredvalues

To evaluate the validity of the values measured withthe system, we performed ten measurement trials using thesystem and an industrial microscope, and compared theaverage values. The difference in the measured values forthe industrial microscope and the proposed system wasbetween 0.002 and 0.02 mm, as shown in Fig. 15. Differ-ences in the measurement conditions occurred, becausemeasurements under the conditions with the base block asthe object to be measured were difficult when using theindustrial microscope. However, the surface of the objectto be measured and the base block used in the setup had fewirregularities because they were polished. As a result, thedifference in the measured values due to the differences inthe measurement conditions described above can be ex-pected to be minimal, and the values may be consideredvalid.

We also calculated the deviation (difference for eachmeasured value with respect to the average value) for tenmeasurements in order to check the precision over time.Figure 16 shows the results. Whereas the deviation in themeasured values for the industrial microscope was between

Fig. 13. Feature point extraction.

Fig. 14. Appearance of experimental equipment. Fig. 15. Experimental results.

46

–0.007 mm and 0.010 mm, the deviation for the measuredvalues under the authors’ system was between –0.014 mmand 0.013 mm. The standard deviations were 0.005 mm and0.009 mm, respectively. We also confirmed that the meas-ured values for the industrial microscope were about asprecise with regard to the level of dispersion.

5.3 Scanning processing time

When performing measurements of the same kind asin the previous section in the experimental environmentdescribed above, the processing time was from 0.357 to0.433 s (maximum value and minimum time values whenprocessing the same sample ten times). However, the proc-essing time does not include the movement time for theobject to be measured.

5.4 Discussion

(1) Direction of movement of the object to bemeasured

In the experiment described here, the object to bemeasured was moved at an angle of γ = –45° (Fig. 5).Surface recognition of the experimental sample, with facesparallel to the x and y axes, was possible. The movementangle when the object in question has a shape consisting offaces parallel to the x axis or the y axis is valid at γ = ±45°.

(2) Baseline length selection

Figure 17 shows the results of comparing the meas-ured values based on differences in the selected baselinelength (Fig. 7: D, Dx, Dy).

In the comparison of the measured values for anindustrial microscope described in the previous section

(5.2), the difference in the measured values when using abaseline length of Dx ranged from 0.002 mm to 0.02 mm,a relatively small difference.

Based on these results, we concluded that using amovement direction along the x axis (Fig. 7: Dx) providedthe highest precision for measuring the feature points of aface virtually parallel to the x axis in the camera’s field ofview. This confirmed the validity of the method for select-ing the baseline length in Section 4.2.

6. Conclusions and Future Topics

6.1 Summary of conclusions

We developed a quasi multi-baseline-stereo measure-ment method using single-camera stereo measurement andclarified the following items through experiments.

(1) By using two images taken before and after themovement of an object fed along a diagonal in the camera’sfield of view, stereo measurements with reduced occlusioncould be obtained.

(2) We improved precision by using the baselinelengths in the three directions separately, as necessary forthe face to be measured.

(3) The space requirement for camera attachment isminimized because of the configuration of the one-camerasystem. Compared to the setup for three-camera stereomeasurements with three cameras, the number of cameras,lenses, and other photographic equipment as well as themechanical parts used for the equipment setup are reducedto one-third. Furthermore, an image input board supportingonly one camera can be selected, achieving a cost reductionof about 50%.

Fig. 16. Deviation in measurements of the proposedsystem and industrial microscope. Fig. 17. Calculation of baseline length.

47

6.2 Future topics

We plan to evaluate the validity of the movementdirection of the object to be measured, and the method forselecting the baseline length and parallax while expandingthe measurement target, and determine the optimal valuesfor the system.

Acknowledgment

We express our gratitude to all persons in the KosuiLaboratory who made this research possible. A portion ofthe research was supported by a 2004 IMS Center-HUTOPgrant, by a 2004 grant from the Ministry of Education,Sports, Science, and Culture High-Technology ResearchCenter (HRC), and by funding from JST and NEDO.

REFERENCES

1. Inoguchi S, Sato K. 3D imaging techniques for meas-urement. Shokodo; 1990. (in Japanese)

2. Kuwashima S. Special editions, Newest trends of 3Dimage measurement and analysis. Image InformationIndustrial 1999;31:17–22. (in Japanese)

3. Aoki K, Kaneko T. Pose detection of 3D object byrange images from moving stereo vision. IEICETrans Inf Syst 2003;J86-D-II:72–83. (in Japanese)

4. Sumi Y, Ishiyama Y, Tomita F. Real-time free-formobject tracking using a model-based approach forstereo vision systems. IEICE Trans Inf Syst2001;J84-D-II:1693–1700. (in Japanese)

5. Ejiri M. Industrial applications of image processing(Examples). Fuji Techno Systems; 1994. (in Japa-nese)

6. Koshimizu H. Today’s automated visual inspectionsystems. Techno Systems; 1990. (in Japanese)

7. Deguchi K, Akiba I. A linear algorithm for monocularstereoscopy of moving objects. Trans SICE1990;26:714–720. (in Japanese)

8. Watanabe T, Kusano A, Fujiwara T, Koshimizu H. 3Dmeasurement by a single camera and its use in aprecise baseline detection algorithm for electronicdevice inspection. Proc IWAIT 2006, p 42–47.

9. Watanabe T, Kusano A, Fujiwara T, Koshimizu H.Flatness inspection of terminal leads by single stereomeasurement system. MIRU 2006, p 1012–1017. (inJapanese)

AUTHORS (from left to right)

Takashi Watanabe (nonmember) graduated from the Department of Mechanical Engineering, Fukushima Vocational andTechnical College, in 1988. Since 2000 he has been a researcher working on artificial intelligence in the Department ofInformatics, Chukyo University. He began the second half of his doctoral studies in information science at the University ofKyoto in 2004. He is now engaged in research on measurements in three dimensions.

Akira Kusano (nonmember) graduated from the Department of Information Technology, Taira Vocational and TechnicalCollege. Since 2005 he has been a researcher working on artificial intelligence in the Faculty of Information Science andEngineering, Chukyo University. He is now engaged in research on the inspection of electronic components.

Takayuki Fujiwara (member) completed his doctorate in information science at Chukyo University in 2003 and becamea lecturer there. He is engaged in research on facial image processing and recognition in two and three dimensions, and its usein systems that generate similar faces. He holds a D.Inf.Sc. degree.

48

AUTHORS (continued)

Hiroyasu Koshimizu (senior member) graduated from the Department of Electrical Engineering, Yamanashi University,in 1970 and completed his doctoral studies at Nagoya University in 1975. He holds a D.Eng. degree. After becoming a lecturerat Nagoya Institute of Technology, and then in urban engineering in the city of Nagoya, he was appointed a professor in theDepartment of Education, Chukyo University. In 1990 he became a professor in the Department of Informatics of the sameuniversity, and in 1994 he became a professor in the Graduate School of Informatics Research. He became head of theDepartment of Informatics in 2004, and a professor in the Department of Information Science and Engineering in 2006. He hasalso served as dean of the Department of Information Science and Engineering. His fields of research include image processing,pattern recognition, computer vision, artificial intelligence for vision, and industrial applications of these topics. He receivedthe Odawara Prize (VIEW 2002, VIEW 2005), and the Prize for Most Outstanding Paper (Society for Arts and Sciences, 2002).He is a senior member of IEEJ (2003), and a member of IEICE, JSAI, SICE, the Japanese Academy of Facial Studies, and theInstitute of Image Information and Television, among others.

49