VISUALIZATION AND SART FOR 3D RECONSTRUCTION OF SOLDER BALLS

1Masatoshi Chang-Ogimoto (荻本雅俊), 2Chiou-Shann Fuh (傅楸善), 3Hung-Yi Chen (陳弘毅),

4Cheng-Shih Wong (翁丞世), 5Tsun-An Hsieh (謝尊安)

Department of Computer Science and Information Engineering, National Taiwan University,

Taipei, Taiwan 1b01902115@ntu.edu.tw, 2fuh@csie.ntu.edu.tw, 3xpineapple@hotmail.com, 4mob5566@gmail.com,

5pups003c@gmail.com

ABSTRACT This paper presents a 3D reconstruction technique to build 3D models using a series of X-ray images, particularly for reconstructing solder balls on Printed Circuit Boards during their manufacturing process. We discuss existing methods, emphasizing on algebraic methods such as the Simultaneous Algebraic Reconstruction Technique (SART). Keywords Computed Tomography; 3D reconstruction techniques; solder balls; Simultaneous Algebraic Reconstruction Technique

1. INTRODUCTION Computed Tomography (CT) for 3D reconstruction is a way to analyze the shapes and the internals of objects non-invasively. It aims to search for defects in objects during the manufacturing process without needing to dissect and destroy these objects. This is similar to building a digital 3D model of a person’s internals using Computerized Axial Tomography (CAT) scan to look for illnesses or broken bones without needing surgery.

We have received several X-ray images of Printed Circuit Boards (PCB) from Test Research, Inc. (TRI德律科技股份有限公司 ), a company specializing in testing and inspecting PCBs. The images have been formed by placing the PCBs between the X-ray source and the X-ray detector. The source projects rays towards the PCBs at various angles, and the detector captures the image on the other side. This process is illustrated in Fig 2. Each X-ray image is encoded as a grayscale image, where each pixel intensity value represents how much a section of an object attenuates the ray. In other words, the darker the pixel is, the thicker the section of the object and thus absorption is.

These X-ray images show an array of solder balls, and we would like to develop an efficient method to reconstruct these into 3D models. This would allow us to analyze the solder balls so that we can search for defects (such as small holes or voids within the solder

balls, too small or too large solder balls, missing solder balls, displaced solder balls) during the manufacturing process of the PCBs.

Analyzing PCBs for defects early on is essential since the company can react to them as soon as possible. It saves time and cost compared with shipping them to the users and shipping it back after the users encountered problems.

(a) (b)

Fig 1: Two of the 64 X-ray images of a PCB for when 𝜽 = 𝟑𝟓°. (a) 𝝓 = 𝟎° (1st image). (b) 𝝓 = 𝟏𝟐𝟗. 𝟑𝟖° (24th image). We have focused our attention on the 9

(𝟑  ×  𝟑) solder balls on the top left since others go out of view as we change the 𝝓 value.

Fig 2: Diagram showing the method of acquiring the X-ray images (theta=35o, SOD: Source Object Distance =

18 mm, SID: Source Image Distance=202 mm).

2. AQUIRING X-RAY IMAGES Fig 2 shows how Test Research, Inc. created the X-ray images for the PCBs. The X-ray tube is located at the top. The object of interest is located Source to Object Distance (SOD=18 mm) below the tube. The X-ray detector is located below the object and is placed Source to Image-Receptor Distance (SID= 202 mm) below the tube. The rays get projected at an angle of 𝜃 (theta=35o) towards the object and the detector. They have rotated the object and the detector around in a circle and produced images for every 𝜙 (phi) angles until they got an image from 0° to close to 360° (interval of 2.8o = 360o/128 projections).

For our research, we have used a set of 128 X-ray images where 𝜃 = 35° and with approximately 2.8° 𝜙 increments. The SOD and SID are approximately 18.4mm and 201.5mm, respectively. Each of these images has 1496 × 1496 pixels.

Fig 3: Diagram showing the rotation of the X-ray source and detector in terms of different 𝝓 values.

3. COMPUTED TOMOGRAPHY TECHNIQUES There are various techniques to perform 3D computed tomography (CT) reconstruction for sets of X-ray images. 3.1. Laminography Laminography involves performing a logical AND on these images after taking the binary threshold of the images [1]. The Filtered Back-Projection (FBP) algorithm involves running projections back through the image.

Fig 4: Resulting image from Laminography.

3.2. Algebraic Methods for Reconstruction We would like to focus our attention on various algebraic methods for 3D reconstruction. The Algebraic Reconstruction Technique (ART) [2] takes a subset that contains only one pixel and updates the volume after each pixel.

𝑓9:;<= = 𝑓9:

; +𝑔: − 𝑓9:

;A9B=

𝑁

where 𝑞 is the iteration number, 𝑔: is the measured data for a projection, 𝑓9:

;A9B= is the sum of the reconstructed

elements along the ray, 𝑁 is the number of reconstructed elements, and 𝑓9: is the element along the 𝑗 th line forming the projection ray 𝑔:.

This is based on the Gauss-Seidel Method that solves the system of linear equations [2].

The Simultaneous Iterative Reconstruction Technique (SIRT) is an improvement of the ART method. Each subset contains all pixels from all projections.

𝑓9:;<= = 𝑓9:

; +1𝑘

𝑔G 𝑓9:;

9,:

𝑁G

I

GB=

where 𝑘 is the total number of projections that pass through element 𝑓9:, 𝑔G is the sum of measured data for a projection, 𝑝  is the projection that passes through 𝑓9:, and 𝑁G is the number of elements in a projection.

This is based on the Jacobi Method for solving the system of linear equations [2].

We can compare the properties of the ART and SIRT methods by comparing the Gauss-Seidel Method [3] and the Jacobi Method [4] for solving system of linear equations. As an example, we would like to solve a system of linear equation that consists of three equations and three unknowns as follows:

5𝑥= − 𝑥L + 2𝑥N = 12 3𝑥= + 8𝑥L − 2𝑥N = −25 𝑥= + 𝑥L + 4𝑥N = 6

Then, we try to solve for the three unknowns:

𝑥= =12 + 𝑥L − 2𝑥N

5

𝑥L =−25 − 3𝑥= + 2𝑥N

8

𝑥N =6 − 𝑥= − 𝑥L

4

The first iteration for the Gauss-Seidel method is:

𝑥= = 0 𝑥L = 0 𝑥N = 0

Then, we use the results from the first iteration to solve for the second:

𝑥= =12 + 0 − 2 0

5=125

𝑥L =−25 − 3 12

5 + 2 08

= −16140

𝑥N =6 − 125 − −16140

4=6132

Next, we take the results from the second iteration to complete the third iteration:

𝑥= =12 + −16140 − 2 61

325

=333400

≈ 0.83

𝑥L =−25 − 3 333

400 + 2 6132

8= −

47371600

≈ −2.96

𝑥N =6 − 333400 −

−47371600

4=26011280

≈ 2.03

On the other hand, if we were to use the Jacobi method to solve the previous equation, the first iteration is:

𝑥= = 0 𝑥L = 0 𝑥N = 0

Unlike the Gauss-Seidel Method, the Jacobi

Method does not take the solution of a variable and immediately plug it into the next equation within an iteration. Instead, it solely uses the results from the previous iteration. Therefore, the second iteration is:

𝑥= =12 + 0 − 2 0

5=125

𝑥L =−25 − 3 0 + 2 0

8= −

258

𝑥N =6 − 0 − 0

4=64

The third iteration is:

𝑥= =12 + −258 − 2 6

45

=4740

≈ 1.18

𝑥L =−25 − 3 12

5 + 2 64

8= −

7320

≈ −3.65

𝑥N =6 − 125 − −258

4=269160

≈ 1.68 The actual solution for this system of linear equation is:

𝑥= = 1, 𝑥L = −3, 𝑥N = 2

Clearly, after the third iteration, the Gauss-Seidel Method already has the results close to the actual solution, which explains how the ART converges faster than the SIRT. However, the quality of the SIRT is higher than that of ART since ART suffers from salt-and-pepper noise [5]. This noise arises from the nature of approximation as illustrated with the above Gauss-Seidel Method where it gets the result of one equation and immediately plugs it into the next equation within an iteration. Although SIRT converges longer, as illustrated by the Jacobi Method, it does eventually produce smoother reconstruction.

To further illustrate how the ART method works for 2D reconstruction using the mathematical formula as presented above, we have an image where the pixels have the values 𝑓= = 5 , 𝑓L = 7 , 𝑓N = 6 , and 𝑓S = 2 as follows:

Fig 5: An array of pixels and their additive projections

(ART).

Assume we do not know the actual values of the pixels; we need to compute those values. We do know that when we project horizontal, vertical, and diagonal rays through the image, we can sum up the values of the pixels as shown in Fig 5. With those values, we can start the reconstruction process using an array of pixels with value of 0.

Fig 6: The process starts with setting all the pixels to 0.

We start with the vertical projection.

For this iteration, we project rays vertically. Using the equation for ART, we can estimate the values of the pixels:

𝑓= = 0 +11 − 02

= 5.5

𝑓L = 0 +9 − 02

= 4.5

𝑓N = 0 +11 − 02

= 5.5

𝑓S = 0 +9 − 02

= 4.5 This results in the following configuration:

Fig 7: We update the values of the pixels and then

continue on to the horizontal projection.

The next iteration takes the horizontal projection and results in:

𝑓= = 5.5 +12 − 10

2= 6.5

𝑓L = 4.5 +12 − 10

2= 5.5

𝑓N = 5.5 +8 − 102

= 4.5

𝑓S = 4.5 +8 − 102

= 3.5 This results in:

Fig 8: We continue updating the pixel values and

perform the diagonal projections.

And then we take the two diagonal projections:

𝑓= = 6.5 +7 − 102

= 5

𝑓L = 5.5 +13 − 10

2= 7

𝑓N = 4.5 +13 − 10

2= 6

𝑓S = 3.5 +7 − 102

= 2 Finally, we get the reconstructed image:

Fig 9: The final result for ART.

If we perform reconstruction for the same image with SIRT, then we start with an array of pixels with value of 0 and we take all of the projections for each iteration.

Fig 10: We again start with an image with empty pixels

and project for all angles.

𝑓= = 0 +13×

7 − 0 + 11 − 0 + 12 − 02

= 5

𝑓L = 0 +13×

9 − 0 + 12 − 0 + 13 − 02

= 5.67

𝑓N = 0 +13×

11 − 0 + 8 − 0 + 13 − 02

= 5.33

𝑓S = 0 +13×

8 − 0 + 9 − 0 + 7 − 02

= 4

which results in the image below:

Fig 11: We update the pixel values and project again.

𝑓= = 5 +13×

7 − 9 + 11 − 10.33 + 12 − 10.672

= 5.17

𝑓L = 5.67 +13×

9 − 9.67 + 12 − 10.67 + 13 − 112

= 6.11

𝑓N = 5.33 +13×

11 − 10.33 + 8 − 9.33 + 13 − 112

= 5.33

𝑓S = 4 +13×

8 − 9.33 + 9 − 9.67 + 7 − 92

= 3.33

Since SIRT converges slower than ART, it can take several more iterations until it gets close to the original values.

Finally, the Simultaneous Algebraic Reconstruction Technique (SART) is the combination of the ART and SIRT methods to bring both fast convergence and good results.

𝑓:9 = 𝑓:

9T= +𝜆𝑝9 − 𝑓I

9T= 𝑤9IAIB=

𝑤9IAIB=

GW∈YZ

𝑤9:GW∈Y[𝑤9:

𝑗 = 1,2, … , 𝑁

where 𝜆 is the relaxation factor, 𝑤9: is the weighted coefficient, and 𝑃[ is all the projection values in one degree.

4. SIMULATION WITH UNITY 3D To get a sense of the positioning of the solder balls with respect to the X-ray source and detector, we have utilized Unity 3D [6]. Although Unity 3D is a game engine, it is still useful for creating quick prototype simulations for our research since it provides means of creating C# scripts and creating simple lighting and primitive objects.

Another advantage of using Unity 3D is that it provides support for Google Cardboard VR [9]. Unity 3D and the Cardboard VR SDK provides a means of compiling the game into Android or iOS virtual reality apps so that we can have a stereoscopic view of our creation. Then, we can place our smartphones inside the Cardboard VR headset so that we can see the solder balls in stereo vision and 3D. This is useful since it allows us to preview our creation in 3D. With smartphones’ built-in accelerometer and gyroscope, it is easy for us to rotate around the virtual solder balls.

First, we have created a simple 3D plane in the scene and applied the X-ray images as the texture. Then, we created spheres and placed them in a 3  ×  3 grid. We have positioned them so that they are above where the solder balls are located on the 3D plane. Next, we have created a directional light which we can change the angle of the overall scene lightning.

Fig 12: Determining the solder ball and X-ray

positioning using Unity 3D.

We have tested with three X-ray images: 𝜙 = 0.0°, 𝜙 = 120.94 , and 𝜙 = 239.06° projection images. For each of these images, we have positioned the solder balls and pointed the directional light at a specific angle (theta = 35o) so that they cast shadows right on top of the X-ray solder ball image. We found out that we can cast shadows in correct positions if we rotated the directional light at 0° for the 0.0° image, 120° for the 120.94°, and 239° for 239.06°. This confirms that the information on the positioning and the angle for the X-ray images matches with our 3D simulation.

Fig 13: Close-up view of the spheres as solder balls and their casted shadows by rotating around eye optical axis.

Fig 14: Panning left and right to view around the solder balls. The center of the plane is the pivot point for the

360° view.

Fig 15: Viewing through an orthogonal projection

camera shows coplanarity of the balls by tilting up and down.

Fig 16: View from the top by tilting up and down.

After this, we would like to determine how well the shadows align with the X-ray images. We can do this by using the fitting ratio:

𝑋 ∩ 𝑆𝑋 ∪ 𝑆

where 𝑋 is the solder ball in the X-ray image, and 𝑆 is the shadow casted by the virtual spheres. A value of 0 means that there is no overlap and 1 means there is a perfect overlap. Using Unity 3D’s Lightmapping tool, we can “bake” the lighting and shadows so that it converts the dynamic shadows into a shadow map, where it is saved as a texture image as shown in Fig 18. We can use Adobe Photoshop CS6 to preview the overlap of the shadows from the shadow map with the original x-ray image as in Fig 19 [7]. Then, using OpenCV library [8], we can create a binary threshold images of the x-ray and shadow map images, perform an binary AND and OR, and then get the fitting ratio of 85.56%. Each of the fitting ratio for individual solder balls appears floating above the 3D balls in the virtual space.

Fig 17: Calculating the fitting ratio by dividing the

intersection by the union (𝑿∩𝑺𝑿∪𝑺

)

Fig 18: Shadow map of the balls projected onto the

plane.

Fig 19: The original x-ray image with 30% opacity

placed on top of the shadow map.

5. EXTRACTING 2D PROJECTIONS Next, we have written code to extract multiple projections of the X-ray images. To test our methods, we have created a simple image to create the projections for. This image of size 149  ×  149 pixels consists of white primitive shapes on a black background: a square, two triangles, a circle, and a rectangle. We have created this using Adobe Photoshop CS6 with its shape drawing tools. We can think of a single pixel of having 1 unit of height and 1 unit of width.

Fig 20: Image used for testing our projection extraction

methods.

When we project the 2D image with rays, we can say that each ray has a width of 1 pixel, so projecting rays vertically and horizontally through the image just requires us to add up the values of the of the pixels. For example, Fig 21 shows that the a ray of 1 pixel width (represented as a gray bar) is projecting horizontally (0° projection). We can simply add 6 + 2 + 8 = 16 so that the projection for that specific ray results in a value of 16. We do this horizontally for all rows (0° projection) and vertically (90° projection). The output is shown in Fig 22. The higher the value (the whiter) of the output pixels are, the more the ray intersects a think or many objects.

Fig 21: Projecting a ray (in gray) of 1 pixel width

horizontally through a 𝟑  ×  𝟑 pixel image.

Fig 22: The resulting projections for 0° and 90° projections.

However, projecting rays at any other angles can get complicated. The amount of influence a pixel has on the ray depends on how much the ray overlaps the pixel. For example, if the ray overlaps half of a pixel, then the ray gets half the value of the pixel. We do this for all the pixels that the ray intersects and acquire the sum of the partial pixel values for each ray.

Fig 23 shows an illustrated example. Each of the pixels have values as represented as the numbers. The gray bar is the projected ray with 1 pixel width. The red dots represent the intersection points between the ray and the pixel edges. We can use these dot positions to calculate shape and area for each pixel that the ray covers. For example, if we connect the dots the ray creates a hexagonal shape when it passes through the pixels with values 3, 2, and 9. These hexagonal shapes covers 92% of each of these pixels. Therefore, we get the 92% value of these pixels (i.e. 3 × 0.92=2.76). The ray creates triangular shapes for pixels with values 1, 6, 8, and 7. These triangles cover 25% of these pixels. Therefore, the total projection value extracted for that particular ray is:

3 + 2 + 9  ×  0.92 + 1 + 6 + 7 + 8  ×  0.25 = 18.38

Fig 23: Example of ray projection at 45°.

We did this by writing a script in Python. The script calculates the projection angles using simple line

equations (determine the slope and line intercepts). Then, it gets the coordinates of the intersection of the ray and the pixel edges. For the pixels that the line intersects, it gets the intersection points for that particular pixel and calculates the size of the area that the shape the points create using the Shoelace algorithm [10]. Next, we add up all the partial pixel values for each ray. After we compute this for all projection angles, we get the largest value we got from the rays and divided it by 255 (maximum value of a pixel of an 8-bit image). We take this number and divide all of the values we have acquired by that number so all of the values are between 0 to 255, inclusive. Finally, we create an image from these values.

Fig 24: Final sinogram output for the projections. Each

pixel row represents each of the 90° projections.

Due to the complexity of getting the partial values for each pixels, we discovered that our script is inefficient. It can take about 3 hours to compute 90° projections even if we use 7 processes.

Instead of computing different ray projections, we have tried to rotate the image itself and simply added the vertical and horizontal projections.

Fig 25: Computing projections for angles 0°, 45°, 90°,

and 135° using the image rotation method.

Rotating the image can cause some distortions to the shapes since this is a low-resolution image. Distortions could be caused by aliasing problems or white pixels from objects disappearing during rotation. However, these kinds of distortions are not noticeable, and we believe that the rotation method can be a quick approximation of the ray projections.

6. 3D VISUTALIZATION USING POINT CLOUD We have used some 2D slices of the solder balls [11], where each slice represents a layer of the balls, to create a 3D visualization using point cloud. For each of the slices, we have used OpenCV’s Canny Edge Detector in Fig 26 to find the contours of the balls [12]. Then, we have converted each of the contours’ pixels into a point cloud using Point Cloud Library (PCL) [13].

Using point cloud can help us visualize the 3D spherical shapes of the solder balls as shown in Fig 27. Furthermore, if we can detect problems such as holes or voids in the solder balls for the image slices, then it is also possible to construct the shape of the holes or voids inside the point cloud of the solder balls.

Fig 26: Using the Canny Edge Detector to extract the

contours of the balls at each layer.

Fig 27: Point cloud of the 9 solder balls gives us a way

to visualize the balls in 3D.

7. CONCLUSION

We have presented several methods of reconstructing and visualizing the solder balls using x-ray images projected at various angles. The SART is the method that builds on top of the general ART and SIRT reconstruction methods, combining fast convergence and high-quality output. Using Unity 3D, we can quickly visualize the positions and sizes of the solder balls. It is also possible to build a Google Cardboard VR smartphone app to have a stereoscopic view of this prototype simulation. Utilizing point cloud is also a simple way of viewing the solder balls in 3D.

ACKNOWLEDGEMENT This research was supported by the Ministry of Science

and Technology of Taiwan, R.O.C., under Grants MOST 103-2221-E-002-188 and 104-2221-E-002-133-MY2, and by Test Research, Lumens Digital Optics, and Lite-on.

REFERENCES

[1] S. Gondrom, S. Schröpfer, FhG ITFP, Saarbrücken, “Digital Computed Laminography and Tomosynthesis - Functional Principles and Industrial Applications,” http://www.ndt.net/article/v04n07/bb67_11/bb67_11.htm

[2] Frank Tabak, “Robust Algorithms for Discrete Tomography,” Technical Report, Delft Institute of Applied Mathematics, Delft University of Technology, 2012.

[3] Wolfram MathWorld, “Gauss-Seidel Method," http://mathworld.wolfram.com/Gauss-SeidelMethod.html, 2016.

[4] Wolfram MathWorld, “Jacobi Method," http://mathworld.wolfram.com/JacobiMethod.html, 2016.

[5] A. C. Kak and Malcolm Slaney, “Principles of Computerized Tomographic Imaging,” 1988.

[6] Unity Technologies, “Unity,” http://unity3d.com/, 2016. [7] Adobe, “Photoshop CS6,” http://ww.adobe.com/, 2016. [8] OpenCV, “OpenCV,” http://opencv.org/, 2016. [9] Google, “Google Cardboard,”

https://vr.google.com/cardboard/index.html, 2016. [10] Charles L. Hamberg and Ronald Vavrinek, “Shoelace

Algorithm,” Illinois Mathematics and Science Academy, 2016.

[11] Y. B. Zhang, “3D Reconstruction of Solder Balls,”

submitted to IPPR Conference on Computer Vision, Graphics, and Image Processing, Keelung, Taiwan, 2016.

[12] OpenCV, “Canny Edge Detection,”

http://docs.opencv.org/master/da/d22/tutorial_py_canny.html, 2016

[13] Point Cloud Library, “Point Cloud Library,”

http://pointclouds.org/, 2016