MOTION TRACKING OF BRACHIATION ROBOT BASED ON LABVIEW

USING SEAMLESS INTEGRATION TECHNOLOGY

1 Henry Liu (劉又豪), 2 Chiou-Shann Fuh (傅楸善)

1 Department of Mechanical Engineering,

2 Department of Computer Science and Information Engineering,

National Taiwan University, Taipei, Taiwan

E-mail: b03502079@ntu.edu.tw fuh@csie.ntu.edu.tw

ABSTRACT

We report on the realization of the real-time

synchronous image acquisition and the image processing

for detecting the motion of the brachiation robot. To

derive the best performance, we cooperate two platforms,

LabVIEW and MATLAB. LabVIEW provides the

environment to control the brachiation robot, and

MATLAB has powerful image-processing functions.

Furthermore, by using COM technology, slow processing

speed caused by the integration of two platforms could

be eliminated. Finally, we conducted experiments to

investigate the performance of the proposed method. The

webcam successfully captured the motion of the robot in

the real-time system.

Keywords: LabVIEW, Real-time visual processing,

MATLAB, COM technology

1. INTRODUCTION

Brachiation robot is a typical bio-inspired robot of

gibbons. While analyzing the dynamic motion of

brachiation robot, some necessary measurements, such as

angle, acceleration and mass center, are required for the

calculation. To collect these data, the equipment like

inertial measurement unit (IMU) is commonly used.

However, several difficulties need to be addressed

while utilizing this device. First of all, IMU consists of

two elements, which are accelerometer and gyroscope.

Fig. 1 presents several crucial parameters of brachiation

robot model. We need angle θ1 and the angle θ3 to

derive the current posture of brachiation robot. θ1, refers

to the angle between the

Fig. 1 Brachiation robot model.

hand and the horizontal branch, and θ3 refers to the

angle between the arm and the body. These results come

from calculating the integration of the angular velocity

from the gyroscope and the angle between two

acceleration vector [1]. Nevertheless, the results are

generally not ideal due to the intense sensitivity of two

instruments.

Filters are implemented for improving the poor

performance caused by the sensitivity, but sometimes it

takes even more time to be familiar with the filter. Take

Kalman filter for example, although the filter does

enhance the results, several experiments must be done to

derive parameters needed by the filter equation, and it’s a

painstaking process [2]. Furthermore, there are only a

few I/O pins left on myRio, which is the embedded

device on the brachiation robot, after the basic

requirements. Since IMU takes five I/O pins to connect

with myRio, the measurements we could detect are

limited.

Therefore, a proposed method is given to facilitate

the whole process. A real-time image acquisition is

realized by the virtual instrument design software

LabVIEW, which is the original software that used to

control the robot so additional development time for

other software could be saved. On the other hand, the

image-processing algorithm is designed in Matlab and

implemented into LabVIEW with COM technology

instead of using Matlab Script Node, a commonly used

method in LabVIEW, to enhance the processing speed.

2. LABVIEW OVERVIEW

LabVIEW (Laboratory Virtual Instrumentation

Engineering Workbench), developed by the National

Instruments, is a system design platform and

development environment. It uses a graphical

programming language which is easy to understand and

simplifies the process. LabVIEW gives the design

concept a more accurate picture since it implements a

data-flow paradigm in which the code is not written, but

instead drawn or represented graphically similar to a

flowchart diagram. (Fig. 2) [3]. Functions in LabVIEW

are stored in virtual instruments (VIs) which has two

essential parts: the front panel for displaying or

controlling

Fig. 3 Example of NI-IMAQdx.

the parameters, and the block diagram where the

algorithms are designed by using the graphical

components. Due to the user-friendly graphical interfere,

the design process is much more straightforward,

reducing the programming time. Furthermore, with

plenty of image processing and machine vision functions

or toolkits, LabVIEW is available for both image

acquisition and image processing.

3. SEAMLESS INTEGRATION BETWEEN

LABVIEW AND MATLAB

3.1. Comparison of different approaches

There are two approaches for LabVIEW to acquire

the images: NI-IMAQdx and using another programming

language. NI-IMAQdx is the built-in image processing

toolkit that offers image-acquisition and processing

functions in Virtual Instrument environment. The whole

toolkit covers the complete process of image processing.

From controlling the camera to processing images, NI-

IMAQdx has similar functions to handle the

requirements (Fig. 3).

Since some image processing algorithms are pretty

complicated and require excellent computation ability,

using graphical programming language might face the

difficulty on managing the program. Thus, using other

programming language is a preferable way for LabVIEW

to achieve the target. Among all languages, it is common

to use MATLAB programming language as a solution.

MATLAB has three ways to cooperate with LabVIEW:

using MATLAB Script node

directly calling MATLAB ActiveX server

using COM technology.

However, [4] shows that to realize a real-time system,

which requires faster response speed and higher

application efficiency, COM technology is the best

choice.

As for NI-IMAQdx approach and COM

technology, [5] gives the conclusion that COM

(a)

(b)

Fig. 2 Example of LabVIEW. (a) Front panel. (b) Block diagram.

Fig. 5 COM compiler.

technology is more flexible and has better performance,

and it can accomplish image acquisition and camera

control excellently. Furthermore, COM technology

spends less time and memory compared to NI-IMAQdx

method. Therefore, it is a better choice for measurements.

3.2. COM Technology Overview

Component Object Model(COM), which created

by Microsoft Company, is a system connects various

platform, creating binary software components that can

interact. It is also the foundation of the ActiveX function

we call in the LabVIEW. The main idea of COM is that

it gives the rule to regulate how a component forms. The

designers are asked to provide the well-defined interface

to deal with the detail of realization without showing. By

reference counting, the component itself could handle the

memory distribution which is a huge difference in several

languages. In this application binary interface (ABI), the

data and machine code are packaged and operated in a

specific way according to the structure of COM. Since a

component are implemented with several interfaces, a

variety of platform could execute the code and access its

functionality. The dynamic interface negotiation in COM

is addressed by the Query Interface.

3.3. Realization of COM Method

Fig. 4 shows the system flow chart of COM. The

whole process is divided into two parts. The first half

executes in the MATLAB, and the last half is

accomplished in the LabVIEW. The primary target of

MATLAB is to design the image processing algorithm

for the motion tracking. Complicated image processing

could be simplified since MATLAB has a built-in image

processing toolbox. Furthermore, with COM technology,

the MATLAB script files are compiled into the binary

COM components, and the advantage could be preserved

in the LabVIEW.

To compile the MATLAB script file into COM

component, the Library Compiler could be found by

entering “deploytool” in MATLAB command line and

the compiler dialog would appear (Fig. 5). Choose the m-

file which stores the image-processing algorithm for

packaging. Note that the m-file for COM component

could only accept the type of function. Furthermore, the

title of the function has to be identical to the name of the

m-file. Also, it is recommended that the code for image

processing should be examined twice to ensure the

correct results since that it is nonadjustable after

implementing into the LabVIEW.

After compiling the m-file into COM component,

we could derive a folder which contains two main files: a

BAT file and a DLL file. The BAT file is for initializing

the installation, and the DLL file is the COM component.

Each file would be used through the process. First of all,

it is necessary to register the component into the

computer before we can load the file into LabVIEW. Find

the “_install.bat” in the folder and double-click it. This

registration could also be accomplished by using the

“regsvr32” command. Open “start/run” interface and

type “Regsvr32 file_path\function_name.dll” command,

and wait for the installation. Note that the file path where

the COM component file locates could not contain

Chinese characters or error would show up afterward.

Furthermore, the file path of COM components is not

allowed to change after the installation, or it will not be

found in the LabVIEW.

In the final part of the COM-component application, we

are going to call the COM component into the LabVIEW.

Fig. 6 gives a brief block diagram of calling COM

component. The process and principle of calling COM

components are similar to VB environment.

Fig. 4 System flow of COM technology.

Fig. 6 Block diagram of calling COM.

Fig. 7 Complete block diagram.

However, some specific difference must be noticed

to avoid compile error. Loading the DLL file into

LabVIEW will not appear on the Front Panel, so the file

must be called in the Block Diagram. Open the Functions

Panel and find the “ActiveX” block in the

“Communication” category. The “Automation Open.vi”

is the function that we can choose the ActiveX class.

Nevertheless, the class would show only if we installed

the DLL file in the previous process, or it will not show

up in the list.

The next step is to operate the function we

designed. Use “Invoke Node.vi” in the ActiveX block to

activate the function. This vi provides both input and

output terminals and variable “nargout,” which indicates

the number of the output parameter. If the MATLAB

function would return output variables, the value of

nargout should not be zero. When calling COM

component, the parameters being input from LabVIEW

would be converted into a MATLAB array form for the

operation of the inner function. The output results of

Fig. 8 System flowchart.

(a) (b)

Fig. 9 Hardware. (a) Brachiation robot. (b) Logitech

C922 Pro Stream Webcam.

MATLAB function would return to LabVIEW in a

particular form, which is called “Variant.” By using

“Variant to Data.vi,” this specific type allows LabVIEW

to transfer the results into assigned data type such as

double, integer and array. Note that the MATLAB

function determines the type of output data. If one selects

the wrong type for the variant to transfer, LabVIEW will

show the error message. Finally, after the whole process

is finished, the sub-vi “Close Reference.vi” is responsible

for the disconnection with the DLL file and cleaning up

the temporary pointer to the object. The complete

program diagram of motion tracking, which contains the

function of acquiring the image from the webcam and

processing the frame through the MATLAB function, is

shown in Fig. 7.

4. SYSTEM DESIGN

4.1. Hardware System

The system is presented in Fig. 8. The user PC runs

LabVIEW to control the brachiation robot’s motion. The

brachiation robot would receive the signals from the PC,

which is addressed by the microprocessor (MyRIO-1990,

National Instruments, Fig. 9 (a)). In the meanwhile,

MyRIO would transfer the signals to the command of

actuators, and the brachiation robot would

Fig. 10 Algorithm of image processing.

start to swing. Since the orientation of the whole robot

could not be measured by the encoder of the actuator.

Extra sensor is needed. However, instead of considering

the IMU which generally used in robots, we propose a

real-time system that use the webcam to grab images of

the robot as the feedback. The webcam we choose is

Logitech C922 Pro Stream Webcam, a high-quality

webcam that satisfies our need (Fig. 9 (b)). By applying

the computer vision, the feedback sensor would not be

influenced by the sensitive fluctuation resulted from the

brachiation robot’s movements, and therefore ensure the

stability and accuracy of the measurements. Also, the

brachiation robot saves at least five I/O pins that an IMU

needs, so these pins are available for other purposes.

4.2. The algorithm of Motion Tracking

To detect the motion, we set signs with bright color,

red, on the brachiation robot’s arm. Furthermore, the

white background is necessary for reducing the

environmental noise. Fig. 10 shows the flowchart of

image processing. We design the algorithm in the

following steps, some steps are processed in LabVIEW,

and others are operated with MATLAB function.

Grabbing the image frame through a Logitech

C922 webcam with LabVIEW IMAQdx.

Split the frame into three independent color

planes.

Transfer three color planes into MATLAB array

formats

To detect target linkage, give three layers

different weights and subtract each other,

deriving a single layer.

Use the median filter to reduce the noise and

enhance the target.

Calculate the global image threshold through

Otsu’s method to binarize grayscale picture.

Execute the morphology process. We operate

opening, which is the dilation of the erosion, in

this section. Use rectangle kernel to erode the

small region and fill the crack within the target

with dilation.

Read the region properties with MATLAB

function “regionprops.” The results show

properties of all regions it finds. The maximal

region is the final target we aim to focus.

Read the orientation of the target region through

regionprops and derive corresponding properties

such as center position for plotting.

If external noises interfere results significantly, a

simple filter could be implemented again to

reduce the effect.

5. EXPERIMENTS AND RESULTS

The experiments would be presented from two

major perspectives. In the first part, we will compare the

measurements resulted from IMU and the webcam. As

for the second part, we will examine whether the whole

process could be facilitated by using COM technology.

5.1. Examination of Accuracy

To ensure the result coming from the algorithm

above is precise and better than that of the IMU,

preceding examination is designed and applied before we

start the official experiment. First, we mount the camera

on a shelf and make sure the direction of view is correctly

corresponding to the red mark on the robot by setting

both the camera and the mark to the same height and

using the spirit level for fine-tuning. Note that even if we

use another camera, as long as the height of the camera

and the robot remain the same, the distance between the

camera and the robot will not affect the orientation since

that this feature is scale-invariant. Next, a series of value

is assigned as the angle that the robot arm tilt from the

initial position i.e. θ1. We capture images in different θ1,

and measure the actual angle with a protractor (shown in

Fig. 11). In the meantime, the vi that contains the

algorithm is also executed and giving the instant

angle measurement. The comparison of two

measurements is shown in Table 1. The error between

measurements is around 0.5 degree, which indicates the

great accuracy the algorithm provides.

5.2. Comparison between IMU and the webcam

After confirming the accuracy of the algorithm, we

begin to examine whether the performance of the

webcam is better than that of the IMU. The discussion

would mainly focus on two situations: normal

brachiation and brachiation with undesired disturbance.

The robot needs to swing in regular order so that

we could record the data and compare. We set a

continuous sinusoidal wave as a command on the actuator.

The frequency of this wave is 0.75 Hertz, which is close

to the natural frequency of this robot. Therefore, when the

robot swings with this command, it can gradually swing

high and remain stable. We operate and record the

experiment for 60 seconds, and capture the data in a

specific time section for comparison.

Fig. 12 shows two charts of different situations.

The chart (a) refers to normal brachiation while the chart

(b) presents the brachiation under disturbance. Each chart

gives the relationship of the desired angle θ1 as well as

time, and consists of three types of data. The green dotted

line represents the raw data receiving from the

IMU while the blue line refers to the IMU data that

filtered by using Complementary Filter. The

complementary filter is the most common filter that

applied to IMU since that a complicated calculation is not

necessary and could easily understand. It combines both

the data from the gyroscope and the accelerometer and

performs high-pass and low-pass filtering on each data

[6]. Last but not least, the red line refers to the results

from the webcam.

While doing the normal brachiation, as chart (a)

shows, the IMU data without filtering is pretty noisy,

experiencing fluctuations through the whole process.

However, the transient response is fast and able to capture

the peak value. The IMU data with filtering has a better

waveform than the previous one because it eliminates

most fluctuations. Nevertheless, there exists one major

problem of this data. The transient response is not fast

enough for this system due to the time spending on the

operation of the Complementary filter. Even though we

use Timed loop structure, which can specify how fast one

loop should iterate, to speed up our program, it is not

sufficient for the requirement. Therefore, in the chart (a),

we could tell that the rise time of the wave is too large,

missing the moment when the robot reaches the peak, and

then the peak value of the IMU data is always lower than

the actual one. We can reduce the error through

adjustment on the

(a) (b) (c)

Fig. 11 Snapshots of three angles.

Angle (deg.)

Method

Small Medium Large

Hand measure 2.0 18.8 36.1

Algorithm

Calculation 2.48 18.15 36.52

Table 1 Angle Comparison.

(a)

(b)

Fig. 12 Performance of two situations.

Fig. 13 Coordinates of the Brachiation robot.

weighting factors in the Complementary filter. However,

it can never reach the actual value. As for the

measurement from the webcam, it presents an excellent

performance under normal brachiation. The waveform is

the smoothest one since that the webcam is mounted on

the shelf instead of the robot, excluding the effect of most

fluctuations. Furthermore, with the help of the computer

computing power, the transient response is sufficiently

fast, and the peak value of the robot is perfectly captured.

When it comes to the disturbance, there are two

types of disturbances in our robot. The first type is

fluctuations of the motion caused by the mechanisms. For

example, while stretching the sliding mechanism, the

robot would experience the drop force. The magnitude of

this force may change with a wide range of command,

giving the system various degree of the influence.

To IMU, this type of disturbances has a dominant

impact on the IMU. Since the IMU measures the angle

with the instant data of acceleration and angular speed,

every time the drop force occurs, the value would

increase( or decrease) dramatically, affecting the

accuracy. On the contrary, this kind of disturbance does

not cause much effect on webcam data since the webcam

is not mounted on the robot. Also, we paint the red mark

on the first linkage connected with the branch, so it

receives less impact when the disturbance occurs at the

lower linkage. From the chart (b), there are two moments

when the disturbance occurs. Both the IMU data are

affected by the disturbance acutely. The filter could not

operate well in this situation since the period is too short,

and the value of the gyroscope, which response speed is

higher than the accelerometer, takes control of the results.

However, the webcam data shows excellent immunity to

the disturbance. The waveform is still as smooth as

normal, and the value during the period of the disturbance

remains unchanged.

Fig. 14 MATLAB script node.

Secondly, for simplicity, we assume that our robot

swings in pure brachiation on a single plane (x-y plane)

that is perpendicular to the view of the webcam (Fig. 13).

This assumption implies the constant distance

between the red mask and the webcam. However, in the

perspective of biology, instead of keeping the body on the

single plane, gibbons swing with a slightly tilted posture.

The reason is that they swing by one hand holding the

branch at a time during the brachiation. The asymmetric

body tends to tilt to keep the center of mass of the body

on the x-y plane. This phenomenon results in periodic

displacement of z-coordinate, a value that is undesirable

to our assumption. It also occurs in our robot, affecting

the performance of the two methods.

The displacement of z-coordinate influences the

accelerometer in IMU. The accelerometer measures the

acceleration of three dimensions. If the brachiation

motion is not purely on the x-y plane, the bias of the

composite acceleration vector would appear, reducing the

accuracy. Also, the gyroscope is sensitive enough to

detect undesired angular velocity coming from the

displacement. Therefore, the results would be

underestimated. As for the webcam, the periodic

displacement enforces the surface of the red mark not to

be parallel with the x-y plane. The actual red mask that

the webcam captures becomes the projection of the real

mask, changing the shape and area. In our algorithm, the

deformation of the red mask may lead to the error on the

calculation. However, due to the hand design, the

displacement of z-coordinate is limited by the mechanism

constraints, so the deformation has little effect on the

accuracy, about 0.5 to 1 degree.

5.3. Performance with and without COM technology

The previous experiments confirm the advantage of

using the webcam. In this section, we examine the power

of COM technology. As mentioned above, there are three

ways for LabVIEW to cooperate with MATLAB, and the

MATLAB Script Node is the most popular way since it’s

very convenient to use. Just call

the function, type the MATLAB code into the frame, and

set the input and output terminals for other function to

connect (shown in Fig. 14). However, a convenient

method implies higher workload for the system, reducing

the operating efficiency.

To illustrate the efficiency of these two method, we

create two vi with identical structure. One vi uses the

COM technology while the other chooses the MATLAB

Script Node. To acquire the information about the

processing speed, we use a built-in function called

“Calculate Frames per Second.vi,” and run the vi

separately. The testing format is 320*240 pixels /30fps

with MJPG. Fig. 15 shows the comparison of two

methods. The fps of using COM technology remains 30

after few iterations, while the fps of using MATLAB

Script Node drops to half of the acquired fps after the

operation. Therefore, we can conclude that the COM

technology works faster than the MATLAB Script Node.

6. CONCLUSION

In this paper, we propose a method to realize the

real-time synchronous image acquisition and image

processing. By using MATLAB, we could take

advantage of powerful image-processing functions

instead of using the built-in LabVIEW vi. Furthermore,

to enhance the processing speed for real-time operation,

we use the COM technology as the connection between

LabVIEW and MATLAB. The experiments confirm the

better performance while using this method compared to

others such as MATLAB Script Node. The next stage is

to detect the whole body with the webcam and

reconstruct the model using this designed system.

Furthermore, with COM technology, we would refine the

algorithm in MATLAB to enhance the ability to resist

disturbances and reduce the false rate.

REFERENCES

[1] Pengfei Gui, Liqiong Tang, Subhas

Mukhopadhyay, “MEMS based IMU for tilting

measurement: Comparison of complementary and

kalman filter based data fusion,” IEEE 10th

Conference on Industrial Electronics and

Applications (ICIEA), 2015.

[2] Viktor P. Semenov, Vladimir V. Chernokulsky,

Natalya V. Razmochaeva, “Investigation of the

Kalman filter in problems of increasing the

accuracy of measurements using MEMS (9 axis)

motion sensors,” IEEE Conference of Russian

Young Researchers in Electrical and Electronic

Engineering (EIConRus), 2018.

[3] Chance Elliott, Vipin Vijayakumar, Wesley Zink,

Richard Hansen, “National Instruments LabVIEW:

A Programming Environment for Laboratory

Automation and Measurement,” 2007.

[4] Yupeng Xu, Feihong Yu, “Reasearch on the image

acquisition and camera control of machine vision

camera based on labview,” Fifth International

Conference on Intelligent Human-Machine Systems

and Cybernetics, 2013.

[5] Wuni Xu, Lanxiang Zhong, Dingyuan Wang,

“Image Processing Based on Seamless Integration

Technology Between LabVIEW and MATLAB,”

Intern ational Conference on Information,

Networking and Automation (ICINA) , 2010.

[6] S. Colton, “The Balance Filter,” 2007.

Fig. 15 Comparison of frames per second.