A Quadrotor Flight Mode For Tracking

Based On Computer Vision Using ROS

Ricardo da Mota Salvador ∗ Leonardo de Assis Silva ∗∗

Vitor Henrique de Moraes Escalfoni ∗

Raquel Frizera Vassallo ∗

∗ Departamento de Engenharia Eletrica, Universidade Federal doEspırito Santo, ES, (e-mails: ricardomsalvador50@gmail.com,

vitor.hme@gmail.com e raquel@ele.ufes.br)∗∗ Coordenadoria de Eletrotecnica, Instituto Federal do Espırito Santo

- Guarapari, ES, (e-mail: leonardo.assis@ifes.edu.br)

Abstract: Unmanned Aerial Vehicles (UAVs) are particularly interesting for covering largeareas or observing regions from a privileged angle of view. However, many applications stillrequire direct human control. The addition of visual feedback can enable these vehicles toperform tasks more autonomously. In this context, this work proposes an autonomous flightmodule, based on computer vision, with embedded processing, to follow a moving target inexternal environments. Interesting results are obtained through real tests done with a quadcopterthat uses Pixhawk for low-level control and a Raspberry Pi for high-level control.

Keywords: Computer Vision, Visual Feedback, ROS, Unmanned Aerial Vehicle, Drone,Pixhawk.

1. INTRODUCTION

Due to its versatility, Unmanned Aerial Vehicles (UAVs)have been used for several applications in which advan-tages can be obtained when flying over a particular regionto visualize or monitor it. Such applications, generally, de-mand visual information for its realization, thus requiringan embedded camera on the UAV.

Some of the most common applications are: air patrol (e.g.,rescue teams support) (Apvrille et al., 2014); equipmentand structures inspection (e.g., wind generators, trans-mission lines, oil and gas pipelines inspection) (McAreeet al., 2016; Sato and Anezaki, 2017); rural or urban areasmonitoring (Reinecke and Prinsloo, 2017); and supportingground teams or land vehicles on situations in which theyhave movement restrictions due to obstacles, ground orvisual constraints, and risks of getting closer to a place ofinterest (Sa et al., 2014). The works just mentioned usuallyrequire a manual control for handling the UAV. Insertingan autonomous control would probably increase the UAVsfunctionality and optimize the developed solutions.

In many cases, whenever it’s desired to execute an ap-plication autonomously, visual information is commonlyvery useful. In Zakaria et al. (2016), for example, a vi-sual servoing control was implemented using optical flow,which allows tracking moving target objects. However, thecomputational processing required for these applicationsusually demands a ground support base. In a differentapproach, the detection of markers can be used to indicatetasks to be performed (Carvalho et al., 2016) or attitudesto be taken (Nitschke, 2014). These works use the markersto perform homography calculation, which provides spatialinformation to plan the vehicle’s movement. The first one

presents a simulation for landing according to the mark’sdetection, and the second one performs a multi markerscontrol system in an internal environment, with a groundstation for processing data.

The possibility of autonomous control makes the equip-ment more comprehensive and modular, facilitating itsinsertion in larger projects, such as in the context of digitalcities (Barreto and de Souza, 2017) and IoT (Motlaghet al., 2017). The vehicle ceases to require direct or in-direct commands from humans operators and becomes aself-sufficient tool to accomplish a given task. In Butzkeet al. (2016), it is proposed an autonomous control, usingfixed markers and cooperation between aerial and groundrobots. The research showed interesting results, however, itcontemplates only internal environments, fact emphasizedby its authors. This scenario also occurs in Lugo andZell (2014), which developed a similar system, tested ondifferent types of routes, but always in a controlled internalenvironment. This highlights the need for specific controlapproaches for large areas and/or areas under the influenceof adverse environmental conditions.

In this context, this work proposes a flight control modulecapable of following a mobile target, while generates posi-tion and orientation commands for the UAV, referenced toa fixed local frame. This feature allows the equipment tobe used in various environments and applications, such ascooperation with other robots or its integration into largersystems, such as Smart Spaces or even Smart Cities. An-other goal of this work is to make the flight module capableof dealing with imperfections intrinsic to the system, suchas the low GPS accuracy in certain situations, and externalinfluences of the environment, like light variation duringsunny days and/or the incidence of moderate winds on the

DOI: 10.17648/sbai-2019-1114692232

vehicle. Thus, the proposed flight module should allow theplatform to chase moving targets in outdoor environments.To validate our work, experiments with several routes werecarried out and the results are discussed further in thispaper.

Our work is presented in this paper with six sections.Section 2 presents the description of the used quadrotor.The computer vision algorithms and the developed flightmodule are explained in detail in Sections 3 and 4. Theresults are presented and discussed in Section 5, and finallythe conclusion and future work are discussed in Section 6.

2. SYSTEM OVERVIEW

We used as UAV, a quadrotor controlled by a Pixhawkboard (Pixhawk, 2019), which is an open source platform.We also embedded a magnetometer, an accelerometer, abarometer and a GPS module, to enhance the systemcontrol.

Additionally, visual feedback is implemented, using anembedded microcomputer (Raspberry Pi 2 Model B+).In order to perform such procedure, and to communi-cate with the Pixhawk, the system is based on Linux(Ubuntu ARM) and adopts the framework ROS Indigovia MAVROS (MAVLink protocol). Images are acquiredby a RaspiCam v1.3, connected to the Raspberry Pi,and a camera stabilizer (Tarot Gimbal) improves imageacquisition.

Four 880 Kv brush-less DC motors are powered by a 4 cellbattery (5200 mAh), which gives the UAV a 15 minutesflight autonomy. This battery also powers the Pixhawk, theRaspberry and the Gimbal. Figure 1 shows the quadrotor,developed for this project.

Figure 1. Quadrotor built for this project.

3. IMAGE ACQUISITION AND PROCESSING

The images are processed by the Raspberry Pi after theRaspiCam acquire them. The result is an attitude com-mand to the UAV. The Pixhawk is responsible for sendingsuch command and ensuring its execution. Therefore, toget a proper visual control, it is necessary to obtain goodimages for processing. This is often a hard task due to thestrong influence of some elements such as vibration andelectromagnetic interference caused by the motors.

To minimize such influence, a camera stabilizer (Gimbal)is used, attempting to maintain constant two of threeorientation components of the RaspiCam (roll and pitch).However, adding this system changes the relationshipbetween the reference frames of the camera and the UAV.

Therefore it also changes the way the obtained data fromeach image and the sensors are associated. Section 4 fullydiscuss this topic.

The camera (RaspiCam) performs the acquisition of animage with 480p resolution and 10 fps rate. The 44.0 ×44.0 cm2 marker on Figure 2 is adopted as the targetfor the tracking task. The acquisition is done with theROS package raspicam node (fpasteaus GitHub reposi-tory, 2015).

Once the pattern on the marker is detected, its ori-entation and position in the camera frame are esti-mated via homography. This is done by the ROS packagear track alvar (ROS.org., 2017) every time the markeris found in an image. If the detection fails, the systemdoes not inform a new orientation and position data tothe subsystem described in Section 4. Consequently, themicrocomputer continues informing the Pixhawk the lastpoint where the marker was detected as the destination forthe UAV.

Figure 2. Marker adopted for tracking.

The size of the marker and the camera resolution weredefined after analyzing: (i) the processing time for imageswith different resolutions, (ii) the maximum height fromwhere the UAV is able to detect the marker, (iii) the targetspeed and (iv) the camera field of view. Such analysis isfully presented on Section 5.

4. DATA PROCESSING

After detecting the marker, the Raspberry Pi should sendthe Pixhawk a new pose for the UAV, in order to keep itflying above and aligned with the marker.

Because we use a camera stabilizer, the position and ori-entation estimated and returned by ar track alvar shouldbe corrected before it is sent to the Pixhawk.

Figure 3 shows the reason for that: the visual data ex-tracted from the marker P and the UAV D are not in thesame reference frame.

While the orientation and position D of the UAV aredefined at take-off, in the global reference frame W , thevisual data P from the marker is represented in the cameraframe C.

In order to work with those two types of information ona proper manner, both of them need to be compared inthe same reference frame (W , due to convenience). Thiscan be done by transforming the visual data, known inthe camera frame C, to be represented in the global frameW , using a rotation matrix RWC and a translation matrixtWC .

DOI: 10.17648/sbai-2019-1114692233

Figure 3. Scheme illustrating the real scenario and coordi-nate frames.

We may assume that the origin of camera frame C iscoincident to the origin of the UAV frame D, since thedistance between C and D is insignificant when comparedto the one between D and W . Thus, we can consider thattWC is the position of the UAV in frame W .

However, because we use the Gimbal to not lose sight ofthe marker when the UAV moves, the orientation of thecamera can not be directly considered as the same of theUAV.

The camera is always looking down-wise. Therefore, theaxis X and Y , in frame C, are at the same plane XY ofthe coordinate system W , sharing axis Z. The differencebetween these two frames, in terms of rotation, is only atyaw. The UAV yaw angle θD defines such rotation, thatis, RWC is given by the following rotation matrix

RWC =

[cos(θD) − sin(θD) 0sin(θD) cos(θD) 0

0 0 1

]. (1)

Thus, it is possible to find the marker position XW =(xW , yW , zW )T at W from C (XC) using the relation

XW = RWCXC + tWC . (2)

Finally, it is important to remark that the packagear track alvar uses a coordinate system A, similar to aplane coordinate system, which is different from a quadro-tor. To solve that and to represent, in fact, the pose of themarker in C, two consecutive rotation must be performed,180◦ on axis Y and 90◦ on axis Z. Figure 4 shows thismechanism.

Figure 4. Frame changing transformations from coordinatesystem A to coordinate system C.

All these transformations together can be described as

XC = RZ(90◦)RY (180◦)XA =

[0 1 01 0 00 0 −1

]XA, (3)

where XA is the marker position on the ar track alvarframe.

In that way, it is possible to inform the Pixhawk the correctposition QW , to where the UAV should go from the initialposition of the mark, with

QW = XW +

[00zo

], (4)

with zo being the height from where the UAV sees themarker while tracking it.

For the orientation control, an important aspect inherentfrom quadrotors is considered. For the on-board controller,the only rotation that matters is the rotation around axisZ (yaw). As a result, from three obtained orientationcomponents (roll, pitch and yaw) only yaw is directly used.

The great advantage of using the Gimbal, is that, althoughit prevents the camera from being considered as a fixedpart of the UAV, it keeps the position and yaw informationidentical between the camera and the quadrotor.

Therefore, once we have the UAV/Camera yaw (θD at W )and the marker yaw (θP at C), we can find the final UAVorientation θQ using the relation

θQ = θD + θP , (5)

as can be seen in Figure 5.

Figure 5. Scheme illustrating the UAV final position andorientation control.

A final treatment regarding the yaw angle θQ must bedone, due to the way that the controller deals with theincoming angles, since one complete turn consists on a−180◦ to 180◦ range. As angle θQ is a composition ofvariables, the resulting value may extrapolate such limits,generating unpredictable or even dangerous control actionsthat can destabilize or damage the equipment.

To reach the acceptable range, the angles are adjustedusing trigonometric identities. This also facilitates the de-cision on the direction that the UAV turns, i.e., clockwiseor counter clockwise.

DOI: 10.17648/sbai-2019-1114692234

At the end of this process, the Raspberry Pi sends theposition QW and orientation θQ to the Pixhawk as thenew desired pose, so the UAV can perform the tracking,following the target.

5. EXPERIMENTS AND RESULTS

In order to validate our system, we performed a set ofoutdoor flights, exposing the UAV to wind and light vari-ations. For that, we attached the marker (Figure 2) to thetop of a terrestrial robot (MobileRobotsTM Pioneer 3-AT)and set the robot to perform three different trajectories:(i) a circular shaped 1 , with 3.80 meters of radius andlinear velocity of 600 mm/s; (ii) a zigzag shaped, withan angular velocity of 30 degree/s and linear velocityof 700 mm/s; (iii) and an eight shaped, with maximumvelocity of 600 mm/s. Linear and angular velocities weredetermined according to the Pioneer’s limitations.

Tracking a moving target depends on variables such asthe time needed to process the images, the UAV relativeheight, the speed of the target in the image and the fieldof view of the camera. To follow a fast moving target itis important to have a processing time as low as possible.But besides that, the camera field of view also influencesthe maximum speed that the target can take - the higherthe UAV flies, the less chance of missing the moving target.

In this scenario, two crucial factors should be highlighted,the size of the marker and the resolution of the acquiredimages. For example, a small marker will present smalldetails, which are challenging to detect when the UAVis flying higher. On the other hand, bigger markers canbe inconvenient depending on the situation, e.g., movingin narrow passages. Hence, we decided to work with amarker that covers exactly the top of the robot, with44.0 × 44.0 cm2.

To find the most proper image resolution, we placed theUAV above the marker on different heights and tested tworesolution values. Table 1 shows the detection rate of themarker in the images (assuming that the marker is alwayspresent on the scene) and the average processing time forboth resolutions.

Table 1. Detection.

480p (640× 480)

10m 15m 16m 20m 24m

Detection (%) 100 91.4 52.5 0.0 0.0Processing Time (ms) 104.4

720p (1024× 720)

10m 15m 16m 20m 24m

Detection (%) 100 100 100 100 82.7Processing Time (ms) 382.2

Based on the results shown in Table 1, we have adopteda resolution of 480p, since the relative height of theexperiments described in this paper is 10m. This valueof resolution allowed better performance regarding the

1 This experiment was recorded and can be viewed on:https://youtu.be/hB3IHAB07I0

processing time, maintaining the same detection rate whencompared to the higher resolution.

Once the marker parameters are defined, we can establishthe experiment protocol. First of all, on manual mode, wetake off the UAV and flight it until the target is detected.Afterwards, the visual feedback module is activated andit takes over the flight control. Our goal is to keep thehorizontal distance between the UAV and the marker closeto zero, and the height close to 10m (zo), which is set bythe user.

Figure 6 shows the terrestrial robot odometry, the UAVposition and the expected trajectory for the circularshaped route. The position data are colored according tothe control mode of the flight: manual mode, in red; visualmode (controlled by visual feedback), in blue. It can benotice that the UAV followed properly the marker, even inthe presence of wind and light variation.

0

5

10

−6

−4

−2

0

2

4

0

2

4

6

8

10

Y [m]X [m]

Z [m

]

MANUAL Control

VISUAL Control

Desired Trajectory

Pioneer Trajectory

Figure 6. Terrestrial robot odometry, UAV position andthe expected trajectory for the circular shaped route.

We can also note that the trajectory presents imperfections(noise), when compared to the expected one. Table 2 showsthe horizontal and vertical errors quantitatively.

Table 2. Trajectory error from GPS data, forthe circular shaped route experiment.

ErrorMean

Error(cm)Standard

Deviation(cm)Max

Error(cm)

Horizontal 1.03 14.48 34.49Vertical 42.44 38.37 117.11

However, it is worth mentioning that the UAV movementwas reconstructed based on GPS data, which has intrinsicmeasuring errors, since there is an incertitude associated tothe sensor - 2m, horizontally (Modules., 2011). Such phe-nomenon influences the results, which does not correspondto the real performance of the system.

DOI: 10.17648/sbai-2019-1114692235

To confirm that hypothesis, we recorded the experimentwith a camera on the ground, at the center of the terrestrialrobot trajectory. Figure 7 shows the expected route, ingreen; the UAV route, given by GPS data, in red; and theroute calculated from the visual data extracted from thecamera, in blue.

0 2 4 6

X [m]

-5

-4

-3

-2

-1

0

1

2

Y [m

]

Trajectory from camera

Trajectory from GPS

Pioneer trajectory

Figure 7. Expected route, in green; UAV route, givenby GPS data, in red; and route estimated from thecamera images, in blue.

As we can see on Figure 7, the real trajectory of theUAV presents less noise when compared to the one givenby the GPS data. Even though there is a GPS error,the UAV follows the marker successfully. We may explainthat by the following: when the GPS misses the UAVposition at W (global coordinate frame), it equally missesthe marker position. That is, the control is performedrelatively between the UAV and the marker, reducing theinterference caused by inaccurate sensor measurements.

Furthermore, we have analyzed the visual feedback con-trolling system, in which it is possible to obtain thedifference between the marker and the camera positionfrom ar track alvar (tracking error). From this data wehave calculated the euclidean distance between the currentposition and the one expected, the same coordinated Xand Y , with 10m high (axis Z). The euclidean error iscalculated point-wise and shown in Table 3.

Table 3. Euclidean distances between UAV realand expected position.

Mean Distance (cm)

Trajectory X Y Z 3D

ZigZag 43.12 1.18 6.71 65.22Circle 61.43 10.70 0.68 66.70Eight 55.74 3.13 3.19 61.33

Table 3 shows that, as well as the trajectory error, thetracking error is also small. The distance from the UAV tothe marker was, on average, 0.61m in X, 0.11m in Y and10.007m in Z, which is near the expected. It can also benoticed that axis X presents the largest distance, due tothe fact that the largest velocity component of the movingtarget, from the camera perspective, is on axis X in all ofthe experiments. That happens because the UAV tends toalign itself to the orientation of the marker, adjusting the

axis X of the camera and the robot moving direction. Inthe zigzag shaped experiment, the tracking errors are dueto translational movement errors, because when the robotturns, there is only rotational movement, which maintainsthe marker in the image’s center, reducing the trackingto only the minimization of the difference between thevehicles angles.

Besides, we can imply that these obtained errors arisefrom the embedded vision system limitations, causing anestimated delay of 104.4ms between the sample acquisitionand the command given by the Raspberry (Table 1).

The behavior of the yaw angle given by UAV data in zigzagand circle shaped experiments is shown in Figure 8. Suchdata are colored according to the control mode: manual,in green, and visual feedback, in lilac.

Figure 8. Behavior of the yaw angle given by UAV data incircle and zigzag shaped experiments.

00:00 00:15 00:30 00:45 01:00 01:15 01:30 01:45−200

−150

−100

−50

0

50

100

150

200

Flight Time [mm:ss]

Yaw

[degre

es]

YawMANUAL ControlVISUAL Control

(a) Circular shaped trajectory

00:00 00:15 00:30 00:45 01:00 01:15 01:30

−140

−120

−100

−80

−60

−40

−20

0

20

40

Flight Time [mm:ss]

Yaw

[degre

es]

YawMANUAL ControlVISUAL Control

(b) Zigzag shaped trajectory

We can notice, from Figure 9(a), that the values of theyaw angle regarding the control mode, based on visualfeedback, presents a linear increase, except on 0:17 minand 1:00 min. At such instants there is a step from +180◦

to −180◦, due to the way the Pixhawk deals with theincoming angles, as described on Section 4. This linearbehavior is a reflection of the circular movement performedby the robot, that is, the UAV does not only follow theposition of the marker, but also its orientation.

Finally, on Figure 9(b), it is possible to note a similarphenomenon for the zigzag shaped experiment. The par-ticularity of this trajectory relies on the fact that the UAVexecutes rectilinear and rotation movements separately.We can see that at instants 0:36 min and 0:51 min theUAV turns positively (to the right) and negatively (to theleft), respectively. Between such instants, the yaw angleis almost unaltered, in other words, the UAV follows a

DOI: 10.17648/sbai-2019-1114692236

straight line. The small variations on this rectilinear sec-tions (e.g., from 0:40 min to 0:51 min) are justified byexternal factors.

6. CONCLUSION AND FUTURE WORKS

In this article, an autonomous flight module for outdoortarget tracking was proposed. The goal is to keep anUAV flying above a moving target marker regardless ofthe route it makes, through a controller based on visualfeedback. The develop system should control the positionand orientation of the UAV relative to the tracked marker.

The proposed module was tested on external flights andpresented satisfactory results, even in situations of windand light variation. The visual control stands out for reach-ing an average horizontal error of 53.66cm and verticalerror of 3.53cm (trajectory error), for all experiments, inaddition to maintaining an approximate height of 10mfrom the target marker. Such a result can be consideredpromising and suggests that the proposed system providesthe vehicle with the ability to track a moving target.

As this project works with a local reference for positioncontrol and final orientation, it allows the UAV to beeasily included in larger projects, on which several robotsshare the same local reference, assisting, for example,applications of cooperation between robots or assumingthe UAV as a component of a digital city, capable ofmeeting generic demands that share the same service needsprovided by this type of agent.

In addition, mobile target tracking using computationalvision demands a lot of processing power, especially whenit comes to agile targets or that have a trajectory withabrupt variations of direction and speed. In this context,it is possible to identify a future need for the project,since the current processing power presents itself as aperformance constraint.

One possible solution would be to transfer the embeddedprocessing to a ground base, to which only the compressedimage is sent and, in response, the UAV would obtain thedesired position and orientation. With this optimization,the system should maintain its ability to cover large areas.However, it would need a wide-ranging data exchangetechnology such as Wi-Fi, WiMax or even GSM.

Another intended future work is the implementation ofa search routine for the marker, both for mission startsituations and for cases where the quadrotor loses sight ofthe pattern. A feasible approach for this search mode is toascend the vehicle, in a simple or helical route, in order togradually increase the field of view of the camera and alsoincrease the chances of locating the marker.

ACKNOWLEDGMENT

The authors would like to thank Fundacao de Amparoa Pesquisa e Inovacao do Espırito Santo (FAPES) for thefinancial support through Projeto Universal No. 504/2015.

REFERENCES

Apvrille, L., Tanzi, T., and Dugelay, J.L. (2014). Au-tonomous drones for assisting rescue services within

the context of natural disasters. In General Assemblyand Scientific Symposium (URSI GASS), 2014 XXXIthURSI, 1–4. IEEE.

Barreto, O.F.G. and de Souza, M.R. (2017). As cidadesdigitais e o teletrabalho. In Anais do CongressoBrasileiro de Processo Coletivo e Cidadania, 4.

Butzke, J., Gochev, K., Holden, B., Jung, E.J., andLikhachev, M. (2016). Planning for a ground-air roboticsystem with collaborative localization. In Robotics andAutomation (ICRA), 2016 IEEE International Confer-ence on, 284–291. IEEE.

Carvalho, J.P., Juca, M., Menezes, A., Olivi, L., Bessa,A., and Marcato, A. (2016). Landing a uav in a dy-namical target using fuzzy control and computer vision.In Anais do XXI Congresso Brasileiro de Automatica(CBA’2016).

fpasteaus GitHub repository (2015). raspicam node.https://github.com/fpasteau/raspicam_node.

Lugo, J.J. and Zell, A. (2014). Framework for autonomouson-board navigation with the ar. drone. Journal ofIntelligent & Robotic Systems, 73(1-4), 401–412.

McAree, O., Aitken, J.M., and Veres, S.M. (2016). A modelbased design framework for safety verification of a semi-autonomous inspection drone. In Control (CONTROL),2016 UKACC 11th International Conference on, 1–6.IEEE.

Modules., U.G. (2011). Neo-6. https://www.u-blox.com/sites/default/files/products/documents/NEO-6_DataSheet_(GPS.G6-HW-09005).pdf.

Motlagh, N.H., Bagaa, M., and Taleb, T. (2017). Uav-based iot platform: A crowd surveillance use case. IEEECommunications Magazine, 55(2), 128–134.

Nitschke, C. (2014). Marker-based tracking with un-manned aerial vehicles. In Robotics and Biomimet-ics (ROBIO), 2014 IEEE International Conference on,1331–1338. IEEE.

Pixhawk (2019). Px4 autopilot. https://pixhawk.org/.Reinecke, M. and Prinsloo, T. (2017). The influence of

drone monitoring on crop health and harvest size. In2017 1st International Conference on Next GenerationComputing Applications (NextComp), 5–10. IEEE.

ROS.org. (2017). ar track alvar. http://wiki.ros.org/ar_track_alvar.

Sa, F.B., Cypriano, M.F., Pereira, F.G., Vassalo, R.F., andFrizera Neto, A. (2014). Planejamento de Trajetoriapara um Robo Movel usando Imagens capturadas porum VANT. In Anais do XX Congresso Brasileiro deAutomatica (CBA’2014).

Sato, S. and Anezaki, T. (2017). Autonomous flightdrone for infrastructure (transmission line) inspection(2). In 2017 International Conference on IntelligentInformatics and Biomedical Sciences (ICIIBMS), 294–296. IEEE.

Zakaria, O., Alaa, B., Fouad, M., and Hicham, M. (2016).Deterministic method of visual servoing: robust objecttracking by drone. In Computer Graphics, Imaging andVisualization (CGiV), 2016 13th International Confer-ence on, 414–422. IEEE.

DOI: 10.17648/sbai-2019-1114692237

Powered by TCPDF (www.tcpdf.org)