Visuelle Perzeption für Mensch- Maschine Schnittstellen€¦ · Edgar Seemann, 09.11.2009 14 Computer Vision for Human-Computer Interaction Research Group, Universität Karlsruhe

Interactive Systems Laboratories, Universität Karlsr uhe (TH)Edgar Seemann, 09.11.2009 1

Visuelle Perzeption für Mensch-Maschine Schnittstellen

Vorlesung, WS 2009

Prof. Dr. Rainer StiefelhagenDr. Edgar Seemann

Institut für AnthropomatikUniversität Karlsruhe (TH)

http://[email protected]

[email protected]


Programming

Assignments

WS 2009/10

Dr. Edgar Seemann

[email protected]

Edgar Seemann, 09.11.2009 3

Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(TH

)cv

:hci

Termine (1)

3

TBA21.12.2009

Stereo and Optical Flow18.12.2009

Termine Thema19.10.2009 Introduction, Applications

23.10.2009 Basics: Cameras, Transformations, Color

26.10.2009 Basics: Image Processing

30.10.2009 Basics: Pattern recognition

02.11.2009 Computer Vision: Tasks, Challenges, Learning, Performance measures

06.11.2009 Face Detection I: Color, Edges (Birchfield)

09.11.2009 Project 1: Intro + Programming tips13.11.2009 Face Detection II: ANNs, SVM, Viola & Jones

16.11.2009 Project 1: Questions

20.11.2009 Face Recognition I: Traditional Approaches, Eigenfaces, Fisherfaces, EBGM

23.12.2009 Face Recognition II

27.11.2009 Head Pose Estimation: Model-based, NN, Texture Mapping, Focus of Attention

30.11.2009 People Detection I

03.12.2009 People Detection II

07.12.2009 Project 1: Student Presentations, Project 2: Intro11.12.2009 People Detection III (Part-Based Models)

14.12.2009 Scene Context and Geometry


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(TH

)cv

:hci

Performance Measures


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(TH

)cv

:hci

Performance Measures

� Measuring the performance of object recognition algorithms is not trivial

� There different measures depending on the application

1. For classification (i.e. yes/no decision, if object is present or not)� ROC (Receiver-Operating-Characteristic)

2. For localization (i.e. detecting the object’s position)� RPC (Recall-Precision-Curve)� DET (Detection Error Trade-Off)


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(TH

)cv

:hci

Classifying a hypothesis

� When comparing recognition hypotheses with ground-truth annotations have to consider four cases:

� Example:

Prediction: Yes No Yes No

Case: TP FN FP TN


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(TH

)cv

:hci

ROC

� Used for the task of classification� Measures the trade-off between true positive rate

and false positive rate:

� Example:� Algorithm X detects 80% of all cups (true positive

rate), while making 25% error on images not containing cups


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(TH

)cv

:hci

ROC

� Each prediction hypothesis has generally an associated probability value or score

� The performance values can therefore plotted into a graph for each possible score as a threshold

1

1

15% TPR, 1% FPR

35% TPR, 3% FPR

60% TPR, 11% FPR

72% TPR, 30% FPR

85% TPR, 68% FPR

100% TPR, 100% FPR

True positive rate

false positive rate

ROC

heaven

ROC

hell


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(TH

)cv

:hci

Recall-Precision

� For localization ROC is not appropriate, since the number of hypothesis varies

� We therefore define:� Recall: percentage of objects found

� Precision: percentage of correct hypotheses

classification localization


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(TH

)cv

:hci

Plotting Recall-Precision

� RPC are typically plotted on a recall vs. 1-precision scale:

1

1

15% recall, 99% precision

35% TPR, 97% precision



72% TPR, 70% precision

1-precision

recall


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(TH

)cv

:hci

Detection Error Trade-Off

� DET measures the number of false detections per tested image window with respect to the miss-rate (1 – recall)

� Used for a sliding window based detector

� Disadvantages:� Chart more difficult

to read (e.g. log-scale)� Depends on the number

of windows tested(i.e. image size, slidingwindow parameters)

� Does not measure the performance of the complete detection system including non-maximum suppression


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(TH

)cv

:hci

Further measures

� In order to express the performance in a single figure, the following measures are common:� Equal Error Rate (EER)

� The point where the errors for true positives and false positives are equal (i.e. points on the diagonal from (0,1) to (1,0))

� Not well-defined for RPC

� Area Under Curve (for ROC)� The area under the curve ;-)

� Mean Average Precision (for RPC)� Average precision values� Sometimes measured only at pre-defined recall values

� False Positives per Image (FPPI)


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(TH

)cv

:hci

Tracking Measures

� For tracking there are performance measures considering object id changes [Bernadin&Stiefelhagen 2008]


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(TH

)cv

:hci

Localization and Ground-Truth

� For localization, the test data is mostly annotated with ground-truth bounding boxes

� It is often not obvious when to count a hypotheses as true or false detection� Misaligned hypotheses

� Double detections

ground-truth

detection


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(TH

)cv

:hci

Comparing hypotheses to Ground-Truth

� Comparison measures1. Relative Distance

2. Cover and Overlap

1. Relative distance


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(TH

)cv

:hci

Comparing hypotheses to Ground-Truth

2. Cover and Overlap

� There is no standard for which values to choose

� Sometimes used as threshold:� Cover> 50%, Overlap>50%, Relative Distance <0.5

� Double detections are counted as false positive


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(TH

)cv

:hci

Assignments


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(TH

)cv

:hci

Organisatorisches

� Gruppe 3� Tingyun, Zhang� Ning, Zhu

� Gruppe 4� Andreas, Wachter� Sina, Martens� Sijie, Shen

� Gruppe 8� Alexander, Herzog� Stefan, Bürger� Jonathan, Wehrle

� Gruppe 9� Marc, Essinger

� Gruppe ?

� Gruppe 5� Christian, Wittner� Matthias, Steiner� Christoph, Weber

� Gruppe 6� Nils, Adermann� Jan, Issac

� Gruppe 7� Alexander, Wirth� Dimitri, Majarle� Paul, Märgner

� Gruppe 1� Patrick, Grube� Benjamin, Hohl� Bastiaan, Hovestreydt

� Gruppe 2� Sebastian, Bodenstedt� Mchael, Heck


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(TH

)cv

:hciQuestions, Answers, Discussions …

� Mailing List� [email protected]


This Lecture

� Overview of programming assignments

� Short Intro into Programming� C++

� Documentation: Thinking in C++ http://www.mindviewinc.com/Books/

� Qt� Documentation: http://doc.trolltech.com

� Includes many tutorials


Qt Documentation

� http://doc.trolltech.com/4.5/index.html


Assignment 1

� Skin-Color Detection� Detect skin color pixels as accurate as possible

� Data set contains images from different lighting conditions

Data

Ground-Truth


The whole thing

� Goal:1. Develop the algorithm

2. Visualize the results

3. Do a thorough quantitative evaluation

4. Present your results in front of the class


It’s a competition

� View it as a competition against the other students

� Don’t just make it work more or less

� I want to see the best possible results

� Apply all tricks you can imagine

� No cheating!!!


Current directory structure

� See README.TXT


Some more details

� Training Set:

� The file trainingset.idl lists these files:� "/home/student/Programming/data/christian1.png";

� "/home/student/Programming/data/cond2-alicia.png";

� "/home/student/Programming/data/robo-edi.png";

� …


Test Set

� Test set is defined in testset.idl� Ground-Truth is defined in testset-groundtruth.idl:

� "/home/student/Programming/data/cond2-john.png": (507, 121, 508, 122):255, (508, 121, 509, 122):255, (509, 121, 510, 122):255, (510, 121, 511, 122):255 …

� "/home/student/Programming/data/cond2-muntsin.png": (306, 202, 307, 203):255, (307, 202, 308, 203):255, (308, 202, 309, 203):255, (309, 202, 310, 203):255 …

� "/home/student/Programming/data/petra1.png": (282, 244, 283, 245):255, (283, 244, 284, 245):255, (284, 244, 285, 245):255, (285, 244, 286, 245):255, (286, …

� "/home/student/Programming/data/rainer1.png": (324, 180, 325, 181):255, (325, 180, 326, 181):255, (326, 180, 327, 181):255, (327, 180, 328, 181):255, (328, …

� "/home/student/Programming/data/robo-klaus.png": (365, 95, 366, 96):255, (366, 95, 367, 96):255, (367, 95, 368, 96):255, (368, 95, 369, 96):255, (369, 95, 370 …

� "/home/student/Programming/data/robo-rainer.png": (163, 65, 164, 66):255, (164, 65, 165, 66):255, (165, 65, 166, 66):255, (166, 65, 167, 66):255, (167, 65, 16 …


Your Task

� Produce an .idl file, which specifies for each pixel in the image, the probability of being skin colored� i.e. specify a 1x1 rectangle for each pixel

� Annotool helps to display results at different confidence levels


Quantitative Evaluation

� For the evaluation, we have two Python scripts� Directory: evaluation

� ./fpr-rec-skin.py testset-groundtruth.idl result.idl

� Computes true positive and false positive rate for all thresholds and writes it to plotdata.txt

� Directory: plotting� ./plotSimple.py ../evaluation/plotdata.txt

� Plots the results from plotdata.txt


Presentation

� Shortly present what exactly you have implemented

� Show the performance plot for different implementations / parameter choices � What worked best?

� What did not work?

� What problems did you encounter?

� What were the lessons learned?

� Each group has approximately 8 minutes


VirtualBox: getting data in/out

� Internet access should work through NAT, if not:� ifconfig –a //shows all network interfaces

� sudo dhclient ethX //request IP address

� alternatively edit /etc/network/interfaces

� Shared Folders� Menu -> Devices -> Shared Folders -> Machine

Folders

� Mount folder:� sudo mount.vboxsf SHARE_NAME /mnt

� alternatively edit /etc/fstab


Assignment 2

� People Classification


Programming Intro


C++

� Hello World

main.cpp:

#include <iostream> // contains cout, cin, cerr …

int main(int argc, char** argv)

{

std::cout << “Hello World\n”;

std::cout << “Number of arguments: “ << argc << std::e ndl;

if (argc>1)

std::cout << “First argument: “ << argv[1] << std::en dl;

return 0;

}


Headers and Source

mainwindow.h:

#include <iostream>

class MainWindow

{

private:

int memberVariable1;

int memberVariable2;

public:

MainWindow();//constructor

~MainWindow();//destructor

void setValue1(int val);

int getValue1();

};

mainwindow.cpp:

#include “mainwindow.h”

MainWindow::MainWindow()

{}

MainWindow::~MainWindow()

{}

void MainWindow::setValue1(int val)

{

memberVariable1 = val;

}

int MainWindow::getValue1()

{

return memberVariable1;

}

� Header: defines class structure / api

� Source: the implementation


Compiling and linking

� The manual way� g++ -c main.cpp mainwindow.cpp

� Compiles main.cpp and mainwindow.cpp into .o files� g++ -o MainProgram *.o

� Links .o files into an executable


Qt: Creating a GUI

mainwindow.h:

#include <iostream>

#include <QWidget>

#include <QLabel>

#include <QVBoxLayout>

class MainWindow : public QWidget

{

Q_OBJECT

private:

QLabel* imageWidget;

public:

MainWindow(QWidget* parent =0);

void open(const char* file);

};

mainwindow.cpp:


MainWindow::MainWindow(QWidget* parent) : QWidget(parent)

{

QVBoxLayout* layout = new QVBoxLayout();

imageWidget = new QLabel();

layout->addWidget(imageWidget);

setLayout(layout);

resize(320, 240);

show();

}

void MainWindow::open(const char* file)

{

QImage image(file);

imageWidget->setPixmap(QPixmap::fromImage(image));

}


Adding window to mainmain.cpp:

#include <iostream>

#include “MainWindow.h”

#include <QApplication>

int main(int argc, char** argv)

{

QApplication app(argc, argv);

MainWindow window;

if (argc>1)

window.open(argv[1]);

return app.exec();

}


Linking libraries

� In order to use Qt, we have to link against the qt libraries

� Manual way:� g++ -c *.cpp –I/path/to/headerfiles

� g++ -o MainProgram *.o –L/path/to/library –lName

� Qmake (the Qt build system):1. qmake –project (create project file: dirname.pro)

2. qmake (create a Makefile)

3. make (execute Makefile)

4. make clean (to delete built files)

� To add a new file/class you have to edit dirname.pro and repeat step 2 and 3


LD_LIBRARY_PATH

� -L parameter tells linker, where to look for libraries� g++ -o MainProgram *.o –L/path/to/library –lName

� Run-Time� Dynamically linked libraries are linked at start up� Dynamic libraries may be moved after linking

i.e. we have to define search path� export

LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/path/to/library

� You may define LD_LIBRARY_PATHin your .bashrc, then you don’t have to set it after each login

� Search path for system libraries is typically already defined in /etc/ld.so.conf


Include Guards

mainwindow.h:

#ifndef MAINWINDOW_H

#define MAINWINDOW_H

#include <iostream>#include <QWidget>#include <QLabel>#include <QVBoxLayout>

class MainWindow : public QWidget{

Q_OBJECT

private:QLabel* imageWidget;

public:MainWindow(QWidget* parent =0);

void open(const char* file);};

#endif

� Avoid to include a header file multiple times


Pointers and References

� Essentially the same thing, but� References cannot be null

� References are syntactically handled as objects

� Example:� QImage& img1 = open1(file);

� QImage* img2 = open2(file);

� img1.getPixel(0,0);

� img2->getPixel(0,0);

� (*img2).getPixel(0,0);

� Please: Avoid using pointers!!!


Memory management

� If we create objects with new, we have to delete them� Otherwise we have a memory leak

� There are nice tools to detect memory leaks e.g. valgrind

� Example� Object* obj = new Object()

� delete obj;

� Object* array = new QObject[20]; //points to the fi rst element

� delete[] array;

� Delete is typically called in the destructor� Exception:

� Qt GUI elements typically use pointers, however youdon’t have to worry about memory management


Tips� Avoid calling “new”

� Object obj(params)

� Creates object in the current scope

� Object is automatically destroyed if obj is out of scope

� Object* obj = new Object(params)

� Creates object on the heap

� Object needs to be explicitly deleted: delete obj;

� Avoid objects as return values, instead pass references� QImage open(const string& file) vs.

� void open(const string& file, QImage& open)

� Removes copying overhead (even though compiler may optimize this)

� This is also the solution to multiple return values


Const correctness

� Consider the following function signatures1. open(string filename)2. open(string& filename)3. open(const string& filename)

1. Bad: filename is passed by value, i.e. involves a copy of the string object

2. Good: filename is passed as a referenceBad: filename may be altered in the function

3. Assures that filename is only read in the functionUse const where ever possible


The QImage class

� QImage provides:� Reading and writing of various image formats

� QImage img(filename)

� Creating an empty image� QImage img(w, h, QImage::Format_ARGB32)

� Access to image data� QRgb pixel = img.getPixel(x,y)

� int width = img.width()

� int height = img.height()

� QImage smallImage = img.scaled(w, h, Qt::AspectRatioMode, Qt::TransformationMode )

� uchar* bits = img.bits()

� QRgb pixel = img.getPixel(x,y)


QRgb

� QRgb represents a RGB value� QRgb pixel = qRgb(100,200,150)

� int red = qRed(pixel)

� int green = qGreen(pixel)

� int blue = qBlue(pixel)

� Grayscale images are stored as RGB, with r=g=b for all pixels� QRgb grayPixel = qRgb(100,100,100)


OpenCV

� Okapi is based on OpenCV 2.0

� OpenCV is an open source computer vision library containing a large number of existing algrithms� Image Processing:

� Edge Detection

� Interest Point Detection

� Morphological Operations

� Machine Learning� SVM

� Boosting

� Optical Flow

� Stereo Computation

� Tracking


Images as matrices

� Load imageMat img = imread(image_fname);

� Load image as grayscale imageMat maskimg = imread(mask_fname, CV_LOAD_IMAGE_GRAYS CALE);

� Access image elementsint pixelValue = img.at<uchar>(y,x);

Vec3b& pixelValue = img.at<Vec3b>(y,x);

int red = pixelValue[2];

int green = pixelValue[1];

int blue = pixelValue[0];


STL

� The C++ Standard Template Library provides many useful functions/classes/containers etc.

� Documentation can be found at http://www.sgi.com/tech/stl

� Examples:� std::vector

� std::sort

� std::search

� Tip: “using namespace std;” avoids the additional typing of std:: (only do this in .cpp files)


Containers

� Vectors� Provide dynamic arrays

� Example:� std::vector<int> numbers; //create vector of intege rs

� numbers.push_back(5); //add 5 to vector

� int val = numbers.back();

� int val = numbers[0]; // array style access

� Iterators are a generalization of pointers� Example:

� std::vector<int>::iterator it;

� for (it=numbers.begin(); it!=numbers.end; ++it)std::cout << *it;


Sorting

� Sorting� std::vector<double> numbers;

…std::sort(numbers.begin(), numbers.end());

� Comparators/Functors� class compMag : public binary_function<double, doubl e,

bool>{

bool operator()(double x, double y){

return fabs(x) < fabs(y);}

};

std::sort(numbers.begin(), numbers.end(), compMag());


Signals and Slots

� GUI events in Qt are handled via so-called signals and slots

� Signals correspond to events

� Slots correspond to event handlers

� signals and slots are connected by the following command:� QObject::connect(button, SIGNAL(clicked()), this,

clickHandler());

� Internally Qt does some magic to make this work (you should not bother)


Examplemainwindow.cpp:


MainWindow::MainWindow(QWidget* parent = 0) : QWidget(parent)

{

QVBoxLayout* layout = new QVBoxLayout();

QPushButton* open = new QPushButton();

layout->addWidget(open);

QOjbect::connect(open, SIGNAL(clicked()), this, ope n());

setLayout(layout);

resize(320, 240);

show();

}

void MainWindow::open()

{

QString filename = QFileDialog::getOpenFileName(this , “Open", QDir::currentPath());

QImage image(filename);

imageWidget->setPixmap(QPixmap::fromImage(image))

}


Scripting

� Typically every computer vision tasks involves a large number of parameters, which have to be tested

� It is often extremely useful to use your programs solely from the command line with a GUI� Allows batch processing

� Allows scripting

� …

� Consequently try to separate GUI and functionality as good as possible

� Automate learning, testing

� Doing things manually does not pay off on the long run


Visualization

� Visualization often helps to understand what your code is doing (and what it is doing incorrectly)

� Possibilities:� Write a GUI

� Render an image and store it to disk

� Write data to a file and use some other tool to visualize them


libAnnotation

� Classes:� AnnoRect : represents a single annotation rectangle� Annotation : represents all annotation rectangles for an image� AnnotationList : represents a set of annotations (i.e. for a

complete data set)

� Example:� AnnotationList list(filename);� Annotation& anno = list[i];� AnnoRect r(x1,y1,w,h,score);� anno.add( r );� list.save(fileoutName);

� There is also a python implementation (evaluation: AnnotationLib.py)


Plotting

� Different kinds of 2D plots are very common in computer vision� ROC, RPC� Histograms� etc.

� Possibilities� Write your own plotting routines� Use Qt-based plotting (e.g. QtiPlot)� GnuPlot� Matlab/Octave� Matplotlib (Python-based)


Matplotlib

� I have written a small script for you, which allows plotting of RPC curves:

./plotSimple.py data.txt

� Data.txt has to have the following format (and has to be sorted by score already):

Column1 Column2 Column3precision recall score


Example

Example:……0.919075 0.878453 245.1910.918605 0.872928 247.2710.923977 0.872928 247.7230.923529 0.867403 248.576

…

…


End of Lecture

Documents

Visuelle Perzeption für Mensch- Maschine Schnittstellen€¦ · Edgar Seemann, 09.11.2009 14 Computer Vision for Human-Computer Interaction Research Group, Universität Karlsruhe