AMI community of interest meeting

David van Leeuwen, Stephan Raaijmakers, Wessel Kraaij

AMI community of interest meeting

Automatic segmentation of meeting recordings

Automatic segmentation of meeting recordings2

TNO is active in five core areas

Facts & Figures:- Annual turnover: 553 Mio euro- Employees: 5100


A unique Dutch ICT innovation centre

About TNO ICT• Established: 1 January 2003 • Bundling of former KPN Research with TNO’s ICT

related departments• One of the largest ICT knowledge centres in Europe

Features and unique selling points• Independent • Frontrunner• Multidisciplinary:

• Conceptual and hands-on• Technical, economical and sociological• In-depth Telecom and IT expertise

Key figures• Annual turnover: EUR 40 Mio• 375 professionals• 10 high-quality patents per year• Locations in Delft, Groningen and Enschede


Indexing meeting recordings

• Characteristics of meetings:• Lack of structure• Low information density• Rich in non-verbal cues

• Challenge:• identify segments• annotate segments

• TNO focus:• robust features• multi-level segmentation• low tech requirements


Application scenario

• Enabling step: building a browsable meeting recording archive:• multimodal analysis:• video analysis (e.g. motion zones), • speech analysis (e.g. diarization, laughter detection) • transcript analysis (e.g. topic segmentation, summarization,

sentiment analysis).

• Usage Scenario: searching/filtering interesting segments, • As soon as meeting segments have been detected and annotated,

they can be exploited for any a search or summary generation application.

• E.g. all positive comments on the company’s new flagship product expressed by marketing consultant “Joe. ”.


TNO proposition

• TNO seeks participation of COI members for a mini-project for the application of multi-level segmentation and annotation of meeting (or lecture) recordings.

• A feasibility study of the application of TNO technology for CoI member product line.

• Technologies ready for evaluation in a mini-project:• Speaker diarization: who spoke when (practical when

speakerphones or central microphones are used)• Topic segmentation: would like to perform a test on an

archive with real data.• Sentiment classifcation: state of the art labeling performance,

would like to perform a test on meeting data• Motion zone classification: finding hot spots


Speaker Diarization

• Answers the question Who spoke When?

• No prior information from participants required• no training necessary• but absolute identity therefore not resolved

• we can use speaker recognition technology for this

• Useful for finding out• who talks most• who interacts with whom• who says important things (using transcript)• …


TNO solution for Speaker Diarization

• Technology• requires unobtrusive distant microphones• uses acoustic properties of the voice• uses direction of signal

• if multiple microphones are available

• Performance• is evaluated in NIST Rich Transcription benchmark

evaluations• is among the best performing teams• good co-operation with these teams


Hot spot segmentation

• Motion: pixels different from background estimation• Motion is measured in zones• Per person 3 zones:

• head• hands• close-up camera

• Motion gives indication about speaker activity

• Gesture activity• Head movement• Cue for ‘hot spots’


Feature browser: motion zones

Hot spot?


Subjectivity in meeting transcripts

• Focal point of TNO ICT: sentiment analysis in texts• Determine if texts (like movie reviews) are positive, negative

or neutral (global sentiment classification; see paper)• Determine local sentiment (phrase level)• Find subjective and objective statements

• For AMI: apply subjectivity detection to speech transcripts• Align subjectivity with hot spot information and speaker

segmentation

• Integrated browser for multimodal sentiment cues:• Motion (gestures)• Speaker information• Subjectivity information


Topic segmentation

• Automated division of texts (like meeting transcripts) into separate topics• Main topics• Fine-grained subtopic structure• A Machine Learning problem: learn on basis of segmented texts

• AMI data is hard• Low interannotator agreement• Highly technical and overlapping vocabulary

• TNO: two approaches• An approach based on Conditional Random Fields, using sequential

(contextual) information, optimized for standard error metrics• An SVM-based approach, optimized for a new and better error

metric• Both approaches significantly outperform the baseline LCSEG

algorithm, a well-known and quite good algorithm


Meeting transcript (ground truth)

• ==========:1• so um the thing we have to know is you already know what we're going to do , you also read

what this the things or , not yet , okay . so um , yeah , it has to be original , trendy , user-friendly that's what we're going to design . uh first we have uh uh three steps of uh making the the remote control . fir the first thing is th the functional design , that's very important . we have to look what the needs are , the effects of the functional design , and and how the mm the the remote control works , so that's where we're going to look in the functional design , it's for the f next meeting .

• ==========:2• yes . • the the second thing is the conceptual design , that's what it that's uh the spe the

specifications of the components and the properties and the specifications of the user interface . and we have to look what uh the market is doing for what kind of uh remote controls are in the market . and the third thing is uh the detailed design um and that's exa yeah , you know what it is , it's exactly how it looks and whatever . okay so uh no , this is a these are two smartboards , with the uh f uh s an introduction of that one .

• and you already saw you know all that that you here can put uh things in the the red project

uh map . folder , okay .

Documents

AMI community of interest meeting