44
Eureka Presentation ROSSANO SCHIFANELLA, Paloma de Juan, Joel Tetreault, Liangliang Cao @ACM Multimedia 2016, Amsterdam DETECTING SARCASM IN MULTIMODAL SOCIAL PLATFORMS

Detecting Sarcasm in Multimodal Social Platforms

Embed Size (px)

Citation preview

Page 1: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

ROSSANO SCHIFANELLA, Paloma de Juan, Joel Tetreault, Liangliang Cao

@ACM Multimedia 2016, Amsterdam

DETECTING SARCASM IN MULTIMODAL SOCIAL PLATFORMS

Page 2: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

SARCASM

Page 3: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

Because who doesn’t love finishing the slides late at night the day before the talk… #acmmm2016 #hangover

Page 4: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

WHAT IS SARCASM?

LITERAL INTENDED≠

Page 5: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

Great day today

Page 6: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

LEXICAL and LINGUISTIC MARKERS

Page 7: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

INTERJECTIONS, INTENSIFIERS, HYPERBOLES

Well, really great day today

Page 8: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

Great day today !?!?!?!?

PUNCTUATION

Page 9: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

CONTEXT

Page 10: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

Great day today! #epicfail

HASHTAGS

Page 11: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

Great day today! #winning

HASHTAGS

Page 12: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

Great day today! 😭 ⛈ ☔

EMOJIS

Page 13: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

Great day today! 😍 🏖☀

EMOJIS

Page 14: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

Great day today

Third car accident in a mile!

1

2

PREVIOUS POSTS

@RSCHIFAN

@RSCHIFAN

Page 15: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

AUTHOR PROFILE, PROPENSITY TO SARCASTIC UTTERANCES

Great day today

Well this is not stressful at all #sarcasm

1

3

@RSCHIFAN

@RSCHIFAN

2 I looooove Trump’s hair! #sarcasm@RSCHIFAN

Page 16: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

SOCIAL MEDIA IS MULTIMODAL

Page 17: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

METADATA VISUALS

TEXT

Page 18: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

Great day today

Page 19: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

Great day today

Page 20: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

Text+Image

Image as a contextual clue

Page 21: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

POSTS CONTAINING #SARCASM OR #SARCASTIC

DATA

517K 63K 20K99% 40% 7.56%

TEXT+IMAGE TEXT+IMAGE TEXT+IMAGE

Page 22: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

CHARACTERISE THE ROLE OF IMAGESStudy of the interplay between textual and visual components

1

Page 23: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

-100 posts per platform -Two questions:

A. Is the text enough? B. Does the image help?

MANUAL ANNOTATION IS THE TEXT ENOUGH?

YES NO

DO

ES T

HE IM

AG

E H

ELP

?

YE

SN

O

Page 24: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

-Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.

TAKEAWAYS

-100 posts per platform -Two questions:

A. Is the text enough? B. Does the image help?

MANUAL ANNOTATION IS THE TEXT ENOUGH?

YES NO

DO

ES T

HE IM

AG

E H

ELP

?

YES

NO

TEXT-ONLY

Page 25: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

-Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.

TAKEAWAYS

-100 posts per platform -Two questions:

A. Is the text enough? B. Does the image help?

MANUAL ANNOTATION

It was a beautiful spring day today! I almost went out outside in shorts it was so nice! 😭 😣 😭 #spring #sarnia #winter #allgonein24hours

Page 26: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

-Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.

-Text+Image: multimodality is key

TAKEAWAYS

-100 posts per platform -Two questions:

A. Is the text enough? B. Does the image help?

MANUAL ANNOTATION IS THE TEXT ENOUGH?

YES NO

DO

ES T

HE IM

AG

E H

ELP

?

YES

NO

TEXT+IMAGE

Page 27: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

-Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.

-Text+Image: multimodality is key

TAKEAWAYS

-100 posts per platform -Two questions:

A. Is the text enough? B. Does the image help?

MANUAL ANNOTATION

Seriously cute cat just wandered into my garden, sweet little thing 😍 #cat #photogenic #cute #garden

Page 28: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

-Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.

-Text+Image: multimodality is key

TAKEAWAYS

-100 posts per platform -Two questions:

A. Is the text enough? B. Does the image help?

MANUAL ANNOTATION

So happy I brought the nice weather back with me...

Page 29: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

-Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.

-Text+Image: multimodality is key -Not Sarcastic: #sarcasm is not always

sufficient to mark the content as sarcastic, users have often their own definition of sarcasm that is close to humour, fun, silly content.

TAKEAWAYS

-100 posts per platform -Two questions:

A. Is the text enough? B. Does the image help?

MANUAL ANNOTATION IS THE TEXT ENOUGH?

YES NO

DO

ES T

HE IM

AG

E H

ELP

?

YE

SNO NOT SARCASTIC

Page 30: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

COLLECT A GROUND TRUTH FOR SARCASMA. Evaluate the impact of visuals as a source for context B. Identify sarcastic posts with a high level of agreement

CHARACTERISE THE ROLE OF IMAGESStudy of the interplay between textual and visual components

1

2

Page 31: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

ASK THE CROWD!

1K POSTS

5 JUDGEMENTS

Page 32: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

SECOND EXPERIMENT

For all the posts that are judged not sarcastic in the previous step, show the text and the image

FIRST EXPERIMENT

Show only the textual component of a post

Page 33: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

Text+Image 37,4%

Text Only 37,8%

Not Sarcastic 24,8%

Text+Image 44,5%

Text Only 23,6%

Not Sarcastic 31,9%

\

Page 34: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

COLLECT A GROUND TRUTH FOR SARCASMA. Evaluate the impact of visuals as a source for context B. Identify sarcastic posts with a high level of agreement

DETECT SARCASMSVM Fusion+Deep learning fusion approaches

CHARACTERISE THE ROLE OF IMAGESStudy of the interplay between textual and visual components

1

2

3

Page 35: Detecting Sarcasm in Multimodal Social Platforms

HOW CAN WE DETECT SARCASM IN MULTIMODAL POSTS?

1 SVM

Page 36: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

-LEXICAL -SUBJECTIVITY -1,2-GRAMS -WORD2VEC -COMBINATION

NLP FEATURES VISUAL SEMANTIC FEATURES

-YFCC100M DATASET -1,570 CONCEPTS VIA CONVOLUTIONAL NEURAL

NETWORK -EACH CONCEPT AS A ONE-HOT FEATURE

Page 37: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

-LEXICAL -SUBJECTIVITY -1,2-GRAMS -WORD2VEC -COMBINATION

NLP FEATURES VISUAL SEMANTIC FEATURES

-YFCC100M DATASET -1,570 CONCEPTS VIA CONVOLUTIONAL NEURAL

NETWORK -EACH CONCEPT AS A ONE-HOT FEATURE

+

LINEAR SVM

+FEATURES VECTOR

FUSION

Page 38: Detecting Sarcasm in Multimodal Social Platforms

HOW CAN WE DETECT SARCASM IN MULTIMODAL POSTS?

2 DEEP

LEARNING

Page 39: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

1 K. Chateld, K. Simonyan, A. Vedaldi, and A. Zisserman. Return of the devil in the details: delving deep into convolutional nets. In BMVC, 2014.

Adapted Visual Representation1 (trained on ImageNet)

NLP Multilayer Perceptron (based on unigrams)

CO

NC

ATE

NA

TIO

N

LAY

ER

NO

N-L

INE

AR

LA

YE

RS

SARCASM DETECTION

Page 40: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

EVALUATIONGOLD SET

2K EXAMPLES

Page 41: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

D-50 D-80 D-100

baseline (1,2-grams) 81.7 82.5 80.2

baseline + VSF +6% +6.3% +4.3%

D-50 D-80 D-100

baseline (1,2-grams) 88.8 86.0 84.4

baseline + VSF -0,04% +2.1% +6.2%

SVM

1

Page 42: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

D-50 D-80 D-100

baseline 77 74.6 74.8

baseline + AVR +1% +5.1% +3.7%

D-50 D-80 D-100

baseline 75.8 74.6 75.5

baseline + AVR +2.4% +1.4% -1%

DEEP LEARNING

2

Page 43: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

Future Directions

SVM: IMPROVE THE FUSION METHOD, ADD SEMANTICS

DEEP LEARNING APPROACH: A LOT TO DO!

VISUAL SENTIMENT CONCEPTS

AUTOMATIC GENERATION OF SARCASTIC IMAGE CAPTION

SARCASTIC CONVERSATIONAL BOTS

Page 44: Detecting Sarcasm in Multimodal Social Platforms

Eureka Presentation

Questions?

@rschifan

http://www.di.unito.it/~schifane

[email protected]

THANKS FOR THE VERY INTERESTING TALK!