Upload
rossano-schifanella
View
147
Download
2
Embed Size (px)
Citation preview
Eureka Presentation
ROSSANO SCHIFANELLA, Paloma de Juan, Joel Tetreault, Liangliang Cao
@ACM Multimedia 2016, Amsterdam
DETECTING SARCASM IN MULTIMODAL SOCIAL PLATFORMS
Eureka Presentation
SARCASM
Eureka Presentation
Because who doesn’t love finishing the slides late at night the day before the talk… #acmmm2016 #hangover
Eureka Presentation
WHAT IS SARCASM?
LITERAL INTENDED≠
Eureka Presentation
Great day today
Eureka Presentation
LEXICAL and LINGUISTIC MARKERS
Eureka Presentation
INTERJECTIONS, INTENSIFIERS, HYPERBOLES
Well, really great day today
Eureka Presentation
Great day today !?!?!?!?
PUNCTUATION
Eureka Presentation
CONTEXT
Eureka Presentation
Great day today! #epicfail
HASHTAGS
Eureka Presentation
Great day today! #winning
HASHTAGS
Eureka Presentation
Great day today! 😭 ⛈ ☔
EMOJIS
Eureka Presentation
Great day today! 😍 🏖☀
EMOJIS
Eureka Presentation
Great day today
Third car accident in a mile!
1
2
PREVIOUS POSTS
@RSCHIFAN
@RSCHIFAN
Eureka Presentation
AUTHOR PROFILE, PROPENSITY TO SARCASTIC UTTERANCES
Great day today
Well this is not stressful at all #sarcasm
1
3
@RSCHIFAN
@RSCHIFAN
2 I looooove Trump’s hair! #sarcasm@RSCHIFAN
Eureka Presentation
SOCIAL MEDIA IS MULTIMODAL
Eureka Presentation
METADATA VISUALS
TEXT
Eureka Presentation
Great day today
Eureka Presentation
Great day today
Eureka Presentation
Text+Image
Image as a contextual clue
Eureka Presentation
POSTS CONTAINING #SARCASM OR #SARCASTIC
DATA
517K 63K 20K99% 40% 7.56%
TEXT+IMAGE TEXT+IMAGE TEXT+IMAGE
Eureka Presentation
CHARACTERISE THE ROLE OF IMAGESStudy of the interplay between textual and visual components
1
Eureka Presentation
-100 posts per platform -Two questions:
A. Is the text enough? B. Does the image help?
MANUAL ANNOTATION IS THE TEXT ENOUGH?
YES NO
DO
ES T
HE IM
AG
E H
ELP
?
YE
SN
O
Eureka Presentation
-Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.
TAKEAWAYS
-100 posts per platform -Two questions:
A. Is the text enough? B. Does the image help?
MANUAL ANNOTATION IS THE TEXT ENOUGH?
YES NO
DO
ES T
HE IM
AG
E H
ELP
?
YES
NO
TEXT-ONLY
Eureka Presentation
-Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.
TAKEAWAYS
-100 posts per platform -Two questions:
A. Is the text enough? B. Does the image help?
MANUAL ANNOTATION
It was a beautiful spring day today! I almost went out outside in shorts it was so nice! 😭 😣 😭 #spring #sarnia #winter #allgonein24hours
Eureka Presentation
-Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.
-Text+Image: multimodality is key
TAKEAWAYS
-100 posts per platform -Two questions:
A. Is the text enough? B. Does the image help?
MANUAL ANNOTATION IS THE TEXT ENOUGH?
YES NO
DO
ES T
HE IM
AG
E H
ELP
?
YES
NO
TEXT+IMAGE
Eureka Presentation
-Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.
-Text+Image: multimodality is key
TAKEAWAYS
-100 posts per platform -Two questions:
A. Is the text enough? B. Does the image help?
MANUAL ANNOTATION
Seriously cute cat just wandered into my garden, sweet little thing 😍 #cat #photogenic #cute #garden
Eureka Presentation
-Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.
-Text+Image: multimodality is key
TAKEAWAYS
-100 posts per platform -Two questions:
A. Is the text enough? B. Does the image help?
MANUAL ANNOTATION
So happy I brought the nice weather back with me...
Eureka Presentation
-Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.
-Text+Image: multimodality is key -Not Sarcastic: #sarcasm is not always
sufficient to mark the content as sarcastic, users have often their own definition of sarcasm that is close to humour, fun, silly content.
TAKEAWAYS
-100 posts per platform -Two questions:
A. Is the text enough? B. Does the image help?
MANUAL ANNOTATION IS THE TEXT ENOUGH?
YES NO
DO
ES T
HE IM
AG
E H
ELP
?
YE
SNO NOT SARCASTIC
Eureka Presentation
COLLECT A GROUND TRUTH FOR SARCASMA. Evaluate the impact of visuals as a source for context B. Identify sarcastic posts with a high level of agreement
CHARACTERISE THE ROLE OF IMAGESStudy of the interplay between textual and visual components
1
2
Eureka Presentation
ASK THE CROWD!
1K POSTS
5 JUDGEMENTS
Eureka Presentation
SECOND EXPERIMENT
For all the posts that are judged not sarcastic in the previous step, show the text and the image
FIRST EXPERIMENT
Show only the textual component of a post
Eureka Presentation
Text+Image 37,4%
Text Only 37,8%
Not Sarcastic 24,8%
Text+Image 44,5%
Text Only 23,6%
Not Sarcastic 31,9%
\
Eureka Presentation
COLLECT A GROUND TRUTH FOR SARCASMA. Evaluate the impact of visuals as a source for context B. Identify sarcastic posts with a high level of agreement
DETECT SARCASMSVM Fusion+Deep learning fusion approaches
CHARACTERISE THE ROLE OF IMAGESStudy of the interplay between textual and visual components
1
2
3
HOW CAN WE DETECT SARCASM IN MULTIMODAL POSTS?
1 SVM
Eureka Presentation
-LEXICAL -SUBJECTIVITY -1,2-GRAMS -WORD2VEC -COMBINATION
NLP FEATURES VISUAL SEMANTIC FEATURES
-YFCC100M DATASET -1,570 CONCEPTS VIA CONVOLUTIONAL NEURAL
NETWORK -EACH CONCEPT AS A ONE-HOT FEATURE
Eureka Presentation
-LEXICAL -SUBJECTIVITY -1,2-GRAMS -WORD2VEC -COMBINATION
NLP FEATURES VISUAL SEMANTIC FEATURES
-YFCC100M DATASET -1,570 CONCEPTS VIA CONVOLUTIONAL NEURAL
NETWORK -EACH CONCEPT AS A ONE-HOT FEATURE
+
LINEAR SVM
+FEATURES VECTOR
FUSION
HOW CAN WE DETECT SARCASM IN MULTIMODAL POSTS?
2 DEEP
LEARNING
Eureka Presentation
1 K. Chateld, K. Simonyan, A. Vedaldi, and A. Zisserman. Return of the devil in the details: delving deep into convolutional nets. In BMVC, 2014.
Adapted Visual Representation1 (trained on ImageNet)
NLP Multilayer Perceptron (based on unigrams)
CO
NC
ATE
NA
TIO
N
LAY
ER
NO
N-L
INE
AR
LA
YE
RS
SARCASM DETECTION
Eureka Presentation
EVALUATIONGOLD SET
2K EXAMPLES
Eureka Presentation
D-50 D-80 D-100
baseline (1,2-grams) 81.7 82.5 80.2
baseline + VSF +6% +6.3% +4.3%
D-50 D-80 D-100
baseline (1,2-grams) 88.8 86.0 84.4
baseline + VSF -0,04% +2.1% +6.2%
SVM
1
Eureka Presentation
D-50 D-80 D-100
baseline 77 74.6 74.8
baseline + AVR +1% +5.1% +3.7%
D-50 D-80 D-100
baseline 75.8 74.6 75.5
baseline + AVR +2.4% +1.4% -1%
DEEP LEARNING
2
Eureka Presentation
Future Directions
SVM: IMPROVE THE FUSION METHOD, ADD SEMANTICS
DEEP LEARNING APPROACH: A LOT TO DO!
VISUAL SENTIMENT CONCEPTS
AUTOMATIC GENERATION OF SARCASTIC IMAGE CAPTION
SARCASTIC CONVERSATIONAL BOTS
Eureka Presentation
Questions?
@rschifan
http://www.di.unito.it/~schifane
THANKS FOR THE VERY INTERESTING TALK!