Predicting the “Stars of Tomorrow” on Social Media

Predicting the Stars of Tomorrow on Social Media

Wen-Huang Cheng ()

Multimedia Computing Lab (MCLab)Research Center for Information Technology Innovation (CITI),

Academia Sinica, Taipei, Taiwanwhcheng@citi.sinica.edu.tw

Presented at on 10 May 2017

2

Academia Sinica ()

The highest national research institute in Taiwan with about 1,000 professors (60 in EE/CS)

3

located in Nangang, Taipei

Multimedia Computing Lab (MCLab)

4

http://mclab.citi.sinica.edu.tw

We are social

5

Real World Digital World

Nanit Baby Monitor

9

Social Signals

Leading Social Networks

13[Ref] http://www.smartinsights.com/social-media-marketing/social-media-strategy/new-global-social-media-research/

14

Sociology and Human Interaction With the huge number of people who are involved nowadays with

social networks, it is very interesting to note how they are influenced by each other in many different ways. e.g., identity in the age of social media

15[Ref] http://edition.cnn.com/2015/10/05/health/being-13-teens-social-media-study/index.html

100 years after

100 years ago

Social Popularity Prediction General Popularity Prediction: Predicting the popularity

score of a new social media post by combining post content (photo, text or video) and user cues

17

Score: 4.9

?

Model

A new post

Predicted Popularity

Training Images

5.6 2.3

5.1 2.8

7.8 3.1

History data

Why is it important? wide applications and high business value

e.g., predicting the Stars of Tomorrow (top popular models) within the fashion Industry using social media

18[Ref] Style in the Age of Instagram: Predicting Success within the Fashion Industry using Social Media, CSCW 2016.

Fashion Model Directory (FMD) profile page

Can you tell who will be the top?

People are desired for knowing the future

19

[Ref] https://www.oreilly.com/ideas/inside-the-washington-posts-popularity-prediction-experiment

20

https://tianchi.shuju.aliyun.com/competition/index.htm

21

22

7

23

Our Related Publications Sequential Prediction of Social Media Popularity with

Deep Temporal Context Networks, IJCAI 2017. Time Matters: Multi-scale Temporalization of Social

Media Popularity, ACM Multimedia 2016 (full paper). Unfolding Temporal Dynamics: Predicting Social Media

Popularity Using Multi-scale Temporal Decomposition, AAAI 2016.

SocialCRC: Enabling Socially-Consensual Rendezvous Coordination by Mobile Phones, Pervasive and Mobile Computing, 2016.

24

What Makes A Post Popular?

25

[Ref]What Makes an Image Popular? WWW, 2014.

What Makes A Post Popular?

Features for prediction Post content

e.g., visual sentiment features (color and texture)

26

[Ref]Analyzing and predicting sentiment of images on the social web, ACM Multimedia 2010.

What Makes A Post Popular?

Features for prediction User cues

e.g., followers (a users follower count), friends (how many users a user follows), statuses (a users current total post count), user time (a users account creation time), etc.

27

A friend graph:

6

1

What Makes A Post Popular? Features for prediction

User cues (topological features) e.g., closeness centrality, the average length of the shortest

path between the node and all other nodes in the graph

28

12

3

45

6 7

Closeness Centrality

29

12

3

45

6 7

12

3

45

6 7

0.5

0.67

0.75

0.460.75

0.46 0.46

Latent Factor Models

The popularity prediction task is formulated as a matrix completion problem of filling in the missing entries of a partially observed matrix.

30

known popularity

to be estimated

Our Observations: Time Matters

31[Ref] http://www.adweek.com/socialtimes/best-time-to-post-social-media/504222

Temporal Modeling for Popularity

To incorporate the temporal evolving structures in popularity prediction

32

The popularity evolving at multi-granularities with different patterns

33

Challenge 1: Temporal Evolving

Multi-granularities Characteristics of Popularity Dynamics

Challenge 2: Data Noise

Popularity patterns are covered in very noisy behavior data or information

Popularity distribution on time series

Our Solution#1 [AAAI16]:Incorporating Multi-Scale Temporal Decomposition

35

popularity matrix time scales

Solver: Multiple Update Rule (D.Lee and Sebastian.Seung 1999)

Datasets Data Sets

Over 1.8M photos Over 70K users Views, User profile, Photo stream Metadata, Images, Annotations

Settings

36

User-specific Dataset (UsD)Users 400Images 600K

Photo-mix Dataset (PmD)Users 70K Images 1200K

Experiments

Metric: Spearman Correlation Time scales:

period, week, month, season period: morning (8:00am-12:00am), lunch time (12:00am-14:00pm),

afternoon (14:00am-17:00pm), dinner time (17:00am- 20:00pm), evening (20:00am-24:00pm) and sleeping (0:00am-8:00am)

37

Our Solution#2 [MM16]:A Multi-scale Temporalization (MT) Framework

38

Algorithm: Multi-scale Temporalization (MT)

39

Optimization Updating Steps

Optimization Updating Steps

Experiments

40

Our Solution#3 [IJCAI17]:Deep Temporal Context Networks (DTCN) We address the problem as a sequential prediction task, where the input is

a user-photo sequence (with time order) while the output is the popularity of a future photo (a photo before its publication on social media)

41

Experiments

Prediction performances on TPIC17-100K, 200K, and 400K datasets Metric: Spearman Ranking Correlation

42

More Influential Factors: Cultures A voting survey of the 2014 TripAdvisor's Top 10 Attractions in Japan by visitors from

different countries shows how much the favorites for attractions can vary among people from different regions, i.e., different cultures.

43[Ref] 2014 TripAdvisors Top 20 Attractions in Japan: http://www.tripadvisor.com/pages/- HotSpotsJapan.html.

Foursquare Datasethttps://sites.google.com/site/yangdingqi/home/foursquare-dataset

Individual check-ins data of the more than 10 million users on Foursquare

44

A Pilot Study: Understanding Foursquare Venue Popularity in Taiwan

45

Performed by Mr. Mrinal Kanti Baowaly in 2016

Taiwan vs. USA Venue Distribution of Top 10 Categories

46

Taiwan

USA

More Influential Factors: Personalization What Your Facial Features Say About Your Personality (MM13)

47

personality report

facial image

Learning Relevance by Neighbor Voting

48

[Ref] X. Li, C.G.M. Snoek, M. Worring, Learning tag relevance by neighbor voting for social image retrieval, Proc. ACM Intl. Conf. Multimedia Information Retrieval (MIR), 2008.

More Influential Factors: Personal Fashion Flavor

49

[Ref] Fashion Analysis: Current Techniques and Future Directions, IEEE Multimedia, 2014.

Urban Tribes: Analyzing Group Photos from a Social Perspective [CVPR12]

50

Urban tribe: the term to describe subcultures of people who share common interests and tend to have similar styles of dress, to behave similarly, and to congregate together. (coined by French sociologist Michel Maffesoli in 1985)

Which groups of people would more likely choose to interact socially? (a) and (b) or (a) and (c)?

Clothing Fashion Analysis

51

"i-Stylist: Finding the Right Dress Through Your Social Networks," MMM 2017.

"A Framework of Enlarging Face Datasets Used for Makeup Face Analysis," BigMM 2016.

"What are the Fashion Trends in New York?" MM 2014. (Grand Challenge Prize)

"Clothing Genre Classification by Exploiting the Style Elements," MM 2012.

Clothing fashion is a reflection of the society of a period The global fashion apparel market today has surpassed

1 trillion US dollars since 2013, and accounts for nearly 2 percent of the world's Gross Domestic Product (GDP)

52

Trend Analysis for the Clothing Fashion

OurworkreceivedMultimediaGrandChallengeAwardin2014ACMMultimediaConference.

Applications: Fashion is becoming mobile first with apps that help track down must-have clothes, accessories and shoes - theguardian.com

LIKEtoKNOW.it The Netbook

Snap Fashion

The Hunt

http://www.fashiontv.com/videos/fashion-weeks

Construct a fashion show dataset

Source:NewYorkFashionWeeks

Color Cut

Pattern Head decoration

major elements forfashion style investigation

key factors for discovering fashion trends: coherence (frequently occur within a fashion week) uniqueness (occur much more often in a fashion week than in other fashion weeks)

http://www.fashiontv.com/videos/fashion-weeks

Detect the presence of catwalk models over all video frames

...

...

...

...

...

...

http://www.fashiontv.com/videos/fashion-weeks

...

...

...

...

...

...

...

...

...

...

...

...

Identify distinct catwalk models

...

...

...

...

...

...

Identify distinct catwalk modelsExtract model location and the full-body image

Collect full-body image of catwalk models

Catwalk Models

e.g.NYFWAutumn/Winter2014Positiveset Negativeset

e.g.allcatwalkmodelsatNYFWexceptforAutumn/Winter2014

Divide the collection of full-body images into two sets

DistributionalclusteringtechniqueW.H.Chengetal.,"LearningandRecognitionofOnPremiseSigns(OPSs)fromWeaklyLabeledStreetViewImages,"IEEETran.onImageProcessing(TIP),2014.

Query Image

Query Image

Color Analysis Texture Analysis Color + Texture Analysis

Query ImageQuery Image

Color Analysis Texture Analysis Color + Texture Analysis

Query Image Query Image Query Image

Color Analysis Texture Analysis Color + Texture Analysis

Query Image Query Image Query Image

Color Analysis Texture Analysis Color + Texture Analysis

Query Image Query Image Query ImageSpring/Summer

2011Spring/Summer

2013Spring/Summer

2013

Spring/Summer 2011 Spring/Summer 2013 Spring/Summer 2013

Predicting Occupation via Human Clothing and Contexts [ICCV11] Diving into the recognition of high-level semantic

categories of human such as occupations

61

Recognizing City Identity via Attribute Analysis of Geo-tagged Images [ECCV14] A set of 7 high-level attributes is used to describe the spatial

form of a city (amount of vertical buildings, type of architecture, water coverage, and green space coverage) and its social functionality (transportation network, athletic activity, and social activity).

62

From Scene Attributes to City Attributes 102 scene attributes are defined. Each of the city attribute classifier is modeled as an ensemble of SVMs.

63

Spatial Analysis of City Attributes The city perception map visualizes the spatial distribution of the 7 city

attributes in different colors and exhibits the visitors and inhabitants own experience and perception of the cities, while it reflects the spatial popularity of places in the city across attributes.

64

Attribute-Based City Identity Recognition

65

66

Sociological understanding of humans and human interactions is fun but still a long way to go!

67

ACM Multimedia 2017http://www.acmmm.org/2017/

Grand ChallengeSocial Media Prediction (SMP):Predicting the Stars of Tomorrow on Social Mediahttps://social-media-prediction.github.io/MM17PredictionChallenge/

Organizers

Wen-Huang Cheng

Academia Sinica

Bo Wu

Chinese Academy of Sciences

Yongdong Zhang

Chinese Academy of Sciences

Tao Mei

Microsoft Research Asis

68

Yahoo! Datasethttp://webscope.sandbox.yahoo.com/

69

YFCC100M This dataset contains 100 million media objects and

explain the rationale behind its creation. This list is compiled from data available on Yahoo! Flickr.

70

Two photos of real world scenes from photographers in the YFCC100M dataset.

YFCC100M

Global coverage of a sample of one million photos from the YFCC100M dataset.

71

Yelp Dataset Challengehttps://www.yelp.com/dataset_challenge

72

Visual Genome Datasethttps://visualgenome.org/

73

Instagram Datasethttp://www.emilio.ferrara.name/datasets/

74

ICWSM-16 DatasetInternational Conference on Web and Social Media

http://www.icwsm.org/2016/datasets/datasets/

75

76

General Chairs Program Chairs

Wan-Chi SiuHong Kong Polytechnic University

Chia-Wen LinNational Tsinghua University

Wen-Huang ChengAcademia Sinica

Gene CheungNational Institute of Informatics

vcip2018.org

Lets exchange ideas!

77

whcheng@citi.sinica.edu.tw

Wen-Huang Cheng

wenhuangcheng