Emerging Topic Detection on Twitter based on Temporal and Social Terms EvaluationKDD 2010 Workshop on Multimedia Data Mining

Chin Hui Chen (陳晉暉 )

Author

• Mario Cataldi Università di Torino, Torino, Italy• Luigi Di Caro Università di Torino, Torino, Italy• Claudio Schifanella Università di Torino, Torino, Italy

Agenda

• Introduction• The Main Steps • Content Extraction• User Authority• Content Aging Theory• Selection of Emerging Terms• From Emerging Terms to Emerging Topics

• Experiments and Evaluation

Introduction

• Twitter.com• 75 million users on December 2009.• 6.2 million new accounts/per month (2-3 per second)

• People post tweets for …• Daily chatter • Conversations• Sharing information/URLs• Reporting news

Introduction (con’t)

• One of the founders of Twitter.com …

• A low level information news flashes portal.

Introduction (con’t)

• Target : Extract the emerging topics.• Process : • Content Extraction• User Authority• Content Aging Theory• Selection of Emerging Terms• From Emerging Terms to Emerging Topics

Agenda

• Introduction• The Main Steps • Content Extraction• User Authority• Content Aging Theory• Selection of Emerging Terms• From Emerging Terms to Emerging Topics

• Experiments and Evaluation

Step 1: Content Extraction

• Target : Tweets => Vector• t-th considered interval :

• Each tweet => tweet vector

Content Extraction (con’t)

where , = vocabulary size.

where , is the term freq value of the x-th vocab terms in j-th tweet, and returns the highest term freq value of the j-th tweet.

Step 2: User Authority

• Target : Which User is Important ?

• Define an author-based graph G(U,F) , where U is the set of users and F is the set of directed edges.

follower

User Authority (con’t)

User Authority (con’t)

• Compute Authority • => PageRank

User Authority (con’t)

Step 3: Content Aging Theory

• Target : Find Emerging Term.

• An Emerging keyword can be viewed as a semantic unit which links to a very recent news event.

• Chien Chin Chen, Yao-Tsung Chen, Yeali S. Sun, Meng Chang Chen: Life Cycle Modeling of News Events Using Aging Theory. ECML 2003

• See each term as a living organism:• With nourishment => life cycle is prolonged. => high energy• Without nourishment => die => low energy

Content Aging Theory (con’t)

• Term with high energy => important currently• Term with low energy => out of favor

• So, we need to know how to compute Nutrition and Energy.• Content Nutrition• Content Energy

Content Aging Theory (con’t) – Content Nutrition• Each food brings a different calory contribution depending on

its ingredients.• Different tweets containing the same keyword generate

different amount of nutrition.• Define the amount of nutrition :

Content Aging Theory (con’t) – Content Energy• Now we obtained the nutrition of a semantic unit => map into

energy => effective contribution (how much it is emergent).

• Hot Terms :

• Emergent Terms :

Content Aging Theory (con’t) – Content Energy

Content Aging Theory (con’t) – Content Energy• Define s = number of previous time slots.

Step 4: Selection of Emerging Terms• Target : How to select emerging keywords.• 1. Supervised

• ( )• 2. Unsupervised• Dynamically sets the critical drop• CoSeNa: a Context-based Search and Navigation System

Step 5: From Emerging Terms to Emerging Topics

• Target : Find Emerging Topics!

• Define topic as a minimal set of a terms semantically related to an emerging keyword.

• “victory”• Nov 2008 : “elections”, “Obama”, “USA” • Feb 2010 : “football”, “superbowl”, “New Orleans Saints”

• Method : co-occurrences

From Emerging Terms to Emerging Topics• 1. Generate Correlation Vector

• a. the keyword k as query.• b. the set of tweets containing k as relevance feedback.• c. relying on probabilistic feedback mechanism.

From Emerging Terms to Emerging Topics• 2. Construct Topic Graph

Keyword-based topic graph :

Thinning.

From Emerging Terms to Emerging Topics• 3. Topic Detection and Ranking

From Emerging Terms to Emerging Topics• Find SCC (Strongly Connect Component) :

• Emerging Topic as a subgraph representing a set of keywords semantically related to term z within the time interval.

Use DFS.

From Emerging Terms to Emerging Topics• Ranking

•

From Emerging Terms to Emerging Topics

Experiments and Evaluation

• Dataset : • 15 days (between 13th and 28th of April 2010)• More than 3 millions of tweets ( 10k/hr )• More then 300k different keywords

Real Case Study

• Set r = 15 mins , time slot s = 200. (2 solar days)• Result :

History Worthiness• Analyze two diff number of considered slots, s=100 and s=200.

History Worthiness (con’t)• “morning” => periodic events

History Worthiness (con’t)

• Life status of a keyword depends => number of time intervals.• Temporal relevance of the retrieved

topics. (Relevance是跟時間有關 )

Conclusion

• 1. Formalized the Keyword Life Cycle.• (now => frequently , past => rare)• 2. Study the Social Relationships.• 3. Formalized the Keyword-based Topic

Graph.

Appendix

• Twitter Search• Google Real Time Search