ExplorationBy Brian Klug

Presentation Overview:• High level overview (Presentation,

Message Syntax)• Communication Paradigms (SMS, Web, API,

e.t.c.)• API Ease of Use (Spidering, Search,

Aggregation)• Growth and Future (Stats, Trends, Goals)• Case Studies (Iran, Michael Jackson,

AT&T)

Exploration

• 140 character or less microblog updates or “tweets” about almost any subject– Encourages high frequency updates on subjects limited to the context of 140

characters. Concise, short, granular, timestamped.

• Described as “SMS of the Web”– Service is unique cross section of Instant Messaging, Blogging, and public

Forums and creates a new niche– Updates aren’t limited to a particular subject, but can be personal in nature,

from celebrities (@aplusk), about news (@nytimes) or media (@cnnbrk), events, for customer support (@comcastcares) and PR (@ATTNews), for marketing and advertising, and even automated (@tweetawatt)

• Real-time search functionality– Allows one to watch real-time a particular topic, a popular subject, or search

• Aims to be “Pulse of the Planet”– http://www.techcrunch.com/2009/07/16/twitters-internal-strategy-laid-bare-to-be-the-pulse-of-the-

planet/ 2

What is Twitter?

Exploration

• Typical User View (CEO of Twitter)– Web View is shown– Note that this is often

not how most people actually use the service, but more on that later

3

High Level Overview - Presentation

Status Updates

User Bio and following/follower counts

@[user] indicates response to other user

Via @[user] (or RT @[user) indicates a re-tweet. The user chooses to repeat something of note/interest.

Exploration

• Typical Home View– Example user (AiLab)– Fictitious, created

solely to demonstrate common syntax and functionality

4

High Level Overview - Presentation

Update field, where one updates with a new status

Update with a link to latitude and longitude google maps of present location

Posts can link to images taken directly from mobile devices

Using a #[tag] example to make it easier for other posters searching to find this topic

Topics (hash tags) becoming popular based on volume

Other users tweets show up in my timeline threaded with my updates

Clicking brings us to Google mapsClicking brings us to TwitPic to see the image

Exploration

• Huge variety in clients– Updates via SMS – “Texts”– Huge variety of 3rd party client

applications (>250 according to http://twitdom.com/)

• iPhone OS• BlackBerry OS• Windows Mobile• Nokia Symbian• Palm Pre - WebOS• All Desktop OSes

– Directly to API from command line– RSS feeds– Automated triggered programs– Directly from browser (Firefox, etc)– Media Players, Entertainment

Consoles (Xbox 360, PS3, e.t.c. coming soon)

5

API allows 3rd party apps/use

Exploration

• More than half of all tweets are published from mobile devices – Take-away: this makes

the service natively ubiquitous and empowers users to discuss things happening around them in real-time, often with media such as pictures and geolocation tags.

6

Mobile Use is Key to Growth

Exploration

• 1-line Bash+Curl update posting– curl --basic --user username:password --data status="Having fun with cURL"

http://twitter.com/statuses/update.xml

• Extremely open, documented API– http://apiwiki.twitter.com/Twitter-API-Documentation – Search, Trends (Current/Daily/Weekly), Statuses (Public, Friends, User, Mentions),

Social Graph data (array of UserIDs a user is following), all returned from API

• Rate Limiting – Two APIs– REST API (Core Twitter Data: Timelines, Status, User info)

• Limited to 150 API calls/hour– Search API (Direct access to Twitter Search and Trends data)

• Not Published: “The number is quite a bit higher … We do not give the exact number because we want to discourage unnecessary search usage.”

• Streaming/firehose API– Access to real-time data is available on request, however, it is in infancy (Alpha testing). Obviously the

sheer volume of data is nontrivial enough to require special provisions for access.• http://apiwiki.twitter.com/Streaming-API-Documentation

7

API makes it easy to gather data

Exploration

• Searching Twitter for a string– http://apiwiki.twitter.com/Twitter-Search-API-Method%3A-search – Query Syntax: http://search.twitter.com/search.format

• EG: http://search.twitter.com/search.atom?q=twitter • Format: JSON or ATOM (XML)

– Also location aware based on user-defined location (updated either via location aware client like iPhone or manually)

• http://search.twitter.com/search.atom?geocode=40.757929%2C-73.985506%2C25km

– Returns tweets by users located within a given radius of the given latitude/longitude

8

Twitter API Examples

Exploration

• Feasible Methodologies– 1: Leverage the generous Search API rate limits and search wildcards which

return as large a percentage of the current posted tweets as possible, store in database.

– 2: Attempt to gain access to “Firehose”/”Streaming” API and monitor real-time twitter stream, simply store all we can in database

• Primary Benefits– Identity of each update already provided by unique UserID – Identity tracking

is trivial save some users who have duplicate accounts– Users have already collectively decided on a schema for passing-on and

tracking topics/ideas with #[tags] and @[replies]. Finding where a idea or key point started is almost trivial given each update’s timestamp. Clear social network graph relevancy

– API provides clean and completely sanctioned access to user generated tweets and trending information

– Huge user base cross section demographic (age, location, language, socioeconomic status)

– Fast update frequency (10-20 thousand tweets/minute) 9

Spidering Twitter

Exploration

• Hundreds of sites leveraging Twitter API to deliver data, mashups, user stats, trending news, e.t.c.– TweetMeme.com – Aggregation of most popular and RT’ed links on Twitter

based on different categories. Effectively a link mining app for delivering news that is trending in popularity

– Twist.flaptor.com – Graphical volume trends of the “trending topics” on twitter and examples of realtime tweets which reference them

– Search.twitter.com – The most obvious and robust, delivers realtime stream of tweets with matching hashtags, links, or topics based on a standing search.

– ReTweet mappers (very many): ReTweet Mapper, retweet radar, ReTweetist, Twidentify All provide either graphical or statistical ways to analyze which topics people find most worth spreading.

10

Twitter Stats via API - examples

Exploration

• Because of 140 character limitation, link shortening services have exploded in popularity– Idea is to use a redirect to shorten an otherwise long URL to a short one

• EG http://ai.arizona.edu/gallery/people/endSem/spring2007/IMG_5388.JPG -> http://is.gd/2boCm

– Started with Tinyurl.com in 2002 before Twitter• Originally to purposefully obfuscate destination links. Abused frequently

for pranks (rickrolling, shock images), spam or malware distribution. Now popularized for character-limited applications.

– Growing concern for “Great Linkrot Apocalypse”• http://joshua.schachter.org/2009/04/on-url-shorteners.html

• Relevant takeaway– Huge opportunity to monetize link mining through shortening services– Official Twitter link shortening service bit.ly already employing data mining on

links for news and for-sale to marketing– Already possible to track stats on shortened click-through URLs by

appending a + to the URL (eg “bit.ly/info/44u4E+”)• http://www.wired.com/epicenter/2009/07/twitter-url-service-bitly-says-no-to-ads-yes-to-data-minin

g-news/

11

Link Shortening

Exploration

• Currently undergoing explosive growth– Feb 09 – <10 million monthly unique visitors– March 09 – 19 million – April 09 – 32 million– June 09 – 44.5 million

• ComScore Stats– 1,382% year over year growth in Feb 09– Relatively flat age distribution of Twitter users– Shows promise for continued global growth

• Frequency of Updates– >1,000,000 tweets/hour

• Note the drop to 0; caused by DDOS– 10-20 k tweets/minute

• http://www.tweespeed.com/

• Ranks 14 in Alexa stats12

Current Growth

Exploration

• Leaked internal documents:– Key Point: Sees 1 billion users

by 2013• http://www.techcrunch.com/2009/07/15/twitter

s-financial-forecast-shows-first-revenue-in-q3-1-billion-users-in-2013/

• Long term goals:– Already stated: Become self-

acclaimed “pulse of the planet”– Real-time searching of tweets

and users

• Long term use:– Twitter as a news wire– Twitter as citizen journalism– Twitter as a PR tool– Twitter as a source for link

mining and social analytics– Real-time sentiment tracking– “Hosebird” stream API 13

Future

Exploration

• ComcastCares Frank Eliason: (@comcastcares)– Pioneered the use of Twitter to engage in direct customer care to interact with

Comcast customers. Watches standing searches of “Comcast” and “Comcrap” (and other pejorative corruptions of Comcast) and replies directly to customers to mitigate issues

– Virtually instant high-level help and dialogue with someone empowered to fix problems on Comcast’s side.

– Spawned a flurry of media attention and led other corporations to make twitter a means of reaching out to customers and dealing with complaints

• Horizon Realty (July 28, 2009)– Woman sued over a tweet:

– Horizon Realty sues on the basis of libel for $50,000, claims: “We’re a sue first, ask questions later kind of an organization”.

– Result: Horizon Realty trends on twitter for almost three days straight, generating more bad press and attention. Horizon Realty likely loses a lot more than $50,000 from Twitter backlash. A painful lesson in social media.

14

Case Studies

Exploration

• Zappos.com (May 6, 2008)– Zappos.com CEO Tony Hsieh: Company’s most followed and active twitter user uses

the site to:• Share some details about what he’s up to personally and professionally.• Provide some behind the scenes info regarding what its like to work at Zappos.com

.• Launch a Twitter contest asking people to help them rewrite their confirmation emai

ls.• Incubate an idea for polling customers on Twitter.• Explain why he is using Twitter.

– http://www.bivingsreport.com/2008/zapposcom-a-twitter-case-study/ – Amazon Acquires Zappos.com for $850 million (July 22, 2009)

• AT&T Blocks 4chan.org (July 27, 2009)– AT&T Broadband inexplicably starts blocking access to 4chan.org’s /b/ and /r9k/ boards– AT&T and #ATTCensorship tags instantly skyrocket Twitter trending topics, creates instant online

outcry against AT&T and organized movements to complain/attack/DDOS AT&T directly start.– Less than three hours later, AT&T realizes it has inadvertently blocked img.4chan.org as part of an

effort to mitigate a DDOS attack on 4chan which was “actively degrading” AT&T customer service. – #ATTfail, #ATTcensorship and other similar tags dominate twitter trends for the remaining week

• (AT&T is a puzzling constant trend on Twitter due to continual complaining about network coverage, outages, iPhone handset exclusivity, e.t.c.)

15

Case Studies

Exploration

• Michael Jackson Death– TMZ.com reports Jackson is hospitalized, in “cardiac arrest” at 4:30 PM PST

• http://www.tmz.com/2009/06/25/michael-jackson-rushed-to-the-hospital/1#c19669675

– Immediately, “Cardiac Arrest” and “Michael Jackson” skyrocket to the top of trending on Twitter, knocking “Farrah Fawcett” out of the top spot

– Nearly half hour passes before mainstream media acknowledges Jackson has been hospitalized, hesitates to acknowledge “cardiac arrest”

• [Shameless Aside: I was online as this erupted and noticed news at least 5 minutes before I saw it on mainstream media TV]

– Me: “Intense, I'm watching the BBC now, but I saw news about Michael Jackson's cardiac arrest on twitter a good 5 minutes before I saw it on TV 2:06 PM Jun 25th from twhirl”

– Resulting fallout: Michael Jackson (“RIP MJ,” “King of Pop,” “Thriller,” “Michael Jackson died”) trends strongly, nearly overwhelms Twitter’s servers, and represents over 30% of tweets for the next week, doubles Twitter’s update frequency to nearly 25,000 updates/minute.

16

Case Studies

Exploration

17

Michael Jackson “Effect”

TweeSpeed Daily View

TweeSpeed Weekly View

Jackson’s Death

http://www.tweespeed.com/

Exploration

• All too familiar examples– Iran citizen journalism Twitter revolution

• (Unfortunately pushed out of the spotlight due to the Michael Jackson death)

– US Presidential Election• Obama campaign successfully leverages Twitter/Social Networking

• Perhaps most chilling example of Twitter’s power leveraging masses of real time data– Jayaram (Twitter’s new VP of Operations) told of being in the Twitter offices in

San Francisco on March 30, when the Twitter engineers noticed that the word “earthquake” had suddenly started trending up. They didn’t know where the earthquake was. Several seconds later, their building started to shake. The earthquake had been in Morgan Hill, 60 miles south of San Francisco, and the tweets about the shaker reached the office faster than the seismic waves themselves.

• http://www.webpronews.com/blogtalk/2009/05/07/twitter-looking-to-tweet-its-way-into-the-world-of-search

18

Case Studies