Embed Size (px)
DESCRIPTION
Citation preview
My name isYoji SHIDARA.
Enishi Tech Inc.Small Company.Only 2 staffs.
Small Office.(CEO’s flat)
I’m from Sapporo!
Sapporo
Tokyo San Fransisco!9.5hrs1.4hrs
@darashi (me)
Sapporo is a beautiful provincial city of Japan.
Delicious foods are waiting for you!
Do you knowbuzztter.com?
http://buzztter.com
buzz-phrases (auto update)
Full-text Search Results(auto update)
Servers are in my room!next to my bed
1,336,444 PVs / Month
1,932,823 tweets/day (Japanese)607,749 tweets/day (English)
We crawls:
(We’re now focusing on Japanese users due to API limits)
We have:
Server 1:Phenom 9950Memory 8GBSSD 80GB + HDD 80GB
Server 2:PhenomII x4 955 Memory 8GBHDD 500GB
@buzztter
Bot for English was banned because it tweeted
spam as hot:-<
Buzztter:
buzz-phrase detection
+real time full text
search
... like official “trending topics”
buzztter released before the official
one launched!
buzztter supports English and Japanese!
1-2 minutes for buzz analysis par a
language.
HOT notification bot is available.
crawlermodule
Analyzermodule
WebFrontend
Notification module
Fulltext Search
BuzzDAS: Buzz Data Analysis System
System Diagram:Rough Sketch
System Diagram: Crawler
crawler
crawlcontroller
MQ
Users DB
RabbitMQ
EventMachine
PostgreSQL
crawl scheduling
langugage guesserlanguage,
post frequency
libtextcat
Typhoeus
MQ
crawler
importerfulltext search engine
analyzer
pagecache
Web Service API
notifier bot
Web Frontend
groonga
Rails
Sinatra
Twitter4R
BuzzDAS: Buzz Data Analysis System
net/irc
memcached
varnish
ReferenceIndex
RecentIndex
keeps post in 24 hours keeps recent 1 hour
phrase extractortokenizer
detects change of phrase occurrence
analyzer
Try http://buzztter.com and give me your feedback please!If you Interested in these keywords:
or our BuzzDAS Engine,please contact me!My name is Yoji Shidara.
groonga, AMQP, RabbitMQ, Typhoeus, EventMachine, PostgreSQL, libtextcat,Sinatra, Rails, Twitter4R, net/irc, memcached, PrefixSpan ,MeCab, TF-IDF, ...
