Plagger the duct tape of internet

Plaggerthe duct tape of the Web

Tatsuhiko Miyagawa [email protected]

Six Apart, Ltd. / Shibuya Perl MongersXML Developers' Day #9

Tatsuhiko MiyagawaTatsuhiko Miyagawa 2006/11/24 XML developers' day #92006/11/24 XML developers' day #9

IRC#plagger-ja

chat.freenode.net(iso-2022-jp)


アウェイっぽいので自己紹介


Tatsuhiko Miyagawa




http://www.vox.com/


What is Plagger?






PluggableRSS/Atom

Aggregator


Why Pluggable?

Just for a feed aggregation?


History


2002 Apr.baseball2rss

http://search.cpan.org/dist/WWW-Baseball-NPB/


2003 Oct.rss2javascript

http://blog.bulknews.net/cookbook/blosxom/rss/rss2js.html


2004 Sep.bloglines2ipod

http://bulknews.net/lib/utils/bloglines2ipod/



2004 Oct.rss2audiobook

http://bulknews.net/lib/utils/rss2audiobook/


2005 Aug.bloglines2gmail

http://svn.bulknews.net/repos/public/bloglines2email/trunk/


#!/usr/bin/perluse strict;use warnings;use LWP::UserAgent;use XML::RSS;

my $url = "http://example.com/rss.xml";my $agent = LWP::UserAgent->new;my $xml = $agent->get($url)->content;my $rss = XML::RSS->new; $rss->parse($xml);

for my $item (@{$rss->items}) { # do something with $item}



my $url = "http://example.com/rss.xml";my $agent = LWP::UserAgent->new;my $xml = $agent->get($url)->content;my $rss = XML::RSS->new; $rss->parse($xml);



"HTTP リクエストがエラーになったら ?"



my $url = "http://example.com/rss.xml";my $agent = LWP::UserAgent->new;my $res = $agent->get($url);if ($res->is_error) { die "Bah." }my $xml = $res->content;my $rss = XML::RSS->new; $rss->parse($xml);



"Atom も読みたい !"


#!/usr/bin/perluse strict;use warnings;use LWP::UserAgent;use XML::RSS;use XML::Atom::Feed;

my $url = "http://example.com/rss.xml";my $agent = LWP::UserAgent->new;my $res = $agent->get($url);if ($res->is_error) { die "Bah." }my $xml = $res->content;if ($res->content_type =~ /atom/) { my $feed = XML::Atom::Feed->new(\$xml);} else { my $rss = XML::RSS->new; $rss->parse($xml);}


If-Modified-Since で帯域節約したい！



my $url = "http://example.com/rss.xml";my $agent = LWP::UserAgent->new;my $local = cache_path_for($url);my $res = $agent->mirror($url, $local);if ($res->is_error) { die "Bah." }my $xml = $res->content;…


" 壊れたフィードもパースしたい！ "


Etc., etc.


他にも


rss2opml

http://aruntx.com/software/rss2opml/


rss2pdf

http://rss2pdf.com/


rss2atom

brian.wanamaker.com/mybicycle/2004/02/rss2atom.html


atom2rss

http://www.2rss.com/software.php?page=atom2rss


rss2ical

http://bura-bura.com/blog/archives/2004/06/22/rss2ical/


Bloglines2opml

http://mycvs.org/wp/wp-content/wp-transform.php


rss2gmail

http://www.cs.utexas.edu/~karu/gmailrss/


rss2imap

http://rss2imap.sourceforge.jp/


ebay2rss

http://www.2rss.com/software.php?page=ebay2rss


svn2rss

http://twiki.org/cgi-bin/view/Codev/Svn2rss


<any>2<any>

Where either of <any> is RSS|Atom|OPML


This is ridiculous.


Different Languages,Different Bugs.No hackability



via http://www.atmarkit.co.jp/fnetwork/rensai/5minplagger/02.html


IRC, Eject, GrowlMSAgent, SSTP …

Filter Publish

StripRSSAdTruePermalinkEntryFullTextPipeThumbnailFindEnclosuresFetchEnclosureSpamAssassinRSSLiberalDateTimeURLBLResolveRelativeLink…

GmailDeliciousPDFMTFeedPlanetSpeech…

Notify

BloglinesConfigOPML, XOXOFile, DBI, FOAF …

Mixi, FrepaPOP3, iCaliTunes, AmazonYouTube …

Subscription

CustomFeed



Filter Publish



Notify



Subscription

CustomFeed



Filter Publish



Notify



Subscription

CustomFeed



Filter Publish



Notify



Subscription

CustomFeed


Just like UNIX pipe

Subscribe OPML | StripRSSAd | ResolveRelativeLink \| Publish Feed --type=Atom


I believe RSS has the potential to be the “UNIX pipe of the internet” …

Ray OzzieCTO of Microsoft

http://rayozzie.spaces.live.com/blog/cns!FB3017FBB9B2E142!285.entry


"the Unix shell for Web 2.0"


組み合わせの数

{CustomFeed,Subscription}/*.pm: 35

{Publish,Notify}/*.pm: 37

35 * 37 = 1295


Plagger Core features


RSS/AtomAuto-Discovery


Support Feed formatsRSS 0.91 to Atom 1.0(XML::Feed + hacks)


Support parsingBroken feeds(XML::Liberal)


HTTP optimizationsIf-Modified-Since /

gzip(URI::Fetch)


Podcast / VideocastSupport

(RSS 2.0 & Atom 1.0)


PhotocastMedia RSS

iTunes RSS*


非同期ダウンロードcURL, wget,

HTTP::Parallel & HTTP::Async*


完全な国際化Unicode & Timezone


Access tobrowser Cookies

IE, Safari, Firefox and w3m


Screen-scrapingVia CutomFeed::*


Stackable Plugins


Rule-basedDispatch of Plugins


Plagger のインストール




省略http://plagger.org/trac/wiki/PlaggerQuickStart


Plaggerクイックチュートリアル


Email クライアントでRSS フィードを読みたい！(Gmail, Thunderbird)


rss2email.yaml

plugins: - module: Subscription::Config config: feed: - http://bulknews.vox.com/library/posts/atom.xml - http://bulknews.typepad.com/blog/ - module: Publish::Gmail config: mailto: [email protected]


RSS in Gmail


HTML + Images


Feed Image (Logo / Buddy Icon)


Search


Auto grouping (“Conversations”)


Diff


オフライン（飛行機）でもフィードを読みたい！


rss2email.yaml

plugins: - module: Subscription::Config config: feed: - http://bulknews.vox.com/library/posts/atom.xml - http://bulknews.typepad.com/blog/ - module: Filter::FindEnclosures - module: Filter::FetchEnclosure config: dir: /tmp - module: Publish::Gmail config: mailto: [email protected] attach_enclosures: 1


Offline Mode

POP3 + Thunderbird


" フィードのリストをYAML で管理するのメンドウ "


opml2email.yamlplugins: - module: Subscription::OPML config: url: http://example.com/subscription.opml - module: Publish::Gmail config: mailto: [email protected]

# subscription.opml<?xml version="1.0"?><opml><outline title="Subscriptions"> <outline title="miyagawa" type="rss" xmlUrl="http://bulknews.typepad.com/blog/atom.xml" /> <outline title="miyagawa on Vox" type="rss" htmlUrl="http://bulknews.vox.com/" /></outline></opml>


"OPML を手動で編集するのはテラダルス "


filesub2email.yaml

plugins: - module: Subscription::File config: url: file:///path/to/subscription.txt - module: Publish::Gmail config: mailto: [email protected]

> cat subscription.txthttp://bulknews.typepad.com/blog/atom.xmlhttp://bulknews.vox.com/>

Tatsuhiko MiyagawaTatsuhiko Miyagawa 2006/11/23 XML Developers' Day #92006/11/23 XML Developers' Day #9

"1000 フィード以上読んでると自分のマシンで読むのは帯域のムダ。 Bloglines にクロールさせたい。 "


bloglines2email.yaml

plugins: - module: Subscription::Bloglines config: username: [email protected] password: blahblahblah - module: Publish::Gmail config: mailto: [email protected]


" それ Livedoor Reader で "



plugins: - module: Subscription::LivedoorReader config: username: [email protected] password: blahblahblah - module: Publish::Gmail config: mailto: [email protected]


"Gmail / Thunderbird からdel.icio.us にブクマしたい !"



plugins: - module: Subscription::Bloglines config: username: [email protected] password: blahblahblah

- module: Widget::Simple config: widget: delicious

- module: Publish::Gmail config: mailto: [email protected]



" それはてなブックマークで "




- module: Widget::Simple config: widget: hatena_bookmark_users




"RSS 広告ウザス "




- module: Widget::Simple config: widget: delicious

- module: Filter::StripRSSAd



Quick tourFor more plugins


Plugin phases (types)

SubscriptionAggregatorCustomFeed

FilterPublishSearchNotify






Subscription

load subscriptions(list the feeds/URLs to aggregate)


Subscription::Config - module: Subscription::Config config: feed: - http://www.yapcchicago.org/feed/ - http://tokyo.yapcasia.org/blog/


- module: Subscription::OPML config: url: http://www.example.com/subs.opml

# subs.opml <opml> <outline xmlUrl="http://www.yapcchicago.org/feed/" /> <outline htmlUrl="http://tokyo.yapcasia.org/blog/" /> </opml>

Subscription::OPML


- module: Subscription::File config: url: file:///path/to/subscription.txt

% cat subscription.txt http://www.yapcchicago.org/feed/ http://tokyo.yapcasia.org/blog/ %

Subscription::File


Subscription::XOXO - module: Subscription::XOXO config: url: http://www.example.com/subscription.html

# subscription.html <ul class="xoxo"> <li><a href="http://www.yapcchicago.org/feed/">YAPC::NA</a></li> <li><a href="http://tokyo.yapcasia.org/blog/">YAPC::NA</a></li> </ul>


Subscription::Bookmarks

Read bookmarks file of IE, Firefox and Safari






Filter

Normalize / Repair feed metadataUpgrade feed content

Filter feed content using text filtersInvoke some action on entries


Filter::EntryFullText

本文なしのフィードをアップグレード個別 HTML を取得して正規表現 / XPath


Filter::TruePermalink

リダイレクト URL などを Canonicalize(e.g. http://…/go.php?url=….)


Filter::FindEnclosures

コンテンツからエンクロージャを抽出<a href="http://…./foo.mp3">episode

#1</a>






Publish

Publish aggregated entry to online servicesreBlogging

Convert feeds to other formats


Publish::Feed

Republish feed in RSS/AtomGood to use with scrapers


Publish::MT

Reblog entries using MT XML-RPC


Publish::MTWidget


Publish::Email

text/plain, multipart/alternativePluggable email protocols

(SMTP, SMTP Auth, IMAP, Maildir …)


Publish::iCal

Publish iCal feeds out of RSS/Atom


Publish::Excel

仕事の合間に !






Search

Index aggregated entries on search engines


Search::Spotlight


Search::Estraier

Uses HyperEstraier XMLRPC node API


Search::Lucene*

Use Lucene WebService API (OpenSearch 1.1 and Atom)






Notify

Notify feed updates in various ways


Notify::Growl


Notify::Balloon


Notify::MSAgent


Notify::Eject

Supports: Windows, Linux, FreeBSD and Mac OSX!


Notify::Pizza


Notify::Pizza

Now it does Sushi too!


Plagger に対する誤解


それ Plagger でできるよ

それプラズマで説明できるよ

http://d.hatena.ne.jp/sugarcut/20061117/p1

それ Pla 脳



何でもできる

… わけじゃない



Plagger is a"Feed" aggregator


Plagger::Subscription

Plagger::Feed

authoridlinktitletagsurlentries

Plagger::Feed


Plagger::Feed


Plagger::Entry

authoridlinkpermalinktitletagsenclosures


ピザ Pla の意義





"the Unix shell for Web 2.0"


"RSS is the Standard IO for the Web"


"RSS is the Standard IO for the Web"


Feed formats

RSS 0.91RSS 2.0RSS1.0 / RDF

Atom 1.0

JSON

iCal

OPML

XOXOXBEL

Sitemaps

attention.xml

Amazon API

Google APIOpenSearch

AtomPP

GData


rvw:

Enclosures

Feed Vocabulary / Extensions

Photocast

iTunes RSSMedia RSS

Dublin Core

FOAF

microformats



Plagger = The duct tape of the web.


Example:Location metadata


N 37.7782 W 122.3973


GeoRSS

<georss:point>37.7782 -122.3973</georss:point>

<georss:where> <gml:Point> <gml:pos>37.7782 -122.3973</gml:pos> </gml:Point></georss:where>

xmlns:georss="http://www.georss.org/georss"xmlns:gml="http://www.opengis.net/gml"


RDF geo vocabulary

<foaf:based_near> <geo:Point> <geo:lat>35.678</geo:lat> <geo:long>139.770</geo:long> </geo:Point></foaf:based_near>

xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#"


Flickr geo tag

<media:category scheme="urn:flickr:tags">geo:lat=37.7782 geo:lon=-122.3973</media:category>

xmlns:media="http://search.yahoo.com/mrss/"


GeoURL

<meta name="ICBM" content="37.7782, -122.3973" />


geo microformats

<div class="geo"> <span class="latitude">37.7782</span> <span class="longitude">-122.3973</span></div>


Links to Google Maps

<a href="http://maps.google.com/maps?q=37.7782,-122.3973&z=16">Link to Google Maps</a>


Eznavi mail


Photo EXIF


adr

<div class="adr"> <div class="street-address">548 4th St.</div> <span class="locality">San Francisco</span>, <span class="region">CA</span> <span class="postal-code">94107</span> <div class="country-name">U.S.A.</div></div>


We don't careabout format diffs.


GeoRSS

<georss:point>37.7782 -122.3973</georss:point>

<georss:where> <gml:Point> <gml:pos>37.7782 -122.3973</gml:pos> </gml:Point></georss:where>

Namespace::GeoRSS


RDF geo vocabulary

<foaf:based_near> <geo:Point> <geo:lat>35.678</geo:lat> <geo:long>139.770</geo:long> </geo:Point></foaf:based_near>

Namespace::Geo


Flickr geo tag

<media:category scheme="urn:flickr:tags">geo:lat=37.7782 geo:lon=-122.3973</media:category>

Filter::geotagged


GeoURL

<meta name="ICBM" content="37.7782, -122.3973" />

Filter::GeoURL


geo microformats

<div class="geo"> <span class="latitude">37.7782</span> <span class="longitude">-122.3973</span></div>

Filter::Microformats::geo


Links to Google Maps

<a href="http://maps.google.com/maps?q=37.7782,-122.3973&z=16">Link to Google Maps</a>

Filter::ExtractMapsLinks


Eznavi mail

Filter::ExtractMapsLinks


Photo EXIF

Filter::FetchEnclosure +Filter::ExtractEXIF


adr

<div class="adr"> <div class="street-address">548 4th St.</div> <span class="locality">San Francisco</span>, <span class="region">CA</span> <span class="postal-code">94107</span> <div class="country-name">U.S.A.</div></div>

Filter::Microformats::adr +Filter::Geocoding::US


Publish::KMLPublish::GoogleMaps

Publish::Feed(with geotags)


Everything's donein plugins

= Clean & extensible.


Plaggerdev. Status


Version0.7.13


Coming Soon …


iTunes RSS support


Geo extensions


Enclosure processors

ffmpeg, Sync::PSP, Sync::iPodVideo


Pluggable summarizer

& text formatter

Lingua::EN::Summarize, Text::Original, HTML::WikiConverter, HTML::FormatText


Rich Media metadata

ID3 tag in enclosuresLinks to imdb.com / amazon.com

hReview microformats


Calendar Support

iCal parser & emitterhCalendar microformats.ics attached in emails

Sync::SyncML


Email refactoring

text/plain, iso-2022-jp supportPluggable storage engines


http://plagger.org/

Planet, Mailing List, IRCBug Tracking, SVN repository


#plagger-ja on freenode


Join Us!


Thank you

Questions?

Technology

Plagger the duct tape of internet