1. WEB 1

2. about me PacketBlackHole, OnePointWall, , secroid CTF CTF( (Agent IV) Winny TV IPA 2010 2 3. Agenda secroid easy webscrap zip de kure P2P3 4. crawler4 5. URL URLHTML 5 6. goo(NTT)Google 2003 20052010 2009 Librahack 2010 Web Yahoo Japan Google 2010 Google98% NSAPRISM 20136 7. 2010 Librahack () http://librahack.jp/ robots.txt () robots.txt URL IP BAN, AS BAN 1 page/ HTTP/1.1 keep alive 21 7 8. http://dic.nicovideo.jp/a/ban8 9. 1 wget 2 UA 3 Cookiereferer 4 5 Cookie 6 IP 7 8 9 10 ) 9 10. wget LinuxMacOSCygwin,WindowsCUI wget r l 5 h http://www.yahoo.co.jp/ robots.txt10 11. UA (UserAgent) HTTP OS Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, likeGecko) Chrome/27.0.1453.94 Safari/537.36 http://www.openspc2.org/userAgent/ 11 12. referer HTTPWebURL URLreferer Referer: http://www.example.co.jp/12 13. wget wget + UA LWP phantomjs firefox+mozrepl+tcpproxy13 14. TLS/SSL (SSL) HTTP Proxy+SSL Socks Socks Proxy + SSL Tor Tor + SSL SPDY SPDY Proxy + SPDY Tor + SPDYProxySPDYSSLSOCKS14 15. Torhttps://www.torproject.org/http://ja.wikipedia.org/wiki/Tor15 16. SPDY GoogleHTTPSSLSSLSSL Googletwitter16 17. capcha cookie IP UA robots.txt JavaScript17 18. 18 19. 19 20. scraping20 21. Perl WeScraper HTML::TreeBuilder Python Ruby WSH (Windows Script Host) jQuery and more middle wares http://www.scrapy.org/ http://nokogiri.org/21 22. (1/2) grep() grep grep or egrep wget -O - http://hamusoku.com/archives/7927364.html |grepblogimg|sed s/.* src="//g|sed s/" .*//g cat te.html|perl -e $/="