Visualize NGiNX log with GoogleMap
Preview:
Citation preview
- 1. Visualize NGiNX log with GoogleMap
- 2.
- 3. purpose ! NGiNX ! !
- 4.
- 5. ! Internet ! ! GoogleMap
- 6. ! ngx_http_geoip_module ! Google BigQuery ! () GeoFUSE
- 7. GeoFUSE? ! GPL ! mbasa/GeoFuse GitHub :
https://github.com/mbasa/geofuse ! GeoFUSE :
http://geofuse.georepublic.net/geofuse/ ! ! (GUI)
- 8.
- 9.
- 10. overwiew GeoFUSE Big QueryNGiNX 1.Logging 2.Transfer
3.Query 4.CSV
- 11. 1.Logging ! NginxGeoIP ! GeoIP C APIGeoIP Datale !
nginx--with-http_geoip_module ! Nginx.confGeoIP Datale ! GeoIPIP
http { geoip_country /usr/local/share/GeoIP/GeoIP.dat; geoip_city
/usr/local/share/GeoIP/GeoIPCity.dat; real_ip_header
X-Forwarded-For; set_real_ip_from 10.0.0.0/8; log_format ltsv
"geoip_country_name:" "$geoip_city_country_name" "t" #
"geoip_country_code3:" "$geoip_city_country_code3" "t" # JPN/USA
"geoip_city:" "$geoip_city" "t" # "geoip_latitude:"
"$geoip_latitude" "t" # "geoip_longitude:" "$geoip_longitude" # ;
}
- 12. 2.Transfer ! NginxBigQuery ! uentd ! in_tail !
uent-plugin-google-cloud ! NGiNXltsv ! BigQueryjson ()
- 13. 3.Query ! textPayload json ! JSON_EXTRACT
JSON_EXTRACT_SCALAR ! (SUM,AVG,MIN,MAX) SELECT ROW_NUMBER() OVER
(ORDER BY count DESC) AS ID, COUNT (*) AS count,
JSON_EXTRACT_SCALAR(textPayload,'$.geoip_longitude') AS lon,
JSON_EXTRACT_SCALAR(textPayload,'$.geoip_latitude') AS lat,
AVG(JSON_EXTRACT_SCALAR (textPayload,'$.response_time')) AS
avelage_time FROM [HOGE.nginx_access_20150402] GROUP BY lat,lon
ORDER BY count DESC
- 14. 4.csv ! BigQuerycsv ! Geo-FUSE ! Geo-FUSE ! ! ID !
lon.lat() ID,count,avelage_time,lon,lat
1,3135686,0.2498652794178637,139.6900,35.6900
2,1596097,0.2536400474791719,139.7514,35.6850
3,182082,0.23747259653015085,135.5200,34.6864
4,117914,0.22794176617882336,121.5250,25.0392
- 15.
- 16. ! EC ( IP) ! (Wi-)
- 17.