Upload
stephen-thair
View
3.288
Download
4
Embed Size (px)
DESCRIPTION
How to measure web performance covering different techniques, the important of context, real user measurement and synthetic etc
Citation preview
Steve ThairSeriti Consulting
@TheOpsMgr
MEASURING WEB PERFORMANCE
Every measurement of web performance you will ever make will be
wrong
MY DEFINITION OF WEB PERFORMANCE
“The human perception of duration is both subjective and variable”
http://en.wikipedia.org/wiki/Time_perception
“The human perception of duration is both subjective and variable”
“PERCEPTION IS VARIABLE…”
Go read Stoyan’s talk! http://velocityconf.com/velocity2010/public/schedule/detail/13019
Web Performance
Subjective Objective
Subj
ectiv
e“Qualitative techniques”
Case Studies
Focus Groups
Interviews
Video Analysis
Surveys
Obj
ectiv
e“Quantitative techniques”
Javascript
Navigation timing
Browser Extensions
Custom Browsers
Proxy timings
Web Server mods
Network sniffing
“I keep six honest serving-men
(They taught me all I knew);
Their names are What and Why and When
And How and Where and Who.”Rudyard Kipling, The Elephant’s Tale
Journey
Page
Object
WHAT LEVEL DO YOU MEASURE?
CHOOSE YOUR METRIC!
https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html
4 Key “Raw” Metrics• Time to First Byte (TTFB)• Render Start Time• DOMContentLoaded• Page (onLoad) Load Time (PLT)
What about “Above the Fold” time?• How long to “render of the static stuff in
the viewable area of the page”?Limitations of AFT– Only applicable to lab setting– Does not reflect user perceived latency based on functionality
http://assets.en.oreilly.com/1/event/62/Above%20the%20Fold%20Time_%20Measuring%20Web%20Page%20Performance%20Visually%20Presentation.pdf
ApdexStatisticalMetrics
Counts/Histograms
Raw Metrics
WHAT OTHER METRICS?
Apdex (t) = (Satisfied Count + Tolerated Count / 2)
/ Total Samples• A number between 0 and 1 that represents “user satisfaction”
• For technical reasons the “Tolerated” threshold is set to four times the “Satisfied” Threshold so if your “Satisfied” threshold (t) was 4 seconds then:
• 0 to 4 seconds = Satisfied 4 to 16 seconds = Tolerated over 16 seconds = Frustrated.
http://apdex.org/
PERFORMANCE IS MULTI-DIMENSIONAL
Multiple MetricsFor Multiple URLSFrom Different LocationsUsing Different ToolsAcross the LifecycleOver Time
The importance of CONTEXT
ContextTime of Day
Browser
Addons & Extensions
Cached objects
Wired, WiFi, 3G
Location Bandwidth
Latency
Operating System
Antivirus
Device
Resolution
Who? User Experience
DevelopersTesters
WebOps“The Boss”
When?
SDLC
Design (UX)
Develop
Build (CI)QA
Prod Ops
WHERE – DEPENDS ON THE HOW & WHY…Sy
nthe
tic v
ersu
s Rea
l-U
ser
“Real User”
Firewall / Load-Balancer
Web Browser
WiFi or 3G(Reverse) Proxy Server
Network “Sniffer”
Smartphone
InternetProxy Server
SPAN
port
or
Netw
ork t
ap
Web ServerSynthetic Agent
User/Browser metrics Server-based metrics
Signal/Noise Ratio increases….
The Synthetic Versus
Real-User Debate
“…it's a question of when, not if active monitoring of websites for availability and performance will
be obsolete.”- Pat Meenan
“Because you’re skipping the “last mile” between the server and the user’s
browser, you’re not seeing how your site actually performs in the real world”
- Josh Bixby
“You can have my active monitoring when you pry it
from my cold, dead hands…”
- Steve Thair
http://blog.patrickmeenan.com/2011/05/demise-of-active-website-monitoring.htmlhttp://www.webperformancetoday.com/2011/07/05/web-performance-measurement-island-is-sinking/
http://www.seriticonsulting.com/blog/2011/5/21/you-can-have-my-active-monitoring-when-you-pry-it-from-my-co.html
Observational StudyVersus
Experiment
Experiment versus Observational Study• Both typically have the goal of detecting a relationship between the
explanatory and response variables.
Experiment• create differences in the explanatory variable and examine any resulting
changes in the response variable (cause-and-effect conclusion)
Observational Study• observe differences in the explanatory variable and notice any related
differences in the response variable (association between variables)
http://www.math.utah.edu/~joseph/Chapter_09.pdf
• “Watching” what happens in a given population sample
• We can only observe… and try to infer what is actually happening
• Many “confounding variables”
• High signal to noise
• Correlation
Observational Study = Real-User
ContextTime of Day
Browser
Addons & Extensions
Cached objects
Wired, WiFi, 3G
Location Bandwidth
Latency
Operating System
Antivirus
Device
Resolution
• We “design” our experiment
• We chose when, where, what, how etc
• We control the variables (as much as possible)
• Lower signal to noise
• Causation*
• “Watching” what happens in a given population sample
• We can only observe… and try to infer what is actually happening
• Many “confounding variables”
• High signal to noise
• Correlation
Observational Study = Real-User Experiment = Synthetic
* OK, real “root cause” analysis will probably take a lot more investigation,
I admit… but you get closer!
So which one is better? Neither.
Complementary not Competing“…Ultimately I'd love to see a hybrid model where
synthetic tests are triggered based on something detected in the data (slowdown, drop in volume, etc) to
validate the issue or collect more data.- Pat Meenan
Real-User Monitoring detect a change in a page’s performance
API Call to Synthetic
Controlled Test and compare to baseline.
From Observation… To Experiment…By controlling the variables
Use RUM as “Reality Check”
Obj
ectiv
e“Quantitative techniques”
Javascript
Navigation timing
Browser Extensions
Custom Browsers
Proxy timings
Web Server mods
Network sniffing
Back to the “How”…
7 WAYS OF MEASURING WEBPERF
1. JavaScript timing e.g. Souder’s Episodes or Yahoo! Boomerang*
2. Navigation-Timing e.g GA SiteSpeed
3. Browser Extension e.g. HTTPwatch
4. Custom browser e.g. 3pmobile.com or (headless) PhantomJS.org
5. Proxy timing e.g. Charles proxy
6. Web Server Mod e.g. APM solutions
7. Network sniffing e.g. Atomic Labs Pion
COMPARING METHODS…
Metric JavaScriptNavigation-Timing API
Browser Extension
Custom Browser
Proxy Debugger
Web Server Mod
Network sniffing
Example Product WebTuna SiteSpeed HTTPWatch 3PMobileCharles Proxy
APM Modules
Pion
"Blocked/Wait" No No Yes Yes Yes No NoDNS No Yes Yes Yes Yes No No
Connect No Yes Yes Yes Yes No YesTime to First Byte Partially Yes Yes Yes Yes Yes Yes
"Render Start" No No Yes Yes No No NoDOMReady Partially Yes Yes Yes No No No"Page/HTTP Complete"
Partially Yes Yes Yes Yes No Partially
OnLoad Event Yes Yes Yes Yes No No NoJS Execution Time Partially No Yes Yes No No No
Page-Level Yes Yes Yes Yes Partially Partially PartiallyObject Level No No Yes Yes Yes Yes Yes
Good for RUM? Yes Yes Partially No No Partially YesGood for Mobile? Partially Partially Partially Partially Partially Partially Partially
Affects Measurement Yes No Yes Yes Yes Yes No
Measurement Method
JAVASCRIPT TIMING – HOW IT WORKS
unLoad Event var start = new
Date().getTime()Stick it in a Cookie Load the next page
onLoad Eventvar end = new
Date().getTime()
PLT = end - start
Send a beaconbeacon.gif?time=plt
https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html
PROS & CONS OF JAVASCRIPT TIMING
• Pro’s
• Simple
• Episodes/Boomerang provide custom timing for developer instrumentation
• Cons
• Relies on Javascript and Cookies
• Only accurate for 2nd page in journey
• Can only really get a “page load metric” and a partial TTFB metric
• “Observer effect” (and Javascript can break!)
Metric JavaScript
Example Product WebTuna
"Blocked/Wait" NoDNS No
Connect NoTime to First Byte Partially
"Render Start" NoDOMReady Partially"Page/HTTP Complete"
Partially
OnLoad Event YesJS Execution Time Partially
Page-Level YesObject Level No
Good for RUM? YesGood for Mobile? Partially
Affects Measurement Yes
NAVIGATION-TIMING – HOW IT WORKS
onLoad Eventvar end = new
Date().getTime()
var plt = now - performance.timing.
navigationStart;
Send a beaconbeacon.gif?time=plt
NAVIGATION TIMING METRICS
https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html
PROS & CONS OF NAVIGATION-TIMING• Pro’s
• Even simpler!
• Lots more metrics
• More accurate
• Cons
• Need browser support for API
• IE9+ / Chrome 6+ / Firefox 7+
• Relies on Javascript (for querying API & beacon)
• “Observer effect”
• Page-level only
MetricNavigation-Timing API
Example Product SiteSpeed
"Blocked/Wait" NoDNS Yes
Connect YesTime to First Byte Yes
"Render Start" NoDOMReady Yes"Page/HTTP Complete"
Yes
OnLoad Event YesJS Execution Time No
Page-Level YesObject Level No
Good for RUM? YesGood for Mobile? Partially
Affects Measurement No
A BIT MORE ABOUT GA SITESPEED… • Just add one line for basic, free, real-user monitoring!
_gaq.push(['_setAccount', 'UA-12345-1']);_gaq.push(['_trackPageview']);_gaq.push(['_trackPageLoadTime']);
• Sampling appears to vary (a lot!)
• 10% of page visits by design but reported 2% to 100%
• Falls back to Google Toolbar if available (but NOT javascript timing)
• Will probably make you think perf is better than it really is…
10/04/2023(C) SERITI CONSULTING, 2011 41
10/04/2023(C) SERITI CONSULTING, 2011 42
10/04/2023(C) SERITI CONSULTING, 2011 43
BROWSER EXTENSION – HOW IT WORKS
Write a browser extension…
That subscribes to a whole lot of API event listeners…
https://developer.mozilla.org/en/XPCOM_Interface_Reference
PROS & CONS OF BROWSER EXTENSIONS
• Pros
• Very complete metrics
• Object and Page level
• No javascript (in the page at least)!!!
• Great for continuous integration perf testing
• Cons
• Getting users to install it…
• Not natively cross-browser
• Some browsers don’t support extensions
• Especially mobile browsers!
• “Observer effect”
MetricBrowser
Extension
Example Product HTTPWatch
"Blocked/Wait" YesDNS Yes
Connect YesTime to First Byte Yes
"Render Start" YesDOMReady Yes"Page/HTTP Complete"
Yes
OnLoad Event YesJS Execution Time Yes
Page-Level YesObject Level Yes
Good for RUM? PartiallyGood for Mobile? Partially
Affects Measurement Yes
CUSTOM BROWSER – HOW IT WORKS
Take some open source browser
code
Like WebKit or the Android Browser
PROS & CONS OF CUSTOM BROWSER
• Pros
• Great when you can’t use extensions / javascript / cookies ie. For mobile performance e.g. 3Pmobile.com
• Great for automation e.g. http://www.PhantomJS.org/
• Good metrics (depending on OS API availability)
• Cons
• Requires installation
• Maintaining fidelity to “real browser” measurements
• “Observer Effect” (due to instrumentation code)
MetricCustom Browser
Example Product 3PMobile
"Blocked/Wait" YesDNS Yes
Connect YesTime to First Byte Yes
"Render Start" YesDOMReady Yes"Page/HTTP Complete"
Yes
OnLoad Event YesJS Execution Time Yes
Page-Level YesObject Level Yes
Good for RUM? NoGood for Mobile? Partially
Affects Measurement Yes
PROXY DEBUGGER – HOW IT WORKSChange browser to use debugging Proxy e.g.
Charles or Fiddler
Debugging proxy records each
request
Export data to log
PROS & CONS OF PROXY DEBUGGER
• Pros
• One simple change to browser config
• No Javascript / Cookies
• Can offer bandwidth throttling
• Cons
• Proxies significantly impact HTTP traffic
• http://insidehttp.blogspot.com/2005/06/using-fiddler-for-performance.html
• No access to browser events
• Concept of a “page” be problematic…
MetricProxy
Debugger
Example ProductFiddler Proxy
"Blocked/Wait" YesDNS Yes
Connect YesTime to First Byte Yes
"Render Start" NoDOMReady No"Page/HTTP Complete"
Yes
OnLoad Event NoJS Execution Time No
Page-Level PartiallyObject Level Yes
Good for RUM? NoGood for Mobile? Partially
Affects Measurement Yes
6 Keep-Alive connections per SERVERVersus
8 Keep-Alive connections TOTAL per PROXY(Firefox 7.0.1)
WEB SERVER MOD – HOW IT WORKS
Write a webserver Mod or ISAPI filter
Start a timer on Request
http://www.apachetutor.org/dev/request
PROS & CONS OF WEB SERVER MOD
• Pros
• Great for Application Performance Management (APM)
• Can be used in a “hybrid mode” with Javascript timing
• Measuring your “back-end” performance
• Can be easy to deploy*
• Cons
• Limited metrics, ignores network RTT and only sees origin requests
• “Observer Effect” (~5% server perf hit with APM?)
• Concept of a “page” be problematic…
• Can be a pain to deploy*
MetricWeb Server
Mod
Example ProductAPM
Modules"Blocked/Wait" No
DNS NoConnect No
Time to First Byte Yes"Render Start" No
DOMReady No"Page/HTTP Complete"
No
OnLoad Event NoJS Execution Time No
Page-Level PartiallyObject Level Yes
Good for RUM? PartiallyGood for Mobile? Partially
Affects Measurement Yes
NETWORK SNIFFING – HOW IT WORKS
Create a SPAN port or network
tap
Promiscuous mode packet
sniffing
PROS & CONS OF NETWORK SNIFFING• Pros
• No “observer effect” (totally “passive”)
• Very common “appliance-based” RUM solution
• Can be used in a “hybrid mode” with Javascript timing
• Can be easy to deploy*
• Cons
• Limited metrics and only sees origin requests
• Not “cloud friendly” at present
• Concept of a “page” be problematic…
• Can be a pain to deploy*
MetricNetwork sniffing
Example Product Pion
"Blocked/Wait" NoDNS No
Connect YesTime to First Byte Yes
"Render Start" NoDOMReady No"Page/HTTP Complete"
Partially
OnLoad Event NoJS Execution Time No
Page-Level PartiallyObject Level Yes
Good for RUM? YesGood for Mobile? Partially
Affects Measurement No
SUMMARY
• Performance is subjective (but we try to make it objective)
• Performance is Multi-dimensional
• Context is critical
• “Observational Studies AND Experiments”
• Real User Monitoring AND Synthetic Monitoring
• 7 different measurement techniques each with Pros & Cons
@LDNWEBPERF USER GROUP!• Join our London Web Performance Meetup
• http://www.meetup.com/London-Web-Performance-Group/
• Next Wednesday 16th Nov - 7pm – London (Bank)
• WPO case study from www.thetimes.co.uk!
• Follow us on Twitter @LDNWebPerf
• #LDNWebPerf & #WebPerf
QUESTIONS?
http://mobro.co/TheOpsMgr