Embed Size (px)
Citation preview
Acceptance Testing for DigitalLand Mobile Radio Systems
Presented by:
Jay Jacobsmeyer, P.E.Pericle Communications Company
Karl Reardon, P.Eng.Planetworks Consulting
IWCE 2010 - W07 Acceptance Testing
So What’s New with Digital?Many more systems, software, and technology between the user and their expectations
Users who expect the “same performance …. but better”
Analog Audio Circuits
Modulator
Transmitter
Antennas / Radio Path
Receiver
Demodulator
Analog Audio Circuits
Analog Audio Circuits
Multi-Level Modulator
Transmitter
Antennas / Radio Path
Receiver
Multi-Level Demodulator
Analog Audio Circuits
Audio Vocoder
Data Packetization/Encoding
Data Decoding and Assembly
IP Backhaul Routing / Transport
Audio Vocoder
Audio Vocoder
Analog Digital
Expectations
New
“Radio Was Fun UntilThey Put Software In It”
- Anonymous
Acceptance Test Elements
RF Coverage
System Capacity
Reliability
Resiliency /Fault Recovery
Features and Functionality
Management and Administration
RF Coverage Testing
VocoderIMBE™ by DVSI is the P25 Vocoder– Model-based like virtually all vocoders
Optimized for Low Bit Rate Apps (< 4.8 kbps)
Breakdown– 20 ms frame (50 frames per second)
– 88 vocoder bits
– 56 error control bits (unequal protection)
– 48 signaling bits
– 192 bits x 50 frames/s = 9600 bps
BER Estimator Used for Frame-by-Frame Decisions– Smoothed estimator weighs past, present, future frames
– Has implications for performance metrics
ATP Implications
Four Choices for Each Vocoder Frame– Accept frame (even if some errors detected)– Repeat frame– Adaptively smooth (interpolate) frame– Mute frame
BER is Helpful, But– Frame Error Rate is More to the Point– I.e., the same BER can occur for a variety of
frame error distributions
Multi-Level Modulation
C4FM is a Form of 4-ary FSK
2 Bits per Symbol, Four Possible Carriers
fc
11 10 00 01
1.2 kHz
ATP ImplicationsModulation Fidelity is an Option for Testing
Can Be Alternative to BER– Is a measure of eye closure– Does not require taking a channel out of service– Tracks very well to BER
BER & MFD
RF Coverage PerformanceRF Fundamentals Still Apply to Digital– Mean C/(I+N) in fading must still achieve minimum level
But Digital Introduces New Elements– Ability of forward error correction to deal with error events– Especially error events caused by delay spread– Vocoder performance with bit errors / burst errors– Vocoder performance in presence of audio noise– Channel entry and exit criteria (roaming algorithms)– Infrastructure data routing and assembly algorithms– Backhaul network performance
These New Elements Can Create Poor PerformanceDespite Seemingly Adequate C/(I+N)
Characteristics of the SignalMultipath Fading Dominates the Mobile Radio Channel– Fade rate is a function of the doppler frequency, V/λλλλ– E.g., for 860 MHz at 60 mph, fade rate is roughly 75 Hz– Amplitude is assumed to be Rayleigh-distributed– Channel can be time-dispersive due to delay spread
Radio Specifications are Typically for Mean Signal– Ideally, corresponds to a particular delivered audio quality (DAQ)– Additional margin is needed to operate in fading– Delay spread performance often poorly addressed or ignored
E.g., Typical P25 Phase I Radio (C4FM):– 12.5 kHz channel– Static sensitivity = -118 dBm (5% BER)– Fading sensitivity = -105 dBm (DAQ = 3.4)
Rayleigh Fading(V = 30 mph, f = 850 MHz)
- 4 0
- 3 0
- 2 0
- 1 0
0
1 0
Am
plit
ud
e n
orm
aliz
ed t
o m
ean
, d
B
0 . 0 0 0 . 0 5 0 . 1 0 0 . 1 5 0 . 2 0 0 . 2 5 0 . 3 0
Time in seconds
Multipath Delay SpreadSimulcast Networks Aside, the Signal at the Antenna is ReallyMultiple Reflected Signals with Random Delays– Extent of problem varies with topography
Delay Spread Causes Bit Errors Even When Signal is Strong– Weak correlation with signal strength remains– Line-of-sight paths tend to have lower delay spread
Unlike Simulcast, the Delay Profile is Time-Varying– Adaptive equalizers are effective tool– No guarantee manufacturer is using one, however
I.e., Performance is Implementation-Dependent– And it is unlikely that the mfr. has tested for all possible
environments
Coverage Test Prerequisites
Complete All Radio Site Acceptance Tests– Including site noise, desense, and intermod tests
Verify Backhaul Network Routing and Stability– Backhaul BER, packet drop rates, jitter etc.
Verify Switch Configuration and Fucntionality
Verify Radio Configuration Parameters– Including roaming parameters, channel lists, site
adjacency lists
Understand and Account for Localized Noise andInterference Sources
What to Measure?
RSS = Received signal strengthBER = Bit error rateFER = Frame error rate (= block error rate)DAQ = Digital audio quality
Closest to UserExperience
RSS BER FER DAQFarthest from User Experience
•Method •Pros •Cons
•Signal Level •+ Test can be automated•+ Specialized test equipment not req’d•+ Aligns well with coverage prediction
•- Does not consider localized noise andinterference environments•- Does not verify digital system componentsand overall system operation•- Far removed from user experience (i.e. maynot be reflective of real voice quality)•- Typically only used to measure one pathdirection
•Bit Error Rate or•Frame Error Rate
•+ Test can be automated•+ Good measure of C/(I+N)•+ Verifies some of the digital elements
•- Specialized test equipment required•- Does not verify all system elements•- Somewhat abstract from actual userexperience•- Typically only used to measure one pathdirection
•Voice (DAQ)Testing
•+ Reflects end user experience•+ Incorporates roaming impacts•+ Automated data collection•+ Specialized test equipment not req’d•+ Exercises all system elements
-- Data evaluation is manual , subjective andtime consuming, or-- Automated using expensive software•- Difficult to ‘mine’ data to identify problemorigins (should they exist)
Signal Strength
Signal Strength to be Measured in All Cases– Adequate C/N is necessary, but not sufficient condition– Measured simultaneously with other parameters
Figure of Merit = Mean Signal Power in Fading– Must be linear average, not log average– At least 50 subsamples over at least 40 wavelengths– Fast receiver needed (25 -100 readings per second)– Receiver sensitivity should match or exceed user radio– Also must have good selectivity & dynamic range
For More on This Subject– See Jacobsmeyer & Weimer, “How to Conduct a Drive
Test Survey,” IWCE 2007
Bit Error Rate Testing(Or Frame Error Rate Testing)
Remove RF Channel from Service and Place in Test Mode– May be problematic for busy, operational systems– Need to verify consistent interpretation of the test bit sequence and value
of status bits to avoid baseline errors
Digital analyzer or Instrumented Radio Used for Drive Test– Consider any differences between analyzer or instrumented radios and
the radios which will be used
Collect Data– Configure for sufficient settling time prior to each measurement to
ensure valid BER tests– Traditional sampling rates (> 40 wavelengths) for measurements in fading
environments still apply– At normal speeds, the above sampling period will collect sufficient bit
samples to be statistically valid
Analyze Data– Use processes recommended in TSB-88 to segment and analyze test data
Voice Quality Testing1. Select and Use an Exclusive Talkgroup2. Automated Voice Play and Recording Preferred
– Duration consistent with a typical voice call– Harvard sentences are appropriate– Use both male and female voice recordings
3. Collect Data Uniformly– Collect at least one test recording per tile– Eliminate duplicates per tile by arbitrary rule
4. Score Data– Train listeners for consistent scoring– Score each recording per DAQ definition– Note: Audio scoring criteria should be agreed to prior
to test execution as the TSB -88 definitions subject tointerpretation
DAQ & MOS Definitions
Excellent.Impairmentsimperceptible.
MOS 5Speech easily understood. Occasionalnoise/distortion.
DAQ 4.0
Good.Impairmentsperceptiblebut notannoying.
MOS 4Speech understandable with repetitiononly rarely required. Somenoise/distortion.
DAQ 3.4
Fair. Slightlyannoying.
MOS 3Speech understandable with slight effort.Occasional repetition required due tonoise/distortion.
DAQ 3.0
Poor.Annoying.
MOS 2Understandable with considerable effort.Frequent repetition due to noise/distortion.
DAQ 2.0
Bad. Veryannoying.
MOS 1Unusable. Speech present but notunderstandable.
DAQ 1.0
DefinitionMOSDefinitionDAQ
MOS = Mean Opinion Score. Note that repetitions are not part of definition. No correlationbetween DAQ and MOS is implied by this table.
Scoring Methods
Even With Training, Some Variance Occurs
One Method that Works:– Decide on passing threshold (E.g., DAQ = 3.4)– Pick three scorers:
» Customer operations» Customer technical» Vendor technical
– Score PASS if majority (2 of 3) score above threshold
Decide on Definition of Repeats– Purely subjective?– Or actually allow for a second call at each tile, but how
many repeats are allowed, 10%, 20%?– Or, avoid repeat problem by defining pass as DAQ 4.0
PESQ
Problem: Manual Scoring is Labor-Intensive– Training is required– Variability of scores is tough to eliminate
Possible Solution = PESQ– Perceptual Evaluation of Speech Quality (PESQ)– Standardized as ITU-T Recommendation P.862– Commercial software available (appx. $12k/license)
Issue: PESQ Scores Do Not Match DAQ Scores– Option 1: Agree on mapping from PESQ to DAQ– Option 2: Abandon DAQ and use PESQ score threshold
Audio File Samples
Sample 1
Sample 2
Sample 3
Pass/Fail CriteriaAnd Use of Statistics
Estimate for SAR(Service Area Reliability)
SART
T
where
T is the number of tiles passed
T is the total number of tiles
p
t
p
t
(%) %
= 100
Minimum Number of Samples
ofnz p p
d=
−α / ( )22
2
1
Where p is the value of the SAR, zα/2 is the argument of theunit normal distribution for a confidence of 1-α and d isone-half of the confidence interval [5]. Substitute zα for“greater than” test. This expression results from the normalapproximation to the binomial distribution for large n.
Arguments of Unit Normal
2.58
1.96
1.64
Zαααα/2
1.6495%
99%
90%
Confidence Level
2.33
1.28
Zαααα
Example(d = .02, p = 95%)
ConfidenceWindow Test
Greater
Than Test
788
456
321
n
643
321
195
n
2.58
1.96
1.64
Zαααα/2
1.6495%
99%
90%
Confidence
Level
2.33
1.28
Zαααα
Greater Than test for these parameters means that if the measured SARis at least 97% and we collect at least 643 samples, then we are at least99% confident that the actual SAR is above 95%.
Some Purchasing Pitfalls
Pitfall #1 - Failure to Specify Measurable Performance Standards
Pitfall #2 - Failure to Specify Standards that Measure Digital VoiceQuality Directly– Signal strength is not enough– Digital radio requires subjective measure of voice quality
Pitfall #3 - Failure to Specify the Statistical Test– “Greater Than” or “Confidence Window”?– Number of samples required?
Pitfall #4 - Failure to Include Test Plan in the Contract– Otherwise, endless finger-pointing when things go wrong– Test for the actual user operating states
Summary
An Acceptance Test Plan Must– Use measurable standards,– Specify precisely how each standard is to be measured,– Include the statistical test to be used, and– Be part of the system contract
Clarify the Standards Where Necessary
Other Acceptance TestElements
System Capacity
System Capacity Goes Beyond the Radio Link
Capacity Often Difficult to Measure Directly– Sheer volume of messages– Unavailability of load generators– Distribution of load sources
Scope Dependent on Whether System Is DeliveredAs Components, Subsystems, or Turnkey
Paper Analysis and Discrete System Tests AreOften the Only Practical Method
System Capacity Specs.1. Define the Voice and Data Usage Models
– Use busy hour traffic estimates– Consider fault / system recovery message loads– Watch for coordinated traffic events (shift starts, etc)
which may be outside the normal busy hour
2. Define the Call Models (Group, Individual,Interconnect, Etc.) and Members
3. Define the Number and Approximate Locations
4. Determine Loads Per Channel, Site, Backhaul Link,Router, and System Switch
System Capacity Testing
Distribute Between Factory and System Acceptance Tests
Use Commercial Network Test Tools to Stress BackhaulComponents
Use Development Simulators (if available) to Test SystemLoad Capacity
Appropriate Suite of Tests are Unique to Each System
System Reliability/ Resiliency
Consider Interaction Between All Systems– Momentary packet loss in one subsystem can cause lengthy
reset in another– Identification of error situations which cause ‘thrashing’
(two subsystems with independent retransmissionalgorithms)
– Identification of race conditions
Acceptance Tests Should Be Developed ThroughIdentification and Subsequent Test of Each of theFailure Modes
Features and Functionality
Centralized Conformance Testing for OpenStandards Assists With This Area
However, Conformance Tests Still Not at StageWhere These Tests Can Be Relegated to Others– Subtle differences between operation in different
frequency bands– Different manufacturer interpretations of specifications– Manufacturers opt to implement different options within
the specifications
Management & Administration
Consider Information You Require to Support andManage the Network– Channel, site, link and system traffic, loading and
blocking statistics and reports– Outage and error reports
Acceptance Procedures Should IncludeGeneration of Applicable Reports Using DataGenerated During Execution of Other ATP’s
Consider Need to Consolidate Reports and DataFrom Disparate Subsystems
Final ThoughtThe view is nice but you really should only have to go out once.
Q&A
Early Land Mobile Radio - 1924Source: Bell Labs, http://www.bell-labs.com/history/75/gallery.html
References[1] W.C.Y. Lee, Mobile Communications Design Fundamentals, 2nd Ed.,Wiley, 1993.
[2] W.C.Y. Lee, “Estimate of Local Average Power of a Mobile Radio Signal,”IEEE Transactions on Vehicular Technology, February, 1985, pp. 22-27.
[3] M. M. Peritsky, “Statistical Estimation of Mean Signal Strength in aRayleigh Fading Environment,” IEEE Transactions on Communications,November, 1973, pp. 1207-1213.
[4] G. C. Hess, Land Mobile Radio System Engineering, Artech House, 1993.
[5] R. J. Larsen, M. L. Marx, An Introduction to Mathematical Statistics and itsApplications, Prentice-Hall, 1986, pp. 281.
[6] TIA-TSB-88-C, “Wireless Communications Systems – Performance inNoise and Interference-Limited Situations, etc.” 4/29/2008.
Points of Contact
Jay M. Jacobsmeyer, P.E.Pericle Communications Company1910 Vindicator Drive, Suite 100Colorado Springs, CO 80919(719) 548-1040Fax: (719) [email protected]
Karl Reardon, P.Eng.Planetworks ConsultingSuite 650 - 220 Cambie StreetVancouver, BC, Canada V6B 2M9Main: +1-604-638-3000Fax: [email protected]
