Embed Size (px)
Citation preview
TALLINN UNIVERSITY OF TECHOLOGY
Faculty of Information Technology
Code: IRT70LT
QoS Implementation on Network Devices
Master Thesis
Student: Konstantin Muhhin
Student‟s code: 990715 IVIM
Instructor: Avo Ots
Presented: 3.06.2010
Tallinn 2010
2
REFERAAT
Käesolev magistritöö “Teenusekvaliteedi tagamise meetodid võrguseadmetes” käsitleb
erinevaid teenuse kvaliteedi garanteerimise (QoS) meetodeid, millega lahendatakse IP liikluse
probleeme, mis on seotud ülekoormatud andmeside kanalitega.
Töö eesmärgiks oli välja selgitada millist tüüpi teenuse kvaliteedi tagamise meetodid on
olemas ja millised sobivad paremini IP liikluse töötlemiseks.
Võrgusimuleerimise tarkvara katsetamise käigus saadud tulemused näitasid, et ratsionaalne ja
läbimõeldud IP liikluse juhtimine aitab vältida IP pakettide kadumise probleeme ülekoormatud
andmeside kanalites.
Töö on kirjutatud inglise keeles 80 leheküljel ning sisaldab 63 joonist ja 5 tabelit.
Võtmesõnad: QoS, congested channel, packets drops, congestion management, fair queuing,
packets queuing, packets treatment, packets scheduling.
3
ABSTRACT
The master‟s thesis paper “QoS implementation on network devices” examines different types
of QoS that could be implemented on a congested or narrow customer‟s IP channels.
The main objective of work is to consider what kind of congestion control mechanisms exist
and how they can be deployed on network devices to minimize or eliminate congestion for
important IP traffic flows. I haven‟t only described some basic principles of congestion
avoidances and control mechanisms, but also shown how these mechanisms work, using
network simulation environment and gave a brief analysis of some solutions used in a practical
part.
The thesis paper is written in English and consists of 80 pages and contains 63 figures and 5
tables.
Keywords: QoS, congested channel, packets drops, congestion management, fair queuing,
packets queuing, packets treatment, packets scheduling.
4
АННОТАЦИЯ
Данная дипломная работа на тему “Реализация методов гарантированного
обслуживания на сетевых устройствах” представляет собой рассмотрение различных
методов качества обслуживания, которые могут быть применены на загруженных или
ограниченных по полосе пропускания IP каналах.
Целью работы являлось ознакомление с различными методами качества обслуживания
(QoS) и различными вариантами их применения на сетевых устройствах. Используя
симулятор построения сетей, я попытался показать различные методы
конфигурирования сетевого оборудования, а также методы обработки интересующего
трафика на загруженных различным IP трафиком коммуникационных каналах. Из
полученных результатов был сделан вывод, что грамотное и рациональное
использование технологии гарантированного обслуживания способствует уменьшению
загруженности канала для критических типов трафика.
Дипломная работа написана на английском языке и состоит из 80 страниц, включая 63
иллюстраций и 5 таблиц.
Ключевые слова: QoS, congested channel, packets drops, congestion management, fair
queuing, packets queuing, packets treatment, packets scheduling.
5
ACKNOWLEDGEMENT
The idea of choosing the theme for my master‟s thesis came to me while working in Estonian
Communication Company. We deal with design of different communication solutions, based
on IP and other types of communication technologies. Our company provides different
connection types based on a various physical types of local loops (radio, copper or optical
lines). Some of our customers take serious care about their connections and monitor every
byte going through the IP channels we provide. Every problem connected with channel
congestion perceives on a very serious level. Other customers start to think about their
connection quality only after congestion has already occurred and channel overflowed. As a
rule they try to solve this problem with existent channel upgrading (channel bandwidth
expansion). Yes, it‟s one of the easiest ways to avoid congestions on channels, but not
absolute. Sometimes data channel or local loops wouldn‟t be easily upgraded due to different
reasons. New upgraded local loop connection cost can dramatically change the enterprise
budget or require a long time to build new connection link (as an example optical cable).
I think that theme raised in my thesis is rather urgent and actual timely, not only for the end
customers, who use ISP local loops as a connection channel between their and ISP networks,
but also for communication Service Providers, who are interested in guaranteeing the quality
of provided services. During familiarization period I have read a lot of materials, which helped
me understand the basic principles of QoS methods, and showed that QoS theme is rather wide
and can be considered from different points of view. I have chosen the part of network, where
the channel congestion appearance probability is rather high.
In my master thesis I tried to show how problems with slow or congested channels could be
solved or partially minimized without data channels (local loops) upgrading, which can be
costly and time consuming for the customer.
I also would like to thank my thesis paper instructor, who has accepted the idea of my work
and further supported and helped me with thesis writing.
Tallinn, May 31, 2010
Konstantin Muhhin
6
TABLE OF CONTENTS
REFERAAT ................................................................................................................................ 2
ABSTRACT ................................................................................................................................ 3
АННОТАЦИЯ ........................................................................................................................... 4
ACKNOWLEDGEMENT .......................................................................................................... 5
TABLE OF CONTENTS ............................................................................................................ 6
ILLUSTRATION INDEX .......................................................................................................... 8
TABLES ................................................................................................................................... 10
GLOSSARY ............................................................................................................................. 11
INTRODUCTION .................................................................................................................... 13
MASTER THESIS IDEA AND STRUCTURE ....................................................................... 14
1. QOS AS STRUGGLE METHOD WITH LINK CONGESTION ........................................ 16
1.1 INTERNET AS A MIX OF MULTITUDE TRAFFIC TYPES ..................................... 16
1.2 CONGESTION IS THE MAIN REASON OF USAGE QOS ........................................ 17
2. IP PACKETS TREATMENT METHODS........................................................................... 22
2.1 PACKETS THREATMENT POSIBILITIES ................................................................. 22
2.2 BASIC PACKETS DESCROPTORS ............................................................................. 24
2.3 CLASSIFICATION AND MARKING AT LAYER 2 (OSI) CONCEPT...................... 25
2.4 CLASSIFICATION AND MARKING AT LAYER 3 (OSI) CONCEPT...................... 27
2.5 IP PACKETS MATCHING, MARKING AND CLASSIFICATON METHODS ......... 31
2.6 CLASSIFICATION AND MARKING ON MPLS LEVEL ........................................... 33
3. QUEUING AS CONGESTION MANAGEMENT MECHANISM .................................... 34
3.1 BASIC PRINCIPLES OF QUEUING ............................................................................ 34
3.2 FIFO – FIRST AND THE SIMPLIEST QUEUING MECHANISM ............................. 35
3.3 PRIORITY QUEUING STRATEGY ............................................................................. 35
3.4 CUSTOM QUEUING STRATEGY ............................................................................... 36
3.4.1 WEIGHTED ROUND ROBIN QUEUING ............................................................. 37
3.5 WEIGHTED FAIR QUEUING (WFQ) .......................................................................... 38
3.5.1 CLASS-BASED WEIGHTED FAIR QUEUING .................................................... 39
3.6 LOW LATENCY QUEUING MECHANISM ............................................................... 40
4. CONGESTION AVOIDANCES MECHANISMS .............................................................. 42
4.1 TAIL DROPPING – EASIAST RESPONSE TO CONGESTED INTERFACE ........... 42
4.2 RANDOM EARLY DETECTION MECHANISM (RED) ............................................ 42
4.3 WEIGHTED RANDOM EARLY DETECTION MECHANISM .................................. 44
4.4 CLASS-BASED WRED ................................................................................................. 45
4.5 DSCP-BASED WRED.................................................................................................... 46
4.5.1 DSCP-BASED WRED CONFIGURATION EXAMPLE ....................................... 46
4.6 TRAFFIC POLICING AND SHAPING ......................................................................... 48
5. BASIC STEPS OF QOS IMPLEMENTATION .................................................................. 50
7
6. PRACTICAL PART /CONFIGURING AND QOS IMPLEMENTING EXAMPLES/ ...... 54
6.1 PRACTICAL TASK 1. /MATCING AND PRIORITAZING TRAFFIC/ ..................... 54
6.2 PRACTICAL TASK 2 /TRAFIC POLICING AND SHAPING METHODS/ ............... 58
6.3 PRACTICAL TASK 3 /PRIORITIZING ICMP PACKETS/ ......................................... 61
6.4 PRACTICAL TASK 4 /MARKING PACKETS WITH DSCP VALUE/ ...................... 67
6.5 PRACTICAL PART ANALYSIS................................................................................... 74
CONCLUSION ......................................................................................................................... 77
LIST OF REFERENCES .......................................................................................................... 79
8
ILLUSTRATION INDEX
Master thesis structure .............................................................................................................. 14
Figure 1. Internet traffic growth and variety [1] ....................................................................... 16
Figure 2. Internet users in the world [16] ................................................................................. 17
Figure 3. Traffic congestion on Router output Interface [7] ..................................................... 18
Figure 4. Congestion occurrence points [17] ............................................................................ 19
Figure 5. Bandwidth and Delay parameters comparing with water pipe [3] ........................... 20
Figure 6. IP Packet treatment methods [4] .............................................................................. 22
Figure 7. Layer 2 Data packet structure (with CoS) [6] .......................................................... 26
Figure 8. Typical usage of CoS on trunk (dot1.q) interface of the switches or routers [17] ... 26
Figure 9. IP packet structure (with ToS and DSCP) [6] .......................................................... 28
Figure 10. IPP and DSCP compatibility [17] ........................................................................... 29
Figure 11. DSCP bits in IP header [7] ...................................................................................... 29
Figure 12. Common QoS mechanism between ISPs [17] ........................................................ 31
Figure 13. Packets inspection with NBAR [7] ......................................................................... 32
Figure 14. MPLS marking with EXP bits [7] ........................................................................... 33
Figure 15. Simplest queuing mechanism FIFO [7]................................................................... 35
Figure 16. Priority queuing (PQ) [7] ........................................................................................ 36
Figure 17. Custom Queuing (CQ) [7] ....................................................................................... 37
Figure 18. WRR example [7] .................................................................................................... 37
Figure 19. Weighted Fair Queuing (WFQ) mechanism [7] ...................................................... 38
Figure 20. CBWFQ architecture [7] ......................................................................................... 39
Figure 21. LLQ – combining queuing mechanism [7] ............................................................. 40
Figure 22. LLQ Architecture [7] ............................................................................................... 41
Figure 23. Tail dropping mechanism [7] .................................................................................. 42
Figure 24. TCP flow before and after RED implementation [7] .............................................. 43
Figure 25. CBWRED mechanism [7] ....................................................................................... 45
Figure 26. DSCP based WRED (EF-Expedited forwarding) [7] .............................................. 46
Figure 27. Packets drops probabilities [17] .............................................................................. 47
Figure 28. Configured classes WRED parameters ................................................................... 48
Figure 29. Differences between traffic Policing and Shaping [13] .......................................... 48
Figure 30. Classical QoS duties distribution on a network [17] ............................................... 52
Figure 31. Networks connection design (L3VPN) between two Enterprise offices [17] ......... 54
Figure 32. Connection between two Enterprise offices in GNS environment .......................... 55
Figure 32. Class-maps configuration in GNS3 environment .................................................... 55
Figure 33. Policy-map configuration in GNS environment ...................................................... 56
Figure 34. Output interface policy-map configuration ............................................................. 56
Figure 35. Matching voip packets ............................................................................................. 58
Figure 36. Strict priority for voip class-map ............................................................................. 58
9
Figure 37. Connection link congestion by R1 router ................................................................ 59
Figure 38. ICMP packets drops on a connection link ............................................................... 59
Figure 39. Two steps of traffic output policing ........................................................................ 59
Figure 40. Packet drop eliminating by class-based shaping mechanism .................................. 60
Figure 41. Policing and shaping configuration ......................................................................... 60
Figure 42. GNS network diagram ............................................................................................. 61
Figure 43. IP NBAR protocol-discovery statistic ..................................................................... 62
Figure 44. IP NBAR protocol-discovery statistic on a congested link (output interface) ........ 63
Figure 45. Latency difference between congested and uncongested situation ......................... 63
Figure 46. Policy-map configuration result .............................................................................. 64
Figure 47. QoS influence in ideal case ..................................................................................... 64
Figure 48. ICMP packets behaviour ......................................................................................... 65
Figure 49. Two types of policy-maps ....................................................................................... 66
Figure 50. ICMP packets behavior ........................................................................................... 66
Figure 51. Packets marking strategy network diagram [17] ..................................................... 68
Figure 52. Example strategy for traffic classification and markings on ISP level [7] .............. 69
Figure 53. Enterprise network traffic marking in GNS enviroment ......................................... 70
Figure 54. Router R3 incoming statistic ................................................................................... 70
Figure 55. Router R3 incoming statistic on a congested incoming interface ........................... 70
Figure 56. Traffic matching configuration ............................................................................... 71
Figure 57. Setting DSCP values to each traffic class ............................................................... 71
Figure 58. IP packets marking process ..................................................................................... 72
Figure 59. Matching and scheduling already marked packets .................................................. 72
Figure 60. Scheduling process on a router WAN interface ...................................................... 73
Figure 61. Accounting process on CE and PE routers .............................................................. 73
Figure 62. Some congestion control methods comparison [17] ............................................... 75
Figure 63. QoS startegy by various connection types [17] ....................................................... 75
10
TABLES
Table 1. Commonly used Layer 2 marking .............................................................................. 27
Table 2. DSCP and EXP values compatibility ......................................................................... 33
Table 3. IP traffic classes approximation .................................................................................. 68
Table 4. ISP QoS strategy and customer output policy ............................................................ 69
Table 5. Enterprise‟s network inner policy ............................................................................... 71
11
GLOSSARY
ACL - Access-list
AF - Assured Forwarding
BE - Best Effort
CBWFQ - Class-Based Weighted Fair Queuing
CBWRED - Class-Based Weighted Random Early Detection
COS - Class of Service
CPE - Customer-Premises Equipment
CQ - Custom Queuing
DiffServ - Differentiated Services
DSCP - Differentiated Services Code Point
DSL - Digital Subscriber Line
GTS - Generic Traffic Shaping
EF - Expedited Forwarding
EIGRP - Enhanced Interior Gateway Routing Protocol
FIFO - First In, First Out
ICMP - Internet Control Message Protocol
IntServ - Integrated Services
IP - Internet Protocol
IPTV - Internet Protocol Television
ISP - Internet Service Provider
LAN - Local Area network
L3VPN - Layer 3 Virtual Private Network
LLQ - Low Latency Queuing
MCA - Mission Critical Application
MPLS - Multi Protocol Label Switching
MQC - Modular QoS Command Line Interface
NBAR - Network Based Application Recognition
NNI - Network-to-Network Interface
PE - Provider Edge
PHB - Per Hop Behavior
12
PQ - Priority Queuing
PSTN - Public Switched Telephone Network
QOS - Quality of Service
RED - Random Early Detection
RSVP - Resource Reservation Protocol
TCP - Transmission Control Protocol
TOS - Type of Service
UDP - User Datagram Protocol
VOD - Video on Demand
VPN - Virtual Private Network
VoIP - Voice over IP
WFQ - Weighted Fair Queuing
WRED - Weighted Random Early Detection
WRR - Weighted Round Robin
13
INTRODUCTION
Nowadays telecommunication as well as IT technologies present a rather rapid level of
development and it is very hard to trace all the changes which take place in IT and
telecommunication world. Recourses of Internet are growing wider and more customers have
possibilities to have a rather wide broadband access to the Internet and separate connections
between branch offices (secure connections). But some small business enterprises don‟t have
possibilities to have a wide Internet or LAN‟s connection‟s (L3VPN) local loop links. In my
master‟s thesis I would like to touch upon that theme, which is very sensitive for enterprise
networks that use the IP protocol as transport for Internet or other data communication
channels. As I have previously mentioned, IT-world become wider and wider, emerging with
different types of applications, which usually need to be transmitted through the Internet or
other types of IP channels. As practice shows networks with a narrow bandwidth of WAN
links very often overload with IP traffic going through local loop channel and some
applications start working with problem (packets delays or packets drops appearance).
One possible solution could be elimination of any congestion on a local loop channel with a
allowing to IP packets as much channel resources as it needed. It is a good solution, however,
as I have previously mentioned not the ideal especially nowadays, where companies are trying
to constrain their budget.
Another possible solution is trying to take occurred congestion under control. Firstly we need
to understand why data channel congestion occurs? What kind of traffic disturbs the normal
IP channel work? Perhaps some types of traffics are not as significant and could be temporary
eliminated or put in queue (buffered or prioritized). So the first step of problem solution is
trying to clear up the traffic types going through the data channel and then determine which
types of traffic are critical and need to be scheduled or sending first and which traffic is
tolerant to delays or packet drops. Only with correct and proper traffic types differentiation
congested IP channel characteristic improvement can be achieved and originated congestion
could be taken under control.
14
MASTER THESIS IDEA AND STRUCTURE
The Main idea of my master thesis work is to find solutions, which could prevent congestions
or minimize it influence on a slow IP data channels. In order to find a better better solution I
have tried to understand why congestion occurs on an IP channels in general and if congestion
could be prevented or avoided by using different types of Quality of Service mechanisms
(QoS). In a first introductory part of my work I tried to present some reasons of congestion
appearance as well as Qos implementation necessity and significance. In a middle part of my
work I have focused on different mechanisms of packet treatment, also were considering basic
ideas of packets classifying and marking on a different OSI levels (data link layer and network
layer). Further I gave definition to several types of congestion management and congestion
avoidance mechanisms. In a practical part I tried to show how different QoS methods can be
implemented and configured on a network devices and how these methods can eliminate or
reduce congestion on IP channels for some important or interested traffic types. Figure below
briefly illustrates the approximate structure of presented thesis.
Master thesis structure
15
In presented work I have focused on enterprises WAN connections, which connect with the
other brunch offices LAN networks through the service provider network resources.
Popularity of these private network solutions nowadays grows due to the strict security
requirement and simplicity of it realization (L3VPN networks).
In the final part I tried to give brief analysis of each used congestion management methods,
and presented some advantages and drawbacks of each.
To configure and test some of the QoS technique I used GNS3 software (Graphical Network
Simulator). It is freeware software, which allows design different types of network diagrams
as well as test them, like it would deployed in a real network environment. I used next Cisco
Internetwork Operating Systems (IOS): Cisco 3725 - 3700 Software (C3725-
TPGEN+IPBASE-M), Experimental Version 12.4, Cisco 2691 - 2600 Software (C2691-
ENTSERVICESK9-M), Version 12.4(13b). Cisco IOS versions were used only for
acquaintance and not for further distribution. I also used PRTG Traffic Grapher (V6.2.2.983),
which helped me collect information about router interfaces congestion and allowed monitor
latency for ICMP packets.
16
1. QOS AS STRUGGLE METHOD WITH LINK CONGESTION
1.1 INTERNET AS A MIX OF MULTITUDE TRAFFIC TYPES
During last year‟s interpretation of Internet dramatically changed. Let‟s recall, that Internet
has evolved from a medium used primarily for research and by academia to a global
infrastructure used by business and user communities. Traditional IP networks employ a best-
effort approach in service delivery, where the network attempts to deliver as many packets as
possible, without ensuring any particular performance levels. The best-effort (BE) paradigm
has proved to be very successful as long as the applications using the network resources are
not sensitive to delay or packet‟s loss and the load of the network are relatively small. The
fundamental design principle of the best-effort Internet is to keep the core network simple and
move the “intelligence” into end systems. The resulting the evolution of Internet and IT field
in general has brought new challenges. A variety of new applications have emerged, with
requirements that are different from those for which the Internet was originally designed. Let‟s
see the figure, where we can see how Internet traffic was grown during five last years and how
it will rise further.
Figure 1. Internet traffic growth and variety [1]
The figure above reflects interesting facts about how different types of traffic are distributed
along the whole traffic contributed by users. We can notice, how changing the presence of
real-time or time sensitive types of traffic in time scale. Of course we see growth and it will
grow further due the popularity of multimedia applications being within the Internet.
Another aspect that brings challenges of changing the fundamental principles of Internet is the
popularity. Every year more and more people start using recourses of Internet, what in one‟s
turn brings to the Network overflow. The figure below reflexes the popularity of Internet over
the world.
17
Figure 2. Internet users in the world [16]
As we can mention by year 2010 (nowadays) the amount of Internet users already reached 1.6
billion. It is around 24.2% of the total population of the world. And how we can observe the
further growing resources are rather actual perspective. Because of growing the popularity of
Internet and appearing more and more multimedia and other time-sensitive applications, which
are became the part of Internet or any other network resources (L3VPN connections), the
certain services need to be more predictable and quality-based. Consequently, researches have
been trying to redesign Internet so that it can simultaneously support different applications
types and satisfy their diverse service requirements. This brought forth the concept of Quality
of Service for Networking and Internet.
1.2 CONGESTION IS THE MAIN REASON OF USAGE QOS
Today with the increasing speed of data connection channels increases the amount of different
traffic types going through those channels as well. As we already know some types of traffic
are extremely sensitive to delays, packet losses and jitters, and these critical recruitments are
the main reason of the qualitative IP traffic transport necessity. Of course, how we can guess, I
talk about video and voice traffic, which is become more and more transportable through the
IP networks. Therefore the task of providing quality transport of time-sensitive traffic types
18
become more popular due increasing amount of voice and video traffic on regular IP channels
(side by side with a regular data traffic).
The necessity in real-time media formats such as audio, video, and gaming has certainly
increased the existing bandwidth requirements, as has applications development, embedding
more media-rich functionality in basic desktop programs such as word processing and yielding
larger files. Real-time audio- and videoconferencing over data networks have created a need
for the same real-time quality-of-service (QoS) guarantees that the telecom industry has
enjoyed since the inception of digital telephony. Ultimately the goal of a QoS framework is to
ensure that data are transferred in a deterministic manner that at least meets the performance
requirements of each service being delivered [15].
For over a decade, the Internet community has developed a lot of research efforts investigating
various approaches and mechanism in order to offer better network services, than the existing
best-effort [15]. But let‟s start with the simple definition of congestion occurrence. As we can
all understand if the channel is empty or some free space is available for certain traffic we
don‟t need any special treatment mechanism to provide efficiency for interesting traffic. So
need of implementing QoS mechanisms is actual only when congestion occurs or in other
words: “link congestion cause the main reason of QoS implementation on a IP networks”
Figure 3. Traffic congestion on Router output Interface [7]
On a figure above we can see, how different types of IP packets are arriving to the LAN
interfaces (virtual or physical) of the router and also we see how these packets exit the router
through the WAN interface. It‟s not hard to guess, that capacity of WAN interface is limited
and not enough to treat all incoming packets. So the situation, when the network device
interface doesn‟t have enough link capacitance called congestion (interface or link
congestion). A simple analogy with congestion on a communication channel would be pouring
water from one bottle to another. If we pour water into the second container faster than the
neck can accommodate, the water will overflow and run down the side of the bottle. We can
temporarily solve the problem by pouring water into a funnel, which would hold the extra
water, but eventually, if we pour the syrup quickly, the funnel will fill up and overflow as
well.
19
Let‟s try to differentiate the main reasons, which can causes congestion on an enterprise or
corporate Network.
Congestion can be caused by Speed Mismatch
Speed mismatches are the most common reason for congestion. It is possible to have persistent
congestion when traffic is moving from a LAN to a WAN, such as when traffic moves from a
high-speed LAN environment (100Mbps or 1Gbps) to lower-speed WAN links (for example
1Mbps or 10 Mbps). Speed mismatches are also common in LAN-to-LAN environments
when, for example, a 1000-Mbps link feeds into a 100-Mbps link. [7]
Congestion can be caused by Aggregation
The second most common source of congestion is points of aggregation in network. Typical
points of aggregation occur in WANs when multiple remote sites feed into a central site. In a
LAN environment, congestion resulting from aggregation often occurs at the distribution layer
of networks where the access layer devices feed traffic to the distribution layer switches (as
shown on Figure 4)
Figure 4. Congestion occurrence points [17]
But to start talk about IP congestions and congestion preventing methods let‟s investigate the
most important network traffic characteristics, which are used if someone starts to discuss IP
packets transport. Recall the main parameters, which are used to analyze and describe IP
traffic on a network:
Bandwidth (BW) - is a measure of available or consumed data communication resources
expressed in bit/s or multiples of it (kbit/s, Mbit/s etc).
20
Delay - specifies how long it takes for a bit of data to travel across the network from one node
or endpoint to another.
Jitter - delay‟s fluctuation or variation, when packets travel through the data channel
Packet Loss - Specifies the number of packets dropped by the network during transmission.
The channel bandwidth definition is very close with definition of water pipe, where the
bandwidth is the width of the pipe, and delay is the length.
Figure 5. Bandwidth and Delay parameters comparing with water pipe [3]
The time IP package transfer through the channel - Transmit time [s] = packet size [bytes] /
BW [bytes/s]. For example, lets calculate the IP packet (IP packet size: 64kbyte) transmitting
time through the 64kbps bandwidth channel.
BW=64000 bit/s,
Packet size (PS) = 64*8=512 (bit)
Transmit Time = 512/64000 = 0.008 s
Logically, that packet transmitting time will be increased due the IP-packet size is increase as
well. But if we assume, that IP packet needs to be transported from source to destination
through the different SP networks, through the dozens of network equipment and
communication channels the IP packet transmitting time can be dramatically increased or in a
worse case packet can be dropped.
In conclusion of the first introductory chapter I would like present three basic QoS paradigms
that can be used on modern IP networks.
Best-effort model (BE): The best-effort model does not use QoS. If it is not important
when or how packets arrive. The best-effort model treats all network packets in the
same way, so an emergency voice message is treated the same way a digital
photograph attached to an e-mail is treated.
21
Integrated services (IntServ): IntServ can provide very high QoS to IP packets.
Essentially, IntServ defines a signaling process for applications to signal to the network
that they require special QoS for a period and that bandwidth should be reserved. With
IntServ, packet delivery is guaranteed. However, the use of IntServ can severely limit
the scalability of a network. IntServ (hard QoS) uses signaling in which the end-hosts
signal their QoS needs to the network. Although IntServ is capable of providing a very
strong service semantic, maintaining per-flow state information induces major
disadvantages. Most of all, a significant processing overhead is incurred in each router,
as it has to maintain state information for each flow. The resulted lack of scalability
hampers the adoption of IntServ on a large generalized scale. [15]
Differentiated services (DiffServ): DiffServ provides the greatest scalability and
flexibility in implementing QoS in a network. Network devices recognize traffic
classes and provide different levels of QoS to different traffic classes. The DiffServ
design overcomes the limitations of both the best-effort and IntServ models. The
DiffServ model is described in Internet Engineering Task Force (IETF) RFC 2474 and
RFC 2475. DiffServ can provide an “almost guaranteed” QoS while still being cost-
effective and scalable. For example, DiffServ groups all TCP flows as a single class,
and allocates bandwidth for that class, rather than for the individual flows as hard QoS
(IntServ) would do. In addition to classifying traffic, DiffServ minimizes signaling and
state maintenance requirements on each network node. [7]
22
2. IP PACKETS TREATMENT METHODS
2.1 PACKETS THREATMENT POSIBILITIES
To decrease the probability of packets drops and make IT packets transit through the network
more reliable many of different methods and algorithms were developed. Firstly I would like
to show, which types of IP packet treatments can by employed on network devices.
Figure 6. IP Packet treatment methods [4]
On a figure above we can see how IP packet can be treated before being transmitted through
the connection link and also some mechanism of preventing congestion.
There are next basic methods, which can prevent the congestion on a output interfaces:
Admission Control;
Packets Classification and Marking ;
Packets Policing;
Scheduling (Queuing);
Traffic shaping;
Link Efficiency Mechanisms.
Because of possibilities to use such kind of treatment mechanisms, we can manipulate with IP
packets and configure different policy rules according to our network requirements. We don‟t
need to use all this tools at the same time, but some of them can be rather useful to struggle
against congested connection links.
In my degree work I want to try describing some of these QoS mechanisms and demonstrating
some configuration examples, which will be implemented on a Cisco network devises, using
Graphic Network Simulator software (GNS3). Once again I would like to repeat, that some
QoS mechanism start to be activated only on a congested network environment and it is very
23
useful deploy QoS mechanisms on a networks, where some delay or delay-sensitive
applications are used. The meaning of QoS is very difficult to define, but logically we can
guess that it somehow associated with prioritizing (something should be done first, and other
processes have to wait until prioritizing processes are finished). It is very similar with our life,
where people prioritize their business and action during some busy period of time, trying to do
most important things first. As I started to familiarize the QoS theme I have understood that it
is very interesting and actually very urgent theme due the eagerness plenty of applications start
working in IP world (telephony, video, gaming etc.).
Further I would like continue with the conception of congestion and offer some methods to
eliminate congestion at all or take channel or network congestion under control.
Existent communication channel upgrade (channel bandwidth expansion);
Try to use some prioritizing methods (congestion control or congestion avoidance
mechanisms) to regulate the traffic flows through the network or communication
channel.
The first method is the easiest and doesn‟t need any effort to analyze and investigate what was
the reason of connection link overflow. Actually it is rather good and logical method, but
doesn‟t guarantee that after some period of time traffic congestion appears again. Although
this type of solution sounds simple, increasing bandwidth could be expensive and could take a
time for implementation. Sometimes some geographical or technological limitations can be
obstacle on a way to channel being upgraded. For these situations QoS methods could seems
very attractive as a one of the solution, which can help partly eliminate congestion.
The second method, as I already told above, allow network administrator to clarify the reasons
of congestion and using different types of congestion control methods to solve existing
problems.
Here are the examples of some basic QoS mechanism, which could be used for congestion
problem elimination or minimization:
Queuing mechanism (priority queuing, custom queuing and etc.);
Congestion avoidances (Random Early Detection);
Link Efficiency mechanism (payload compression, header compression).
The ideas of some QoS mechanisms consist of IP packet determination and separating them
from each other (how IP packets will by distinguish from each other?). The different technique
can be implemented for this purpose. About these techniques we will speak later.
24
Queuing manages congestion to provide bandwidth and delay guarantees. In any network
where some applications require differentiated levels of service, traffic must be sorted into
different classes and if needed quality of service (QoS) technique will be applied to these
traffic types. Classification and queuing are critical functions of any successful QoS
implementation. Classification allows network devices to identify traffic as belonging to a
specific class with the specific requirements determined by an administrative QoS policy.
After network traffic is sorted (classified), individual packets can be marked (colored) so that
other network devices can apply QoS features uniformly to those packets in compliance with
the defined QoS policy. Queuing dispatches the packets according to their markings.
Classical process of IP packet treatment on a network device could by presented according the
next sequence:
1. Identification of traffic pattern;
2. Traffic classification;
3. Traffic prioritization;
4. QoS method selection (queuing, compression or others);
5. Appling crated policy rule to the network interface.
2.2 BASIC PACKETS DESCROPTORS
As was already mentioned before the QoS can be applied only if packets classification or
packet marking were implemented before. QoS needs a material to work with to distinguish
and separate applications on a network. This role performs classification.
Classification is the process of identifying traffic and categorizing that traffic into classes.
Classification uses a traffic descriptor to categorize a packet within a specific group to define
that packet. Traffic descriptors that are typically used include:
Incoming interface;
IP precedence (IPP);
Differentiated services code point (DSCP);
Source or destination address;
Application port.
Marking is the QoS feature component, that color a packet so it can be identified and
distinguished from other packets in QoS treatment and also can be understandable for other
networks, if packets passing different IP networks (other ISP networks)
25
Traffic descriptors that are typically used include these:
Link layer:
CoS (IEEE 801p focuses on support QoS over VLANs and trunk ports
(802.1Q));
Multiprotocol Label Switching (MPLS) experimental (EXP) bits;
Network layer:
DSCP;
IP precedence (IPP);
First of all it‟s needed take a briefly look on the structure of the packet on a different level on
OSI model to understand the anatomy of IP packet and frame, and clear up how we can
manipulate with them. QoS can by deployed on different layers of OSI model, so in the next
paragraphs I will take a look on data packet structure on a different OSI layers and see which
part of IP packet or data frame could be used for marking traffic flows.
Packet or frame marking allows subsequent network devices to easily distinguish the marked
packets or frames as belonging to a specific class. After the packets or frames are identified as
belonging to a specific class, QoS mechanisms can be uniformly applied to ensure compliance
with administrative QoS policies.
2.3 CLASSIFICATION AND MARKING AT LAYER 2 (OSI) CONCEPT
First of all we need to understand what is the different between classification and marking.
Classification – By this method we are trying inspect one or more aspects of a packet to see
what that packet is carrying.
Marking – by this method we write information to a packet to easily identify it on other
network devices.
On figures below we can see which part of data frame could have information about marking
on a layer 2 OSI model concept.
26
Figure 7. Layer 2 Data packet structure (with CoS) [6]
Class of service marking works only on trunk interfaces (dot1.Q), because of congestion
appearance probability on an uplink interfaces.
Figure 8. Typical usage of CoS on trunk (dot1.q) interface of the switches or routers [17]
802.1Q is a standard defined by IEEE for implementing VLANs in a layer 2 switching
environment. The CoS marking uses the three 802.1p user priority bits and allows a Layer 2
Ethernet frame to be marked with eight levels of priority (values 0–7). Three bits allow for
levels of classification, allowing a direct correspondence with IP version 4 (IPv4, IP
precedence) type of service (ToS) values. Because of this kind of data link layer marking is
media technology specific, namely Ethernet media technology specific, this data link layer
Data Link
address
Source
IP addr
Dest.
IP addr
Application DATA Dest.
port
Source
port
Layer2 Layer 3 Layer 4 Layer 5-7
CoS - Class of Servise
MPLS EXP
27
marking can only be used within an Ethernet LAN environment and cannot be carried over to
a non-Ethernet environment.
Ethernet trunk CoS uses 3 bits in Ethernet frame header “user priority portion of tag field –
802.1q/p CoS”
Table 1. Commonly used Layer 2 marking
Binary Decimal Typical use
111 7 Reserved (Network traffic: STP, different routing protocols: OSPF,
BGP etc.)
110 6 Reserved (Automatically mark themselves)
101 5 Voice
100 4 VIDEO
011 3 Voice signaling (~10kbps)
010 2 High data
001 1 Low data
000 0 Best Effort
Figure above illustrates how CoS marking could be distributed along the different types of
traffic on Ethernet networks. There is one disadvantage of using CoS markings is that frames
lose their CoS markings when transiting a non-802.1Q to a non-802.1p link. As soon as the
packet encounters Layer 3 forwarding, either with a router or a Layer 3 switch, the old
Ethernet header gets discarded and the CoS field will be lost. Therefore, a ubiquitous
permanent marking should be used for network transit.
2.4 CLASSIFICATION AND MARKING AT LAYER 3 (OSI) CONCEPT
In compare with Layer 2 model, Layer 3 is the end-to-end Quality of Service model. Layer3
QoS is very important because it works in IP world, where no information about frame header
and therefore no information about CoS field.
The original TCP/IP standard defined a ToS byte. The first implementation of marking using
the ToS byte was IP Precedence (IPP). IPP only used the left three bits. The IPP marking
strategy is very similar with the CoS marking strategies.
As we can see on figure below ToS bits are added between layer 3 and layer 4 of OSI model.
28
Figure 9. IP packet structure (with ToS and DSCP) [6]
The DiffServ model decided to use the ToS field to be the DiffServ field using 6 bits from this
field, but prior to DiffServ using this particular field, IPV4 already had a QoS mechanism used
on this field called IP precedence. IP precedence uses a 3-bit system, so for backward
compatibility, were decided to use the 6-bit DSCP field as two 3-bit systems, the class selector
(CS) and the drop probability (DP). Because it is now a 3-bit system, the class selector lines
up nicely with the IP precedence. So as such IP precedence is now equal to the class selector,
and if we have a 5 in IP precedence, we will have the 5 in the class selector. This will allow us
to easily map IP precedence services into a DSCP environment and vice versa.
Data Link
address
Source
IP addr
Dest. IP
addr
Application DATA Dest.
port
Source
port
Layer2 Layer 3 Layer 4 Layer 5-7
ToS – Type of service
DSCP
IP Precedense (IPP)
29
Figure 10. IPP and DSCP compatibility [17]
DSCP uses 8 bits (6 usable bits) for deploying QoS on layer 3 connections. First three bits are
used for PHB (per hop behavior) bits (major bits), next 3 bits are defined packets drop
probability and the last 2 bits are used for rather new technology of admission control, where
regular PC and router communicate with each other and PC receive information about router
and data link congestion.
The IETF defines the following PHBs:
Default PHB: Used for best-effort (BE) service (bits 5 to 7 of DSCP equal 000);
Expedited Forwarding (EF) PHB: Used for low-delay service (bits 5 to 7 of DSCP
equal 101);
Assured Forwarding (AF) PHB: Used for guaranteed bandwidth service (bits 5 to 7 of
DSCP equal 001, 010, 011, or 100);
Class-selector PHB: Used for backward compatibility with non-DiffServ-compliant
devices (RFC 1812-compliant devices; bits 2 to 4 of DSCP equal 000);
As I have already mentioned before, DSCP uses 8 bits, where only 6 bits are used effectively
for QoS. This portion of bits gives us 64 different variations of markings.
Figure 11. DSCP bits in IP header [7]
30
Figure 11 illustrates the bits that are used in DSCP marking, where
„aaa‟ – is the binary value of the class
„dd‟ – is drop probability (DP)
Packets with this common DSCP marking will be granted the assured forwarding (AF) per-
hop behavior. In an assured forwarding per hop behavior we can guarantee bandwidth, but not
priority, and it allows access to extra bandwidth if it is available. There are four standard
classes of assured forwarding from AF1, AF2, AF3, and AF4.
The drop probability (DP) uses only the left 2 bits and can be one of 3 values:
1. High drop preference: 11;
2. Medium drop preference: 10;
3. Low Drop preference: 01.
Let‟s see a brief example of two types of IP packets with the same Per Hop Behavior bits and
with different Drop Probability (DP) bits:
AF4: 10011000
AF4: 10010000
We can see the different of two AF4 marking in the Drop Probability bits (11 vs. 10). That
means, that packet with the higher (11) Drop Probability bits will be dropped first if
congestion situation appears.
EF marking doesn‟t have drop preference at all. It means that EF bits all the time looks the
same: 101110, where 101 mean EF bits and 110 – drop preference (unchangeable value). We
have do understand, that IPP and DSCP methods of marking are just only the marking. Only
Enterprise network requirements dictate what kind of rules and policies will be used further on
a network. But anyway due the DSCP marking works along the whole way of movement
trough the networks, marking types of course need to be standardized. This logical
standardization gives possibility for different Internet Service providers use the common
politics of QoS strategy and provide their common customers Differentiated services based on
PHB mechanisms.
31
Figure 12. Common QoS mechanism between ISPs [17]
Common and contractual QoS politics of different ISPs gives the wide possibilities and
flexibility for data traffic going through and between ISP. For IP packets it doesn‟t matter how
it will transfer from point A to point B, but important that QoS policy will be the same during
the whole route of IP packet and PHB strategy could assist in this cooperation‟s.
2.5 IP PACKETS MATCHING, MARKING AND CLASSIFICATON METHODS
As were already discussed, QoS mechanism intends separating IP packets to different classes
to be further treated or prioritized. The methods of separating packets to different classes
called packet classification. Classification is the ability to recognize and distinguish between
different traffic streams. This is the basic fundamental of QoS, because in a QoS environment
the main idea is providing differentiated services, so in order to provide differentiated services
the first thing that need to be done is a possibility to identify which type of traffic need to be
treated, according arranged rules.
Marking gives the ability to mark (color) a frame or a packet, so that it can be easily identified
and distinguished later from other packets for QoS processing. There are two places were
marking procedure could be employed (L2-CoS (3 user priority bits) and L3-ToS (1 byte).
At once arise a question, how (with which methods) packets could be classified. There are
many attributes, many ways of identifying packets for further classification. We can use
access list for classifying, were we can us different parameters of IP packets (IP address
destination, source, port numbers) and group the interesting packets or we can use IP
precedence value, IP DSCP values, MPLS experimental bits and other characteristics of the
packets. Selection process on predefined packets characteristics called matching.
32
Actually, matching mechanism is the first step in a whole differentiated process. As I told
already packet could be matched with different methods: input interface, access-list etc; or we
can use already marked packet, match them and over color with new ToS or DSCP value.
NBAR (Network Based Application Recognition) mechanism is type of packet matching,
where network device organize the deep inspection of IP packet. The NBAR approach is
useful in dealing with malicious software using known ports to fake being "priority traffic", as
well as non-standard applications using dynamic ports. That's why NBAR is also known as
layer 7 categorization of OSI model [20].
NBAR helps to solve one of the problems of trying to classify modern client servers and web-
based applications. In the old days, every application had a fixed application port number;
Telnet used port 23; FTP used 20, 21; SMTP used port 25. But the problem with modern day
applications is that a lot of the applications nowadays use the browser as the universal
interface. What this means is that we could be using the browser port 80 to check our email.
We could also be using browser port 80 to do chatting or can also be using browser port 80 to
be doing peer-to-peer file sharing. So a traditional router would not be able to find out what
are the applications running behind port 80. NBAR empowers the router with the ability to
investigate and analyze the data packet from layer 4 all the way to layer 7. A deep packet
inspection capability gives ability to really see what protocol is running on our network. Also
NBAR keep track of the statistics of all the traffic identified on network. NBAR basically can
provide an easy way to help identify the applications transiting an interface for classification,
so that the appropriate QoS policies can be implemented on the traffic classes.
Figure 13. Packets inspection with NBAR [7]
So NBAR has the ability to investigate from layer 4 all the way to layer 7. It has the ability to
classify applications based on their dynamic TCP or UDP port numbers that are negotiated
during the connection establishment because NBAR is able to listen into the application and
determine what dynamic TCP or UDP ports are being used to establish the connection (Figure
13). NBAR can discover what are the sub-ports used under the main port number. NBAR can
also be used to configure deep packet inspection up to layer 7 for specific application
attributes such as looking for a specific URL website reference for classification.
33
2.6 CLASSIFICATION AND MARKING ON MPLS LEVEL
Modern Service Providers use the MPLS technology as transport mechanism for providing
rapid and flexible deliverance of IP packets. MPLS is usually uses on SP backbone network
(core level), so it is very important to understand how the marking from the other layers could
by bounded with MPLS marking mechanism. When an IP packet enters an MPLS
environment, the IP packet is encapsulated with the necessary MPLS header information,
which is inserted between layer 2 and layer 3. In IT world it called MPLS header a shim
header. Within this MPLS header, 3 bits called the experimental bits (EXP) are set aside to be
used for Class of Service identification. With 3 bits we can specify up to 8 calls of service.
When IP packet enter the MPLS network, the MPLS routing devices only look at the MPLS
header for forwarding instructions, as such it will not be able to see the QoS parameters that
were set up in the IP precedence or DSCP fields located in the IP header. So by default, the
some network devises copies the three more significant bits of the DSCP or the entire 3-bit IP
precedence field on to these experimental bits. The intention of that is to maintain IP QoS
services in the MPLS environment. But the service provider has the option of changing this
QoS values.
Figure 14. MPLS marking with EXP bits [7]
As I have already told MPLS is transport mechanism on modern SP trunk cables, so
congestion appearance is very undesirable and SP try to control traffic loading on trunk cable
connections and do channel upgrading if it necessary.
Table 2. DSCP and EXP values compatibility
EXP DSCP
7 Reserved CS7
6 Reserved CS7
5 EF
4 AF4x
3 AF3x
2 AF2x
1 AF1x 0 Default
On a table above illustrated how the three most significant bits of the DSCP or the IP
precedence of IP packet could be copied to EXP field.
34
3. QUEUING AS CONGESTION MANAGEMENT MECHANISM
3.1 BASIC PRINCIPLES OF QUEUING
Queuing is designed to accommodate temporary congestion on an interface of a network
device by storing excess packets in buffers until bandwidth becomes available or until the
queue depth is exhausted and packets have to be dropped. Queuing is a congestion-
management mechanism that allows controlling congestion by determining the order in which
identified packets leave an interface based on priorities assigned to those packets. Congestion
management entails creating queues, assigning packets to those queues based on the
classification of the packet, and scheduling the packets in a queue for transmission. Cisco
routers and other network devices support several queuing methods to meet the varying
bandwidth, jitter, and delay requirements of different applications.
As rule queuing consists of two parts:
Hardware queue: Uses FIFO strategy, which is necessary for the interface drivers to
transmit packets one by one. The hardware queue is sometimes referred to as the
transmit queue;
Software queuing system: Schedules packets into the hardware queue based on the
quality of service (QoS) requirements.
One way network elements handle an overflow of arriving traffic is to use a queuing
mechanism and algorithms to sort the traffic and then determine some method of prioritizing it
onto an output link. The most popular are next queuing tools:
First-in, first-out (FIFO) queuing;
Priority queuing (PQ);
Custom queuing (CQ);
Flow-based weighted fair queuing (WFQ);
Class-based weighted fair queuing (CBWFQ);
In the next paragraphs of this chapter I will try to describe the main principles of different
queuing mechanisms.
35
3.2 FIFO – FIRST AND THE SIMPLIEST QUEUING MECHANISM
FIFO is the simplest end easiest queuing algorithm. FIFO provides basic store-and-forward
capability as shown in the next Figure. FIFO is the default queuing algorithm in some
instances, thus requiring no configuration.
First in first out queuing basically queues the packets up in the order in which the packets
arrive.
Figure 15. Simplest queuing mechanism FIFO [7]
One of the benefits of using the first in first out queue mechanism is that it is simple and easy
to implement, but the major drawbacks of FIFO is that it lacks the intelligent scheduling
mechanism, thereby allowing aggressive traffic flows to come in and monopolize the
bandwidth starving other traffics, preventing other traffic from getting access to the
bandwidth.
3.3 PRIORITY QUEUING STRATEGY
Priority Queuing (PQ) is another type of Queuing mechanism, which allows prioritizing traffic
on output WAN interfaces, based on packet characteristics to cause the router to place traffic
into the different number of queues. The queue with the highest priority is serviced first until it
is empty, then the lower queues are serviced in sequence.
During transmission, PQ gives priority queues absolute preferential treatment over low-
priority queues. The most important traffic given the highest priority, will always take
precedence over less important traffic. Packets are classified based on user-specified criteria
and placed in one of the next output queues: one, two, three and four - based on the assigned
priority. Packets that are not classified by priority fall into the normal queue (no priority).
36
Figure 16. Priority queuing (PQ) [7]
Figure above shows the main idea of priority queuing, where high level mark packets have the
precedence over the lower ones.
As I have already mentioned before, that packets can be classified by the following packet
characteristics:
Protocol type;
Incoming interface;
Packet size;
Fragments;
Access control list (ACL);
Without doubt we can say that PQ suite very well for time sensitive services like VOIP or
Video over IP, but when we choose PQ as a queuing mechanism on a network device, we have
to understand, that lower-priority traffic is often denied bandwidth in favor of higher-priority
traffic. The use of PQ could, in the worst case, result in lower-priority traffic never being
transmitted (high-priority traffic can fill available bandwidth of the existing link). To avoid
such kind problem, we can use other types of queuing mechanism, like custom queuing or
traffic shaping technique to rate-limit the higher-priority traffic. About these mechanisms I
will talk a little bit later.
3.4 CUSTOM QUEUING STRATEGY
Custom queuing is rather popular queuing mechanism in a data transporting networks, where
is not important the time of delivery, but the accuracy of delivery IP packet. Custom queuing is
an egress queuing tool that allows classifying traffic into various queues based on the types of
information that can be selected by any matching tools. These properties include transport or
application protocol, port numbers, differentiated services code point (DSCP) or IP Precedence
markings etc.
37
The main idea of these methods is the arrangement that involves choosing all elements in a
group equally in some rational order, usually starting from the top to the bottom of a list and
then starting again at the top of the list and so on. In IT word it is called “round-robin
mechanism”, where one packet is taken from each queue and then the process repeats. Figure
below illustrates round robin process [7].
Figure 17. Custom Queuing (CQ) [7]
If all packets are the same size, all queues share the bandwidth equally. If packets being put
into one queue are larger, that queue will receive a larger share of bandwidth. But different
types of configuration can avoiding that one queues takes more bandwidth over another one.
The goal of custom queuing is to allocate bandwidth proportionally among various classes of
traffic.
3.4.1 WEIGHTED ROUND ROBIN QUEUING
Another type of queuing, which can give somehow prioritization above the other queues is
WRR. The weighted round robin (WRR) algorithm provides prioritization capabilities for
round-robin queuing as shown in Figure 18.
Figure 18. WRR example [7]
38
As we can see in WRR, packets are accessed round-robin style, but queues can be given
priorities called “weights.” For example, in a single round, four packets from a high-priority
class might be dispatched, followed by two from a middle-priority class, and then one from a
low-priority class. It gives us more channel capacitance resources for a first queue, where four
packets need to be transported first.
Some implementations of the WRR algorithm provide prioritization by dispatching a
configurable number of bytes each round rather than a number of packets. The Cisco custom
queuing (CQ) mechanism is an example of this implementation.
3.5 WEIGHTED FAIR QUEUING (WFQ)
If we once again take a brief look on the previous considered queuing mechanisms, we can
find next drawbacks in each mechanism:
FIFO causes starvation and IP packets delay;
PQ causes starvation to lower priority traffic;
CQ causes long delays.
A weighted fair queuing mechanism eliminates some of the shortcomings of the other queuing
mechanisms. Weighted fair queuing basically can be thought of as round robin mechanism
starting with small packets. The reason why WFQ chose small packets as the preferred packet,
because associated the idea of small packets to interactive traffic, delay-sensitive packets.
Weighted fair queuing mechanism will automatically sort the different traffic flows into the
respective queues. As I already mentioned, weighted fair queuing is like round robin, but with
the preference to start with small packets. So the “round robinness” of the weighted fair
queuing prevents high volume flows from monopolizing the interface bandwidth.
Figure 19. Weighted Fair Queuing (WFQ) mechanism [7]
39
Although WFQ automatically adapts to changing network traffic conditions, it does not offer
the precise degree of control over bandwidth allocation that custom queuing (CQ) and class-
based weighted fair queuing (CBWFQ) offer.
3.5.1 CLASS-BASED WEIGHTED FAIR QUEUING
CBWFQ extends the standard WFQ functionality to provide support for user-defined traffic
classes. Class-based weighted fair queuing uses the weighted round robin mechanism with
user defined traffic classes. Class-based weighted fair queuing can sort the incoming packets
into the different class queues based on the class matching criteria. As we can understand from
the figure below the queuing guarantees the minimum bandwidth for each class, but also gives
a class unlimited access to more bandwidth if more is available. After a queue has reached its
configured queue limit, additional packets to the class causes tail drop or random packet drop
depending on configuration.
Figure 20. CBWFQ architecture [7]
In our example voice packets have a different class and minimum limited bandwidth
characteristic as well. The CBWFQ mechanism calculates weights based on the available
bandwidth. These weights are then used by the CBWFQ scheduling mechanism to dispatch the
packets. We can configure bandwidth guarantees by using one of the following methods:
Bandwidth configuration (as an example kbps);
Percentage of bandwidth (percentage of available interface bandwidth);
Percentage of remaining available bandwidth.
40
3.6 LOW LATENCY QUEUING MECHANISM
As we have already considered Class-based weighted fair queuing uses the weighted round
robin mechanism with user defined traffic classes. The low latency queuing (LLQ) mechanism
combines the class-based weighted fair queuing with a priority queue and this priority queue
does have a limit so as to prevent the priority queue from starving the non-priority queue‟s
access to the network resources.
Neither the basic queuing methods nor the more advanced weighted fair queuing (WFQ)
methods completely solve the quality of service (QoS) problems resulting from converged
network traffic. The following problems remain:
If only a priority queue is used for a voice-enabled network, voice gets the needed
priority. However, data traffic would suffer;
If only custom queuing (CQ) is used for a voice-enabled network, data traffic is
assured of some bandwidth. However, voice traffic would suffer delays;
If WFQ is used, voice still experiences delay even when treated “fairly” by WFQ.
LLQ queuing mechanisms combine the best aspects of existing queuing methods. LLQ is a
combination of class-based weighted fair queuing (CBWFQ), which assigns weights
according to bandwidth, and a priority system based on class that gives delay-sensitive traffic
priority it requires while ensuring that data is serviced efficiently. This solves the potential
starvation problem of the priority queue by including a policing function based on the
configured bandwidth of the priority system. Figure below can illustrate the main concept of
LLQ strategy.
Figure 21. LLQ – combining queuing mechanism [7]
One benefit of LLQ is having a consistent configuration across all media types, irrespective of
the media used. In my example I use the voice as example packets, which have to be
prioritized due the low delay and no jitter requirements.
41
If we want to understand the principle of LLQ we need to look at the architecture of LLQ. As
we can see on figure below incoming packets are forwarded and treated with the low latency
queuing mechanism. The packets received are being identified as the delay-sensitive voice
packets, and as such are being processed by the priority queue. The priority queue has also
identified that the voice packets have not exceeded their bandwidth allocation, and as such
they are being sent to the priority queue buffer.
Figure 22. LLQ Architecture [7]
Let‟s consider some benefits of LLQ mechanism over the others:
Low-latency propagation of packets;
Guaranteed bandwidth;
Entrance criteria to a class can be defined by an ACL or NBAR.
Priority traffic is metered only under conditions of congestion. When the device is not
congested, the priority-class traffic is allowed to exceed it allocated bandwidth. When the
device is congested, the priority-class traffic above the allocated bandwidth is discarded.
42
4. CONGESTION AVOIDANCES MECHANISMS
4.1 TAIL DROPPING – EASIAST RESPONSE TO CONGESTED INTERFACE
As we have already understood, QoS has a lot of different implementations and techniques
how to take a congestion, which already occurred on a connection link, under control. But let‟s
consider another smart mechanism that allow somewhat avoid congestions on a link or
network device interfaces. Congestion avoidance is a technique, where congestion avoidances
algorithms start operate before connection became congested.
When an interface on a router or other network device can‟t transmit a packet immediately, the
packet is queued. Packets are then taken out of the queue and eventually transmitted on to the
interface, but if the arrival rate of the packets to the output interface exceeds the router‟s
ability it causes congestion and packets start building up in the buffer queue. Eventually, the
buffer is full and no more packets can be stored in the buffer queue. Subsequently, additional
incoming packets are dropped. This strategy is called the tail drop strategy (Figure 23).
Figure 23. Tail dropping mechanism [7]
One of the disadvantages of tail drop strategy is that it treats all traffic equal and does not
differentiate between Type of Service. So a delay-sensitive application may suffer from
performance degradation due to packet loss caused by the tail drop strategy. In the next
paragraphs I will consider some solution examples to minimize the disadvantage of tail
dropping strategy.
4.2 RANDOM EARLY DETECTION MECHANISM (RED)
As we have already considered, tail drop occurs when the buffer queue is full and actually tail
dropping can be avoided if next mechanisms will be activated. Random early detection is a
dropping mechanism that randomly drops packets before a queue is full. The dropping strategy
is based on average queue length. When the average queue length is short, packets have a high
43
chance of being welcomed into the buffer, but as the size of the queue increases, RED will
associate a higher drop probability to the incoming packets, so as to prevent the buffer queue
from being full. Random early detection drops packets randomly and we can say that random
early detection can help to slow down aggressive traffic flow. The rationale behind it is that an
aggressive traffic flow sends more packets. So logically it has a higher probability of having
their packets dropped. Random early detection is able to help slow down the aggressive traffic
flow, while giving the less aggressive traffic flow a chance at the bandwidth. Also random
early detection can help avoid what we call the global synchronization problems (Figure
below demonstrates it) associated with tail drop, where all applications suffer the tail drop and
back off at the same time, resulting in a sudden under utilization of the link. Random early
detection also helps to keep the average queue size low because it uses probability to keep the
queue size low. As the queues are filling up, RED will implement a more and more aggressive
drop strategy to prevent the queue from filling up.
RED has three packet dropping modes based on average queue size is less than the minimum
threshold:
When the average queue size is between 0 and the configured minimum threshold, no
drops occur and all packets are queued;
When the average queue size is between the configured minimum threshold and the
configured maximum threshold, random drops occur, which is linearly proportional to
the mark probability denominator and the average queue length;
When the average queue size is at or higher than the maximum threshold, RED
performs full (tail) drop in the queue. This situation is unlikely, because RED should
slow down TCP traffic ahead of congestion. If a lot of non-TCP traffic is present, RED
cannot effectively drop traffic to reduce congestion, and tail drops are likely to occur.
Figure 24. TCP flow before and after RED implementation [7]
Figure above shows TCP throughput behavior compared to link bandwidth on a congested
network scenario where the tail-drop and RED mechanisms are in use on the link.
44
Before RED, when all sessions slow down, congestion on the router interface is removed and
all TCP sessions restart their transmission at about the same time. After that router interface
again quickly becomes congested and caused tail drop. As a result, all TCP sessions back off
again. This behavior cycles constantly, resulting in a link that is generally underutilized.
After RED is applied, RED randomly drops packets, influencing a small number of sessions at
a time, before the interface reaches congestion. Overall throughput of sessions is increased, as
is average link utilization. Global synchronization is very unlikely to occur because dropping
occurs on a randomly selected traffic TCP flows.
4.3 WEIGHTED RANDOM EARLY DETECTION MECHANISM
As we could cleared up RED mechanism is rather good compare with pure tail dropping, but
not the ideal due the dropping process occurs randomly (without any strategy). Weighted
random early detection changes this approach. WRED uses multiple differed RED profiles,
each with their own maximum threshold, minimum threshold, and maximum drop probability
values, and each profile is assigned based on their IP precedence or DSCP values. The
objective of weighted random early detection is to create a bias drop strategy, whereby less
important traffic is dropped more aggressively first, than more important packets.
As was already considered RED profiles are identified by:
Minimum threshold;
Maximum threshold;
Maximum drop probability.
Very interesting fact, that Cisco devices don‟t support RED. Instead, Cisco supports weighted
random early detection (WRED) which combines RED with IP precedence or DSCP and
performs packet dropping based on IP precedence or DSCP markings.
WRED profile selection is based on:
IP Precedence (8 profiles);
DSCP (64 profiles).
It is doesn‟t recommend WRED for any voice queue, although we may enable WRED on an
interface carrying voice traffic. WRED will not throttle back voice traffic because voice traffic
is User Datagram Protocol (UDP)-based. The network itself should be designed not to lose
voice packets because lost voice packets result in reduced voice quality. WRED controls
45
congestion by affecting prioritized traffic other than voice, and avoiding congestion helps to
ensure voice quality.
4.4 CLASS-BASED WRED
Congestion avoidance technique monitors the network interface in an effort to anticipate and
avoid congestion at common network bottlenecks. Congestion avoidance is achieved through
intelligence packet dropping mechanism. The class-based weighted fair queuing (CBWFQ)
system supports the use of WRED inside the queuing system. Each class is queued in it
separate queue and has a queue limit, performing tail drop by default.
The figure below shows two different weighted random early detection profiles that are used
for traffic of two different QoS classes, the BE (Best Effort) class and the premium class. The
BE class traffic has been set a much lower maximum and minimum threshold as compared to
the premium class. So as such BE class will be dropped earlier and more aggressively before
the premium class traffic and when heavy congestion occurs, all the BE class traffic will be
tail dropped in favor of the premium class traffic. To avoid the need for setting other weighted
random early detection parameters in a router, 8 default values have already been defined for
precedence based WRED and 64 default values have been assigned for a DSCP based
weighted random early detection.
Figure 25. CBWRED mechanism [7]
As we can conclude that less important traffic could be assigned a more aggressive drop
strategy, while the more important IP traffic could be assigned a less aggressive drop strategy.
46
4.5 DSCP-BASED WRED
As already was mentioned it can be used DSCP marking for configuring WRED mechanism
on the output queues. EF PHB is suggested for applications that require a hard guarantee on
the delay and jitter. Typically, mission critical applications would require this service and
would be allocated a small percentage of the total network capacity.
In DSCP, the Expedited Forwarding (EF) PHB is identified based on these parameters:
A low departure rate is ensured to provide low delay to delay-sensitive applications;
Bandwidth is guaranteed to prevent starvation of the application if there are multiple
applications using EF PHB;
Bandwidth is policed to prevent starvation of other applications or classes that are not
using this PHB;
Packets requiring EF should be marked with DSCP binary value 101110 (46).
Figure 26. DSCP based WRED (EF-Expedited forwarding) [7]
For the EF DiffServ traffic class, WRED configures itself so that the minimum threshold is
very high, increasing the probability of no drops being applied to that traffic class. It is
expected that EF traffic will be dropped very late (as we see on the figure above, when queue
size reach the value of 36 packets), compared to other traffic classes, and the EF traffic is
therefore prioritized in the event of congestion.
4.5.1 DSCP-BASED WRED CONFIGURATION EXAMPLE
In this paragraph I would like to show the part of Cisco router configuration, where I tried
match already marked with DSCP packets and divide them by different classes. I have
configured two classes: CRITICAL (DSCP values: af41, 42, 43) and BULK_DATA (DSCP
values: af31, af32, af33). I gave to each traffic flow (based on their marking) separate packets
47
drop probabilities. Here presented the part of Cisco router configuration, where we can see
how class-maps and policy-map were created.
! class-map match-any CRITICAL
match dscp af41 af42 af43
class-map match-any BULK_DATA match dscp af31 af32 af33
!
! policy-map DSCP_BASED_WRED
class CRITICAL
bandwidth percent 35 random-detect dscp-based
random-detect dscp 34 128 256 10
random-detect dscp 36 96 256 10
random-detect dscp 38 64 256 10
class BULK_DATA
bandwidth percent 25 random-detect dscp-based
random-detect dscp 26 96 128 10
random-detect dscp 28 64 128 10 random-detect dscp 30 32 128 10
!
As we can mentioned I gave to each class certain bandwidth in percentages from total
bandwidth capability (35% and 25% respectively). Further I attached to each type of DSCP
marked packet drop probability characteristics. There are: minimum-threshold, maximum-
threshold and mark-probability denominator. These parameters define when packets drop can
start with a certain drop frequency (minimum-threshold) and when tail-drop starts dropping all
the packets (maximum-threshold). As mark-probability denominator I choose 10, that means
that one packet from ten, situating in a queue will be dropped (dropping probability 10%).
On a figure below I have illustrated how these probabilities distributed among each other.
Figure 27. Packets drops probabilities [17]
On figure illustrated two configured traffic classes, where each class has his own maximum-
threshold value and mark probability denominator (10). From this figure we can clearly see,
48
that if congestion occur inside the configured classes, the packets will have different packets
drop probabilities, according marking value and configured threshold parameters.
On the next figure I illustrated how configured WRED parameters will look on router.
Figure 28. Configured classes WRED parameters
To finish with DSCP_BASED_WRED configuration, we need to implement configured
policy-map on Router WAN output interface. Using WRED mechanism we can take under
control different type of traffic, separating their by classes and mark them by DSCP or IPP
values. Further these markings help designate packets drop policy for each marked packet.
WRED helps avoid congestion on a channel due to control on created IP packet‟s queues.
4.6 TRAFFIC POLICING AND SHAPING
Further I would like very briefly talk about traffic limitation. I suppose, that it‟s not so hard to
guess, that both traffic policing and traffic shaping are mechanisms used to limit the traffic
rate for a different classes of traffic. In traffic shaping, excess packets are queued to stay
within the desired traffic rate. While in traffic policing, excess traffic is normally dropped to
stay within the traffic rate limit, while in other cases, we can choose to map down the packets
instead of dropping the packets.
Figure 29. Differences between traffic Policing and Shaping [13]
49
Traffic policing and traffic shaping are both traffic conditioning mechanisms in which they try
to ensure that various traffic classes stays within their traffic rate. Traffic policing excess
packets that exceed their allocated traffic rate are dropped, which typically results in TCP
retransmissions. Even though traffic policing normal tendency is to drop excess traffic, we can
also implement a remarking strategy, whereby excess traffic can have it layer 2 or layer 3 QoS
markings marked down to a lower priority. Because of traffic policing tends to discard excess
traffic, it uses fewer buffers compared to traffic shaping. In traffic shaping, the objective is
also to try to ensure that the traffic stays within it traffic rate, but any excess packets are
queued instead of being dropped. Buffering the TCP traffic does not cause the TCP
application to retransmit, but traffic shaping does not have the marking or remarking
capability as that of traffic policing. The schematic difference between both mechanisms we
can see on a figure above. To execute traffic limiting prioresses more efficiently exist class-
based policing and class-based shaping as well, where every individual class of traffic can be
shaped or limited according network requirements and traffic policy rules.
50
5. BASIC STEPS OF QOS IMPLEMENTATION
We have considered different types of QoS mechanisms that could be very useful for
implementation on a congested and slow data connection channels to decrease congestion and
allow some critical traffic types being transported without delays and packet drops. In this
chapter I would like to show how some very simple QoS mechanisms could by physically
implemented on a network devices. As example I have used Cisco devices, as a rather popular
networking product. Similarly way could be configured other brands network devices, but
with a different type of syntax.
The information below summarizes the three steps to follow when configuring QoS using
Cisco MQC (Modular QoS CLI (command line interface)). Each step answers a question
concerning the classes assigned to different traffic flows:
Build a class map: What traffic do we care about? The first step in QoS deployment is
to identify the interesting traffic. This step defines a group of network traffic: a class-
map in MQC terminology with various classification tools: Access Control Lists
(ACLs), IP addresses, IP precedence, IP Differentiated Services Code Point (DSCP),
IEEE 802.1p, Multiprotocol Label Switching Experimental bit (MPLS EXP), and
Network Based Application Recognition (NBAR). In this step, we configure traffic
classification by using the class-map command.
Policy map: What will happen to the classified traffic? We decide what to do with
classified in step 1 traffic classes. This step is the actual construction of a QoS policy.
A policy-map in MQC terminology: choosing the group of traffic (class-map) on
which to perform QoS functions. Examples of QoS functions are queuing, dropping,
policing, shaping, and marking (all these functions were already discussed in my thesis
before). In this step, we configure each traffic policy by associating the traffic class
with one or more QoS features using the policy-map command.
Service policy: Where the policy will be applied? Apply the appropriate policy map to
the desired interfaces, sub-interfaces or any other interfaces. In this step, we attach the
traffic policy to inbound or outbound traffic on interfaces, sub interfaces, or virtual
circuits by using the service-policy command (in a Cisco CLI)
Recall, that I use as example Cisco devices and corresponding Cisco IOS (Internetwork
Operating System) as well. Further I would like to show a brief example of class-map and
policy-map configuration on Cisco routers. For class-map VOIP we match real time protocol
and give to this class 50% of whole bandwidth capacity, the same policy (50% of whole
bandwidth capacity) we arrange for DATA class, where we have matched only http traffic.
51
Router(config)#class-map VOIP
Router(config-cmap)# match protocol rtp audio
Router(config)#class-map DATA
Router(config-cmap)#match protocol http
Router(config-cmap)#exit
Router(config)policy-map MY_POLICY
Router(config-pmap-c)#class VOIP
Router(config)#bandwidth percent 50
Router(config-pmap)# class DATA
Router(config-pmap)# bandwidth percent 50
Router(config-pmap)#exit
Router(config)# interfasce X
Router(config-if)#service policy output MY_POLICY
Another example demonstrates us a little bit different configuring type of policy-map, using
already four different classes. In this case, our task is to enforce a sub-rate (that is, 10 Mbps
virtual pipe on a 1 Gbps link) on a particular link, while offering minimum bandwidth
guarantees to applications such as voice, mission critical applications (MCA), and video
within that virtual pipe as follows:
voice: 1 Mbps
MCA traffic: 2 Mbps
video: 5 Mbps
Remaining bandwidth allocated to BE traffic within the defined 10 Mbps pipe
Below is the sample configuration:
Router(config)# policy-map 10M_traffic
Router(config-pmap)# class VOICE
Router(config-pmap-c)# priority 1000
Router(config-pmap-c)# class MCA
Router(config-pmap-c)# bandwidth 2000
Router(config-pmap-c)# class VIDEO
Router(config-pmap-c)# bandwidth 5000
Router(config)# policy-map PARENT
Router(config-pmap)# class class-default
Router(config-pmap-c)# shape average 10000000
Router(config-pmap-c)# service-policy 10M_traffic
Due to the rather wide packet classification possibilities, network administrators have to
clearly understand, which type of protocol need to be prioritized policed or shaped according
existing networking requirements.
Very simple and classical classification of different protocols on a network looks like that:
52
Real-time: Voice over IP (VoIP) traffic and voice-signaling traffic;
Business-critical: Business traffic important to a typical corporate environment. Some
of the protocols included in this traffic category are: Citrix, SQLNet, Notes, LDAP,
and Secure LDAP. Routing protocols included in this category are egp, bgp, eigrp, and
ospf;
Best-effort: Remaining traffic.
Nowadays providing QoS to different types of traffic is a major goal and in the same time a
major challenge for the ISP and Internet community. As new applications (audio, video, real-
time streaming, interactive applications) have emerged, the „traditional‟ way of handling
packets delivery (BE service) is not able to cope with the new, diversified requirements these
services call for. To overcome some problems, DiffServ approach aims to provide a natural
evolution path from the best effort service model, capable to provide service discrimination,
while preserving the salient feature – scalability, by pushing the complexity at the edge of the
network and keeping the core nodes simple.
On a figure below I have tried to illustrate, how classical QoS mechanism can be usually
implemented on ISP and customer side networks. Figure is divided into two parts (per flow
discrimination and per class discrimination). Per flow discrimination part is a access level,
where usually different kind of congestion control and avoidance mechanisms could be
employed. Core level usually responsible for pure IP traffic transport, were usually no
necessity to use QoS mechanisms, which could reduce the network device efficiency. But
sometimes several limited number of traffic classes exist on a core level network (as an
example: Management (CS6), Voice (EF), Data (AF41) Data2 (AF31) and BE traffic). This
policy (for example CS6 - management) guarantees, that routing protocols functionality never
will be suppressed by other traffic types.
Figure 30. Classical QoS duties distribution on a network [17]
The figure above also shows that implementing different types of QoS technique is very
important particularly on a slow communications links, as local loops, where probability of
congestion appearance is rather high. In my thesis I tried to consider just the part of network
53
(local loops), where per flow QoS technique present the main struggle method with IP channel
congestion.
54
6. PRACTICAL PART /CONFIGURING AND QOS IMPLEMENTING
EXAMPLES/
6.1 PRACTICAL TASK 1. /MATCING AND PRIORITAZING TRAFFIC/
/CONFIGURATION EXAMPLES/
In this practical part I would like to show how IP channel could be divided between different
types of traffic with a minimum throughput per class and how using LLQ mechanism we can
prioritize the delay-sensitive traffic types (for example voip) and guarantee for them
maximum, but limited throughput. Firstly let‟s take a look on a picture below, where I
illustrated two Enterprise offices connected with each other through the SP network.
Figure 31. Networks connection design (L3VPN) between two Enterprise offices [17]
As we can understand from the figure, different types of traffic going through the client CPE
(CE-customer edge) router and Service Provider network as well. Enterprise brunch office has
limited speed connection between CE and PE. So the idea of this task is guarantee some part
of existent data channel bandwidth to classified traffic types. First of all, let‟s clarify with
what types of traffic we have deal and in which proportion classified traffic have to be
divided. Let‟s assume that we have the next network requirements:
1. http and ftp need at least 25% from link capacity;
2. Remote administrator application needs at least 10% from link capacity;
3. SQL server replication needs at least 20% from link capacity;
4. Everything else can share the leftover bandwidth.
Our initial objective is matching the interested types of traffic and classifying them according
presented network requirements. Using the GNS3 software, I have connected two Cisco
routers with each other, assumed that two routers work through the ISP network.
55
Figure 32. Connection between two Enterprise offices in GNS environment
On figure above we can see two Cisco routers. I used Cisco 3725 routers and corresponding
IOS (c3725-advipservicesk9-mz.124-25b.bin). Let assume, that necessary QoS rules need to
be implemented on a branch office router‟s output interface.
In first scenario we deal only with data traffic types and in this step we don‟t need configure
any prioritization for a time-sensitive traffic. As was discussed in Chapter 5, we need to start
with the traffic classification and group our concerned traffic types into different groups
(classes). For matching packets I use NBAR feature, which allow packets deep inspection.
From the figure below we can see how packets matching and classification methods can be
configured.
Figure 32. Class-maps configuration in GNS3 environment
We have the next four class-maps: WEB_TRAFFIC, MSSQL, XWindow and class-default.
I use casual protocols, because this practical part is illustrative and we can use different
protocol types. Further I have created the policy rule (in Cisco world it is a policy-map), where
I have defined, which part of the link throughput is guaranteed for our already created class-
maps (in percentages). On a figure below illustrated policy-map configuration result (our
policy-map name is CB_POLICY)
56
Figure 33. Policy-map configuration in GNS environment
I also have configured fair queuing mechanism on a class-default class, which do the queuing
mechanism on leftover part of bandwidth more flexible and smarter. As we have already
known from theoretical part, queuing mechanism can be realized only on outbound interfaces.
So it means that we need implement our created policy-map on a router R2 outbound interface.
In our example it is FE0/0 interface.
Figure 34. Output interface policy-map configuration
As we can see from this output, our policy-map with our class based policies are correctly
configured on the outbound interface of router R2, what guarantee a class based scheduling
strategy for outgoing packets. This output shows us the real time parameters on configured
57
classes (packets matched, offered rate, drop rate and others). These parameters could be used
for traffic load estimation. Using getting result our QoS strategy and classifying methods
could be changed to achieve desired results if some classes don‟t have enough bandwidth and
packets start dropping. In our case we see only zero values due to absence any types of traffic
in a network.
We also have to remember that percentages, which were configured for different classes, mean
not the maximum limit of the channel bandwidth, but minimum guaranteed bandwidth level.
This means, that if there is available free bandwidth on the link (due some traffic classes are
not aggressive enough), it can be used by other classes. Command line bandwidth percent
guarantees at least (minimum) configured bandwidth value (in percentages from total link
capacity) for desired traffic class.
I have configured QoS mechanism that is very similar with custom queuing. This type of
configuration guarantees some part of total bandwidth for particular traffic classes. But if for
example we have a real-time traffic types, like voice or video, and these delay-sensitive traffic
types need to be transported through the already congested IP channel. What kind of QoS
technique could be useful in this situation? Recall from theoretical part of thesis, how delay-
sensitive traffic could be integrated to the existent policy-map rule. We have to extend already
configured policy-map with a new traffic type class, which will have a guaranteed QoS
mechanism with lowest delay parameters. This QoS technique calls LLQ (Low Latency
Queuing). Let‟s try to configure class-map, which will match the all real-time protocol packets
going through the interface FE0/0, and integrate this class-map to the existing policy-map. We
need to decide how much bandwidth from total link capacitance we guarantee to our critical
delay-sensitive traffic. It depends on how many IP phones we have on a LAN network and
what is the probability of simultaneous voice conversations can take place on data channel.
We have a 10M/10M communication channel and assume, that we guarantee 10% of total
bandwidth (1M/1M) to real-time protocol (voip) traffic.
Our initial strategy will be changed and will look like this:
1. http and ftp need at least 25% from link capacity;
2. xwindows application needs at least 10% from link capacity;
3. SQL server replication needs at least 20% from link capacity;
4. Everything else can share the leftover bandwidth;
5. voip needs the first 10% of bandwidth.
In our case we have to prioritize voip class above the other traffic types. Voip traffic needs to
be pushed in front of the line. It can be configured with the priority command on an existent
policy-map rule. But firstly we need to match voip traffic with a new created class-map VOIP.
58
Figure 35. Matching voip packets
I have configured a new class-map, where NBAR mechanism matches all rtp audio packets.
Further I have integrated new created VOIP class-map to the existent policy-map
CB_POLICY with the strict priority (10% of total bandwidth is guaranteed for voip traffic
packets).
Figure 36. Strict priority for voip class-map
From this output we can see that new class VOIP has been added to policy-map and new class
has strict priority above the other classes. In this case strict priority guarantees only limited
bandwidth for a voip class: 10% from total channel bandwidth and not more. In this case
protocol‟s packets will have probability to be dropped if aggressive voip traffic utilizes whole
reserved part of channel bandwidth.
6.2 PRACTICAL TASK 2 /TRAFIC POLICING AND SHAPING METHODS/
In this practical part I would like to show how we can control traffic going through the
network device interface by shaping and policing mechanisms. I have composed next network
diagram in GNS3 simulation environment and turned on preconfigured traffic generator to
start congest IP channel between router R1 and R2 with different types of IP traffic.
59
Figure 37. Connection link congestion by R1 router
I also installed PRTG Traffic Grapher on my PC (V6.2.2.983), which helped me to be sure,
that configuration I employed influents on a traffic going through the router. From the figure
below we can see how ICMP packets can be dropped due to channel congestion. I start ping
remote IP address (100.100.100.1/24, Traffic Generator FE1/0 interface) from PRTG Grapher.
Figure 38. ICMP packets drops on a connection link
I received next result (Figure above). ICMP packets don‟t go through the congested channel or
respond time is very big. Further I started to police all traffic, going through the router R2 with
the proper type configuration. I did two steps of changing the policing rules in order to see
how our router R2 output (FE 1/0) traffic graph will be changed.
Figure 39. Two steps of traffic output policing
60
Figure 31 demonstrates how output graph changed with traffic policy configuration. It can
mean that free space of remaining bandwidth could be used for a class-base traffic type (for
example ICMP packets, which had problems with the respond time or dropping probability
before). On a next diagram we can see that ICMP packet drops stopped after we have
implemented class-based shaping on router R2 (FE1/0) output interface.
Figure 40. Packet drop eliminating by class-based shaping mechanism
Graph shows that using simple methods of traffic shaping and policing data channel
congestion can be taken under control and guarantee more channel bandwidth for important
and critical traffic types. Certainly our class-default traffic, which was policed, dramatically
suffered and great number of packets belonging to this class was dropped anyway. But as I
have already mentioned, we achieve the availability and scalability of one service to detriment
the others. On figure below we can see how simple policing and shaping methods were
configured on a policy-map rule (peak rate traffic shaping = 512kbps (icmp class), police rate
= 100 kbps (class-default class)).
Figure 41. Policing and shaping configuration
I specially don‟t mention any link speeds, because these methods could be used on a different
links with a different shaping and policing methods. All depends on a situation on a working
network and it recruitments. As we can see my implemented traffic shaping and policing
speeds (CIR) are rather low, because of network environment virtualization. Using two or
61
three virtual routers on a GNS3 environment my computer processor were loaded around 75-
85% of total efficiency.
6.3 PRACTICAL TASK 3 /PRIORITIZING ICMP PACKETS/
Two previous practical examples showed how different QoS methods can be implemented on
a network environment to schedule intensive traffic on the routers output interfaces. In this
part of my work, I would like not only show how routers could be configured, but also how
configured QoS mechanism could influence on an IP packet treatment. On figure below
illustrated classical network topology, which can exist on rather many real-implemented
networks. Let‟s assume, that we have deal with two LAN networks (branch offices). On my
diagram I have used two routers (R2 and R3), installed on different side of the networks.
Figure 42. GNS network diagram
GNS diagram very briefly describes the IP-plan of network. I have connected Traffic
Generator router (R2) with a router R1 and router R1 with R3. Traffic generator function is to
generate configured traffic types and try to load connection link between our branch‟s routers
(R1 and R3). Three routers communicate with each other by EIGRP (AS 100) configured on
it. I also used my personal computer (PC) to be in the same network with our routers. I used
loopback interface to connect my PC directly to router R1 (cloud on figure). The main idea of
this practical part is to determine, that correctly configured QoS strategy on a router (in my
example it‟s router R1) could change the behavior or IP traffic, which leaves router R1
through output interface (FE0/1). My practical part based on GNS3 software, where I can
draw my network diagram, and configure network devices, according our QoS requirements.
Two branch offices routers (R1 and R3) are Cisco routers (Cisco 2961) and Traffic generator
role realize Cisco 3725. To monitor connection and loading on R2 output interface installed
PRTG Traffic Grapher, which helped me to test and be assure, that configured QoS strategy
62
operate as it has to. I put together all network devices on the network diagram (as we can see
on Figure) and assured that built scheme has functionality (routing protocols work well and
ICMP packets going through). Using PRTG, I start ping IP address, situated on the other side
of the data link (R3 WAN interface). Traffic grapher shows us, that ICMP connectivity works
correctly (no other traffic is going through the interconnection link).
Figure 43. IP NBAR protocol-discovery statistic
From router R2 output interface statistic we can see, that only two types of traffic leave the
router output interface (FE0/1) and we don‟t have any congestion on this link. After turning on
Traffic generator different types of traffic were activated on a network. Next figure
demonstrate which types of traffic have congested the line.
63
Figure 44. IP NBAR protocol-discovery statistic on a congested link (output interface)
Here we can see different type of traffic, which are generated with traffic generator. As we
have deal with a virtual infrastructure, some parameters don‟t reflect the real situation that
could be happened on a real network, but nevertheless we can achieve some interesting results.
Figure 45. Latency difference between congested and uncongested situation
Figure above demonstrate how congestion on a link could change ICMP packet behavior. We
see, that on congested link ICMP packets don‟t go through the channel or don‟t go back. And
also those packets, which returned back have increased latency time. As I told already, due all
this network diagram exists on my PC, some network parameters (like ICMP response time)
don‟t reflect real situation. But anyway using some QoS mechanism I have gained positive
results. As I already described in a teoretical part of my thesis, we can for example prioritize
ICMP trafic and return ICMP packets respond time on a initial level. To prioritaize ICMP
pacckets we need to implement trhree steps. Firstly we matc ICMP packets with a NBAR tool,
then created class-map we configure under policy-map and lastly we bound policy-map with a
router R1 outbound interface. I have configured next policy rule.
64
Figure 46. Policy-map configuration result
The part of configuration above shows, how our queuing mechanism were configured. From
this output we can notice, that ICMP packets matching acts, as well as class-default class
matching occur. But the one difference exists between these two classes. One class is a strict
priority class; another class is a custom class-default, where packets drops can take place. As
we can see from the output class-map icmp doesn‟t have any dropped packets.
On next two figures I have captured two ICMP latency graphs, which presented two situations
of possible results.
Figure 47. QoS influence in ideal case
On the firs figure I presented how could look like ICMP packet latency parameters in ideal
networking condition. And on the next figure illustrates the results, which I got after service-
policy was implemented on an outgoing interface.
65
Figure 48. ICMP packets behaviour
From these two outputs we can conclude, that configured QoS strategy influent on ICMP
respond time parameters. Obtained result is not ideal, but it reflects the configured QoS
parameters on router R1.
The main goal of this practical part is to show, how some types of traffic (in our case ICMP
packets) could by prioritized on congested link. There are many different solutions of this case
realization, but I used the easiest one. The idea of used method was prioritize ICMP packets
going from PC and allow ICMP packets to use guaranteed bandwidth of the existing channel.
This guarantee, that ICMP packets will always have prioritization above other types of traffic
(analogy with LLQ).
After policy map (icmp-policy) was configured on a router output interface, PRTG graph was
changed and we could see that ICMP respond time was almost the same as it was before
traffic generator starts generate IP traffic. And as we could mention some types of traffic is
still continued dropping, but as were already discussed before, QoS mechanism isn‟t a magic
wand and can‟t solve all the problems, connected with a congested channel. Some Qos
mechanism can only help preventing packets drops for specific types of traffic (for example
delay-sensitive), which we match and prioritize by ourselves.
As I consider different types of queuing mechanisms, I also decided to compare two queuing
mechanism (custom queuing and custom queuing with LLQ). Actually I used the same
diagram with the same router configuration. I just have created two new policy-maps: first
policy-map with the priority queue (LLQ) for ICMP packets and another with two equivalent
CB queues. Policy-map configuration is illustrated on a figure.
66
Figure 49. Two types of policy-maps
I desided to divide time scale on a next three parts: ICPM packets without QoS, ICMP packets
with CB queuing (no LLQ) and CB queuing with LLC. Next figure ilustrates the ICMP
packets behaviour on these three parts of time sclae.
Figure 50. ICMP packets behavior
As we can see from the figure channel congestion were presented on a channel during all the
time of QoS testing (upper figure). First part of the lower figure illustrates ICMP packets drop
due the channel congestion; second part shows congestion elimination because of usage LLQ
mechanism (strict priority for ICMP packets); and the last part of the figure shows ICMP
67
packet respond time with using class-based QoS mechanism (without any prioritization). As
we can observe, the best result gave the QoS with prioritization. CB queuing mechanism
without prioritization shows good results as well (several packet drops), but this rather good
result were achieved because of only two classes (class-map icmp and class-map class-
default) were configured on a WAN interface. With increasing of different configured classes
on a router less part of total bandwidth is guaranteed for each configured classes due to round
robin mechanism. Consequently, with high possibility ICMP packets dropping starts if we
configure more classes on a router.
6.4 PRACTICAL TASK 4 /MARKING PACKETS WITH DSCP VALUE/
In this practical part I would like to show how different type of traffic can be marked with
DSCP values on Enterprise CE router and sent further through the SP or other network. As
was mentioned before, when any a real-time or delay-sensitive traffic going through the SP
network it is recommended that this type of traffic were prioritized not only on enterprise side
router, but also on SP network. Usually SP networks have four or five classes‟ types to
guarantee some QoS for customers on their network. About these classes I will talk a little bit
later.
Let‟s assume that we need to configure CE routers to guarantee some critical and some
interested traffic were passed through the SP network, according existing classes types on SP
network. Knowing the SP network policy rules, we have possibility to mark outgoing packets,
which exit enterprise network. Also before leaving the routers output interface packets can be
scheduled (queued), according enterprise network requirement.
Next figure illustrate how our considered network diagram can look like and where packet
marking usually take place.
68
Figure 51. Packets marking strategy network diagram [17]
Before classify and mark outgoing packets, which need to be transmitted through the SP
network, we need to clarify two things:
1. What kind of QoS strategy exists on SP network? (existing SP marking policy)
2. How can we mark and schedule packets on a CE router to let SP network treat them
correctly, according our network requirements?
As I already mentioned SP networks usually use around four and five traffic classes. The less
classes on a network, the easier control and monitor it within SP network. So if we want to use
SP classes, we need to define which traffic has to be classified and then marked.
Table 3. IP traffic classes approximation
VOICE VIDEO Bulk Data (http,
ftp)
Mission Critical
Data
Bandwidth Low to
Moderate
Moderate Moderate to
High
Low to
Moderate
Drop-Sensitive High High Low Moderate
Delay-Sensitive High High Low Moderate
Jitter-Sensitive High High Low Moderate
The figure above demonstrates the approximation of the IP traffic classes and their
requirements, which could be used on an enterprise network.
Next figure below illustrates a QoS baseline recommendation for traffic classification and
markings on SP networks. Service providers don‟t use all of them, but they can arrange it own
69
policy and packets treatment rules or summarize them in separate classes. Given example also
shows how L2 marking types (CoS) intersect with a L3 (IPP and DSCP) marking types.
Figure 52. Example strategy for traffic classification and markings on ISP level [7]
Recall that it is all about recommendation and final decision about, how traffic classes will be
treated inside the network depends only on SP network policy. Therefore if network customer
traffic need to be prioritized on a SP level, customer network devices have to mark all the
packets flows, according SP marking policy.
Let‟s try to emulate a network diagram in GNS3 environment and look how traffic classes can
be created and marked with the DSCP values that further will be transmitted through the SP
network.
Firstly assume, that SP provide to all their customers next static QoS strategy and can treat
packets according next DSCP values:
Table 4. ISP QoS strategy and customer output policy
ISP Classes DSCP PHB Enterprise output policy
Controlled Latency 46 EF 15%
Controlled Load_1 34 AF41 30%
Controlled Load_2 26 AF31 20%
BULK DATA 10 AF21 10%
As we can see this table illustrate ISP classes, which DCSP values have to be on customer
packets and also I showed that enterprise network has its own separate policies where basic
character for traffic classes will be percentages from existing WAN channel bandwidth. It will
guarantee that traffic classes will be scheduled in a customer router before leaving interface.
70
Using GNS3 virtual environment I have simulated a network diagram, where also configured
all routers, according our task.
Figure 53. Enterprise network traffic marking in GNS enviroment
I have started my realization with monitoring PE (R3) router. I have assured that data link is
empty and only EIGRP and ICMP packets are active.
Figure 54. Router R3 incoming statistic
Further I have activated Traffic generator and checked, that other type of traffic started
arriving to the R3 incoming interface.
Figure 55. Router R3 incoming statistic on a congested incoming interface
71
How we can see connection link now isn‟t empty and channel is rather congested. Next I
started with matching process on a router R2 LAN interface. I have classified enterprise LAN
traffic in a four groups; let‟s assume that it is enterprise inner policy.
Figure 56. Traffic matching configuration
Figure above demonstrate part of router configuration, were I classified incoming traffic in a
four classes, where each class have to match configured protocol type. Table 5 also illustrates
the packets classification policy on Enterprise LAN level.
Table 5. Enterprise‟s network inner policy
Traffic Classes Protocol description
Controlled Latency rtp audio
Controlled Load_1 eigrp, telnet, ssh
Controlled Load_2 sqlnet, sqlserver, pcanywhere
BULK DATA http, https, pop3, imap,smtp
To start marking our defined classes, we have to create policy-map, where every matched
class will be marked with DSCP value.
Figure 57. Setting DSCP values to each traffic class
On this figure we can see, how different predefined classes can be marked with agreed DSCP
values. Also to finish with marking configuration policy-map MARKING has to be bounded
with router R2 LAN interface (FE0/0).
72
Figure 58. IP packets marking process
Figure above demonstrate, how matching process occure on a router R2 LAN interface. We
can noutice, that class-map CONT_LATENCY didn‟t match any packets (I didn‟ simulate any
real-time protocol traffic), but class-map CONT_LOAD_1 has already matched and marked
565 packets.
So as we can see our interested packets are matsched and we wish, we can use some queuing
or scheduling mechanisms to provide more efficiency on an enterprise WAN interface
(FE0/1). As we have already arranged I use next Qos policy strategy: 10% - EF, 30% - af41,
20% - af31, 10% - af21. Percentages show which part of total channel bandwidth will be used
for traffic classes.
On figures below illustrated how marked packets are matched and scheduled, according our
predefined requirements. Here is the branch office Router R2 class-map and policy-map
configurations, where we can see that our marked packets are matched and scheduled before
leaving router interface.
Figure 59. Matching and scheduling already marked packets
73
This policy-map OUTPUT_POLICY has to be implemented on the Router R2 output
interface. After policy-map was activated I checked does matching and queuing process work
as it have to.
Figure 60. Scheduling process on a router WAN interface
This output shows, that all our classes are configured and scheduling and queuing process is
active. We can see that our new created class EF still empty and for example class AF41 has
already matched 1618 marked (DSCP value af41) packets.
To be sure that our marked packets obtained SP router (PE-R3) I also checked if any marked
packets arrived to router PE FE0/1 interface.
Figure 61. Accounting process on CE and PE routers
74
Information above shows that marked packets (for example af31 (identical IPP =3)), which
leave the CE router are arrived to PE router with the same IPP value.
6.5 PRACTICAL PART ANALYSIS
In a practical part of my thesis, I wanted to demonstrate the usefulness of some Qos
mechanisms. The practical work examines only a small segment of QoS implementation with
more existing QoS mechanisms which could be realized on a customer networks, as well as on
ISP networks. I have used congestion control mechanisms, which allow queue, schedule and
prioritize desired traffic types. From the examples above we can conclude, that it is possible to
guarantee quality of service for some types of traffic, especially for delay-sensitive. For
example, if we have purely delay-sensitive traffic type and congestion has been investigated
on a connection link, delay-sensitive traffic can be prioritized over the others types. If there
are a several types of delay-sensitive traffic on a network, network administrator needs to
determine, which of the critical traffic types are preferable. However, we are running a risk
that some packets from other low-priority flows can feel link capacity starvation in case of
prioritization takes a larger part of total bandwidth.
I have used LLQ technique, where I have limited the priority class by percentages from total
output interface capacity. We can also use CBWFQ mechanism to guarantee a fixed
bandwidth capacity for marked classes, but in this case time-sensitive traffic could suffer.
Therefore, before starting to implement any QoS techniques on a network, rather deep
inspection is required. It needs to be clarified which kind of traffic is critical and needs
prioritizing and the types that are not. All depends on application types, using on a networks
and requirements to each types.
75
Figure 62. Some congestion control methods comparison [17]
On a figure above I compared some congestion management types and demonstrate how
channel bandwidth can be distributed between four traffic classes‟ types. I have arranged for
each class around 25% of total channel capacity and gave some characteristic to each
considered congestion control methods using presented bandwidth sharing conditions.
In conclusion part of my practical part I want to propose in my opinion the easiest and most
optimum variant of QoS strategy, where separate communication channels are used for each
type of service. Nowadays, more and more enterprises try to integrate their services (Internet,
LAN-to-LAN connections and telephone services) to the common IP communication line. The
decision is logical due to the accessibility and lower prices for IP communication services. But
some enterprises up today try separate services using different technologies or different
communication lines for each service type. This is not an economical solution, however
guarantees that in case of any service breakdown, other service work will not be interrupted.
In my opinion, the figure below illustrates rather optimum type of QoS solution for enterprise
networks.
Figure 63. QoS startegy by various connection types [17]
This approach allows enterprise‟s networks use their services separately, which don‟t have the
common point of failure. To provide an extra guarantee in case of failure of one of the services
the backup connections has to be implemented for desirable services. In this case topology of
network diagram above has to be changed and will look somewhat differently. Providing
76
backup and other redundancy solutions to existent connection channels is already another
interesting part of immense and actual QoS theme.
I would like to finish the practical part with a general SWOT analysis of QoS implementation
efficiency.
SWOT
Strengths: Guaranteed and qualitative service for some critical traffic types (LLQ) on a
congested communication channels. Traffic differentiation and classification, according to
traffic type requirements. End to end service quality guarantee (PHB). Universal mechanism,
can work on and with network devices from different brands.
Weaknesses: Takes time for network investigation. Needs qualified and skilled network
administrators for a qualitative QoS implementation. Complicated configuration methods.
Network devices have to support QoS mechanisms (packets matching, marking, scheduling
etc). Network devices cost factor.
Opportunities: IP traffic classification and packet treatment rules determination (traffic
policing, shaping, congestion avoidance mechanisms). IP packets marking possibility and
sending them through the ISP PHB domain. QoS mechanisms flexibility and quick reaction to
network requirements changes.
Threats: Network devices performance overflowing possibilities. Lack of possibility to
guarantee service quality to every type of traffic. Packet drops possibility for some low-
priority traffic classes.
77
CONCLUSION
In presented work I have clarified that implementing several QoS techniques on a slow or
congested local-loop links could be very effective. As I can conclude QoS theme is rather
broad and it‟s practically impossible to consider and examine every nuance of QoS
mechanism in the scope of one thesis. There are numerous books dedicated to QoS theme and
every author tries to show how nowadays QoS methods and algorithms could be changed and
improved. I have described only a fraction and shown some practical methods, where IP
congested channel has been taken under control and significant traffic were treated using some
types of QoS methods.
Practice and real life show that every problem connected with data channel overload on an
Enterprise connection links is unique and have to be considered and investigated separately.
There is no universal solution that could be employed on every slow connection links, but in
some cases QoS methods implementation will help solve serious problems on congested IP
links. I have also understood that QoS technique is a tool, which can only be used for IP
packets treatment and scheduling only on a congested IP channel. QoS mechanism can‟t
guarantee desirable quality of services to every traffic flow, but as per demonstration in a
practical part can guarantee some flow quality for certain traffic types.
Internet Service Providers also try to expand their backbone connection links as far as possible
and usually don‟t interest of using some QoS mechanisms on a core network. The principle is
analogical to that of city traffic, where traffic lights are implemented as temporary solution to
urban traffic congestions. But driving through rural areas eliminates the need of traffic lights
making the trip more efficient. ISP tries to deploy QoS at an area with high links congestion
probability. It is usually ISP access or distribution layer networks, where customer‟s network
connects to ISP network. In these network places congestion can occur and ISP uses different
QoS methods to guarantee appropriate bandwidth for critical and time-sensitive traffic types
(voip or video streaming (IPTV or VOD)), going from the ISP towards the customer‟s
network.
Due to the rapid development in Information Technology area and new IT services
appearance, humans can‟t imagine their lives without connection to the Internet or other
network resources services. Without a doubt IT world makes our lives easier and more
comfortable. We have to recognize that some part of these achievements belongs to QoS
mechanisms, that where developed to effective use of every part of communication network.
I would like to complete my thesis with a common definition of QoS. As we may suspect there
are numerous different QoS definitions, but let‟s try to give the universal and brief
approximation, which reflect the basic meaning and core idea of Quality of Service.
78
“A QoS mechanism is a set of procedures/protocols designed to determine network elements
to service differently the competing applications by following a set of predefined
rules/policies” [15]
79
LIST OF REFERENCES
[1] http://www.satmagazine.com/cgi-bin/display_article.cgi?number=1609810110 (“No End
in Sight for Bandwidth Demand Growth” Satmagazine, by Mark Dankberg)
[2] http://en.wikipedia.org/wiki/Quality_of_service (materials about queuing mechanisms)
[3] “Deploying QoS for Cisco IP and next Generation Networks” (Definitive guide, Vinod
Joseph, Brett Chapman)
[4] “QoS over Heterogeneous Networks” (Mario Marchese, Department of Communications,
Computer and System Science University of Genoa, Italy)
[5] “Качество обслуживания в сетях IP” (Russian translation “IP Quality of Service”,
Srinivas Vegesna)
[6] http://www.cisco.com/en/US/docs/solutions/Enterprise/WAN_and_MAN/QoS_SRND/QoSIntro.html
(Quality of Service Design Overview)
[7] CCNP Cisco Academy course (Optimizing Converged Networks)
[8] http://en.wikipedia.org/wiki/Differentiated_services (Differentiated services)
[9] http://www.gns3.net (Graphical network simulator that allows simulation of complex
networks)
[10] http://www.pingplotter.com/ (Graphically Monitoring tool)
[11] http://www.cisco.com/en/US/technologies/tk543/tk766/technologies_white_paper09186a00800a3e2f.html
(DiffServ - The Scalable End-to-End QoS Model)
[12] http://network.xsp.ru/3_11.php (Cisco QoS for beginners)
[13] http://www.cisco.com Different Cisco devices configuration guides, as an example http://www.cisco.com/univercd/cc/td/doc/product/software/ios122/122newft/122t/122t8/dtnbarad.htm#wp1085738
[14] http://www.cisco.com/en/US/docs/ios/qos/configuration/guide/12_4/qos_12_4_book.html
(Cisco IOS Quality of Service Solutions Configuration Guide, Release 12.4)
[15] “Scalable Solutions for QoS-enabled IP Networks” Avadora Dumitrescu, Thesis for
degree of Doctor of Technology.
[16] http://www.allaboutmarketresearch.com/internet.htm (“INTERNET GROWTH AND
STATS”)
80
[17] Figures were designed using MS Office Visio 2007
[18] http://en.wikipedia.org/wiki/NBAR
[19] “Content Networking Fundamentals” Silvano Da Ros, Cisco Press, Copyright© 2006
Cisco Systems, Inc.
[20] “QoS for IP/MPLS Networks” Santiago Alvarez, Copyright© 2006 Cisco Systems, Inc
