TCP Pacing in Data Center Networks

TCP Pacing in Data Center Networks

1

Monia Ghobadi, Yashar Ganjali

Department of Computer Science, University of Toronto{monia, yganjali}@cs.toronto.edu

TCP, Oh TCP!

2

TCP, Oh TCP!

๏ TCP congestion control

2

TCP, Oh TCP!


๏Focus on evolution of cwnd over RTT.

2

TCP, Oh TCP!



๏ Damages

2

TCP, Oh TCP!



๏ Damages

๏ TCP pacing

2

The Tortoise and the Hare: Why Bother Pacing?

3


๏ Renewed interest in pacing for the data center environment๏Small buffer switches

๏Small round-trip times

๏Disparity between the total capacity of the network and the capacity of individual queues

3


๏ Renewed interest in pacing for the data center environment๏Small buffer switches

๏Small round-trip times

๏Disparity between the total capacity of the network and the capacity of individual queues

๏Focus on tail latency cause by short-term

unfairness in TCP

3

TCP Pacing’s Potential

4


๏ Better link utilization on small switch buffers

4


๏ Better link utilization on small switch buffers๏ Better short-term fairness among flows of

similar RTTs:๏ Improves worst-flow latency

4


๏ Better link utilization on small switch buffers๏ Better short-term fairness among flows of

similar RTTs:๏ Improves worst-flow latency

๏ Allows slow-start to be circumvented๏Saving many round-trip time๏May allow much larger initial congestion window to

be used safely

4

Contributions

5

Contributions

๏ Effectiveness of TCP pacing in data centers.

5

Contributions


๏ Benefits of using paced TCP diminish as we increase

the number of concurrent connections beyond a

certain threshold (Point of Inflection).

5

Contributions





๏ Inconclusive results in previous works.

5

Contributions






๏ Inter-flow bursts.

5

Contributions






๏ Inter-flow bursts.

๏ Test-bed experiments.

5

Inter-flow Bursts

6

Inter-flow Bursts

๏ C: bottleneck link capacity

6

Inter-flow Bursts

๏ C: bottleneck link capacity๏ Bmax :buffer size

6

Inter-flow Bursts

๏ C: bottleneck link capacity๏ Bmax :buffer size๏ N: longed lived flows.

6

Inter-flow Bursts

๏ C: bottleneck link capacity๏ Bmax :buffer size๏ N: longed lived flows.๏ W: packets in every RTT in paced or non-paced

manner.

6

Inter-flow Bursts


manner.

6

0 RTT 2RTT 3RTT

Inter-flow Bursts


manner.

6

0 RTT 2RTT 3RTT

Inter-flow Bursts


manner.

6

0 RTT 2RTT 3RTT

Inter-flow Bursts


manner.

6

0 RTT 2RTT 3RTT

Inter-flow Bursts


manner.๏ X: Inter-flow burst

6

0 RTT 2RTT 3RTT

Inter-flow Bursts


manner.๏ X: Inter-flow burst

6

0 RTT 2RTT 3RTT

Modeling

7

Modeling

7

best case of non-paced

Modeling

7

best case of non-paced worst case of paced

Modeling

7

RTT


Modeling

7

RTT RTT


Modeling

7

RTT RTT


Modeling

7

RTT RTT


Experimental Studies

8


๏ Flow of sizes 1,2, 3 MB between servers and clients.

8


๏ Flow of sizes 1,2, 3 MB between servers and clients.๏ Bottleneck BW: 1,2, 3 Gbps

8


๏ Flow of sizes 1,2, 3 MB between servers and clients.๏ Bottleneck BW: 1,2, 3 Gbps๏ RTT: 1 to 100ms

8


๏ Flow of sizes 1,2, 3 MB between servers and clients.๏ Bottleneck BW: 1,2, 3 Gbps๏ RTT: 1 to 100ms๏ Bottleneck utilization, Drop rate, average and tail

FCT

8

Base-Case Experiment: One flow vs Two flows, 64KB of buffering, Utilization/Drop/FCT

9


9

No congestion


9

No congestion Congestion

0 50 100 150 200 250 3000

50

100

150

200

250

300

Time (sec)

Bottle

neck

Lin

k U

tiliz

atio

n (

Mbps)

pacednon!paced


9


0 50 100 150 200 250 3000

50

100

150

200

250

300

Time (sec)

Bottle

neck

Lin

k U

tiliz

atio

n (

Mbps)

pacednon!paced


9


0 50 100 150 200 250 3000

50

100

150

200

250

300

Time (sec)

Bottle

neck

Lin

k U

tiliz

atio

n (

Mbps)

pacednon!paced


9

38%


0 50 100 150 200 250 3000

50

100

150

200

250

300

Time (sec)

Bottle

neck

Lin

k U

tiliz

atio

n (

Mbps)

pacednon!paced


9

38%


0 50 100 150 200 250 3000

50

100

150

200

250

300

Time (sec)

Bottle

neck

Lin

k U

tiliz

atio

n (

Mbps)

pacednon!paced


9

38%


0 50 100 150 200 250 3000

50

100

150

200

250

300

Time (sec)

Bottle

neck

Lin

k U

tiliz

atio

n (

Mbps)

pacednon!paced


90.1 0.2 0.3 0.40.0630.0390

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Flow Completion Time (sec)

CD

F

pacednon!paced

38%


0 50 100 150 200 250 3000

50

100

150

200

250

300

Time (sec)

Bottle

neck

Lin

k U

tiliz

atio

n (

Mbps)

pacednon!paced


90.1 0.2 0.3 0.40.0630.0390

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1


CD

F

pacednon!paced

38%


1RTT

0 50 100 150 200 250 3000

50

100

150

200

250

300

Time (sec)

Bottle

neck

Lin

k U

tiliz

atio

n (

Mbps)

pacednon!paced


90.1 0.2 0.3 0.40.0630.0390

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1


CD

F

pacednon!paced

38%


1RTT2RTTs

0 50 100 150 200 250 3000

50

100

150

200

250

300

Time (sec)

Bottle

neck

Lin

k U

tiliz

atio

n (

Mbps)

pacednon!paced


90.1 0.2 0.3 0.40.0630.0390

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1


CD

F

pacednon!paced

1 2 3 50.06 70.50.20

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1


CD

F

pacednon!paced

38%


1RTT2RTTs

0 50 100 150 200 250 3000

50

100

150

200

250

300

Time (sec)

Bottle

neck

Lin

k U

tiliz

atio

n (

Mbps)

pacednon!paced


90.1 0.2 0.3 0.40.0630.0390

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1


CD

F

pacednon!paced

1 2 3 50.06 70.50.20

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1


CD

F

pacednon!paced

38%


1RTT2RTTs 2RTTs

Multiple flows: Link Utilization/Drop/LatencyBuffer size 1.7% of BDP, varying number of flows

10


10

0 10 20 30 40 50 60 70 80 90 1000

200

400

600

800

1000

Number of Flows

Bo

ttle

ne

ck L

ink

Util

iza

tion

(M

bp

s)

paced

non!paced

PoIN*


10

0 10 20 30 40 50 60 70 80 90 1000

200

400

600

800

1000

Number of Flows

Bo

ttle

ne

ck L

ink

Util

iza

tion

(M

bp

s)

paced

non!paced

PoIN*

0 10 20 30 40 50 60 70 80 90 1000

0.2

0.4

0.6

0.8

1

Number of Flows

Bo

ttle

ne

ck L

ink

Dro

p (

%)

pacednon!paced

PoIN*


10

0 10 20 30 40 50 60 70 80 90 1000

0.2

0.4

0.6

0.8

1

Number of Flows

Ave

rag

e F

CT

(se

c)

pacednon!paced

N*

PoI

0 10 20 30 40 50 60 70 80 90 1000

200

400

600

800

1000

Number of Flows

Bo

ttle

ne

ck L

ink

Util

iza

tion

(M

bp

s)

paced

non!paced

PoIN*

0 10 20 30 40 50 60 70 80 90 1000

0.2

0.4

0.6

0.8

1

Number of Flows

Bo

ttle

ne

ck L

ink

Dro

p (

%)

pacednon!paced

PoIN*


100 10 20 30 40 50 60 70 80 90 1000

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

Number of Flows

99

th P

erc

en

tile

FC

T (

sec)

pacednon!paced

N*

PoI

0 10 20 30 40 50 60 70 80 90 1000

0.2

0.4

0.6

0.8

1

Number of Flows

Ave

rag

e F

CT

(se

c)

pacednon!paced

N*

PoI

0 10 20 30 40 50 60 70 80 90 1000

200

400

600

800

1000

Number of Flows

Bo

ttle

ne

ck L

ink

Util

iza

tion

(M

bp

s)

paced

non!paced

PoIN*

0 10 20 30 40 50 60 70 80 90 1000

0.2

0.4

0.6

0.8

1

Number of Flows

Bo

ttle

ne

ck L

ink

Dro

p (

%)

pacednon!paced

PoIN*


100 10 20 30 40 50 60 70 80 90 1000

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

Number of Flows

99

th P

erc

en

tile

FC

T (

sec)

pacednon!paced

N*

PoI

0 10 20 30 40 50 60 70 80 90 1000

0.2

0.4

0.6

0.8

1

Number of Flows

Ave

rag

e F

CT

(se

c)

pacednon!paced

N*

PoI

0 10 20 30 40 50 60 70 80 90 1000

200

400

600

800

1000

Number of Flows

Bo

ttle

ne

ck L

ink

Util

iza

tion

(M

bp

s)

paced

non!paced

PoIN*

0 10 20 30 40 50 60 70 80 90 1000

0.2

0.4

0.6

0.8

1

Number of Flows

Bo

ttle

ne

ck L

ink

Dro

p (

%)

pacednon!paced

PoIN*

• Number of concurrent connections increase beyond a certain point the benefits of pacing diminish.


11


110 10 20 30 40 50 60 70 80 90 1000

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

Number of Flows

99

th P

erc

en

tile

FC

T (

sec)

paced

non!paced

N*

PoI

0 10 20 30 40 50 60 70 80 90 1000

0.2

0.4

0.6

0.8

1

Number of Flows

Ave

rag

e F

CT

(se

c)

paced

non!paced

N*

PoI

0 10 20 30 40 50 60 70 80 90 1000

200

400

600

800

1000

Number of Flows

Bo

ttle

ne

ck L

ink

Util

iza

tion

(M

bp

s)

pacednon!paced

N*

PoI

0 20 40 60 80 1000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Number of flows sharing the bottleneck

Bottl

enec

k lin

k dr

op(%

)

pacednonpaced


110 10 20 30 40 50 60 70 80 90 1000

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

Number of Flows

99

th P

erc

en

tile

FC

T (

sec)

paced

non!paced

N*

PoI

0 10 20 30 40 50 60 70 80 90 1000

0.2

0.4

0.6

0.8

1

Number of Flows

Ave

rag

e F

CT

(se

c)

paced

non!paced

N*

PoI

0 10 20 30 40 50 60 70 80 90 1000

200

400

600

800

1000

Number of Flows

Bo

ttle

ne

ck L

ink

Util

iza

tion

(M

bp

s)

pacednon!paced

N*

PoI

0 20 40 60 80 1000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8


Bottl

enec

k lin

k dr

op(%

)

pacednonpaced

• Aagarwal et al.: Don’t pace!

• 50 flows, BDP 1250 packets and buffer size

312 packets

• N* = 8 flows.

• Kulik et al.: Pace!

• 1 flow, BDP 91 packets, buffer size 10

packets.

• N* = 9 flows.

N* vs. Buffer

12

50 100 150 200 2500

200

400

600

800

1000

Buffer Size (KB)

Bo

ttle

ne

ck L

ink

Util

iza

tion

(M

bp

s)

pacednon!paced

N* vs. Buffer

12

50 100 150 200 2500

200

400

600

800

1000

Buffer Size (KB)

Bo

ttle

ne

ck L

ink

Util

iza

tion

(M

bp

s)

pacednon!paced

N* vs. Buffer

1250 100 150 200 2500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Buffer size(KB)

Bottl

enec

k lin

k dr

op(%

)

pacednonpaced

50 100 150 200 2500

200

400

600

800

1000

Buffer Size (KB)

Bo

ttle

ne

ck L

ink

Util

iza

tion

(M

bp

s)

pacednon!paced

50 100 150 200 2500

0.2

0.4

0.6

0.8

1

1.2

1.4

Buffer Size (KB)

Ave

rag

e �C

T (

se

c)

pacednon!paced

N* vs. Buffer

1250 100 150 200 2500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Buffer size(KB)

Bottl

enec

k lin

k dr

op(%

)

pacednonpaced

50 100 150 200 2500

0.4

0.8

1.2

1.6

2

2.4

2.8

Buffer Size (KB)

99

th P

erc

en

tile

�CT

(se

c)

pacednon!paced

50 100 150 200 2500

200

400

600

800

1000

Buffer Size (KB)

Bo

ttle

ne

ck L

ink

Util

iza

tion

(M

bp

s)

pacednon!paced

50 100 150 200 2500

0.2

0.4

0.6

0.8

1

1.2

1.4

Buffer Size (KB)

Ave

rag

e �C

T (

se

c)

pacednon!paced

N* vs. Buffer

1250 100 150 200 2500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Buffer size(KB)

Bottl

enec

k lin

k dr

op(%

)

pacednonpaced

50 100 150 200 2500

0.4

0.8

1.2

1.6

2

2.4

2.8

Buffer Size (KB)

99

th P

erc

en

tile

�CT

(se

c)

pacednon!paced

50 100 150 200 2500

200

400

600

800

1000

Buffer Size (KB)

Bo

ttle

ne

ck L

ink

Util

iza

tion

(M

bp

s)

pacednon!paced

50 100 150 200 2500

0.2

0.4

0.6

0.8

1

1.2

1.4

Buffer Size (KB)

Ave

rag

e �C

T (

se

c)

pacednon!paced

N* vs. Buffer

1250 100 150 200 2500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Buffer size(KB)

Bottl

enec

k lin

k dr

op(%

)

pacednonpaced

Clustering Effect: The probability of packets from a flow being followed by packets from other flows

13


Non-paced: Packets of each flow are clustered together.

13


Non-paced: Packets of each flow are clustered together.

Paced: Packets of different flows are multiplexed.

13

Drop Synchronization: Number of Flows Affected by Drop Event

14


14

NetFPGA router to count the number of flows affected by drop events.


14

0 10 20 30 400

0.10.20.30.40.50.60.70.80.9

1

Number of Flows Affected by Drop Event

CD

F

paced

non!paced

0 20 40 60 800

0.10.20.30.40.50.60.70.80.9

1


CD

F

paced

non!paced

0 50 100 150 200 250 300 350 4000

0.10.20.30.40.50.60.70.80.9

1


CD

F

paced

non!paced

N: 48 N: 96 N: 384



14

0 10 20 30 400

0.10.20.30.40.50.60.70.80.9

1


CD

F

paced

non!paced

0 20 40 60 800

0.10.20.30.40.50.60.70.80.9

1


CD

F

paced

non!paced

0 50 100 150 200 250 300 350 4000

0.10.20.30.40.50.60.70.80.9

1


CD

F

paced

non!paced

N: 48 N: 96 N: 384


Future Trends for Pacing: per-egress pacing.

15


15

6 12 24 48 96 1920

0.2

0.4

0.6

0.8

1

Number of Flows

Ave

rag

e R

CT

(se

c)

per!flow pacednon!pacedper!host + per!flow paced

PoIN*

6 12 24 48 96 1920

200

400

600

800

1000

Number of Flows

Bo

ttle

ne

ck L

ink

Util

iza

tion

(M

bp

s)


PoIN*

6 12 24 48 96 1920

0.20.40.60.8

11.21.41.61.8

22.2

Number of Flows

99

th P

erc

en

tile

RC

T (

sec)


N*PoI

6 12 24 48 96 1920

5

10

15

20

Number of flows

Bottle

neck

Lin

k D

rop (

%)

per!flow pacednonpacedper!host + per!flow paced


15

6 12 24 48 96 1920

0.2

0.4

0.6

0.8

1

Number of Flows

Ave

rag

e R

CT

(se

c)


PoIN*

6 12 24 48 96 1920

200

400

600

800

1000

Number of Flows

Bo

ttle

ne

ck L

ink

Util

iza

tion

(M

bp

s)


PoIN*

6 12 24 48 96 1920

0.20.40.60.8

11.21.41.61.8

22.2

Number of Flows

99

th P

erc

en

tile

RC

T (

sec)


N*PoI

6 12 24 48 96 1920

5

10

15

20

Number of flows

Bottle

neck

Lin

k D

rop (

%)



15

6 12 24 48 96 1920

0.2

0.4

0.6

0.8

1

Number of Flows

Ave

rag

e R

CT

(se

c)


PoIN*

6 12 24 48 96 1920

200

400

600

800

1000

Number of Flows

Bo

ttle

ne

ck L

ink

Util

iza

tion

(M

bp

s)


PoIN*

6 12 24 48 96 1920

0.20.40.60.8

11.21.41.61.8

22.2

Number of Flows

99

th P

erc

en

tile

RC

T (

sec)


N*PoI

6 12 24 48 96 1920

5

10

15

20

Number of flows

Bottle

neck

Lin

k D

rop (

%)



15

6 12 24 48 96 1920

0.2

0.4

0.6

0.8

1

Number of Flows

Ave

rag

e R

CT

(se

c)


PoIN*

6 12 24 48 96 1920

200

400

600

800

1000

Number of Flows

Bo

ttle

ne

ck L

ink

Util

iza

tion

(M

bp

s)


PoIN*

6 12 24 48 96 1920

0.20.40.60.8

11.21.41.61.8

22.2

Number of Flows

99

th P

erc

en

tile

RC

T (

sec)


N*PoI

6 12 24 48 96 1920

5

10

15

20

Number of flows

Bottle

neck

Lin

k D

rop (

%)



15

6 12 24 48 96 1920

0.2

0.4

0.6

0.8

1

Number of Flows

Ave

rag

e R

CT

(se

c)


PoIN*

6 12 24 48 96 1920

200

400

600

800

1000

Number of Flows

Bo

ttle

ne

ck L

ink

Util

iza

tion

(M

bp

s)


PoIN*

6 12 24 48 96 1920

0.20.40.60.8

11.21.41.61.8

22.2

Number of Flows

99

th P

erc

en

tile

RC

T (

sec)


N*PoI

6 12 24 48 96 1920

5

10

15

20

Number of flows

Bottle

neck

Lin

k D

rop (

%)


Conclusions and Future work

๏ Re-examine TCP pacing’s effectiveness:๏Demonstrate when TCP pacing brings benefits in

such environments.๏ Inter-flow burstiness

๏ Burst-pacing vs. packet-pacing.๏ Per-egress pacing.

16

Renewed Interest

17

Traffic Burstiness Survey

๏‘Bursty’ is a word with no agreed meaning. How do you define a bursty traffic?

๏If you are involved with a data center, is your data center traffic bursty?

๏ If yes, do you think that it will be useful to supress the burstiness in your traffic?

๏ If no, are you already supressing the burstiness? How? Would you anticipate the traffic becoming burstier in the future?

18

[email protected]

mailto:[email protected]

mailto:[email protected]

19

Base-Case Experiment: One RPC vs Two RPCs, 64KB of buffering, Latency

20

Multiple flows: Link Utilization/Drop/LatencyBuffer size: 6% of BDP, varying number of

21

Base-Case Experiment: One RPC vs Two RPCs, 64KB of buffering, Latency / Queue Occupancy

22


22


22


22

Functional test

23

Functional test

23

Functional test

23

Functional test

23

RPC vs. Streaming

24

1GE10GE 10GE

RTT = 10ms

Paced by ack clocking

Bursty

Zooming in more on the paced flow

25


26


26

0 10 20 30 40 50 60 70 80 90 1000

200

400

600

800

1000

Number of Flows

Bo

ttle

ne

ck L

ink

Util

iza

tion

(M

bp

s)

pacednon!paced

N*PoI


26

0 10 20 30 40 50 60 70 80 90 1000

200

400

600

800

1000

Number of Flows

Bo

ttle

ne

ck L

ink

Util

iza

tion

(M

bp

s)

pacednon!paced

N*PoI

0 20 40 60 80 100−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1


Bottl

enec

k lin

k dr

op(%

)

pacednonpaced


26

0 10 20 30 40 50 60 70 80 90 1000

0.2

0.4

0.6

0.8

1

Number of Flows

Ave

rag

e F

CT

(se

c)

pacednon!paced

PoIN*

0 10 20 30 40 50 60 70 80 90 1000

200

400

600

800

1000

Number of Flows

Bo

ttle

ne

ck L

ink

Util

iza

tion

(M

bp

s)

pacednon!paced

N*PoI

0 20 40 60 80 100−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1


Bottl

enec

k lin

k dr

op(%

)

pacednonpaced


260 10 20 30 40 50 60 70 80 90 1000

0.4

0.8

1.2

1.6

2

2.4

2.8

3.2

3.6

4

Number of Flows

99

th P

erc

en

tile

FC

T (

sec)

pacednon!paced

PoIN*

0 10 20 30 40 50 60 70 80 90 1000

0.2

0.4

0.6

0.8

1

Number of Flows

Ave

rag

e F

CT

(se

c)

pacednon!paced

PoIN*

0 10 20 30 40 50 60 70 80 90 1000

200

400

600

800

1000

Number of Flows

Bo

ttle

ne

ck L

ink

Util

iza

tion

(M

bp

s)

pacednon!paced

N*PoI

0 20 40 60 80 100−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1


Bottl

enec

k lin

k dr

op(%

)

pacednonpaced

Documents

TCP Pacing in Data Center Networks