View
214
Download
1
Category
Preview:
Citation preview
ENGS 116 Lecture 19 1
Interconnection Networks
Vincent H. Berk
November 14, 2005
Reading for Friday: 8.1-8.7
Reading for Monday: 8.8-8.13
ENGS 116 Lecture 19 2
Project Reports
• Due by Beginning of class next Monday, November 24
• Content
– Introduction and description of the topic
– Coverage of topic: breadth/depth, appropriate background information
– Analysis and discussion
– References: correct citations, proper form
• Writing
– Spelling
– Grammar
– Style and presentation
• Assume that the reader is familiar with basic architecture concepts
• Must use appropriate citations.
• Argument all your decisions.
• Email all source code to <vberk@Dartmouth.EDU>
ENGS 116 Lecture 19 3
Project Presentations
• 15 minutes – absolutely no more – Practice your timing!
• All group members talk
• 24th (4) of November, 1st (4) of December
• Present:– research question
– approach
– results
– conclusions
• EVERYONE ATTENDS these presentations
ENGS 116 Lecture 19 4
Networks
• Common topics of conversation:– direct (point-to-point) vs. indirect (multi-hop)– topology (e.g., bus, ring, directed acyclic graph, star)– routing algorithms– switching (aka multiplexing)– wiring (e.g., choice of media, copper, coax, fiber)
• What really matters:– latency– bandwidth– cost– reliability
ENGS 116 Lecture 19 5
ABCs of Networks
• Starting point: Send bits between 2 computers
• Queue on each end
• Can send both ways (“Full Duplex”)
• Rules for communication? “protocol”
– Inside a computer:
• Loads/Stores: Request (Address) & Response (Data)
• Need request & response signaling
– Name for standard group of bits sent: packet
ENGS 116 Lecture 19 6
A Simple Example
• What is the format of a packet? (Protocol)
– Fixed? Number of bytes?
Request/Response
Address/Data
1 bit 32 bits
0: Please send data from address1: Packet contains data corresponding to request
ENGS 116 Lecture 19 7
Questions About Simple Example
• What if more than 2 computers want to communicate?
– Need computer address field (destination) in packet
• What if packet is garbled in transit?
– Add error detection field in packet (e.g., CRC)
• What if packet is lost?
– More elaborate protocols to detect loss (e.g., NAK, ARQ, time outs)
• What if multiple processes per machine?
– Queue per process
• Questions such as these lead to more complex protocols and packet formats
ENGS 116 Lecture 19 8
A Simple Example Revisited
• What is the format of a packet?
– Fixed? Number of bytes?
Request/Response
Address/Data
2 bits 32 bits
00: Request—Please send data from address01: Reply—Packet contains data corresponding to request10: Acknowledge request11: Acknowledge reply
4 bits
CRC
ENGS 116 Lecture 19 9
Additional Background
• Connection of 2 or more networks: Internetworking
• 3 cultures for 3 classes of networks
– SAN: server (storage) networks, performance
– LAN: workstations, cost
– WAN: telecommunications, long range
• Cost
• Performance (BW, latency)
• Reliability
ENGS 116 Lecture 19 10
Interconnections (Networks)• Examples:
– SAN networks (infiniband): 100s nodes; ≤ 100 meters per link– Local Area Networks (Ethernet): 100s nodes; ≤ 1000 meters– Wide Area Network (ATM): 1000s nodes; ≤ 5,000,000 meters
Interconnect
SW Interface
HW Interface
Node
Link
SW Interface
HW Interface
Node
Link
SW Interface
HW Interface
Node
Link
SW Interface
HW Interface
Node
Link...
...
ENGS 116 Lecture 19 11
Software to Send and Receive
• SW Send steps
1: Application copies data to OS buffer
2: OS calculates checksum, starts timer
3: OS sends data to network interface HW and says start
• SW Receive steps
3: OS copies data from network interface HW to OS buffer
2: OS calculates checksum, if matches send ACK; if not, deletes message (sender resends when timer expires)
1: If OK, OS copies data to user address space and signals application to continue
• Sequence of steps for SW: protocol
– Example similar to UDP/IP protocol in UNIX
ENGS 116 Lecture 19 12
Network Performance Measures
SW Interface
HW Interface
Node
LinkBandwidth
Overhead
Link
SW Interface
HW Interface
Node
LinkBandwidth
Overhead
Link
Bisection Bandwidth
Interconnect
Latency
...
...
ENGS 116 Lecture 19 13
Universal Performance Metrics
Sender
Receiver
SenderOverhead
Transmission time(size ÷ bandwidth)
Transmission time(size ÷ bandwidth)
Time ofFlight
ReceiverOverhead
Transport Latency
Total Latency = Sender Overhead + Time of Flight + Message Size ÷ BW + Receiver Overhead
Includes header/trailer in BW calculation?
Total Latency
(processorbusy)
(processorbusy)
ENGS 116 Lecture 19 14
Simplified Latency Model
• Total Latency ≈ Overhead + Message Size / BW
• Overhead = Sender Overhead + Time of Flight + Receiver Overhead
• Example: show what happens as we vary the following
– Overhead: 1, 25, 500 µsec
– BW: 10, 100, 1000 Mbit/sec (factors of 10)
– Message Size: 16 Bytes to 4 MB (factors of 4)
• If overhead is 500 µsec, how big a message is needed to get > 10 Mb/s of bandwidth?
ENGS 116 Lecture 19 15
0.01
0.1
1
10
100
1000
16 64 256
1024
4096
16384
65536
262144
10485764194304
Message Size (bytes)
Eff
ecti
ve B
an
dw
idth
(M
bit
s/s
ec)
o1, bw10
o1, bw100
o1, bw1000
o25, bw10
o25, bw100
o25, bw1000
o500, bw10
o500, bw100
o500, bw1000
o1, bw1000
o500, bw1000o25, bw1000
o500, bw100
o500, bw10
bw1000
bw100
bw10
o1, bw100
Overhead, Bandwidth, Size
ENGS 116 Lecture 19 16
Measurement: Sizes of Message for NFS
• 95% messages, 30% bytes for packets ≤ 200 bytes• > 50% data transferred in packets = 8KB
Packet size
Cu
mu
lati
ve %
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0 1024 2048 3072 4096 5120 6144 7168 8192
Msgs
Bytes Why?
ENGS 116 Lecture 19 17
HW Interface Issues• Where to connect network to computer?
– Cache consistent to avoid flushes? ( memory bus)
– Latency and bandwidth? ( memory bus)
– Standard interface card? ( I/O bus)
– MPP memory bus; LAN, WAN I/O bus
$
CPU
L2 $
Memory Bus
Memory Bus Adaptor
I/O bus
I/OController
I/OController
Network
ideal: high bandwidth, low latency, standard interface
Network
ENGS 116 Lecture 19 18
SW Interface Issues
• How to connect network to software?
– Programmed I/O? (low latency)
– DMA? (best for large messages)
– Receiver interrupted or received polls?
• Things to avoid
– Invoking operating system in common case
– Operating at uncached memory speed (e.g., check status of network interface)
ENGS 116 Lecture 19 19
CM-5 Software Interface
• CM-5 example (MPP)– Time per poll 1.6 secs; time
per interrupt 19 secs
– Minimum time to handle message: 0.5 secs
– Enable/disable 4.9/3.8 secs
• As rate of messages arriving changes, use polling or interrupt? – Solution: Always enable
interrupts, have interrupt routine poll until no messages pending
– Low arrival rate interrupt
– High arrival rate polling
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100
message interarrival (µsecs)
mes
sage
ove
rhea
d (µ
secs
)
Polling
Interrupts
Time between messages
Overhead
ENGS 116 Lecture 19 20
Network Media
Copper, 1mm thick, twisted to avoidantenna effect (telephone)
Used by cable companies: high BW, good noise immunity
3 parts are cable, light source, light detector.
Twisted Pair:
Coaxial Cable:
Copper coreInsulator
Braided outer conductorPlastic Covering
Fiber Optics
Transmitter– L.E.D– Laser Diode
Receiver– Photodiode
lightsource Silica
Total internalreflectionAir
ENGS 116 Lecture 19 21
Connecting Multiple Computers
• Shared Media vs. Switched: pairs communicate at same time, “point-to-point” connections
• Aggregate BW in switched network is many times that of shared
– point-to-point faster since no arbitration, simpler interface
• Arbitration in shared network?– Central arbiter for LAN?
– Listen to check if being used (“Carrier Sensing”)
– Listen to check if collision (“Collision Detection”)
– Random resend to avoid repeated collisions; not fair arbitration;
– OK if low utilization
(a.k.a. data switching interchanges, multistageinterconnection networks,interface message processors)
Switch
NodeNode
Node Node
Switched Media (CM-5, ATM)
Node Node Node
Shared Media (Ethernet)
ENGS 116 Lecture 19 22
Switch Topology
• Structure of the interconnect
• Determines
– Degree: number of links from a node
– Diameter: max number of links crossed between nodes
– Average distance: number of hops to random destination
– Bisection: minimum number of links that separate the network into two halves (worst case)
• Warning: these three-dimensional drawings must be mapped onto chips and boards which are essentially two-dimensional media
– Elegant when sketched on the blackboard may look awkward when constructed from chips, cables, boards, and boxes (largely 2D)
ENGS 116 Lecture 19 24
Examples of Static Interconnection Network Topologies
c) Star
a) Busb) Linear array
e) Tree
f) Near-neighbor mesh
d) Ring
g) Completely connected
h) 3–cube (hypercube)
ENGS 116 Lecture 19 26
Important Topologies
N = 1024 Type Degree Diameter Avg Dist Bisection Diam Avg D
1D mesh ≤ 2 N-1 N/3 1
2D mesh ≤ 4 2(N1/2 - 1) 2N1/2 / 3 N1/2 63 21
3D mesh ≤ 6 3(N1/3 - 1) 3N1/3 / 3 N2/3 ~ 30 ~ 10
Ring 2 N / 2 N/4 2
2D torus 4 N1/2 N1/2 / 2 2N1/2 32 16
Hypercube n n = LogN n/2 N/2 10 5
ENGS 116 Lecture 19 27
Figure 8.14 A fat-tree topology for 16 nodes.
01
23
00 0 1
45
67
01 1 1
89
1011
02 2 1
1213
1415
03 3 1
0
0
0 10 2
0 3
0
Recommended