JAWS (A Framework for High-Performance Web Servers) TP Version 1.1

JAWS(A FRAMEWORK FOR HIGH-PERFORMANCE WEB

SERVERS)

TP VERSION 1.1

Son [email protected]

Microsoft MVPDevpia Architecture Sysop

EvaCast Study Leader (http://www.EvaCast.net)

04/22/2023http://www.EvaCast.net 2

Motivation

Architect

What’s the Good Web Server?

Throughput

Short Re-

sponse Time

Reli-able

Trans-action

Scala-bility


Motivation

ConcurrencyProtocol

& Filter

I/O Cache


Motivation

Concur-rency

Thread-per-Ses-

sionThread-per Re-quest

Thread Pool

I/O

Syn-chronous

Reactvie

Asyn-chronous

Cache

LRU

LFU

Structured Cache

Hinted Cache

Protocol&

Filter

Read Re-quest

Parse Re-questParse

HeaderPerform RequestLog Re-quest


Seed Paper

James C. Hu, Douglas Schmidt,

JAWS: A Framework for High-Performance Web Servers,

Domain-Specific Application Frameworks: Frame-works Experience By Industry,

John Wiley & Sons, October, 1999.

Adobe Acrobat 7.0 Document


Agenda Motivation Applying Patterns & Frameworks to Web Server

Common Pitfalls of Developing Web Server Overcoming Web Server Pitfalls Relationship Between Frameworks, Patterns & Other Re-

use Technique

The JAWS Web Server Overview of the JAWS Framework Overview of the Design Pattern in JAWS Concurrency Strategy I/O Strategy Protocol Pipeline Strategy File Caching Strategy

Web Server Benchmarking Concluding Remarks


2. Applying Patterns & Frameworks to Web Server Common Pitfalls of Developing Web Server

Software

Overcoming Web Server Pitfalls with Patterns and Frameworks

Relationship between Frameworks, Patterns, and Other Reuse

Techniques


2.1 Common Pitfalls of Developing Web Server SW Excessive low-level detail

Continuous rediscovery & reinvention of incompatible higher-level programming abstraction.

High potential for errors

Lack of portability

Steep learning curve

Inability to handle increasing complexity


2.2 Overcoming Web Server Pitfalls with Patterns & Frame-works Limitations of Class Library

do not capture the control flow and collaboration among families of related SW components.

Class Library reuse often re-invent and re-implement the overall Software architecture

for each new Application.


2.2 Overcoming Web Server Pitfalls with Patterns & Frame-works The Benefits of Patterns

Expert’s Design Experiences Good Methods to use Framework The infrastructure of Framework

The Benefits of Frameworks Expert’s intellectual products Help developers avoid costly re-invention


2.3 Relationship Between Frameworks, Pat-terns, and Other Reuse Techniques

Class Libraries vs Framework

App SpecificLogic

OO Design

EventLoop

DATABASE ADTs

MATH NETWORKING

GRAPHICS GUI

Singleton Strategy

Reactor Adapter

State Active Object

Class Library Component Architecture

Invocations

Selections

Class Library

Design PatternDATABASE

Singleton

GRAPHICS

Adapter

EventLoop

ReactorNETWORKING

Active Object

GUI

State

App SpecificLogic

MATH

ADTs

Callbacks

Invocations

Application FrameworkComponent Architecture


2.3 Relationship Between Frameworks, Pat-terns, and Other Reuse Techniques

Frameworks define “semi-complete” application that embody domain-specific object structures and

functionality. Class Libraries don’t offer explicit guidance to system design Components in a Framework manage

the canonical flow of control within App

Frameworks are active and exhibit “IoC (Inversion of Control)” at runtime. Template Method Pattern (Hollywood Principle)

In Practice, Frameworks, Class Libraries, and Components are complementary technologies.


3. The JAWS Adaptive Web Server

Overview of the JAWS Framework, Overview Design Pattern in JAWS.

4 Core Components in JAWS Concurrency. I/O. Protocol/Pipeline. File Caching.

JAWS Framework Revisited


3.1 Overview of the JAWS Framework


3.2 Overview Design Pattern in JAWS Strategic Pattern

Message Composition/Decomposition Pipe & Filter , Layer, Composite Message

Dispatcher Acceptor –Connector / Reactor / Proactor

Active Object Service Configurator

Tactical Pattern Strategy Adapter State Singleton

Strategic Pattern Over-view

Data

LayerComposite Message

Messages

UnMarshaling

Marshaling

LayerDecomposite

Message

Dispatching


3.2.1 Strategic Pattern

Marshaling / UnMarshaling Composite Message / Layer / Pipe & Fil-

ter

Dispatcher Acceptor-Connector / Reactor / Proactor

Active Object Service Configurator


Composite Message

Layers

+pushPacket()+popPacket()+headerSize()+makeHeader()

PacketInterface

+size()+addHeader()

Packet

+size()+addHeader()+marshal()+unmarshal()

Header

+size()

*

FragmentationLayer

+headerSize()+makeHeader()


Acceptor/Connector Pat-tern


Acceptor Pattern Acceptor is Factory. Creates, Accepts and Activates a new Protocol Handler

Whenever the Event Dispatcher notifies Acceptor that a connection has arrived from a Client.

Event Dispatcher Concurrency

TaskAcceptor

peer_acceptor_accept()

Protocol Handler

peer_stream_open()

Protocol Handler


Reactor Pattern

Reactor

handle_events()register_handler(in handle)remove_handler(in handle)

Event Handler

handle_event(in type)get_handle()

SynchronousEvent

Demultiplexer

select()

Concrete Event Handler


Concrete Event Handler


Handle<<uses>>

handleset

notifies

owns

dispatch


Reactor Pattern decouples the synch demultiplexing and dispatching logic. the Single-Thread Web Server Concurrency model

InputOutput Handler

Sync

hron

ous

IO

Reactor

send_file()recv_file()send_data()recv_data()

Protocol PipelineInputOutput

Asyn

chro

nous

IO

Reactive IO

Initiation Dispatcher

handle_event()register_handler(h)remove_handler(h)

Timer_Queue

schdule_timer(h)cancel_timer(h)expire_timers(h)

Filecache Handle

Reactive IO Handler

handle_input()handle_output()

IO Handle


Proactor Pattern

Proactor

handle_events()

AsynchronousEvent Demultiplexer

get_completion_event()

Concrete Completion Handler

Handle

<<uses>>

*

<<enqueues>>

is associated with

CompletionEvent Queue

AsynchronousOperation Processor

execute_async_operation()

AsynchronousOperation

async_operation()

Initiator

Completion Handler

handle_event()

<<executes>>

<<dequeues>>

<<uses>> <<invokes>>

<<uses>>

<<demultiplexes & dispatches>>


Proactor Pattern (IOCP)

Socket A

Socket B

Socket C

Completion Port

2.2 Create Completion Queue

Thread

Thread

Thread

1. Create Socket

2.1 Create Completion Port

4. Link Socket and Port

5. 1 Send Completion Socket Info

5. 2 Send Completion Packet

Thread Pool

3. Create Thread

6. Process Completion Events

7. Thread Returns Thread Pool


Proactor Pattern decouples the Asynch demultiplexing and dispatching logic. The Asynch variant of the Thread Pool Use IOCP (IO Completion Port)

InputOutput Handler

Sync

hron

ous

IO

Proactor

send_file()recv_file()send_data()recv_data()

Protocol PipelineInputOutput

Reac

tive

IO

Asynchronous IO

Completion Dispatcher

handle_event()register_handler(h)remove_handler(h)

Timer_Queue

schdule_timer(h)cancel_timer(h)expire_timers(h)

Filecache Handle

Proactive IO Handler

handle_read_file()handle_write_file()

IO Handle

Asynch Op

open()cancel()


Active Object Pattern

Future

Scheduler

Dispatch() Insert()

MethodRequest

Can_run()Call()

*

Proxy

method_1()method_n()

ActivationQueue

Insert()Remove()

Servant

method_1()method_n()

creates creates maintains

ConcreteMethodRequest 1

ConcreteMethodRequest N

Client

write to

obtain result from

invoke

1 11 1


Active Object Pattern decouples method invocation from method execution. Thread-per-Request, Thread Pool, Thread-per-Session Model

use Active Object pattern.

Scheduler

http_request()ftp_request()snmp_request()dispatch()

Activation Queue

insert()remove()

Method Object

Protocol Handler

http_request()ftp_request()snmp_request()

Resource Representation

Protocol Pipeline

dele

gate

s

for(;;){ m = aq->remove();dispatch(m); }aq->insert(http)


Service Configurator Pat-tern

Component

+Init()+Fini()+Suspend()+Resume()+Info()

CoreComponent

CoreComponent

ComponentRepository

ServiceConfigurator


Service Configurator Pat-tern Provides System Configuration at Runtime. Dynamically optimize, control and reconfigure

the behavior of Web server strategies a installation-time or during run-time.

Service+init()+fini()+suspend()+resume()+info()

Filecache

Protocol HandlerService Repository

Filter Repository

Cache Strategy Repository

Protocol Pipeline DLL Cache Strategy

Read Request Filter LRU Strategy LFU Strategy

Parse Request Log Request


3.2.2 Tactical Patterns Strategy

Configure Cache replacement strategies without afffecting the core SW Architecture.

Adapter Uniformly encapsulate the operation

of synch, asynch and reactive I/O.

State Event Dispatcher use State

to seamlessly support different concurrency strategies

and both synch and asynch.

Singleton To provide a global point to access object. Exist one copy of Cached Virtual File System.


Review


3.3 4 Components in JAWS

Concur-rency

Thread-per-Ses-


Thread Pool

I/O

Syn-chronous

Reactvie

Asyn-chronous

Cache

LRU

LFU

Structured Cache

Hinted Cache

Protocol&

Filter

Read Re-quest

Parse Re-questParse



3.3 Concurrency Strategy

Design Challenge A large portion of non-I/O related Web server over-

head is due to the Web server’s concurrency strat-egy.

Key Overhead Synchronization Thread/Process Creation Context Switching

Choosing an efficient concurrency strategy is crucial to achieve high-performance


3.3 Concurrency Strategy

Alternative Solution Strategies Thread-per-Request Thread-per-Session Thread Pool Single-Thread

Experiments

JAWS Concurrency Strategy Frame-work


3.3.1 Alternative Solution Strate-gies

Concurreny Strategy of Web Servers

Single Thread Roxen

Process-Based Apach Zeus

MultiThread PHTTD JAWS

Static Factors

Dynamic Factors

H/W Configuration OS Platform

Web Server Use Case

Machine Load

# of simultaneous requests Memory Use

Memory Workload


Thread-per-Request Allows each client request to run concurrently in a separate

thread.

Each Request arrives, a new thread is crated to process request.

This design appropriate synch I/O mechanisms


TaskProtocol Handler

Protocol Handler

Acceptor

Protocol Handler

Protocol Handler


Thread-per-Request (cont’d) Advantage Simplicity Exploit parallelism on multi-processor platform

Limitation The # of running threads may grow without bound, ex-

hausting available memory and CPU resources

Compatible Fields Use Light loaded servers with low latency Don’t use time consuming task based system


Thread-per-Session Allows each client connection to run concurrently

All of these requests are submitted through a connection between each client and a separate thread in the web server process

Advantage Does not spawn a separate thread for each request.

Limitation This model still vulnerable to unbounded resource consumption

as the # of clients grows.


Thread-per-Session (cont’d) Suited for HTTP 1.1 but not HTTP 1.0

HTTP 1.0 support open/operation/close One URL Fetch per Connection

Needs Reconnection Transporting Data Size Limitation URL Size Limitation

HTTP 1.1 Persistent Connection Pipeline Functionality

Many Requests -> Support Serial Response Compressed Data Transporting Use Proxy Server and Cache


Thread Pool Allows up to N requests to execute concurrently

within a pool of threads.

Support prespawning of Thread Creation


Task

Protocol HandlerProtocol Handler

Acceptor Protocol Handler

Protocol Handler


Thread Pool (cont’d) Advantage

Lower overhead that Thread-per-Request. Thread Prespawning

thread creation cost is amortized Pool Size is Fixed

Resource Consume is bounded

Limitation Small Pool Size

New incoming requests be dropped or wait indefinitely. Large Pool Size

Resource Consumption may be no better than using Thread-Per-Request


Single-Threaded All connections and requests

are handled by the same thread of control.

Single-Threaded servers process requests iteratively.

Compatible Filed Asynch or Reactive I/O Multiple Request Inadequate for high volume server

Subsequent requests are blocked until they are reached


Experiments Static adaptivity

Select the concurrency strategy that best meets The static requirements of the system.

multi-threaded concurrency More suited to

A multiple-processor machinethan a uni-processor machine

Dynamic adaptivity Adapt dynamically

to current server condition

In order to accommodate Unantici-pated load Support to increase

the # of available threads in the Pool


JAWS Concurrency Strategy FrameworkEvent Dispatcher

dispatch()

Concurrncy

dispatch()

Task

accept()activiate()enque()deque()svc()

Thread Pool

dispatch()

Thread-per-Connection

dispatch()

Protocol Handler

open()

Acceptor

accept()

task->accept()task->activate()

task->enque()

acceptor->accept()

server->dispatch()

server

task

acceptor

handler


3.4 I/O Strategy Design Challenge

is to devise efficient data retrieval and delivery strate-gies,

collectively referred to as I/O.

Key Overhead Arranging Multiple I/O operation.

Example Synch

Web Transaction involving monetary fund transfer ASynch

Web access to static information (Web page – ASP..)


3.4 I/O Strategy

Alternative Solution Strategies The Synchronous I/O Strategy The Reactive I/O Strategy The Asynchronous I/O Strategy

Experiments

JAWS I/O Strategy Framework


The Synch I/O Strategy Synch I/O describes the model of I/O interaction between a Web

server process and kernel.

Kernel doesn’t return the thread of control to the server until the requested I/O operation either complete , completes partially, or fails.

Advantage (Easy Use)

Disadvantage (Resource Exhaust) If combined with a single threaded concurrency strategy, It is not possible to perform multiple synch I/O operation simultaneously.

When using multiple thread, It is still possible for an I/O request to block indefinitely.


The Reactive I/O Strategy

Reactive I/O alleviates the blocking problems of synchronous I/O without resorting to polling.

Web Server uses an OS event demultiplexing system call.

select() in UNIX WaitForMultipleObjects() in Win32 Refers Reactor Pattern

Disadvantage May not make effective use of multiple CPU.


The Asynch I/O Strategy Asynch I/O simplifies

the de-multiplexing of multiple events in one or more thread of control without blocking the Web server.

Advantage The Web Server not block on I/O request.

Disadvantage Asynch I/O is not available on many OS Platform. more complicated than writing synch program.


Experiments

The Result reveal that each I/O strategy behaves differently

under different load condition.

JAWS I/O Strategy Framework addresses this issue by allowing the I/O Strategy to adapt dynamically

to run-time server condition.


JAWS I/O Strategy Frame-workPerform Request

svc()

fhInputOutput Handler

receive_file()receive_data()send_file()send_data()

complete()error()

InputOutput


Synch IO


Asynch IO


Proactor

Reactive IO


Reactor

Filecache Handle

ioh

io

ioio->send_file(“index.html”)

write(fh)Proactor->send_file(fh)

Reactor->send_file(fh)


3.5 Protocol Pipeline Strategies

Design Challenge Early Web Server

Simply retrieved the requested file. Transferred its contents to the request.

Modern Web Server Support Various File Format

text, image, audio, video.

Introducing Server Side Engine (ASP, JSP..) search engine, map generation, interfacing data-

base, security, financial transaction.


3.5 Protocol Pipeline Strategy

Alternative Solution Strategies Pipeline.

Reflection + Component Configurator.

Examples Advanced Search Engine. Image Servers. Adaptive Web Contents.

JAWS Protocol Pipeline Strategy Frame-work


Using Pipeline

PerformRequest

LogRequest

HTTP/ 1.0 200 OKDate: Thu, 09 Oct 01:26:00 GMTServer: Eva/1.0Last-Modified: Wed, 08 Oct 1997 ..Content-Length : 12345Content-Type : text/html

<HTML><TITLE> Homepage </TITLE> ..</HTML>

any.net - -[09/Oct/1997:01:26:00 -0500]“GET /~jxh/home.html HTTP/1.0

ReadRequest

GET /~jxh/hoem.html HTTP/1.0Connection: Keep-AliveUser-Agent: Mothra/0.1Host: any.netAccept: image/gif, image/jpeg, */*

ParseRequest

GET/users/jxh/public_thml/home.htmlHTTP /1.0

ParseHeaders

CONNECTIONUSER_AGENTHOSTACCEPTACCEPTACCEPT

Keep-AliveMothra/0.1any.netimage/gifimage/jpeg*/*


Static vs Dynamic

Static Configuration HTTP 1.0 Request limited # of processing operation

Marsharing & Demarsharing Demutlipelxing

Dynamic Configuration Unlimited # of Processing operation Requires a lot of process arranging cost.


Example

Advanced Search Engine May construct data filters dynamically.

“(perfomrnace AND (NOT symphonic))”

Requesting for Web pages containing the word “performance” but not the word “symphonic”.

Could be implemented as a pipeline of a positive match component coupled with a negative match component


Example (cont’d)

Image Servers User may request that Image be cropped, scaled, rotated, and

dithered.

Adaptive Web Contents Dynamic Delivery for End User.

PDA User Web Content Overviews and Smaller Image.

Workstation & Personal Computer Full Multimedia Enriched Pages.


JAWS Protocol Pipeline StrategyProtocol Handler InputOutput Handler

Protocol Pipeline

svc()

Read Request

svc()

Filter

svc()

Filter

svc()parse()

Parse Headers

svc()parse()

Perform Request

svc()perform()

Log Request

svc()log()

component->svc()io->receive_data()

Filter::svc()parse()

Filter::svc()perform()

Filter::svc()log()

pipe io

component


3.6 File Caching Strate-gies Design Challenges

Accessing the File System is a significant source of web server overhead.

Cache is a storage medium that provides more efficient retrieval.

Alternative Solution Strategies Least Recently Used (LRU) Caching. Least Frequently used (LFU) Caching. Hinted Caching. Structured Caching.

JAWS Cached Virtual File System Framework


Alternative Solution Strate-gies Least Recently Used Caching Least Frequently Used Caching

Structured Caching Refers the Next Page.

Hinted Caching Analysis of Web page retrieval pattern. Web pages have spatial locality. Using statistical information about links

(hinted). Using Pre-fetching Method. Refers “Hinted Caching in the Web”


Alternative Solution Strate-gies Structured Caching


JAWS Cached Virtual FS Framework

InputOutput

receive_file()send_file()

filename

InputOutput

handle()mapaddr()name()size()open()

Filecache Object

handle()mapaddr()name()size()

HashTable

Filecache

find()fetch()remove()create()findish()

rw_lock[]

Cache Strategy

insert()

LRU Strategy

insert()

LFU Strategy

insert()

fh fo

replace leastrecently used

Replace leastFrequently used

fh->open(file)cvf->open(file,fo)

htcache

cvf


JAWS Cached Virtual FS FrameworkPerform Request

svc()

fhInputOutput Handler


complete()error()

InputOutput


Synch IO


Asynch IO


Proactor

Reactive IO


Reactor

Filecache Handle

ioh

io

ioio->send_file(“index.html”)

write(fh)Proactor->send_file(fh)

Reactor->send_file(fh)


3.7 The JAWS Framework Visited

Concur-rency

Thread-per-Ses-


Thread Pool

I/O

Syn-chronous

Reactvie

Asyn-chronous

Cache

LRU

LFU

Structured Cache

Hinted Cache

Protocol&

Filter

Read Re-quest

Parse Re-questParse



4. Web Server Benchmark-ing Hardware Testbed Software Request Generator

Experimental Results A Summary of Techniques

for Optimizing Web Servers.


4.1 Hardware Testbed System Specification

Two Micron Millerinia PRO2 plus workstation 128M RAM Client – 200 MHZ Server – 180 MHZ ENI-155P-MF-S ATM Card ATM Network running through

a FORE Systems ASX-200BX (bandwith 622 Mbps)

Limitation of LAN emulation mode = 120 Mbps


4.2 Software Request Generator

Experiment Purpose Using WebSTONE Stress Tool

Collect Client and Server Side Metric average server throughput, client latency

Gather Loads and Statistics using various particular files sizes

Refers the next page. Using various concurrency & dispatching strategies

research the impacts of results.


4.2 Software Request Generator

File Access Patterns

Document Size500 bytes5 Kbytes50 Kbytes5M Bytes

Frequency35%50%14%1%


4.3 Experimental Results Using 3 Models Definition of Terms

Throughput Comparisons Latency Comparisons Summary of Benchmark Results

A Summary of Techniques for Optimizing Web Servers


4.3 Experimental Results Using 3 Models

Synchronous Thread-per-Request Using Synchronous I/O

Synchronous Thread Pool Using Synchronous I/O

Asynchronous Thread Pool Using TransmitFile() for Asynchronous I/O TransmitFile() is a custom Win32 Function()


4.3 Experimental Results Definitions of Terms

Throughput The average # of bits received per second by client. Start Time

HTTP Request received by client. End Time

Connection is closed at the client end.

Latency The average amount of delay in milliseconds Start Time

HTTP GET Request received by client. End Time

Completely Received File.


4.3 Experimental Results

Throughput Comparisons.

Connection ↑ - Throughput ↓

As expected, The throughput for each connection de-grades as the connections per second increase.


4.3 Experimental Results Throughput Comparisons.

Refer Figure 3. Experiment Results from 50K Files.

Thread-Per-Request Model Connection load ↑ -> Throughput ↓ rapidly Incurs higher thread creation overhead.

Synchronous Thread Pool Connection load ↑ -> Throughput ↓ more gracefully Uses Pre-spawning thread.



Refer Figure 1 through 5.

TransmitFile() performs extremely poorlyfor small files ( i.e. < 50K bytes)

TransmitFile() is forced to wait while the kernel services incoming re-

quest.

As the Size of the file grows, TransmitFile() rapidly outperforms the synch

model.



Fig 1. 500 Byte File

Fig 2. 5K Byte File



Fig 3. 50K File

Fig 4. 500K File



Fig 5. 5M File


4.4 A Summary of Techniques (for Optimizing Web Server)

Lightweight Concurrency Don’t use Process-based concurrency Mecha-

nism. Using POSIX thread.

Specialized OS Feature Using IOCP

Request Lifecycle system call overhead. Reducing synchronization. Caching files.

Reducing Iterative System I/O Call (open, read, write..)

Using “gather-write” Using writev (Unix System Call)

Pre-computation HTTP response



Logging Overhead File System access

Makes a significant number of I/O Calls.

Synchronization overhead Thread/Process are required to log request to a common

shared log file, access to the log file needs to be synchronized.

Reverse hostname lookups Hostname is typically more useful information in the log

file.

Ident Lookups For identifying User Name Make setting up a new TCP/IP Connection



Transport layer optimizations The listen backlog

Give Queue Size Higher.

Socket Send buffers Should be set to the highest permissible limit. On Solaris, 64KB limit

Nagle’s Algorithms Congestion Control Algorithm Server latency critical App (such as X-Windows)

disable this algorithm.


5. Concluding Remarks


References

POSA1 Layer , Pipe & Filter

POSA2 Component Configurator , Active Object,

Dispatcher (Reactor, Con-Acc, Proactor)

Composite Message Pattern

Documents

JAWS (A Framework for High-Performance Web Servers) TP Version 1.1