Upload
jermaine-cherry
View
35
Download
4
Embed Size (px)
DESCRIPTION
Distributed Data Mining System in Java. Group Member 王春笙,林俊甫,王慧芬. Overview of Project. Project participants 王春笙, 林俊甫,王慧芬. Project Programming Tasks. D92725002 林俊甫 Polling and reply Multicast between client and server Client/Server Socket programming - PowerPoint PPT Presentation
Citation preview
Distributed Data Mining System in Java
Group Member
王春笙,林俊甫,王慧芬
Overview of Project Overview of Project
• Project participants– 王春笙,林俊甫,王慧芬
Project Programming Tasks Project Programming Tasks
• D92725002 林俊甫– Polling and reply Multicast between client and server– Client/Server Socket programming– Client dynamic join and leave mechanism– Multi-thread programming – Synchronization mechanism– Data chunks maintenance and dispatching mechanis
m– Client/Server communication link control
Project Programming Project Programming Tasks(cont’d)Tasks(cont’d)
– Client failure handling• Reassign backup server, if failure client is backup• Restore failure client works (with 王春笙 )
– Server failure handling• Backup Server designate mechanism and logic design
– RMI mechanism (with 王春笙 )– Basic GUI
System Infrastructure System Infrastructure
• System diagram
LAN
Server/Coordinator
Client Client Client
...
Mining data chunk
Mining result
Basic OperationBasic Operation
Server Client1. Polling on port 4444 Group 230.0.0.1@: who is server?
2. Servername: I am the server
3. Connect to <servername, port 4445>
4. Client do: filechunk#
5. ok
6. Client do: next filechunk#
7…..8…..….
Time Time
Listen multicastGroup query and reply Server found;
Connect to the Server
Fork thread to Handle client connection
Receive server’sInstruction, ivokeRMI to get file chunk
Wait for client’sProcessed result,Order client to getAnother file chunk
Port AssignmentPort Assignment
• Port 4444: for multicast
• Port 4445: for TCP/IP socket connection
• Port 4446: for RMI services
Finding A ServerFinding A Server
• Once a client start up, it will query periodically every 3 sec. over the multicast group 230.0.0.1 port 4444 by sending 1 byte string “@” to locating the server host.
• Once a server start up, it will fork a thread to dealing with the query
6. Server failure detect -> if I am backup
go to backup serverprocedure, otherwise
go to step.1.
3.Connect to Server on port
4445
2. Listen forserver response
1. Client Query: who is the Server now?
4. Use RMI Get file chunk from
Server
5. Process data mining and return
result to server
File DispatchingFile Dispatching
• Server maintain a file chunk pool .
• Server will find a available file chunk for client, set it to 1 and order client to get this file chunk by RMI file chunk will be update to 2 when client return result.
• Recovery: When server detects client’s link-broken, it will restore file chunk allocate to client to 0.
• File chunk class is declared as Serializable for RMI message passing to backup server
• File chunk class use Synchronization for concurrent control
FileChunks …………
-1: empty, 0: available, 1: using, 2:used
Backup Server SelectionBackup Server Selection
• Server maintains and assigns unique id for each individual client.
• Unique id is incremented as serial number.
• Client with smallest id is assigned as backup server
• When client failure, server will check if it is the backup server to restart the selection process or not.
Nodes MaintenanceNodes Maintenance
• Server maintain connected client’s records in an ArrayList
• ArrayList is compound with class Nodes, which records client’s detail information.
Key Value
Id Address Port Work on Status
ArrayList: ht
Nodes
RMI ServicesRMI Services
• RMI services is written in independent program because server and client (which acts as backup server) will use it.
• RMI services provides:– Backup server data to backup-server.– Get file chunk from server– Return mining result to server– Receive nodes information from server
Client FailureClient Failure
• Server’s action took:– Recovery– Reassignment – Redo backup server selection if failure nodes
is backup
• Client’s action– Do nothing except one is told by server to act
as backup
Server Failure Server Failure Server S Client BTime Time
Server run backupSelection choose AAs backup
TimeClient A
1.A is told by S thatIt is the backupA invoke RMI to get all Server data
A: Do backup
RMI Get file
RMI reply
2. A periodically Get server services,File chunk data do reply
Client do #
Client do #
do reply
1. B receives instruction as discuss before
Server CrashX X3. Comm.link brokenIs detected, start ServerAction class
2. Comm.Link Broken is detected, multicast query who is the server now?
B Polling @: who is server?4. Create server Socket at 4445, fork threadTo listen to query And wait for connection
A reply: I am the server3. B know A is the backup, re-connect to A
Connect to A:4445
Server/Client Life CycleServer/Client Life Cycle
Server Client
ServerNormal/AbnormalTermination
Normal/AbnormalTermination
evolve
Project Programming Tasks Project Programming Tasks
• D91725001 王春笙– Web log file preprocessing and separating– Web pages traversal sequences parsing– Page items transferring and mapping– Web pages sequential patterns mining – Mining results maintenance – RMI mining results transfer– Mining results lookup and display
Project Programming Project Programming Tasks(cont’d)Tasks(cont’d)
– Backup mechanism • Separate thread backup server files and memory data • Restore failure client works (with 林俊甫 )
– RMI mechanism (with 林俊甫 )– GUI global states refreshment– System integration
• Testing and debugging
Web Log File FormatWeb Log File Format
• User IP
• Date
• Time
• Web pages URL
Web File PreprocessingWeb File Preprocessing
• Select *.htm and *.html pages
• First sort by user ID
• Second sort by time
• Pages sequences separated by time– more than 30 seconds
Chunk Data FilesChunk Data Files• Part*.ppp
• Items.ppp
6023 2 1 1 2 86024 1 1 2066025 7 1 1 1 1 1 1 1 2 5 17 18 19 20 116026 3 1 1 1 144 145 3386027 2 1 1 2 96028 3 1 1 1 2 8 3
/~visualdep/htm/p5b.htm 168/~businessdep/student/picture.html 169/~comedu/inde.htm 170/~account/91tuition.htm 171/~stuaffair/life/procedure-17.htm 172/~stuaffair/life/procedure-25.htm 173
Apriori algorithmApriori algorithm
• 1:find all L1
• 2:generate C2 from L1
• 3:count C2 and find all L2
• 4:k=3
• 5:generate & prune Ck from Lk-1
• 6:count Ck and find all Lk
• 7:if Lk not empty then k++, goto 5
Apriori algorithm Apriori algorithm (cont’d)(cont’d)
• join phase:s1 join s2 if s1(drop first) = s2(drop last)
– s1 join s2 =>
• prune phase:delete a k candidate if any k-1 sub sequence not large
• C & L are stored in hash data structure
},{},,{ 21 absbas
},,{ aba
Mining Result DisplayMining Result Display• Client frequent patterns
– Web page ID– Support– Saved as *.pppl files
• Client frequent patterns– Web page ID– Support– Web page name
Backup MechanismBackup Mechanism
• When backup server selected, that client start a backup thread
• Backup thread loop every 0.5 second
• RMI data transfer– Chunk data file(part*.ppp,items.ppp)– Client information– File chunk information
• determine MaxID and set “in use” to “available”
– Frequent patterns information
System IntegrationSystem Integration
• Java class integration– Server component– Client component– Data mining component– GUI component
• Testing
• Debugging
Project Programming TasksProject Programming Tasks
• D92725001 王慧芬
– Graphical User Interface• Since this is a system working on data mining task
in a distributed way, its GUI provides four panels :– A system console– A result window– A connection table– A graphical network configuration
GUIGUI
• The system console shows how system proceeds
GUI (cont’d)GUI (cont’d)
• The result window displays the progress and results of data mining
GUI (cont’d)GUI (cont’d)• A connection table lists all of the on-line
client connection information
GUI (cont’d)GUI (cont’d)• A connection table consists of 5 fields
– NO: client-server connection id– IP address: client’s IP address– Port: client’s port number– Status: connection status, it could be
• 0: offline 1: online• 2: file transfer from server to client• 3: client is doing data mining• 4: client returns value back to server if data mining finished• 5: client is doing the backup and data mining at the same time
– # chunk works on: if data mining and backup, it indicates the chuck number that the connection works on
GUI (cont’d)GUI (cont’d)• A graphical network configuration follows the
connection table to depict the dynamic network configuration
GUI (cont’d)GUI (cont’d)
• In the dynamic network configuration, we use different client GIFs to express the status :– Offline On-line
– Data mining
– Backup and mining
GUI interfaceGUI interface• mw.showMsg()
– provided by GUI for server/client module to show the console message
• mw.showResultString()– provided by GUI for server/client module to show the re
sults of data mining
• Connection table– modified by server/client module for connection inform
ation– read by GUI every 0.01 second to depict the dynamic n
etwork configuration
GUI designGUI design
• Java swing is used to generate label, text, scrollbar, and table, etc..
• Java AWT 2D painting is used to form the animation of the connection lines in the dynamic configuration panel
• ‘Photo Impact’ and ‘GIF animator’ are used to generate the node icons
• EasyRGB used to tune the color harmonies.
GUI design (cont’d)GUI design (cont’d)• A new thread is forked from the GUI task to work on the
animation of the connection lines in the dynamic configuration panel,
– to read the table
every 0.03 second and
to show the connection
status with a moving
ball.
GUI
Generateconnection
table
Generateresult panel
Generatesystemconsole
Generateconnection
table
animation
InstallationInstallation
• 以執行一個 server ,兩個 client 為例– 建立三個資料夾,此三資料夾 Ser(Server),Cli(Client1),Cli2(Client
2)– 將附檔解壓至 Ser 資料夾,此資料夾內要下載 weblog10.zip 檔,
並解壓– 將附檔解壓至 Cli 與 Cli2 的空資料夾– 開啟二個 dos 視窗 (1,2 號視窗 ) ,進入 Ser 資料夾– 開啟三個 dos 視窗 (3,4,5 號視窗 ) , 3,4 號進入 Cli 資料夾, 5 號
進入 Cli2 資料夾– 1 號視窗執行 compile.bat 批次檔,再執行 rmi.bat– 2 號視窗執行 server.bat 批次檔– 3 號視窗執行 compile.bat 批次檔,再執行 rmi.bat– 4 號視窗執行 client.bat 批次檔– 5 號視窗執行 compile.bat 批次檔,再執行 client.bat 批次檔