SRI RAMAKRISHNA ENGINEERING COLLEGE(An Autonomous Institution ,Affiliated to Anna University , Chennai)An NBA Accredited InstitutionVattamalaipalayam,N.G.G.O. Colony Post,Coimbatore-641022Phone: 0422 - 2460088, 2461588 PROJECT PRESENTATIONNAME : JAYACHANDRAN.R, KARTHIKPRABHU.SDEPARTMENT : COMPUTER SCIENCE ENGINEERINGREGISTER NO : 1301166, 1301160SEMESTER :2rd yr, 3rd semCOLLEGE NAME: SRI RAMAKRISHNA ENGINEERINGCOLLEGE

WEBCAM BASED VIRTUAL KEYBOARDLIST OF FIGURESNAME OF THE FIGURE

Need for virtual keyboard Problem Statement Existing system Drawbacks Virtual keyboard method and Device Visual Panel Finger-Joint Gesture Wearable Keyboard Thumbcode Chording Glove FingerRing TouchStream Multi-Point Touchpad VKey VKB Projection Scurry Senseboard Comparison between Traditional Keyboard andVirtual Keyboard Structure of the Virtual Keyboard system Practical Setup Use Case Diagram Sequence Diagram for Image Acquisition Sequence Diagram for Interrupt Detection in FrameSequenceNAME OF THE FIGURE Sequence Diagram for Finger Extraction using Threshold Algorithm Sequence Diagram for Finger Tip analysis by usingEdge detection Sequence Diagram for Key Extraction Class Diagram for Virtual Keyboard Activity Diagram for Virtual Keyboard Block Diagram for Image Acquisition Snapshot of Image Acquisition Block Diagram for Interrupt Detection in FrameSequence Snapshot of Interrupt Detection Block Diagram for Finger extraction using Threshold Algorithm Snapshot for Finger extraction using Threshold Algorithm Snapshot of Finger tip analysis by Edge Detection Method Snapshot Key Extraction WinRunner Testing Result WinRunner Testing Code System Testing Conclusion Screen Shot for Choosing the EnvironmentLIST OF SYMBOLS

SYMBOL NAMEDESCRIPTIONP(C/x)Posterior Probabilityp(C)Prior ProbabilityP(x/C)Conditional Probabilityp(X)Feature Vectormax P (C\x)Highest Posterior ProbabilityCArray of Pre-stored Key elements

CHAPTER1INTRODUCTION1.1NEED FOR VIRTUAL KEYBOARDHistorically, keyboards are designed for text character printing and text character entry later on in the attached devices. Therefore, keyboard is communication mechanism between human beings and the computing machines and is used to enter textual data and commands. Virtual keyboard are generally assumed to produce the same output as the conventional QWERTY layout produces. The utilization of virtual keyboard appears in space saving situations or requirement in soft programmability of keys or systems avoiding mechanical failure or in movement situations where usability of standard keyboard is limited. Utilization of virtual keyboards in space saving situation is enormous. (e.g.) Text entry in PDAs and cellular phones etc. PDAs and cellular phones dont have a standard keyboard available with them due to limited key on their text entry keypads. Soft key programmability of keyboard is also a considerable factor in the design of virtual keyboards. Similarly virtual keyboards have been designed without mechanical keys. These kinds of keyboards are utilized in hostile environments. Virtual keyboards find their position in transport environments. E.g., rail, plane or automotive. Virtual keyboards are also designed for public kiosks and here those designs are suitable which avoids mechanical failure. Industrial environments and medical environments also require the special requirements from keyboard designs.1.2PROBLEM STATEMENTTo design a vision based Virtual Keyboard which detects interrupt as key recognition instead of mechanical transducer operations of key pressing. Mono-vision video of hand posture for pressing the keys is analyzed. The analyzed hand posture is taken into account under various transactions to estimate the key pressed. Mechanical transducers does two operations for key estimation (key press and key release concepts), while Virtual Keyboard requires only key press operation to estimate the key and not key release operation.

CHAPTER2SYSTEM ANALYSIS2.1EXISTING SYSTEMCurrent virtual keyboards design appears in various forms such as finger-joint wearable sensor gloves, thumb code, accelerometer based inputs, laser projected keyboards and gyroscope based sensing. Each virtual keyboard has certain design characteristics. However, performance parameters for evaluation of keyboards are same as number of discrete keys, response time and failure rate. Other parameters included can be the ability to remap, key symbol mapping and space requirements.2.1.1DRAWBACKS The cost of the most successful Virtual Keyboard design is high due to design and expensive technology. The set up cost ranges from Rs.5000 and above.2.2VIRTUAL KEYBOARDS: METHODS AND DEVICESThe following are the various types and methods of existing Virtual Keyboard system,2.2.1VISUAL PANELThe Visual Panel [15] consists of a camera and a sheet of paper as shown in figure 2.1. The location of the extended index finger in reference to the paper is located with computer vision means. The primary application is a mouse pointer, clicking is achieved by resting the fingertip in its current position for three seconds. The authors demonstrated text entry by interpreting pointer locations as the keys of a keyboard, which were printed on the sheet of paper. An audible notification signals the recognition of a character after the 3 second significance interval.

Figure 2.1. Visual Panel

2.2.2FINGER-JOINT GESTURE WEARABLE KEYPADThe Finger-Joint Gesture Wearable Keypad (FJG) [3] suggests viewing the phalanges of the fingers (besides the thumb) of one hand as the keys on phone keypad as shown in the figure 2.2. The thumb is used to press the virtual buttons. Thumbcode relies on word disambiguation to produce more than 12 characters. Yet the drawback of key-to-symbol mapping might be mitigated by the familiar layout. Also, less complex hand configurations might be less tiring for the user. Just as Thumbcode, FJG has no user feedback method beyond skin contact sensations.

Figure 2.2. Finger-Joint Gesture Wearable Keyboard2.2.3THUMBCODEThe Thumbcode method [14] described in figure 2.3 which defines the touch of the thumb onto the other fingers phalanges of the same hand as key strokes. Consequently there are 12 discrete keys (three for each index, middle, ring finger and pinky). To produce up to 96 different symbols, the role between keys and operators is broken up: The four fingers can touch each other in eight different ways, each basically representing a mode, or modifier key that affects the mapping for the thumb touch. Tactile user feedback is implicit when touching another finger with the thumb. A glove implementation was tested by the author.

Figure 2.3. Thumbcode2.2.4CHORDING GLOVEThe Chording Glove [9] employs pressure sensors for each finger of the right hand in a glove to implement a chording input device. Almost all possible finger combinations are mapped to symbols, making it potentially hard to type them. Additional mode switches, located along the index finger, are used to produce more than the 25 distinct characters as in figure 2. 4. Yet user experiments suggest otherwise: rates of up to 19 wpm are achieved after ten training sessions with no signs of leveling off.

Figure 2.4. Chording Glove

2.2.5FINGERINGFingeRing [4] uses accelerometers on each finger to detect surface impacts. In the wireless version depicted in the figure 2.5 these rings communicate with a wrist-mounted data processing unit. The interaction method is designed for one handed use, but could be extended to two hands with obvious implications. In the current version, the finger movements to produce one character are extensive: two chording patterns have to be typed within a time interval, each consisting of a combination of fingers hitting the surface. Due to this piano-style typing method, users with prior piano experience fare much better with this device; in fact, the full 2-stroke chord mapping is rendered too difficult for novice users.

Figure 2.5 FingeRing2.2.6TOUCHSTREAMThe TouchStream keyboard stretches our definition of a VK as it has keys printed on the surface. Yet the underlying technology permits software configuration of the sensed areas, equal to the multi-point touchpad. Despite conventional touch-typing the TouchStream affords a number of chording patterns as alternatives to modifier keys. These patterns are pressed by one hand (anywhere on the pad) while the other touches the key that is to be modified. Picture of the TouchStream keyboard is shown in the figure 2.6.

Figure 2.6. TouchStream2.2.7MULTI-POINT TOUCHPADDSI Datotech Systems offers one of the few touchpads that reports up to ten surface contacts and their pressure forces independently and simultaneously [1]. While it has not been implemented yet, one could use the 20x15cm large device to transfer the traditional keyboard modality in a one-to-one fashion to an interactive, software-configurable surface as in figure 2.7. Inherent to this device are the same user feedback methods as for any of the devices employing tabletop units: finger surface impacts.

Figure 2.7. Multi-Point Touchpad 2.2.8VKEYVirtual Devices Inc. recently announced a combined projection and recognition VK [12]. Little is known about this device, but their press release suggests that visual sensors (cameras) detect the movement of all ten fingers as shown in figure 2.8. Just as the VKB device, the VKey also consists of a tabletop unit and feedback is the tactile sensation of hitting a surface.

Figure 2.8.VKey2.2.9VKB PROJECTIONThe virtual keyboard technology developed by VKB [13] is a tabletop unit that projects a laser image of a keyboard on any flat surface. Infrared cameras detect key strokes of all ten fingers as shown in the figure 2.9. Word disambiguation techniques are employed despite this 1 DOF mapping. Therefore, our guess is that engagement of all distinct key locations is detected, yet with a fairly low accuracy. These two characteristics in combination should result in fairly good recognition rates. Surface impact of the fingers serves as typing feedback.

Figure 2.9. VKB Projection2.2.10 SCURRY Tiny gyroscopes on each finger are the sensing technology in Samsungs Scurry [10]. The prototype suggests that these finger rings communicate with a wrist-mounted unit where the data is processed. Finger accelerations and relative positions are detected, making it possible to distinguish multiple key targets per finger. Pictorial representation of Samsungs Scurry is shown figure 2.10. A surface impact is required to register a keystroke, making for the primary sensory feedback to the user. Little LEDs on the rings potentially provide additional feedback.

Figure 2.10. Scurry2.2.11 SENSEBOARD The Senseboard [11] consists of two rubber pads that slip onto the users hands. Muscle movements in the palm are sensed (with unspecified, non-invasive means) and translated into key strokes with pattern recognition methods. The only feedback other than characters appearing on a screen comes from the tactile sensation of hitting the typing surface with the finger as in figure 2.11.

Figure 2.11 Senseboard

CHAPTER3SYSTEM REQUIREMENTS3.1HARDWARE REQUIREMENTSProcessor : Minimum o f Intel Pentium RAM : 256 MBMemory Space : 100 MB (fully for this Application)Webcam : Pixel ratio of 320 x 240

3.2SOFTWARE REQUIREMENTSOperating System :Windows Operating SystemMedia Software:Java Media Framework

CHAPTER4LITERATURE SURVEY4.1GESTURE Gesture may be defined as the physical movement of hands, arm, face or body with the intent to convey the information or command. Gesture recognition consists of tracking human movement and interpretation of that movement as semantically meaningful commands. There are three types of gesture. They are mimetic, deictic, and arbitrary. In mimetic gestures, motions from an objects main shape or representative feature. These gestures are intended to be a transparent. Deictic gestures are used to point at the important objects and each gesture is transparent within its given context. These gestures can be specific, general or functional. Specific gestures refer to one object. General gestures refer to class of objects. Functional gestures represent intentions such as pointing to a chair to ask permission to sit. Arbitrary gestures are those whose interpretation must be learned due to their opacity. Although they are not common in a cultural setting.4.2GESTURE ARCHITECTURELike any other pattern recognition system, gesture recognition consists of three components.Gesture acquisition and preprocessing Gesture feature extraction and representationGesture recognition and classification4.2.1GESTURE ACQUISITION AND PREPROCESSINGVariety of transducers or are available for gesture acquisition ranging from discrete components such as magnetic sensor, position sensor, accelerometer sensor to fully developed functional units such as gesture gloves and more sophisticated systems such as mono vision, stereo vision and range sensors etc. Technologically these sensors employ different methodology ranging from basic physics principles to sophisticated imaging and lasers.Vision sensors are installed in mainly two configurations they are mono-vision and stereo vision. Mono vision sensors incorporate one sensing camera naming CCD (charge coupled device) or CMOS (complementary metal oxide semiconductor) with multiple possible interfacing such as USB 2.0, Camera link, Ethernet, etc, for their video signal transmission. Similar kind of acquisition sensors are utilized for stereo vision. However, the primary difference exists in the further interpretation of stereo-imaging.4.2.2 GESTURE FEATURE EXTRACTION AND REPRESENTATION Feature extraction for the purpose of gesture recognition consists of segmentation of image components that contribute to the formation of gesture inputs. Both raster (skin tone blobs, colored gloves, etc.) and vector information (joint geometry, facial animation parameters, etc..) from the basis of feature extraction. Some commonly features for gesture recognition are given below:Image momentsSkin tone blobsColored markersGeometric featuresMulti scale shape characterization Motion History Images (MHI)Motion Energy Images(MEI)Shape SignaturesPolygonal approximation-based Shape DescriptorShape descriptor based upon regions and graphs4.2.3GESTURE RECOGNITION AND CLASSIFICATIONFinally, gesture recognition and classification stage classifies the reported features belonging to certain pre-stored category. Following are the list of gesture recognition or classification methods proposed in the literature so far:Hidden Markov Model (HMM)Viola and Jones algorithmDynamic ProgrammingFuzzy Interference EngineBayesian Classifier Template Matching4.2.3.1 HIDDEN MARKOV MODELDynamic gestures prolong over certain duration of time, due to which gestures usually appear in the form of sequences or spatiotemporal information. Ultimately some kind of sequences or spatiotemporal matching process is required for successful gesture recognition. Theoretically, a sequence can be characterized as being generated by some parametric random process can be successfully accomplished by Hidden Markov Model (HMM).4.2.3.2 DYNAMIC PROGRAMMINGDynamic programming is the method of solving the problems exhibiting the properties of overlapping sub-problems and optimal sub structure that takes much less time than nave methods. The term was originally used in the 1940s by Richard Bellman to describe the process of solving the problems where one needs to find the best decisions one after another. By 1953, he had refine this to modern meaning. The word Programming in Dynamic Programming has no particular connection to computer programming at all. A program is, instead the plan for action is produced. For instance, a finalized schedule of events at an exhibition is sometimes called a program. Programming in the sense, finding the acceptable plan of action.4.2.3.3 BAYESIAN CLASSIFIERBayesian classifier is one of the basic classification methods from supervised classification category. Bayes rule is stated as follows: P(C/x) = (p(C) x p(C/x)) /p(x)(1)where p(C) is the prior probability, which in particular can be said the prior probability of gesture. P(x\C) is the class likelihood and is the conditional probability that an event belonging to see has observation x. Likelihood for gesture recognition can be specified as the conditional probability that gesture belonging to class C as the feature vector x. p(x) is the evidence in the sense that a particular feature vector for some gesture appears with this probability. Finally posterior probability p(C/x) is calculated by combining the prior. Likelihood and evidence. For multiple classes, the posterior probability can be calculated as Finally for the minimum error Bayesian classifier selects the class with the highest posterior probability i.e.Select C if P (C\x) = max P (C\x) (2)4.2.3.4 TEMPLATE MATCHINGTemplate Matching is most general class for recognition of a pattern in the pre-stored patterns . Pattern matching operates at both levels raw shapes are recognized or features are initially extracted and later on matched with the pre-stored classes. Pattern matching operates both in spatial as well as in the frequency domain.4.2.3.5 VIOLA-JONES OBJECT DETECTION FRAMEWORKThe Viola-Jones object detection framework is the first object detection framework to provide competitive object detection rates in real-time proposed in 2001 by Paul Viola and Michael Jones. Although it can be trained to detect a variety of object classes, it was motivated primarily by the problem of face detection. This algorithm is implemented in OpenCV as cvHaarDetectObjects().4.3REAL TIME MONO VISION GESTURE BASED VIRTUAL KEYBOARD Real Time Mono Vision Gesture Based Virtual Keyboard System paper presents a novel mono-vision virtual keyboard design for consumers of mobile and portable computing devices such as PDAs, mobile phones etc. Fuzzy for each symbol (rather than cording methods) inherently approaches to gesture recognition are developed to reveal the realization of soft keyboard. Key pressed over the printed sheet keyboard by analyzing the hand and finger gesture captured in the video sequence. Real time system is developed by integrating camera with PDA in the application environment. Reliable results are experienced by the implementation o f the proposed real time mono vision gestured virtual keyboard system.In this project a camera for video capture. Novel gesture recognition based virtual keyboard system is designed. A gesture may be defined as the physical movement of hands, arm, face and body with the intent to convey information or command. Gesture recognition consists the tracking of human movement and interpretation of that movement as semantically meaningful commands, Gesture recognition has the potential to be a natural and powerful tool for intuitive interaction between the human and computer. Gesture recognition has been successfully applied in virtual reality, human computer interaction, game control, robot interaction, remote controlling of home and office appliances, sign language, activity recognition, human behavior, and training systems etc. Gesture recognition system is designed in four stages: gesture acquisition, feature extraction, classification, and learning. Gesture acquisition is accomplished by position sensors, motion / rate sensors, and digital imaging. Feature extraction and classification are real time stages to analyze the acquired gesture while learning stage is off-line activity to learn the relationship between gesture and information or command. a novel gesture recognition based virtual keyboard system which replicates the transducer based keyboard system. Gesture acquisition is accomplished by a mono vision sensor. Suppose the output of the keyboard system is defined asC= {c1, c2, ...,cL} where c1 =A, C2=B etc where as L=63, its the total no of keys on the keyboard.

Traditional keyboard Key stroke hand Transducer Action Character emitted Movement

Gesture Based Virtual keyboard Key stroke Hand Gesture analysis Character emitted Movement Figure 4.1 Comparison between Traditional Keyboard and Virtual KeyboardIn a traditional keyboard, transducer action is performed in electro-mechanical switch function fashion. While mono vision gesture based virtual keyboard analyzes the hand and finger gestures in the video sequence. Concept of making key stroke by both traditional keyboard and gesture based virtual keyboard are similar. Comparison of actions between traditional keyboard and gesture based virtual keyboard are shown in the figure 4.1.Hand video is captured continuously, Then the video is disassemble into the frames. A template is pre-stored in a database for the symbol A,B, etc. Such that the frames will compare with the pre-stored data, stored in a database, when it is matched the corresponding symbol will be printed.

CHAPTER5PROPOSED SYSTEM5.1PROPOSED WORKIn this project, a new perspective to view the problem of Virtual Keyboard is done by using a simple mono vision camera. As the project is fully based on software centric and not hardware centric, therefore project cost is drastically low. The proposed model of the system is shown in the figure 5.1.Steps involved in the proposed system: Human hand finger movement is analyzed from a video sequence. Background objects are eliminated (other than the finger) using Threshold algorithm. Finger tip is analyzed and processed using Edge Detection Method. Finally, key is evaluated from the process of Edge Detection Method.

WebCam

Capturing Key Strokes in the form of Video

Separate Video into Frames

Identify the odd frame (varies maximum) from the frame sequence

Convert frame into Binary Image by Thresholding and extract the finger

Find the edge(x,y) of the finger from the obtained binary image

Extract the relevant character for the coordinate(x,y)EeeFigure 5.1 Structure of the Keyboard system5.2PICTORIAL REPRESENTATIONA Camera is mounted over any desired location. But the only criteria are that the camera should focus the entire keyboard layout. Practical representation of the Virtual Keyboard System setup is shown in the figure 5.2.

Figure 5.2. Practical Setup5.3MODULESProposed Virtual Keyboard system includes the following modules.1. Image Acquisition Video capture Frame extraction2. Interrupt Detection from Frame sequence3. Finger extraction using Threshold Algorithm4. Finger tip analysis by Edge detection Method5. Key Extraction5.3.1IMAGE ACQUISITIONImage acquisition is the process of capturing the human hand finger movement over the keyboard layout in a video format. Interface is created in-between the computer and the simple mono vision camera. Once when the model is executed, the interface connects the application with the Web cam and sequence of frames i.e., video stream is obtained.5.3.2 INTERRUPT DETECTION FROM FRAME SEQUENCEVideo stream (frames) is compared sequentially whenever the frame is processed which varies much from the other still processed. Then grab that image and use it for next process. (Example - whenever key stroke is done image totally varies from its background such frame is grabbed).5.3.3FINGER EXTRACTION USING THRESHOLD ALGORITHMThresholdingis the simplest method of imagesegmentation. From agrayscaleimage, thresholding can be used to createbinary images. Frames are analyzed continuously. When any interrupt occurs, say human finger, the algorithm is made to captures the particular frame and background elimination occurs. Only the interrupt is shown after converting it into a grey scale image.5.3.4FINGER EDGE DETECTION MECHAANISMHuman finger tip or edge is analyzed using the Edge Detection Method. Edge detectionis a fundamental tool inimage processingandcomputer vision, particularly in the areas offeature detectionandfeature extraction, which aim at identifying points in adigital imageat which the image brightnesschanges sharply or, more formally, has discontinuities. Using vector calculation method, if the estimated finger tip is located in the particular vector region then the desired key is estimated.5.3.5KEY EXTRACTIONWhen key is evaluated, it should make readily available with all the text editors and other applications. This is done by overriding the evaluated key through hardware. Robot package in java does this process easily. Importing Robot package, Robot class is easily called with an object. This object is called by two default functions such as keyPressed and keyReleased with a single integer argument.

5.4ADVANTAGES Opens a new door to keyboard based applications. Games can make the maximum utilization of the keyboard by displaying only those keys that are used in the game. Multilingual support: Since the key-displays are reconfigurable, there is no language barrier any more. Touch screen is similar to this implementation, but they do require additional effort and are not ergonomically comfortable. User doesnt have to raise his arm to the monitor every time to use it. Keyboards with any resolution can be built according to users choice. 5.5APPLICATIONS Any place where keyboard is used. As an alternative keypad for mobile phones or smart devices which have a frontal camera. This is possible if the software is converted to J2ME. As an input device for gaming. At places where a computer or device has multi-lingual users like in net cafes. Highly comfortable usage in areas like ATMs, Hospital Bill Checking, Railway Reservation Center, etc.5.6TECHNOLOGIES USED IN PROPOSED SYSTEM5.6.1THRESHOLDINGThresholdingis the simplest method of imagesegmentation. From agrayscaleimage, thresholding can be used to createbinary images. During the thresholding process, individualpixelsin an image are marked as object pixels if their value is greater than some threshold value (assuming an object to be brighter than the background) and as background pixels otherwise. This convention is known asthreshold above. Variants includethreshold below, which is opposite of threshold above;threshold inside, where a pixel is labeled "object" if its value is between two thresholds; andthreshold outside, which is the opposite of threshold inside. Typically, an object pixel is given a value of 1 while a background pixel is given a value of 0. Finally, a binary image is created by coloring each pixel white or black, depending on a pixel's labels.5.6.1.1 THRESHOLD SELECTIONThe key parameter in the thresholding process is the choice of the threshold value (orvalues, as mentioned earlier). Several different methods for choosing a threshold exist; users can manually choose a threshold value, or a thresholding algorithm can compute a value automatically, which is known asautomatic thresholding.A simple method would be to choose themeanormedianvalue, the rationale being that if the object pixels are brighter than the background, they should also be brighter than the average. In a noiseless image with uniform background and object values, the mean or median will work well as the threshold, however, this will generally not be the case. A more sophisticated approach might be to create ahistogramof the image pixel intensities and use the valley point as the threshold. The histogram approach assumes that there is some average value for the background and object pixels, but that the actual pixel values have some variation around these average values. However, this may be computationally expensive, and image histograms may not have clearly defined valley points, often making the selection of an accurate threshold difficult. One method that is relatively simple, does not require much specific knowledge of the image, and is robust againstimage noise, is the followingiterative method:1. An initial threshold (T) is chosen; this can be done randomly or according to any other method desired.2. The image is segmented into object and background pixels as described above, creating two sets:1. G1= {f(m,n):f(m,n)>T} (object pixels)2. G2= {f(m,n):f(m,n)T} (background pixels) (note, f(m,n) is the value of the pixel located in themthcolumn,nthrow)3. The average of each set is computed.1. m1= average value ofG12. m2= average value ofG24. A new threshold is created that is the average ofm1andm21. T = (m1+m2)/25. Go back to step two, now using the new threshold computed in step four, keep repeating until the new threshold matches the one before it (i.e. until convergence has been reached).5.6.1.1ADAPTIVE THRESHOLDINGThresholding is calledadaptive thresholdingwhen a different threshold is used for different regions in the image. This may also be known aslocalordynamicthresholding.5.6.2EDGE DETECTION MECHANISMEdge detectionis a fundamental tool inimage processingandcomputer vision, particularly in the areas offeature detectionandfeature extraction, which aim at identifying points in adigital imageat which the image brightnesschanges sharply or, more formally which has discontinuities.The purpose of detecting sharp changes in image brightness is to capture important events and changes in properties of the world. It can be shown that under rather general assumptions for an image formation model, discontinuities in image brightness are likely to correspond to: discontinuities in depth, discontinuities in surface orientation, changes in material properties and Variations in scene illumination.In the ideal case, the result of applying an edge detector to an image may lead to a set of connected curves that indicate the boundaries of objects, the boundaries of surface markings as well as curves that correspond to discontinuities in surface orientation. Thus, applying an edge detection algorithm to an image may significantly reduce the amount of data to be processed and may therefore filter out information that may be regarded as less relevant, while preserving the important structural properties of an image. If the edge detection step is successful, the subsequent task of interpreting the information contents in the original image may therefore be substantially simplified. However, it is not always possible to obtain such ideal edges from real life images of moderate complexity. Edges extracted from non-trivial images are often hampered byfragmentation, meaning that the edge curves are not connected, missing edge segments as well asfalse edgesnot corresponding to interesting phenomena in the image thus complicating the subsequent task of interpreting the i

CHAPTER6PROPOSED SYSTEM DESIGN6.1UML DIAGRAMSSet of fundamental design concepts have evolved over the past three decades. Each one provides the software designer with the foundation from which more sophisticated design methods can be applied. The choice of what models and diagrams one creates has a great influence on how a problem was encountered, and how a corresponding solution is shaped.Design is defined as a model of the system and continues by converting this model to implementation of the new system. Every complex system is best approached through a small set of nearly independent view of the model. Every models can be expressed at different levels. The best models are connected to reality.In the design process the following UML diagrams are used. Use Case Model Class Diagram Activity Diagram Sequence Diagram6.1.1USE CASE DIAGRAMUse cases are scenario for understanding the system requirements. A use case model can be instrumental in project development, planning and documentation of system requirements. A use case is an interaction between users and a system; it captures the goal of the users and the responsibility of the system to its user.The use case model describes the uses of the system and shows the courses of events that can be performed. In other words, it shows a system in terms of its users and how it is being used from a user point of view. Furthermore, it defines what happens in the system when the use case is performed. In essence, the use case model tries to systematically identify uses of the system and therefore the systems responsibilitiesA use case model also can discover classes and relationships among subsystems of the systems. A use case model can be developed by talking to typical users and discussing the various things they might want to do with the application being prepared. Each use or scenario represents what the user wants to do.The use case model expresses what the business or application will do and not how. Since it is called as what model.A use case is a sequence of transactions in a system whose task is to yield results of measurable value to an individual actor of the system. Since the use case model provides an external view of the system or application, it is directed primarily toward the users or the actors of the system, not its implementers. An actor is a user playing a role with respect to the system. Use case diagram of the proposed Virtual Keyboard System is shown in the figure 6.1.

Figure 6.1Use Case Diagram

6.1.2SEQUENCE DIAGRAMA sequence diagram is made up of objects and messages. Objectives are represented exactly how they have been represented in all UML diagrams as rectangles with the underline in class name within the rectangle. Sequence diagrams map the scenarios described by use cases in step by step detail to define how objects collaborate to achieve his applications goals. A lifeline in a sequence diagram represents an object and shows all its points of interaction with other objects in events that are important Sequence diagrams are easy and intuitive way of describing the behavior of a system by viewing the interaction between the system and its environment. A sequence diagram has two dimensions .The vertical dimension represents time, the horizontal dimension represents different objects. The vertical line is called the objects lifetime. The lifeline represents the objects existence during interaction. An object is shown as a box at the top of a dashed vertical line. Each message is represented by an arrow between the lifelines of two objects. The order in which this messages occur is shown top to bottom on the page. Thus a sequence diagram is very simple and has immediate visual appeal. Lifelines start at the top of the sequence diagram and descend vertically to indicate the passage of time. Interactions between objects- messages and replies-messages are drawn as horizontal direction arrows connecting lifelines. In addition, boxes known as combine fragments are drawn around sets of arrows to mark alternative actions, loops and other control structures. Sequence diagram for Image Acquisition module is shown in the figure 6.2.

Figure 6.2 Sequence Diagram for Image AcquisitionSequence diagram for Interrupt Detection in Frame Sequence module is shown in the figure 6.3.

Figure 6.3 Sequence Diagram for Interrupt Detection in Frame SequenceSequence Diagram for Finger Extraction using Threshold Algorithm module is shown in the figure 6.4.

Figure 6.4 Sequence Diagram for Finger Extraction using Threshold AlgorithmSequence Diagram for Finger Tip analysis by using Edge detection module is shown in the figure 6.5.

Figure 6.5 Sequence Diagram for Finger Tip analysis by using Edge detectionSequence Diagram for Key Extraction module is shown in the figure 6.6.

Figure 6.6 Sequence Diagram for Key Extraction

6.1.3CLASS DIAGRAM In class diagram, there are four classes. They are video to frames, threshold, edge detectionand keyboard interupt handler. The functions for video to fames are frame extracted, buffered image, frame saving and operations are save() and frame extracted(). The functions and operators for threshold are getRGB value, average RGB, binary image and operators are getRGB() and avgRGB(). The functions and operators for edge detection are get finger tip vector and vector matching and opertor is get first black(). The functions and operators for keyboard interupt handler are vector key estimation and hardware overriding and operator is print char(). The Class digram of the proposed system is shown in the figure 6.7.

Figure 6.7 Class Diagram of Virtual Keyboard6.1.4 ACTIVITY DIAGRAMAn activity diagram is variation or special case of machine, in which the states are activities representing the performance of operations and the transitions are triggered by the completion of the operations. Unlike state diagram that focus on the events occurring on a single object as it responds to messages, an activity diagram can be used to model an entire business process. The purpose of an activity diagram is to provide a view of flows and what is going on inside a use case or among several classes. However, an activity diagram can also be used to represent a classs method implementation as well.An activity diagram is used mostly to show the internal state of an object, but external may appear in them. An external event appears when the object is in a wait state, a state during which there is no internal activity by the object and object is waiting for some external event to occur as the result of an activity by another object. The two states are wait state and activity state. The Activity diagram of the proposed system is shown in the figure 6.8.

Figure 6.8 Activity Diagram of Virtual KeyboardCHAPTER7PROJECT DESCRIPTION7.1IMAGE ACQUISITIONImage acquisition is first step of this project where acquisition process is done by using simple mono vision camera. Simple mono vision camera is not a special one just like ordinary camera such as external camera or web cam integrated. Camera should provide atleast 320x240 pixels of image. Camera pixel should be 1.3 or above. More the pixel greater the accurate rate and decreases the error rate.Camera default device driver is required for hardware overriding .When multiple cameras are connected, a window is opened to select required camera. vfw://0 is the key word used to detect multiple connected cameras.Acquisition is in the form of the video. Video is sequence of multiple frames. When video splitted into frames, each frames deserved as images 10-15 frames are created for every second. During the process of image acquisition, all the obtained images are stored in any desired location if necessary. The block diagram of the Image Acquisition is shown in the figure 7.1 and Snapshots are on the figure 7.2.

SURFACEDFDFDFVIDEO CAPTURECAMERA

CONVERTING VIDEO TO FRAMES

Figure 7.1 Block Diagram for Image Acquisition

FRAMES COMBINES TOGETHER AND FORMS A VIDEOVIDEO INTO FRAMESFigure 7.2 Snapshot of Image Acquisition7.2INTERRUPT DETECTION IN FRAME SEQUENCE

Check for maximum variation in 5 framesAt regular intervals (about 10 to 15 times every second), the current frame from video is compared with previous images and grab the image which has maximum variation. The block diagram of the Interrupt Detection in Frame Sequence is shown in the figure 7.3 and Snapshots are on the figure 7.4.

Grab the Frame

FRAMES Yes

PROCEEDS TO NEXT LEVEL No

Figure 7.3 Block Diagram for Interrupt Detection in Frame Sequence

SEQUENCE OF FRAMESINTERRUPT DETECTED FRAMEFigure 7.4 Snapshot of Interrupt Detection7.3FINGER EXTRACTION USING THRESHOLD ALGORITHMImage from the preprocessing sector comprises Red, Green and Blue model i.e., RGB color model. RGB color model is removed from each image and only black and white color model remains. These two colors are called as binary colors and finally a binary image is created.Thus by converting a colored image into binary image helps in calculating and analyzing the fore ground objects. Ultimately background object details are quietly often reduced to greater level. Combining all these red >16, green >8, blue>0 produces binary image. The block diagram of the Finger extraction using Threshold Algorithm is shown in the figure 7.5 and Snapshots are on the figure 7.6.

7.3.1PSEUDOCODE FOR THRESHOLDING

WHITE PIXELIMAGE PIXELRed >16 noyes

Green >8no

Blue >0 yesno

BLACK PIXELyes

Figure 7.5 Snapshot for Finger extraction using Threshold Algorithm

RGB IMAGE BINARY IMAGEFigure 7.6 Snapshot for Finger extraction using Threshold Algorithm7.4FINGER TIP ANALYSIS BY EDGE DETECTION METHODEdge detectionis a fundamental tool inimage processingandcomputer vision, particularly in the areas offeature detectionandfeature extraction, which aim at identifying points in adigital imageat which the image brightnesschanges sharply or, more formally which has discontinuities. The purpose of detecting sharp changes in image brightness is to capture important events and changes in properties of the world. The output image from the process of thresholding is in the form of binary image. The obtained image contains more details about the object and lesser details about the background. Only black pixels are taken into account. First black pixel layer is analyzed from the binary image. The vector value of the pixel is calculated. Corresponding value for the vector is pre-stored. Then the key is evaluated. The figure 7.7 represents the method of Finger tip analysis by Edge Detection.7.4.1PSEUDOCODE FOR EDGE DETECTION METHOD BINARY IMAGE RED COLOURED LINE DEPICTS THE EDGE OF THE FINGERFigure 7.7 Snapshot of Finger tip analysis by Edge Detection Method7.5KEY EXTRACTIONOnce when the Virtual Keyboard model is implemented, evaluated key or the model should act as like as traditional keyboard i.e., when key is evaluated, it should made readily available with all the text editors and other applications. This is done by overriding the evaluated key through hardware. Robot package in java does this process easily. Importing Robot package, Robot class is easily called with an object. This object is called by two default functions such as keyPress and keyRelease with a single integer argument. But in case of this Virtual Keyboard only keyPress function in sufficient for Hardware overriding. The figure 7.8 represents the method of Key Extraction.

Respective x and y coordinate value is calculatedFigure 7.8 Snapshot Key Extraction

CHAPTER 8SYSTEM TESTING8.1 SYSTEM TESTINGSystem testing is the stage of implementation. It aims at testing and ensuring that the system works accurately and efficiently before live operation commences. The logical design and physical design should be thoroughly and continually examined on paper to ensure that they will work when implemented. Thus the system in implementation should be a confirmation that all system works. The testing phase includes entering the sample data to verify whether the system is suitably working to the requirements mentioned. This phase is important in the way that it actually deals with the real data.Software testing is an important element of software quality assurance and ultimate review of specification, design and coding. In testing, the engineer creates a series of test cases that are indented to demolish the software that has been built.8.2OBJECTIVE OF TESTINGThe rules that serve for testing are, Testing is a process of executing a program of executing a program with the intent of finding an error. A good testing is the one that has the high probability of finding undiscovered errors. A successful test is the one that uncovers a discovered error. If testing is conducted successfully according to the above objectives, it will uncover the errors in the software.The various types of testing are: Unit testing Integration testing Validation testing

8.3UNIT TESTINGUnit testing is a procedure used to validate the individual unit of code. A unit is the smallest testable part of an application. The goal of unit testing is to isolate each part of the program and show that the individual parts are correct. A unit test provides a strict, written contract .8.4INTEGRATION TESTINGIntegration testing (sometimes called Integration and Testing, abbreviated I&T) is the phase of software testing in which individual software modules are combined and tested as a group. Integration testing takes as its modules that have been unit tested, group them in larger aggregates, applies tests defined in an integration test plan to those aggregates, and delivers as its output the integrated system ready for system testing.8.5VALIDATION TESTINGValidation testing provides the final assurance that the software meets all functional behaviour and performance requirements. The software once validated must be combined with other system elements. After each validation test cases has been conducted, two possible conditions exist.They are: The function from specification characteristics confirms to specification and is accepted. A deviation from specification is uncovered and a deficiency list is created. The deviation or error discovered at this stage in a project can rarely be corrected prior to schedule completion, it is necessary to negotiate with the customer to establish a method for resolving deficiencies.8.6INTRODUCTION TO WINRUNNERWin Runner facilitates easy test creation by recording how work on applications work as you point and click GUI objects in your applications. You can generate a test script in the C-like Test Script Language (TSL). You can further enhance your test script with manual programming. Win Runner includes the function generator which helps you to quickly and easily add functions to your recorded test.CHAPTER 9CONCLUSION AND FUTURE ENHANCEMENT9.1CONCLUSIONResults showed a very reliable and practical system. The proposed system is less cost due to its software centric mechanism rather a hardware centric mechanism. Performance of the system had been tested over Personal Computer. The data set involved in the development of the system can be easily altered as user requests. Standard data set style is implemented. Response time for Key Extraction is less quite compared to system like Finger-Joint Gesture Wearable Keypad, Thumbcode, etc. 9.2FUTURE ENHANCEMENTFailure rate of the system entirely depends on the light intensity. The system works on dim light also but failure rate is high due to shadowing effects. Future work relies on high efficiency of the system with rid of light intensity.

10.1SCREEN SHOTS

Figure10.1 Screen Shot for Choosing the EnvironmentINPUTOUTPUT

INPUTOUTPUT

2