Touch Screen Input Shapes Recognition in PN Designer

Katarína Jelemenská, Pavel �i�ák Institute of Computer Systems and Networks

Faculty of Informatics and Information Technologies Slovak University of Technology in Bratislava

Slovakia jelemenska@fiit.stuba.sk; cicak@fiit.stuba.sk

Abstract: This paper presents an approach to Petri net objects recognition that has been adopted to develop natural user interface based on Tablet PC stylus and touch screen, so that students have the feeling they are drawing on a sheet of paper rather than on a computer. The user interface was used in an educating tool, designed to support Petri nets education, and proved to be efficient and user friendly. The PN Designer, as the tool is called, helps students to master Petri nets and can also be used to examine the student‘s knowledge. For this purpose the server part has been developed mainly devoted to tests distribution and collection. The tests themselves are taken and evaluated on client computers, not directly on the server. Therefore there will not be a situation where all the solved tests would have been lost due to server malfunction.

Introduction

The evolution of Information Technology has affected every aspect of our lives, including education. Natural user interface has been recognized as one of the important features of modern applications either devoted to computers or various smart devices. Touch screens, integrated in Tablet PCs, represent one of the possible ways to design intuitive user interface. In recent times, Tablet PCs have gained increasing importance in the education process of students (Enriquez et al., 2006, Harris et al., 2008, Jelemenská et al., 2010; Johnson et al., 2006; Jurikovi� et al., 2010, Kowalski et al., 2007b). A lot of applications have been developed to support education process, a majority of which are interactive presentation systems e.g. Classroom Presenter (Anderson et al., 2005, 2006; University of Washington, 2008), DyKnow (Berque, 2006), Ubiquitous Presenter (Wilkerson et al., 2005), Enhanced Classroom Presenter (Jelemenská et al., 2012), or The Presenter (Jelemenská et al., 2011). The interactive presentation systems support mostly handwriting recognition feature that is usually implemented using the built-in operating system functions (most often Windows XP or Windows7). An interesting solution of interactive presentation system brings D&T Presenter (Hlatký et al., 2012) that is focused to support mainly the testability of digital systems field. For this purpose the presentation system was enriched by an automatic topic support and specific libraries for logic circuits design and test.

However, a number of Tablet PC tools have been developed supporting other activities, mostly in education process. The most popular tools are Ink Survey, Rite Pen, Equation Writer, Ink Flash Cards, Send to OneNote 2003, GoBinder Lite, Snipping Tool, and Ink Desktop.

Ink Survey (Kowalski et al., 2007a) is a freeware web-based tool that comes from the School of Mines in Colorado and can be downloaded from the college web site. It uses web-based client-server architecture to create and display the survey questions. An instructor uses the web browser to enter questions into this system. Subsequently, he or she can use these questions and collect responses from students. A student must be registered in the system in order to answer any question. The tool allows response by text or drawings. In case of drawings, the student can use the touch screen and stylus or finger.

Rite Pen (Evernote Production, n. d.) is a commercial application. Its main contribution is the ability to convert any type of screen to drawing canvas. Consecutively, it is possible to use stylus and draw on the screen using touch screen input. Inked areas with handwriting can be recognized and converted into the computer based text. Handwriting recognition of this program is on a very high level. Stroke analysis is almost always correct and a list of alternative conversions can be used as well, where the program displays the most likely alternatives paired with the corresponding percentage probability. Rite Pen does not use in-built API (Application

- 1418 -

Programming Interface) of operating system, and therefore it is not dependent on an installed language version of the system. In addition, the system also supports diacritics.

Equation Writer (Microsoft, 2005a) is a commercial tool dedicated to mathematical expressions recognition. A user can handwrite a math equation, and convert it into a neatly typewritten image, copy and paste it into a report or a presentation. It is one of five tools included in the Education Pack for Tablet PC developed by Microsoft company. The other popular tools from this pack include Ink Flash Cards, Send to OneNote 2003, and GoBinder Lite. The tool Ink Flash Cards enables a student to create flash cards that can be used as a help to learn facts for example to study for an exam. A question is to be handwritten on the front of a card, and an answer on the back. After building a set of cards, the student can review the cards to test his/her knowledge and mark those ones that need to be reviewed again. The tool Send to OneNote 2003 helps to keep all the class notes and research in one place. It allows to send web pages, pictures, and other files to OneNote, and annotate or highlight them with the tablet pen. The tool GoBinder Lite (by Agilix Labs, Inc.) enables to jot down quick reminders about class schedules, project deadlines, and extracurricular activities using the tablet pen.

Snipping Tool (Microsoft, 2005b) is the most popular of the tools included in the Experience Pack for Tablet PC developed by Microsoft company (included in operating system Win7). It enables to use the tablet pen to select a portion of a website, program, or other content on the screen, annotate it, and then save it, copy it, or e-mail it to a friend. Another popular tool from this pack include Ink Desktop that turns the Tablet PC desktop into a to do list, a monthly calendar, or a grid paper, where the user can take notes to be accessed later on. The tool enables to customize ink color or thickness, and also to copy and paste notes into another program. PN Designer

Each of the tools mentioned in the previous section includes some kind of digital ink recognition or manipulation, utilized in a specific area. An educating tool called PN Designer that is presented in this paper belongs to this number. PN Designer was developed to support Petri nets education. Although there are many applications, better or worse, providing the environment for Petri nets design, simulation, and properties analysis, none of the known applications is directly suitable for testing the student’s knowledge and practical skills in designing Petri net. This was the main motivation for designing a new tool, called PN Designer (see Fig.1) that will provide the facilities like test creation, and test evaluation, together with simple, comfortable and user friendly interface, equipped with shape recognition editor. To enable students to draw a Petri net as they were used before on a piece of paper, a novel approach was developed to recognize the Petri net components. The approach will be described in the following section. Petri Net Input To develop a tool for Tablet PC with Windows operating system that will support the use of a stylus as a replacement of mouse there is no need to install any special development software since the operating system, containing the appropriate library, will take care of this function. However, in case the special stylus features should be used e.g. gestures recognition or pen’s pressure recognition, the special Tablet PC libraries should be installed and used to call the special functions. For example, the principle of handwriting recognition is as follows. When the user starts writing, the so called Pen API (Application Programming Interface) begins to capture the movements. The coordinates are transferred to the ink manipulation interface called Ink API. The Ink API draws the movements on the screen and saves them. The saved data are then sent to the Text recognizer to be interpreted as the characters. Similarly, the Gesture recognizer is used to recognize the common gestures. However, to recognize other objects or actions the special recognizers have to be developed.

Petri net is especially suitable for stylus and touch screen interface since it is a graphical and mathematical tool with two kinds of nodes - places and transitions, and edges that can be directed either from place to transition, or from transition to place. For graphical representation the input in the form of a drawing is the most natural one and the limited number of different objects to be recognized makes the proper recognition easier. Our aim was to use the Tablet PC stylus as an ordinary pencil in order to draw a Petri net. This means a user can draw a circle-like shape on the screen and the system will recognize it as a place and replace it with a neat circle. Similarly, a straight line sketched on the screen will be recognized as a transition. To draw an edge the user starts drawing a line from one object (place or transition) to the other. The edge is then created exactly

- 1419 -

as the user drew it. Several shape recognition algorithms have been designed, implemented, and tested to find the most efficient solution for this purpose. The resulting algorithms will be described here.

Drawn Objects Recognition

When drawing Petri net we need to recognize two objects types – place and transition, and one action – the edge. While drawing on the touch screen the points coordinates are captured and analyzed afterwards. In order to recognize a drawn place, we should identify a circle. The circle can be mathematically divided into four sections according to how the x and y coordinates of the circle points behave. Namely, the circle will be divided into the following sections:

1. the x-coordinate grows, the y-coordinate grows 2. the x-coordinate grows, the y-coordinate decreases 3. the x-coordinate decreases, the y-coordinate decreases 4. the x-coordinate decreases, the y-coordinate grows

During the shape recognition it is checked whether at least three of these sections can be found on the user input. If so, the curve is identified as a circle and replaced by a neat circle.

A transition is recognized as a straight line drawn in any direction. Again the recognition is based on the points coordinates analysis. The straight line is mathematically represented by two points selected as an eleventh point from the beginning, and end of the drawn shape respectively. The distance from this straight line is then calculated for all the points in between the two points and compared to the stated limit. A transition is identified in case all the points satisfy the condition and the drawn shape is replaced by a neat rectangle.

In addition to place and transition a criss-cross gesture is recognized as a delete command. The criss-cross gesture is actually composed of two straight lines that intersect. That means it is first necessary to recognize the two straight lines using the above mentioned algorithm and then calculate whether the two lines intersect. In order to recognize it as one object it was necessary to insert a delay after each shape recognition. Obviously this delay affects also the recognition of other object types.

Figure 1: PN Designer user interface

question list

Help

selected question

Edit Tools View Server File

drawn reply

New Document

New Test

Add question icons

Delete question icon

- 1420 -

A Petri net edge is not recognized as an object. Instead, it represents an action - drawing a line from one object (place or transition) to the other. The edge shape follows exactly the curve drawn on the touch screen, but converted into an arrow. Some examples of drawn shapes together with the recognition results illustrates Fig.2. Other PN Designer Features

The above mentioned shapes recognition algorithms were integrated into the educating tool called PN Designer as an integrated part of the graphical Petri net editor. In addition to shapes recognition function the editor provides other basic editing features (see Fig. 3). The objects can be selected using stylus or mouse and

moved anywhere on the canvas. The properties of the selected objects can be edited as well. In addition to drawing Petri net using stylus and touch screen the editor provides also an alternative input. Using the provided menu bars and tool bars the Petri net can be easily edited with traditional computer mouse. Clicking for example a button named “Add transition” the transition mode is activated enabling the user to place any number of transitions onto the canvas. Similarly the place mode and edge mode can be activated as well. In both cases the interaction is very user friendly and made as simple as possible.

The PN Designer can work in learning or testing mode. In learning mode students have the possibility to learn by drawing their own Petri nets, simulating them, analyzing their properties, creating their own test questions and tests, and testing themselves. In this mode the built-in real-time correction mechanism is enabled (disabled during an examination), warning students in case of invalid actions (e.g. connection between two places). In testing mode the students are solving the tests prepared by the teacher. Various kinds of test questions can be used varying from drawing a Petri net with selected attributes, completing a pre-drawn Petri net, redrawing an incorrect Petri net to traditional on-line test questions, including multiple choice, single and multi answer, and matching questions. This gives the teacher the advantage of using a single test, created in one environment, to evaluate students’ theoretical knowledge as well as practical skills in the Petri nets area.

Figure 2: Shape recognition results.

- 1421 -

Conclusions

The PN Designer, designed and implemented as a result of this work, enables students to understand the basic principles of Petri net design and simulation and to gain basic designing skills by means of solving the integrated exercises. Teachers can prepare various tasks that can then be used in the learning and/or assessment process. To support the assessment process the server part was designed to provide test creation, distribution, and automated evaluation facilities.

The PN Designer was implemented and tested in the pilot run in the academic year 2010/2011. The questionnaire inquiring about student attitudes was filled out by 43 out of 60 students involved. The students appreciated especially the fact that they can do the exam in the environment they are already familiar with from the learning process. The PN Designer’s natural user interface is simple, very intuitive and the vast majority of students found it very comfortable. However, the educating tool not only provides the enhanced Petri net editor with the shape recognition function, but also supports Petri net simulation and properties analysis. The results show a great deal of attractiveness for this kind of learning and substantial time saving in the assessment process while preserving the assessment quality. Although there are many applications supporting Petri net design, simulation, and properties analysis, based on the authors’ knowledge, none of the known applications provides facilities like test creation, test evaluation, or shape recognition editor. This is the main enhancement that was brought by our educating tool.

Acknowledgements This work was partially supported by Slovak Science Grant Agency (VEGA 1/1008/12 “Optimization of low-power design of digital and mixed integrated systems“). References Anderson, R., Anderson, R., McDowell, L., Simon, B. (2005). Use of Classroom Presenter in Engineering Courses. FIE '05 - 35th Annual Conference Frontiers in Education Proceedings (pp. T1A – 1-6.). USA: IEEE. Anderson, R. et al. (2006). Classroom Presenter: A Classroom Interaction System for Active and Collaborative Learning. In D.A. Berque, J.C. Prey, & R.H. Reed (Eds.), The Impact of Tablet PCs and Pen-based Technology on Education: vignettes, evaluations, and future directions (pp. 21-30). West Lafayette: Purdue University Press, ISBN: 978-1-55753-434-7.

Figure 3: PN Designer editing menu.

- 1422 -

Berque, D. (2006). An Evaluation of a Broad Deployment of DyKnow Software to Support Note Taking and Interaction Using Pen-Based Computers. Journal of Computing Sciences in Colleges, 21 (6), 204-216. Enriquez, A., Gunawardena, A., Kowalski, F., Kowalski, S., Millard, D.,& Vanides, J. (2006). Innovations in Engineering Education using Tablet PCs – Panel Discussion with Four Institutions. 36th ASEE/IEEE Frontiers in Education Conference Proceedings (pp. S2B-1-2). San Diego, CA: IEEE. Evernote Production, (n.d.). Right Pen 4.0 Guide. Retrieved December 10th, 2011, from http://www.ritescript.com/Manuals/ritePenQSG.aspx Harris, D.A., Krousgrill, C. (2008). Distance Education: New Technologies and New Directions. Proceedings of the IEEE, 96 (6), 917 - 930. doi: 10.1109/JPROC.2008.921612 Hlatký, M., Martinek, V., Gramatová. E. (2012). D&T Presenter - Electronic Interactive System for Design and Test Education. Proceedings of the 15th IEEE Symposium on Design and Diagnostics of Electronic Circuits and Systems (DDECS2012), (pp. 175-178). Piscataway, NJ : Institute of Electrical and Electronic Engineers, Inc., ISBN 978-1-4244-9754-6. Jelemenská K, �i�ák P, Dúcky V. (2011). Interactive Presentation Towards Students Engagement. In: Procedia - Social and Behavioral Sciences, 29, 1645–53. ISSN 1877-0428. Jelemenská, K., �i�ák, P., Koine, P. (2010). The Pen-based Technology towards the Lecture Improvement. In L.G. Chova, D.M. Belenguer, I.C. Torres (Eds.), EDULEARN 2010 Proceedings CD, (pp.6703-6712). Valencia, Spain: IATED. ISBN: 978-84-613-9386-2 Jelemenská, K., Koine, P., & �i�ák, P. (2012). Enhanced Classroom Presenter. In T. Sobh, & K. Elleithy (Eds.), Emerging Trends in Computing, Informatics, Systems Sciences, and Engineering. Lecture Notes in Electrical Engineering, Vol. 151. Dordrecht: Springer, ISBN 978-1-4614-3557-0 . Johnson, D. W., Johnson, R. T., & Smith, K. A. (2006). Active learning: cooperation in the College Classroom. 3rd Ed. Interaction Book Company. Jurikovi�, M. et al. (2010). Using Tablet Computers for Presentation of Slides in the Process of Teaching. In L.G. Chova, D.M. Belenguer, I.C. Torres (Eds.), EDULEARN 2010 Proceedings CD, (pp. 1374–1379). Valencia, Spain: IATED. ISBN: 978-84-613-9386-2 Kowalski, F., Kowalski, S., & Hoover, E.( 2007a). Using InkSurvey: A Free Web-Based Tool for Open-Ended Questioning to Promote Active Learning and Real-Time Formative Assessment of Tablet PC-Equipped Engineering Students. American Society for Engineering Educational Annual Conference and Exposition Proceedings (12 p.). Kowalski, F., Williams, J., Reed, R., & Vanides, J. (2007b). Transforming teaching and learning using Tablet PCs - A panel discussion using Tablet PCs. The 2007 American Society for Engineering Educational Annual Conference and Exposition Proceedings. Microsoft (2005a). Education Pack for Windows XP Tablet PC Edition, Retrieved December 10th, 2011, from http://www.microsoft.com/download/en/details.aspx?id=1153 Microsoft (2005b). Experience Pack for Windows XP Tablet PC Edition, Retrieved December 10th, 2011, from http://www.microsoft.com/download/en/details.aspx?id=13797 University of Washington (2008). Classroom Presenter 3.1. Retrieved December 10th, 2011, from http://classroompresenter.cs.washington.edu/ Wilkerson, M., Griswold, W.G., & Simon, B. (2005). Ubiquitous Presenter: Increasing Student Access and Control in a Digital Lecturing Environment. SIGCSE'05 Proceedings. St. Louis, Missouri, USA.

- 1423 -