Advancing Human-Centric Solutions with Software Engineering Practices

Software Engineering Lab

School of Computer Science and EngineeringSoftware Engineering Lab

Evgeny PYSHKINピシキンエヴゲニー

Ph.D., Senior Associate ProfessorOffice: 342-B E-mail: pyshe@u-aizu.ac.jp

http://u-aizu.ac.jp/~pyshe

Evgeny Pyshkin: Academic Expertise

▪ Undergraduate Courses

• Introduction to Programming

• Programming in C

• Introduction to Data Management

▪ Graduate Courses

• Software Development Paradigms

▪ Areas of Interest

• Human-centric applications▪ Information systems for travelers

▪ Speech processing applications

▪ Educational software

• Software design and engineering

• Software quality assurance

• Software development education

• Art and Humanities

2

http://u-aizu.ac.jp/~pysheSee more:

“We are in era where we are reimagining nearly everything… powered by new devices, plus connectivity, plus new user interfaces, plus beauty…”

Mary Meeker

International Outlook: Academic Links

▪ 2015 – 2019 Lectures and seminars on Software Design and Engineering, Human-Centric Systems, Digital Transformation

• University of Luxembourg

• Belarussian Russian University, Mogilev, Belarus

• Nanjing University of Aeronautics and Astronautics, China

• Ulm University of Applied Sciences, Germany

• Karelia University of Applied Sciences, Joensuu, Finland

• Peter the Great St. Petersburg Polytechnic University, Russia

▪ 2019 Invited talk on Interinfluence of Environmental Integration Architecture

• Aalto University, Finland

▪ 2015 - 2016 Seminars on Human-Centric Computing and Software Engineering, UniversitatPolitècnica de València, Spain

▪ 2015 WC2 Meeting, Talk on Unique Monuments of Buddhist Architecture in Aizu, Meiji University, Tokyo, Japan

3

Student Research Theme Overview (1):Information Systems for Travelers

▪ A Tool for Ancient Map Markup for Extending Traveler Experience• This task is from our ongoing project on designing information system for travelers. Developing a

tool allowing to markup a set of ancient maps, in order to “connect” different historical perspectives of a place and to provide necessary references to actual electronic maps. In a sense, the possible implementation follows a concept of virtual time travel [1, 2].

▪ Ontologies in Travel-Centric Design• The aim of this project is to develop a relevant domain-specific ontology to be used as a

conceptual foundation for designing travel-centric systems. We have to study existing tools for ontology development and investigate the subject domain carefully in order to define an extensible ontology model. The project includes research on currently available tools and libraries for domain-specific ontology design and access [1, 2].

▪ Location Accuracy Management Policies for Location Aware Mobile Applications• The aim of this project is to improve location accuracy for creating better post-trip reports. Our

idea is to combine a general-purpose application (such as Google Timelines) with a companion application aimed at managing location data quality according to policies set by the user [10]. See also the poster “Trace Accuracy Analysis for Improving Sightseeing Trip Timelines” (presented at the APSCIT Meeting 2018, Sapporo, Japan, July 2018).

These projects are parts of the ongoing JSPS funded international project “Advancing interfaces, ontologies and algorithms for traveler-centric information systems supporting geographical, cultural and historical perspectives”.

4

Student Research Theme Overview (2):Speech Processing Apps

▪ Pitch Quality Evaluation in CAPT Tools• Currently we have a couple of prototype native solutions (for Android and for iOS devices) for

speech prosody modeling and visualization that can be used in computer-assisted prosody teaching (CAPT). A particular area of research is to define a good model for evaluation whether a user’s pitch corresponds well/not well to a model pitch. We need to find informative pitch quality metrics enabling language learners to act on the quantitative and qualitative feedback received from the prosody-based language learning software. Description of the current state of our project can be found in [3]. The most recent related work is [4] (to appear, feel free to ask Prof. Pyshkin for the preprint). Description of the current state of our project can be found in [3, 4]. See also the poster “Study Intonation courseware kit for EFL prosody teaching” (presented at the Speech Prosody 2018 conference, Poznan, Poland, June 2018).

▪ Extracting Keywords from Speech: Analysis of Speech Processing Tools• There are numerous solutions for speech processing including our own tool for keyword

extraction from unstructured speech data [5, 6]. However, there is a lack of extensive analysis on whether how these tools are accurate enough. We need to arrange a number of experiments in order to discover the possible drawbacks of such systems and to refine the requirements for improved implementations.

These projects are parts of the ongoing project in collaboration with the CLR of the University of Aizu and with our partners from Peter the Great St. Petersburg Polytechnic University (St. Petersburg, Russia).

5

Student Research Theme Overview (3):Human-Centric and Software Engineering

▪ Wine Connoisseur Knowledge Modeling and Representation• The goal of this project is to define a relevant ontology and convenient tools for managing

information about the wines, and for creating sharable and jointly edited tasting reports. Creating a client application for mobile devices would be preferable. The project requires a preliminary study on currently available solutions [7].

▪ Digitalization in Music Composition: Software and Its Personalization• This theme suggests an area for study of today’s music representation models and tools for music

composition. We want to analyze, how digitalization affects this area, what are possible approaches for better composition software deployment, customization, and personalization. The first goal I to collect user requirements for different classes of music composition software and proceed with investigation on how existing solutions fit these requirements. See [11, 12] to start exploring the topic.

▪ Mobile Software Testing Farm: Health Monitoring of Mobile Devices under Intensive Tests• Within the framework of our ongoing project (in collaboration with Prof. Maxim Mozgovoy from

the Active Knowledge Engineering Lab) on designing an environment for mobile software testing with a particular focus on implementing long-lasting tests on real devices connected to the framework [8, 9]. We defined a conceptual architecture and implemented a working prototype used in the big mobile game project “World of Tennis: Roaring 20’s”. One open problem is monitoring and reporting the device heath in process of testing. See also the poster “Mobile Farm for Software Testing” (presented at the MobileHCI’18 conference, Barcelona, Spain, September 2018).

6

Student Research Theme Overview (4):Human-Centric and Software Engineering

▪ Informational Approach to Building Architecture: Ontology and Personalized Software• The project is about designing the ontology and related software aimed

at creating a personalized environment for learning building architectures, including finding the ways to present particularities of architectural styles, to learn how different styles interact, to introduce particular architectural attractions and their creators to learners and travelers.

▪ Informational Approach to Study Software Metaphors: Possibilities of Class Time Organization• This is a cross-disciplinary research on using metaphors in software

engineering research, development and education. An expected outcome is a possible set of classroom activities and related tutorials for studying metaphors within the scope of software engineering courses and soft skill trainings.

▪ YOU HAVE YOUR OWN CREATIVE IDEA OF YOUR RESEARCH TOPIC?• Come to our lab, and we will be brainstorming it!

7

“Time Machine”: Travel Guides with Old Maps▪ Travel, discover and learn▪ Leveraging current IT features▪ Interactive tools which are hardly

available without computers• A perspective to the ancient views

accessible electronically (which seems to be out of attention of the existing systems) can significantly extend the way to learn history while visiting tourist attractions all around the world

8

▪ We want to develop tools which would allow marking up the old maps available usually as images in order to establish the sets of corresponding points between the old maps and the current electronic maps.

▪ Many open problems• Map incompleteness• Difference in map orientation (conditioned

often by national and cultural particularities)• Different scaling and area shapes• Different object names• Differences in landscapes• Existing natural of artificial objects• Disappeared names and territories

Prosody based Speech Processing Systems

▪ Addressing the domain of developing software for prosody learning• Pitch detection and approximation• Pitch visualization and modeling intonation

styles and presenting them to a learner• Algorithms of signal processing meet

problems of practical phonology• Particularly promising for languages where

the stress is mainly phrasal (like French) or tonic (like Chinese)

• Pitch estimation and post-processing

9

Keyword extraction as a particular task of information retrieval▪ Unstructured

speech data (no metadata available)

▪ Audio and audiovisual content

▪ Automatic information indexing

Mobile Software Quality Assurance Process

▪ Improving testing quality and advancing methodology of resource and time-consuming testing automation focusing on mobile applications:• High variety of human-

computer interaction• Rich user GUI including

non-standard or hand-drawn user controls

• May behave very differently on different devices

• Contains elements that are easy to overlook with manual testing

• Real mobile devices required (not only emulators / simulators)

10

▪ Use case: mobile games• Automated non-native GUI testing

relying on image recognition algorithms

• Organization of parallel testing on multiple devices

• Intensive data exchange• Maintenance of device health and

acceptable battery levels of test devices

Welcome to the Software Engineering Lab

▪ Join our team…

• … if you are creative• … if you want to create high class

software, including software for mobile devices

• …. if you are interested in joining our international projects and working in an international community

• …. if you want to extends your knowledge in software engineering and mobile software development

• …. if you are interested in multi-disciplinary projects in the areas of software engineering, information retrieval, Internet applications, and human-centric systems

11

References1. E. Pyshkin, A. Baratynskiy, A. Chisler, and B. Skripal,

“Information management for travelers: Towards better route and leisure suggestion,” In Proc. of 2016 Federated Conference on Computer Science and Information Systems, Sep 11–14, Gdansk, Poland, 2016, M. Ganzha, L. Maciaszek, M. Paprzycki (eds). ACSIS, Vol. 8, pp. 429–438. DOI: 10.15439/2016F220. Web: https://fedcsis.org/proceedings/2016/drp/224.html

2. E. Pyshkin and M. Pyshkin, “Towards better requirement definition for multimedia travel guiding applications,” 2016 IEEE Symposium Series on Computational Intelligence (SSCI), Athens, Greece, 2016, pp. 1-7. DOI: 10.1109/SSCI.2016.7850189. Web: http://ieeexplore.ieee.org/document/7850189/

3. E. Boitsova, E. Pyshkin, Y. Takako, N. Bogach, I. Lezhenin, A. Lamtev, and V. Diachkov, “StudyIntonation Courseware Kit for EFL Prosody Teaching,” Proc. 9th International Conference on Speech Prosody 2018, 413-417, DOI: https://doi.org/10.21437/SpeechProsody.2018-84.

4. E. Pyshkin, J. Blake, A. Lamtev, I. Lezhenin, A. Zhuikov, and N. Bogach, “Prosody Training Mobile Application: Early Design Assessment and Lessons Learned,” IDAACS-2019, September 18–21, 2019, Metz, France. In press.

5. I. Lezhenin, A. Zhuikov, N. Bogach, E. Boitsova, and E. Pyshkin, “PitchKeywordExtractor: Prosody-based Automatic Keyword Extraction for Speech Content,” In Proc. of the 2017 Federated Conference on Computer Science and Information Systems, M. Ganzha, L. Maciaszek, M. Paprzycki (eds). ACSIS, Vol. 11, pages 265–269. DOI: https://doi.org/10.15439/2017F326.

6. I. Lezhenin, E. Boitsova, V. Diachkov, A. Zhuikov, A. Lamtev, N. Bogach, E. Pyshkin, and I. Krylov, “Automatic Intonation-based Keyword Extraction From Academic Discourse,” Proc. of the 2018 Federated Conference on Computer Science and Information Systems, M. Ganzha, L. Maciaszek, M. Paprzycki (eds). ACSIS, Vol. 15, pages 165–168 (2018). DOI: http://dx.doi.org/10.15439/2018F42.

7. A. Kiselev and A. Kuznetsov, “Developing a Mobile Application for Wine Amateurs,” In Proc. of 2015 International Workshop on Applications in Information Technology, Oct 8–10, Aizu-Wakamatsu, Japan, 2015, The University of Aizu Press, E. Pyshkin and V. Klyuev (eds), pp. 65–67. Web: http://web-ext.u-aizu.ac.jp/labs/is-se/conference_proceedings/iwait-15/iwait-2015-e-proceedings-release.pdf

8. M. Mozgovoy and E. Pyshkin, “Using Image Recognition for Testing Hand-drawn Graphic User Interfaces,” 11th International Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies (UBICOMM 2017), November 12-16, 2017, Barcelona, Spain, IARIA, pp. 25-28. ISBN: 978-1-61208-598-2. Web: https://www.thinkmind.org/index.php?view=article&articleid=ubicomm_2017_2_20_10083.

9. M. Mozgovoy and E. Pyshkin, “Mobile Farm for Software Testing,” In Proc. of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct (MobileHCI '18), ACM, New York, NY, USA, pp. 31-38. DOI: https://doi.org/10.1145/3236112.3236117.

10. A. Kuznetsov and E. Pyshkin, “Approach to Develop an Assistant Application for Controlling Trace Accuracy in Travel Timelines,” Proc. of the 8th International Conference on Advances in Human-oriented and Personalized Mechanisms, Technologies, and Services (CENTRIC 2018), Oct 14-18, Nice, France, IARIA, 2018.

11. P. Théberge, “Digitalization,” In The Routledge Reader on the Sociology of Music, Edited by J. Shepherd and K. Devine, Taylor & Francis, 2015. Web: https://www.taylorfrancis.com/books/e/9781135007911.

12. J. Bresson, C. Agon, and G. Assayag, “OpenMusic: visual programming environment for music composition, analysis and research,” In Proc. of the 19th ACM international conference on Multimedia (MM '11), 2011. ACM, New York, NY, USA, 743-746. DOI: http://dx.doi.org/10.1145/2072298.2072434.