NLP and Public Engagement: the Case of the Italian School Reform

Tommaso Caselli∗, Giovanni Moretti3, Rachele Sprugnoli3\, Sara Tonelli3,Damien Lanfrey♣, Donatella Solda Kutzmann♣

Vrije Universiteit Amsterdam∗, FBK3, University of Trento\, MIUR♣

De Boelelaan 1105 1081 HV Amsterdam NL, Via Sommarive, 18 38122 Povo, Viale Trastevere, 76/a 00153 Romat.caselli@vu.nl, {moretti;sprugnoli;satonelli}@fbk.eu, {damien.lanfrey,donatella.soldakutzmann}@istruzione.it

AbstractIn this paper we present PIERINO (PIattaforma per l’Estrazione e il Recupero di INformazione Online), a system that was imple-mented in collaboration with the Italian Ministry of Education, University and Research to analyse the citizens’ comments givenin #labuonascuola survey. The platform includes various levels of automatic analysis such as key-concept extraction and wordco-occurrences. Each analysis is displayed through an intuitive view using different types of visualizations, for example radar charts andsunburst. PIERINO was effectively used to support shaping the last Italian school reform, proving the potential of NLP in the context ofpolicy making.

Keywords: Text Mining, Information visualization, E-government

1. IntroductionThe engagement of citizens in policy making has becomean important challenge for public administrations in the lastyears since current digital technologies could be adapted topromote forms of direct democracy by means of online sur-veys, asking for the citizens’ opinion on key issues relatedto public life. However, the full potential of state-of-the-art technologies in collecting and analyzing such data hasnot been exploited yet. Online surveys limited to yes/noquestions are somehow easily processed but more elabo-rated questionnaires, involving idea gathering and propos-als collection from citizens, even if easily implementedthrough online forms, raise costs for an effective analysis.Indeed, successful public consultations - especially on top-ics high on the agenda, can end up with thousands of free-text answers, which cannot be analyzed manually. To makethe engagement process effective, reward citizens’ efforts,and produce useful outcomes for the administration, suchamount of information needs to be automatically processedto extract content that can support civil servants and politi-cians in taking decisions. Besides, the extracted contentmust be displayed in an intuitive way, since it should bealso accessed by non-technical users.Research in the automatic analysis of open answers hasmainly focused on data from the marketing and advertis-ing fields (Yamanishi and Li, 2002) with few notable stud-ies in the area of e-government (Ku and Leroy, 2014). Theliterature reports the use of different techniques. For ex-ample, supervised approaches (Giorgetti et al., 2003) andactive learning (Patil and Ravindran, 2015) are employedto perform survey coding, that is matching open-ended an-swers with a short description associated to a code, thusconverting qualitative information (text) into a quantitativeformat (code). In addition, text clustering and topic model-ing methods are applied to explore information contained inopen answers, e.g. to summarize and classify them (Wangand Mulrow, 2014) and to identify the intentions of surveyparticipants (Rosell and Velupillai, 2008).In this work, we present the first attempt to apply state-

of-the-art Italian Natural Language Processing (NLP) tech-niques to the outcome of an online public consultation forpolicy making, with the goal to integrate citizens’ contribu-tions in the Italian school reform. The work is a joint ini-tiative among the Digital Humanites Group at FondazioneBruno Kessler (FBK), the Vrije Universiteit Amsterdam(VUA) and the Italian Ministry of Education, Universi-ties and Research (MIUR). The goal of the work is theautomatic analysis of linguistic data contained in the an-swers given by the participants to the public consultationLa Buona Scuola1 (commonly referred to in social mediaas #labuonascuola).#labuonascuola has been among the biggest on-line publicconsultations ever realized in Europe. Intended to gatherideas, opinions and evidence on issues related to the ongo-ing consisted in an on-line platform with a dedicated sur-vey and several discussion areas, a section for position pa-pers and a calendar of more than 2000 local live debates,whose results have been uploaded to the consultation plat-form. The survey included a set of multiple-choice andsome open questions, to which people could freely answerusing natural language. Open questions were for exampleWhat features should teachers have? or How would youchange the school cycles?. The questionnaire was onlinefor 2 months, from 15th of September to 15th of November2014, and could be accessed by the general public, with re-spondents providing some anagraphical data while keepingtheir replies anonymous.The school reform has been highly debated by Italian pub-lic opinion and citizens, as also shown by the high num-ber of contributes, ideas, replies and local live debates. In-dividual participants in the online questionnaire amountedto 130,000 people. Overall, more than 775,000 free textcomments were provided by the users. While answers toclosed questions were quite easy to process and summa-rize, the large amount of free texts represented a big chal-lenge for MIUR personnel. Indeed, the amount of data pre-vented them from a manual analysis due to time, money,

1https://labuonascuola.gov.it

401

and errors, which can impact on the final recommendationsto policy makers. To address this problem, we developeda system called PIERINO (PIattaforma per l’Estrazione eil Recupero di INformazione Online), which analyzed thefree text answers and made them available in a user-friendlyformat as described in Section 2. The information collectedwas used to assist policy makers in the development of thenew bill for the Italian school system: details on this pointare given in Section 3. In addition, a version of PIERINOhave been developed to deal with a specific topic related toschool, i.e. learning disabilities: this version is describedin Section 4. Finally, Section 5. presents conclusions andfuture work

2. System DescriptionIn this Section we describe data pre-processing togetherwith PIERINO structure and functionalities. Examplestaken from the #buonascuola use case and screenshots areprovided.

2.1. Data Selection and PreparationThe platform takes in input a list of natural language com-ments/answers, grouped by question. In our use case, theanalysis focused on a set of 10 questions, which were con-sidered as highly relevant by MIUR collaborators. Overall,we processed more than 270,000 free text comments, con-taining almost 5,000,000 tokens.Each group of comments is first processed with TextPro2

(Pianta et al., 2008), an NLP suite performing several lin-guistic analyses, and with Mallet3 (McCallum, 2002), asystem for the extraction of topic models. Then, we im-plemented a Web interface to easily navigate the data andquery them. The analyses and the visualization strate-gies were discussed with MIUR and improved iterativelyto meet their requirements, which were basically:

• for each question, provide a concise representation ofthe main topics mentioned in the replies;

• provide an interface which can be intuitively usedwithout the need of technical support;

• allow users to easily make queries to the database;

• give the possibility to see the source comments and notjust the summarized information.

These requirements were addressed in different views pro-vided by PIERINO, as detailed in the following subsection.We rely on Highcharts4 to present the most common charttypes (i.e. bar and line charts), while the most interac-tive and custom data-driven visualizations (i.e. network ofco-occurrences) are displayed with d3.js5 (Bostock et al.,2011).

2http://textpro.fbk.eu/3http://mallet.cs.umass.edu/4http://www.highcharts.com/5https://d3js.org/

2.2. PIERINO functionalitiesPIERINO includes five types of analysis and views. Theplatform functionalities are generic and can be applied toany public consultation dataset with textual data. A limitedinteraction with experts and promoters of the consultationis first necessary in order to set some parameters and for re-fining the results. It is important to stress out that PIERINOdoes not provide “the answers” but facilitates the analysisby experts of large amounts of textual data to support deci-sions. Below we report each functionality of PIERINO byusing #labuonascuola data as an example.The first is the key-concept based view, that displays foreach question the 20 top-relevant concepts mentioned in thecitizens’ answers. These were extracted with the first ver-sion of KD (Moretti et al., 2015), a highly parametrizablekey-concept recognizer6. We assigned a higher weight tolonger concepts (between 2 and 4 tokens), in order to givea higher relevance to specific concepts and discard genericones (usually unigrams). The ranked list of key-concepts isdisplayed through a sunburst, as shown in Figure 1. In thisexample, related to the question “What is missing in whatyou learn at school?”, we observe a clear prevalence of spe-cific subjects such as civics (first ranked), foreign languagesand Italian.The second section is devoted to the analysis of verbs,aimed to establish what actions citizens associate with thevarious issues presented in the consultation. PIERINO al-lows users to select a question of interest and see what arethe most common verbs displayed in a bar chart. In all theanswers, the high frequency of the verbs “dovere” (must)and “fare” (do) stands out as an index of the decisivenessand commitment with which users have expressed theiropinions and the urgency they feel. Figure 2 shows that inthe answers to the question ”Which features of teachers areimportant to you?”, beyond obvious verbs such as “inseg-nare” (teach) and “sapere” (know), there are many verbsencoding participation and relationship such as “coinvol-gere” (involve), “trasmettere” (instill), “relazionarsi” (re-late), “interagire” (interact).The third section analyzes the degree of overlap betweenthe content of the responses to individual questions and alist of keywords provided by MIUR, in order to verify thecorrespondence between the issues on the political agendaand the outcome of the consultation. We report an examplein Figure 3, related to the question about how to train theteachers of the future. The analysis visualized as a radarchart shows a strong overlap between the outcome of thesurvey and the keywords “public competition”, “evalua-tion” and “certification”.The fourth section of PIERINO includes a targeted searchengine, which allows users to look for a term or expres-sion of interest in the participants’ responses, and see themost common words co-occurring with it. An exampleis displayed in Figure 4. Co-occurrence selection maybe refined by specifying various parameters, including thecontext in which to search (right context only, left con-text only or both) and its width (from from 1 to 20 to-kens). Besides, the user can search only for co-occurring

6http://dh.fbk.eu/technologies/kd

402

Figure 1: Visualization of key-concepts for the question: “What is missing in what you learn at school?”).

Figure 2: Most used verbs in the answers to “Which teach-ers’ characteristics are important to you?”.

verbs, nouns and adjectives, or for different part-of-speechcombinations. Examples of part-of-speech patterns arenoun+adjective, e.g. “alunni stranieri” (foreign students),and noun+preposition+noun, e.g. “corso di recupero” (re-medial class). Once the search is completed, users can clickon the co-occurrence nodes to display the citizens’ com-ments in which the terms of interest appear. Additionally,the nodes can be colored by polarity (positive or negative),to show whether citizens associated a positive or negativeopinion to the given term. The polarity scores associatedwith a word are extracted from an Italian prior polarity lex-icon (Sprugnoli et al., 2015).Finally, topics extracted with Mallet are used to show themost discussed issues as expressed by single words (e.g.“merit”) and groups of words (e.g. “work-market”). Top-ics are displayed as a network (Fig. 5). This highlighted,

Figure 3: Overlap between the content of the question“How do we train the teachers of the future?” and key-words provided by MIUR

for instance, that respondents stressed the need to put moreweight on the training and continuing education of teachersbut also on the involvement of families.

3. Impact on Prospect UsersPIERINO can substantially advance the practice of opengovernment. The mere availability of tools like this allowsGovernments and public administration to organize publicconsultations that are more stratified and that are able tocollect finer data for policy design and implementation.

403

(a) (b)

Figure 4: Networks of Co-occurrences: (a) Co-occurrences of the word “lingua” (language) in the answers to question“Whats missing in what you learn at school?”; (b) Co-occurrences of the expression “scuola media” (middle school) forquestion “How would you reshape school cycles?”

(a) General View (b) Detail

Figure 5: Topic Network

One of the major impact of PIERINO for prospect usersis the enhancement of the quality of decision-making col-lecting an entire new class of information, referred to nat-ural language expressions, particularly crucial for complexand non-linear policy problems. Those problems tend tobe characterized by a wide continuum of options, that can-not be collected by traditional survey methods. For exam-ple, a question regarding the possibility of reforming theItalian education cycle (primary, lower secondary and up-per secondary education) allowed the MIUR to understandthat possible answers with apparent widespread agreement(the possibility to reform and possibly eliminate lower sec-ondary education, merging primary and lower secondaryyears) proved to be less agreed than other options, moretechnical and less evident in public debate (such as, for ex-

ample, the possibility of reforming the first two years of up-per secondary education, as in the German education sys-tem). The combined use of key-concepts (Figure 1) andco-occurrence networks (Figure 4) allowed the Ministry tounderstand this and other positions on the subject matterand avoid to subscribe to the easier option expressed bypublic debate.An additional example of impactful use of PIERINO isprovided by the analysis of the answers to the questionconcerning the permanent hiring of 100,000 new teachersand, particularly, to 50,000 new functional teachers. Over130,000 textual answers to the question provided a veryclear address to policy-makers: at primary school level,additional teachers should be used to promote the coexis-tence of two teachers in classrooms with over 20 pupils; at

404

secondary school level, instead, additional teachers shouldbe used to support students with poor performances in theafternoon, with particular respect to schools in areas withlower socio-economic indicators. These and other distinctaddresses by the public have been directly embedded in thelaw finally approved in July 2015.In many cases, it was the combination between key-concepts and co-occurrence data to provide the clearestreading: while key-concepts analysis provides the generalframing to citizens answers, co-occurrence networks al-lowed to investigate the granularity and nuances of answers,a priceless combination of information for policy-makers.

4. Use case: PIERINO-DyslexiaA platform version called PIERINO-Dyslexia has beendeveloped to explore participants’ comments related tolearning disorders on the occasion of the 2015 EuropeanDyslexia Awareness Week.Although the questionnaire did not include specific ques-tions on learning disability (in Italian DSA, Disturbi Speci-fici di Apprendimento), this theme has been mentioned byseveral participants to the survey in relation to differentsubjects such as the training of teachers or the allocationof additional resources. Therefore, we exploited the func-tionalities of PIERINO to analyze the contexts in whichDSA were mentioned and their relevance with respect tothe other topics of the survey.Starting with all the answers given by the citizens, we au-tomatically extract a list of 62,200 key-concepts orderedby relevance. Among these, we checked the ranking ofthe concepts related to DSA. The term DSA is at position397, demonstrating the attention to this issue paid by par-ents and teachers. Concepts with a similar weight are moregeneral expressions such as “difficolta” (difficulty), “soldi”(money) and “scuola superiore” (high school).Citizens have discussed learning disabilities in differentcontexts, which can be analyzed by using the targetedsearch engine based on co-occurrences. For example,by looking at expressions co-occurring with the acronym“dsa”, it is possible to see that participants suggested meth-ods to help the inclusion of dyslexic students in every-daylearning activities such as the introduction of occupationaltherapy in schools and the use of concept maps.The analyses carried out show that the survey participantsare aware of the impact of learning disabilities on the schoolorganization, and propose effective strategies to addressthem. As an example, answering the question ”In whichactivities should non-tenured teachers be employed apartfrom teaching?”, several participants suggest to employthem in supporting classes with children with learning diffi-culties after an adequate training. Another example is givenby many answers to the question ”What features shouldteacher have?” that highlight the importance of empathy,of knowing how to relate with children with learning diffi-culties and their families, and how to manage classes withstudents having different problematic issues. In general,respondents demonstrate a thorough knowledge of learningdisabilities, and in particular of dyslexia, as shown also bythe use of technical terminology. Indeed, participants coulddifferentiate between various types of disorders and, de-

spite knowing the difficulties of schools and teachers, theywant to propose concrete strategies for managing these phe-nomena.

5. Conclusion and Future WorkWe presented PIERINO, the first attempt to integrate NLPand data visualization in the Italian policy making process.The NLP modules in PIERINO have all been benchmarkedagainst standard datasets and successfully applied to othertext types such as the citizens answers. Its analyses con-tributed to the decisions which informed the recently passedbill on the Italian school reform.On the basis of the results obtained through the analysisof public participation results, the Ministry has decided touse an adapted version of PIERINO to a new stream oftexts. In this case, texts are of administrative nature, andrelate to school ex-post impact assessment on products andservices offered by commercial parties through their CSRcampaigns.We are planning to extend the functionalities of the plat-form so as to have a finer-grained understanding of citizens’suggestions and concerns. For instance, useful featuresmay include the possibility to cluster answers per contentand to perform comparisons based on supporting and con-trasting ideas or on the sources. Modality (how the messageis conveyed), negation scope and sentiment analysis shouldbe investigated and improved. Another important aspect forpolicy makers is the possibility to perform real-time analy-sis of the aggregated data by region, as some specific issuesmay vary based on the geographical area. Although thispiece of information is currently not present in the platform,it can be easily retrieved from the respondents metadata andintegrated in the system.

6. AcknowledgementsOne of the authors wants to thanks the NWO Spinoza Prizeproject Understanding Language by Machines (sub-track 3)for partially supporting this work.

7. Bibliographical ReferencesBostock, M., Ogievetsky, V., and Heer, J. (2011). D3 data-

driven documents. Visualization and Computer Graph-ics, IEEE Transactions on, 17(12):2301–2309.

Giorgetti, D., Prodanof, I., and Sebastiani, F. (2003). Au-tomatic coding of open-ended surveys using text catego-rization techniques. In Proceedings of Fourth Interna-tional Conference of the Association for Survey Comput-ing, pages 173–184.

Ku, C.-H. and Leroy, G. (2014). A decision support sys-tem: Automated crime report analysis and classificationfor e-government. Government Information Quarterly,31(4):534–544.

McCallum, A. K. (2002). Mallet: A machine learning forlanguage toolkit.

Moretti, G., Sprugnoli, R., and Tonelli, S. (2015). Dig-ging in the Dirt: Extracting Keyphrases from Texts withKD. In Proceedings of the Second Italian Conference onComputational Linguistics CLiC-it 2015, page 198. Ac-cademia University Press.

405

Patil, S. and Ravindran, B. (2015). Active Learning BasedWeak Supervision for Textual Survey Response Classifi-cation. In Computational Linguistics and Intelligent TextProcessing, pages 309–320. Springer.

Pianta, E., Girardi, C., and Zanoli, R. (2008). The TextProTool Suite. In Proceedings of the sixth international con-ference on Language Resources and Evaluation (LREC2008). Citeseer.

Rosell, M. and Velupillai, S. (2008). Revealing Relationsbetween Open and Closed Answers in Questionnairesthrough Text Clustering Evaluation. In LREC.

Sprugnoli, R., Tonelli, S., Marchetti, A., and Moretti,G. (2015). Towards Sentiment Analysis for HistoricalTexts. Digital Scholarship in the Humanities.

Wang, F. and Mulrow, E. (2014). Analyzing Open-EndedSurvey Questions Using Unsupervised Learning Meth-ods. In Proceedings of the Survey Research MethodsSection of the American Statistical Association (JSM2014).

Yamanishi, K. and Li, H. (2002). Mining open answers inquestionnaire data. Intelligent Systems, IEEE, 17(5):58–63.

406