A MACHINE LEARNING APPROACH TO

SENTIMENT ANALYSIS AND STANCE

DETECTION FOR POLITICAL TWEETS EXPLORING THE INFLUENCE OF IRONY ON THE PREDICTABILITY

OF SENTIMENT AND STANCE

Aantal woorden: 14658

Lot De Kimpe Studentennummer: 01404202

Promotor: Prof. Dr. Els Lefever

Masterproef voorgelegd voor het behalen van de graad Master in de Meertalige Communicatie

Academiejaar: 2017 - 2018

Page 3 of 76

Verklaring i.v.m. auteursrecht

De auteur en de promotor(en) geven de toelating deze studie als geheel voor consultatie

beschikbaar te stellen voor persoonlijk gebruik. Elk ander gebruik valt onder de beperkingen

van het auteursrecht, in het bijzonder met betrekking tot de verplichting de bron uitdrukkelijk

te vermelden bij het aanhalen van gegevens uit deze studie.

Het auteursrecht betreffende de gegevens vermeld in deze studie berust bij de promotor(en).

Het auteursrecht beperkt zich tot de wijze waarop de auteur de problematiek van het onderwerp

heeft benaderd en neergeschreven. De auteur respecteert daarbij het oorspronkelijke

auteursrecht van de individueel geciteerde studies en eventueel bijhorende documentatie, zoals

tabellen en figuren.

Page 4 of 76

Abstract

With the emergence of Web 2.0, easy-accessible microblogging platforms such as Facebook

and Twitter have allowed users to easily share their opinions online. Sentiment analysis and

stance detection, allow a business, organization or political party to gather all these viewpoints

and to find out which sentiment (positive, negative or neutral) a piece of text contains to

optimize their products and services. Despite the fast developments in this field of study,

challenges for the automatic prediction of sentiment and stance labels are still present (Pang et

al., 2008; Kumar & Sebastian, 2012; Mandya et al., 2016). In this research, it was explored how

well a machine learning system performs for sentiment analysis and stance detection on an

English Twitter corpus of 482 political tweets with #Brexit. The manually annotated labels were

compared to the predictions of a machine learning system, considering the possible impact of

irony on the performance of our system. The results show that the system performs fairly well

on sentiment analysis (accuracy of 0,55) and stance detection (accuracy of 0,61). It remains,

however, unclear to which extent irony affects the quality of the automatic predictions. Further

research could specifically focus on the comparison between irony detection and sentiment

analysis or stance detection. (204)

Page 5 of 76

Acknowledgements

First of all, I would like to express my gratitude towards my sister, Lies De Kimpe. As from the

very first letter of my bachelor’s paper, she provided me with professional feedback and

encouraging pep talks. And even now, up until the very last letter of my master’s thesis, she has

always been by my side for support. Without her eye for detail and willingness to answer every

little question, this paper would have not reached the quality it has today.

Secondly, many thanks go to my supervisor, Els Lefever, for her help, patience and support

during the last two years. She has always given me useful advice and motivating compliments,

which encouraged me to complete my thesis successfully. Furthermore, I also wish to thank her

for being such an approachable and kind mentor.

Thirdly, I would like to thank my friends Julie Carton, Lien De Wulf, Bo Van Eetvelde and the

entire group of KLJ people, who were my towers of strength in stressful moments. They were

wonderful in offering my daily dose of distraction in solitary times behind my desk. Special

thanks go to Hanne Christiaens, who was willing to share her recognizable experiences as a

master’s student at VTC with me.

And last but definitely not least, I wish to thank my parents from the bottom of my heart for

allowing me to pursue any possible dream and keeping their endless faith in me.

Page 6 of 76

Table of contents

List of tables and figures ......................................................................................................................... 8

1. INTRODUCTION ........................................................................................................................... 9

2. LITERATURE STUDY ................................................................................................................ 11

2.1 Sentiment analysis ................................................................................................................. 11

2.1.1 Terminology .................................................................................................................. 12

2.2 Approaches to SA .................................................................................................................. 13

2.2.1 Lexicon-based approach to SA ...................................................................................... 14

2.2.2 Supervised machine learning approach to SA ............................................................... 15

2.3 ABSA: Aspect Based Sentiment Analysis ............................................................................ 17

2.4 Sentiment analysis for political tweets .................................................................................. 17

2.4.1 Twitter ........................................................................................................................... 18

2.5 Stance detection ..................................................................................................................... 19

2.6 Irony detection ....................................................................................................................... 20

2.6.1 What is irony? ............................................................................................................... 20

2.6.2 Difficulties and challenges ............................................................................................ 21

3. RESEARCH DESIGN .................................................................................................................. 22

3.1 Research questions and hypotheses ....................................................................................... 22

3.2 Methodology ......................................................................................................................... 23

3.2.1 Data collection ............................................................................................................... 23

3.1.1 Annotation ..................................................................................................................... 24

3.1.2 Experimental approach .................................................................................................. 25

4. RESULTS ...................................................................................................................................... 26

4.1 Results manual annotation ..................................................................................................... 26

4.1.1 Sentiment and topics...................................................................................................... 26

4.1.2 Stance and irony ............................................................................................................ 28

4.2 Results machine learning system ........................................................................................... 30

4.2.1 Sentiment and topics...................................................................................................... 30

4.2.2 Stance and irony ............................................................................................................ 31

4.3 Analysis ................................................................................................................................. 33

4.3.1 Sentiment analysis: tenfold cross-validation scheme .................................................... 33

4.3.2 Sentiment analysis: general overview ........................................................................... 34

4.3.2.1 Impact of irony on the prediction of sentiment ....................................................... 34

Page 7 of 76

4.3.2.2 Error Analysis .......................................................................................................... 34

4.3.3 Stance detection: tenfold cross-validation scheme ........................................................ 39

4.3.4 Stance detection: general overview ............................................................................... 40

4.3.2.1 Impact of irony on the prediction of stance ............................................................. 41

4.3.2.2 Error analysis .......................................................................................................... 42

4.3.5 Comparison sentiment analysis and stance detection .................................................... 43

5. CONCLUSION ............................................................................................................................. 46

6. LIMITATIONS AND FURTHER RESEARCH ........................................................................... 49

APPENDIX 1 ........................................................................................................................................ 54

APPENDIX 2 ........................................................................................................................................ 57

Page 8 of 76

List of tables and figures Table 1 - Sentiment per topic (manual annotation) ............................................................................... 28

Table 2 - Manual annotation sentiment, stance and irony ..................................................................... 30

Table 3 - Presence of irony per topic ..................................................................................................... 30

Table 4 - Sentiment per topic (machine learning approach) ................................................................. 31

Table 5 - Machine learning annotation sentiment, stance and irony ..................................................... 33

Table 6 - Tenfold cross-validation scheme for sentiment analysis ....................................................... 34

Table 7 - Comparison manual and automatic sentiment analysis + irony presence .............................. 35

Table 8 - Precision, recall, F1-score and accuracy of sentiment labels ................................................. 35

Table 9 - Tenfold cross-validation scheme for stance detection ........................................................... 40

Table 10 - Comparison manual and automatic stance detection + irony presence ................................ 40

Table 11 - Precision, recall, F1-score and accuracy of stance labels .................................................... 41

Table 12 - Comparison results sentiment analysis and stance detection ............................................... 44

Table 13 - Precision, recall, F1-score and accuracy of sentiment and stance labels ............................. 44

Table 14 - Accordance of manually assigned sentiment labels with their stance labels ....................... 45

Table 15 - Accordance of automatically assigned sentiment labels with their stance labels ................ 45

Figure 1 - Manual annotation sentiment ................................................................................................ 26

Figure 2 - Topics in tweets .................................................................................................................... 27

Figure 3 - Manual annotation stance ..................................................................................................... 29

Figure 4 - Machine learning annotation sentiment ................................................................................ 31

Figure 5 - Machine learning annotation stance ..................................................................................... 32

Page 9 of 76

1. INTRODUCTION

With the rise of Web 2.0, microblogging websites have increasingly become a valuable

platform for people to express their opinion and sentiment on a certain topic. Blogs, forums and

social media platforms allow users to easily add blogposts, reviews, reactions and ratings to

share their point of view on the internet. Since online opinionated texts offer a wide range of

easy accessible data, businesses, services and political parties are not bound to conduct surveys

or carry out opinion polls (Lui, 2012) to gather the public’s sentiment anymore. Nowadays,

organisations have enough feedback at their disposal to examine how people’s views differ on

a certain product, service or policy.

The system used to extract and analyse the public’s opinion is called sentiment analysis (SA).

Opinion mining or sentiment analysis is defined as “the computational study of opinions,

sentiments and emotions expressed in text” (Kumar & Sebastian, 2012, p.2). A product or

service can be adapted based on the outcome of the sentiment analysis, regarding whether the

general opinion of a certain aspect is neutral, positive or negative. This is beneficial for the

quality of the product or service and therefore the consumer’s satisfaction. Sentiment analysis

can be applied for various purposes: to detect trends on the social media of a business, to

discover what the underlying reason for the success or failure of a certain product is, or to

predict the outcome of a referendum or elections. Whereas SA helps to determine the speaker’s

sentiment in a piece of text, stance detection (SD) aims at extracting the author’s opinion

towards a certain target or entity. SD thus encloses “the task of automatically determining from

text whether the author of the text is in favour or, against, or neutral towards a proposition or

target” (Mohammad & Kiritchenko et al., 2016, p.31). It can be used to reveal weak spots or

positive aspects of a target. A target may be a product, an aspect of a certain service, a person,

a political point of view, an organisation, a policy, et cetera.

Of all social platforms that offer a wide range of accessible opinions and sentiment, Twitter is

notably interesting for both researchers and marketers who can use tweets to easily collect the

opinion of a large audience. This social network is particularly instrumental for politicians and

political parties to evaluate their position within the political landscape. Tumasjan et al. (2010)

indicate that Twitter is used frequently to speculate about politics, since messages mentioning

a party tend to reflect the outcome of, for instance, an election. Consequently, they conclude

that political tweets plausibly give an indication of how the current offline political landscape

is divided.

Page 10 of 76

Besides the advantages of Twitter as an easy accessible source of user-generated data, the

platform has its own specific characteristics. For the very reason that the data is user-generated,

Twitter messages or tweets will often contain language phenomena that are typical for online

messaging (e.g. flooding, abbreviations, emoji’s). Furthermore, as users are limited to write a

message in under 280 characters they are obliged to phrase their messages creatively. As a

result, irony, sarcasm, metaphors or other figurative use of language frequently appear in

tweets, which is challenging to detect due to its creative character. Regardless of the challenging

nature of figurative language such as irony, Reyes et al. (2012) constructed an irony detection

model specifically for short online texts. This model provided valuable insights into figurative

language use on Twitter and tasks such as sentiment analysis. Moreover, Van Hee (2017)

explored the automatic detection of irony on social media and found that a machine learning

approach can lead to good performance in combination with a varied set of information sources.

In the future, this irony detection system could, however, be further optimised. In the framework

of SemEval-2015 Ghosh et al. (2015) explored the determination of sentiment in tweets

containing irony, sarcasm or metaphors. For this purpose, they measured the polarity of tweets

that use creative and figurative language. Their system appeared to be useful for further

research.

In this study, we want to make a contribution to the existing findings on automatic sentiment

analysis on Twitter. It will be further explored how well a machine learning system performs

for sentiment analysis and stance detection on a Twitter corpus of political tweets. Additionally,

we will attempt to provide more insight into the impact of ironic language in tweets on the

predictability of sentiment and stance labels. The manual annotation of sentiment in a Twitter

corpus containing 482 tweets with the hashtag Brexit (#Brexit) will be compared with the

predicted sentiment labels of a supervised machine learning system for sentiment analysis.

Apart from sentiment labels, the tweets will also be provided with a stance label to explore

whether a machine learning system is able to detect (implicit) stance. On the basis of the

collected and analysed data and experimental results, we will try to provide an answer to

following questions: Can we automatically predict sentiment with the help of sentiment

analysis? Which impact does irony have on the predictability of sentiment? Can a machine

learning system detect implicitly expressed stance? In further passages of this study, we will

give an in-depth overview of the current state-of-the-art of sentiment analysis, including ABSA,

stance detection and irony detection.

Page 11 of 76

2. LITERATURE STUDY

2.1 Sentiment analysis

People’s opinions on and reviews about a certain event, product or service can have a big

influence on future customers. The latter often turn to blogs, review platforms and social media

to gather information and advice to form their own opinions. Therefore, sentiment analysis (SA)

is an important field of study. By performing SA, it can be determined whether a piece of text

is neutral, positive or negative. To gain insight into for instance sales figures or election results

it is beneficial for companies and organisations to keep track of their online reputation.

Preferably, SA is performed by an automatic system, since manual annotation would appear to

be a time-consuming, intensive task.

The term sentiment analysis appeared for the first time in 2003 in a study by Nasukawa and Yi

(2003). In the same year, Dave et al. (2003) mentioned opinion mining in their work. Nowadays,

both terms are used interchangeably to denote the same field of study. More specifically, SA is

considered to be “the field of study that analyses people’s opinions, sentiments, evaluations,

appraisals, attitudes and emoticons towards entities such as products, services, organizations,

individuals, issues, events, topics, and their attributes” (Liu, 2012, p.7). As from the year 2003,

this field of study has been well-studied and is still constantly developing.

Nowadays, a range of applications are available to systematically organize opinions, reviews,

ideas on entities and events as well as products. Some applications, for example, arrange

reviews and ratings according to sentiment, and other recommendation systems are able to only

recommend products or advertisements with positive sentiment (Pang et al, 2008, p.8).

Additionally, SA may also be valuable for governments to detect possible opposed or negative

voices. Especially during times of elections, the latest technologies can assist in keeping an

overview of the various points of view on politicians, parties, bills, et cetera. Besides numerous

possibilities for commercial applications, several problems still rise. These need solutions and

therefore make SA a relevant field of study to explore thoroughly and in further detail.

In this study, an in-depth overview will be given of the SA-related terminology in section 2.1.1

and the two approaches to SA in chapter 2.2. In chapter 2.3, Aspect Based Sentiment Analysis

(ABSA) will be explained, whereas in chapter 2.4, the particular nature of SA for political

tweets will be discussed. Lastly, the aspects, difficulties and challenges of irony detection will

be discovered in chapter 2.5.

Page 12 of 76

2.1.1 Terminology

As SA has increasingly been discussed over the last decennia, it is important to keep a clear

overview of which terms are being used in the field of study of Natural Language Processing

(NLP). In literature the terms “opinion mining, review mining and appraisal attraction” (Kumar

& Sebastian, 2012, p.2) are used interchangeably with the concept of SA. According to Liu

(2012), the terms mentioned all differ from each other on a certain level. They can, however,

all be classified under the umbrella term of SA.

In the field of SA, it is essential to make a distinction between subjectivity and subjectivity

analysis on the one hand, and sentiment and sentiment analysis on the other hand. Pang et al.

(2008) believe subjectivity comprises everything that contains a personal opinion, such as

evaluation, emotions and speculations. Subjectivity analysis is used to distinguish facts

(objective sentences) from opinions and sentiment (subjective sentences). Furthermore,

subjectivity and sentiment do not have the exact same meaning. Sentiment implies the

expression of an opinion, which often reveals the speaker’s attitude towards a certain topic. Not

only subjective sentences contain sentiment. The following sentence “The battery of my

Bluetooth speaker only lasts for 8 hours”, is for example an objective sentence with negative

sentiment. The usage only reveals a sense of disappointment by the speaker. By means of

automatic SA, polarity, opinions, emotions and other subjective information can be

automatically detected and analysed (Desmet et al., 2014). It is often used to determine which

sentiment (positive, negative, neutral) belongs to a piece of text.

Moreover, two kinds of opinions can be distinguished, namely regular opinions and

comparative opinions (Liu, p.12). A regular opinion expresses sentiment on a specific feature

or aspect of a product, for example “The battery of my Bluetooth speaker lasts for a very long

time.”. A comparative opinion, however, compares different products or services based on

certain aspects they (don’t) have in common, for instance “My JBL speaker has a further

Bluetooth ranger than my old Philips speaker.”. Specific words that regularly express the same

kind of sentiment can be used to easily recognize opinions. These words are defined as

sentiment words or opinion words (Liu, 2012, p.12). Some opinion words are generally

positively used such as ‘beautiful, nice, lovely’ and others usually have a negative connotation,

such as ‘ugly, stupid, boring’. In addition, there are also sayings or proverbs who are

systematically classified under a certain polarity label. ‘To have a finger in every pie’ for

example, is generally used negatively.

Page 13 of 76

There are different ways to categorize sentiment. For one, SA often makes use of a binary

classification and only assigns the labels ‘positive’ and ‘negative’. This is also defined as

sentiment polarity classification (Pang et al, 2008, p17). In some cases, the degree of positivity

and negativity can be more thorough and complex. Therefore, Boertjes (2011) also defines a

non-binary classification system. When following this system, sentiment can then for example

vary on a scale going from ‘extremely dissatisfied’ to ‘extremely satisfied’. This is also called

multi-class classification (Kumar & Sebastian, 2012, p.4), which implies SA with more than

two sentiment categories. In the latter, the star system is also often applied, in which one star

expresses dissatisfaction and five stars satisfaction. Cambria et al. (2013, p.16) also distinguish

two common sentiment analysis tasks: polarity classification (supra) and agreement detection.

On the one hand, they add other examples to the binary polarity classification, such as ‘thumbs

up’ and ‘thumbs down’ or ‘like’ and ‘dislike’. On the other hand, they introduce agreement

detection as another example of a binary classification system. This task helps to determine

whether two texts hold the same opinion and should receive the same or opposite sentiment

labels.

2.2 Approaches to SA

Sentiment analysis is an interdisciplinary field of study which focusses on web mining

(extracting and analysing information on the web) as well as natural language processing (NLP

or computational linguistics). Kumar and Sebastian (2012) describe four levels at which

sentiment can be determined: feature level, word level, sentence level and document level.

Feature based SA on entity and aspect level (Liu, 2012, p.11) focusses on the opinion on

different aspects or features instead of the sentiment of entire paragraphs, sentences or phrases.

The sentence “My new smartphone has a great camera, but the sound quality is horrible.”, for

instance, cannot be labelled as entirely positive or negative because it reports on two features

of the same product. The evaluation of the sound quality of this new smartphone is negative,

but the camera is evaluated positively. Feature based SA is often seen as the most challenging

level of SA since two or more aspects can be covered within one sentence. On word level, the

polarity label is predicted for each separate word as each word is then considered as a separate

unit, which can hold different sentiment (Kolkur et al, 2015, p.2). According to Kumar and

Sebastian (2012), adjectives are the parts of speech that contain the most explicit sentiment. On

sentence level, a polarity label is given to each separate sentence. A neutral label usually equals

the fact that the sentence does not hold a personal opinion or that the sentence is both positive

Page 14 of 76

as negative which neutralizes the sentiment. For SA on document level, entire documents (e.g.,

product reviews) are labelled positively or negatively as a whole. This level of SA is not

applicable when the document covers multiple products or aspects.

Systems for automatic SA rely on one of two valid approaches for the prediction of a sentiment

label: either a lexicon-based approach or machine learning approach is applied.

2.2.1 Lexicon-based approach to SA

The lexicon-based approach to SA generally makes use of a “dictionary of opinion words to

identify and determine sentiment orientation (positive, negative or neutral)” (Zhang et al, 2011,

p.2). These opinion words are words that each have their own sentiment label, based on the

positive or negative connotation they commonly have. The dictionary, which is also called a

sentiment lexicon or opinion lexicon (Liu, 2012, p.12), is usually completed with synonyms

and antonyms of the opinion words.

However, Zhang et al. (2011) indicate that the lexicon-based approach would lead to a low

recall problem. Due to the specific nature of online language, it is impossible to add every

existing opinion word to the sentiment lexicon. In other words, some words will explicitly

express sentiment, but will not be picked up using the lexicon-based approach. For example, in

the following sentence ‘I haaaate going the dentist.’, the negative expression ‘haaaate’ will most

likely not be detected. These expressions change continuously and, therefore, adding them to

the opinion lexicon would appear to be an endless and time-consuming task.

Moreover, Kumar et al. (2012) explain that the lexicon-based approach does not work on

domain specific level. More specifically, this means that a certain word can have a positive or

negative label depending on the context or domain. The word ‘unpredictable’, for instance, can

be positive when it is used in a film review, but negative when it is used to evaluate the steering

behaviour of a brand new car. So far, lexicon-based methods are not yet able to interpret the

sentiment or meaning of a word depending on the situation. According to Khan et al. (2015), it

would be too labour intensive and time-consuming to manually label each opinion word per

specific context (e.g. film or car review).

Page 15 of 76

2.2.2 Supervised machine learning approach to SA

Machine learning allows computers to automatically carry out analysis tasks by means of self-

learning algorithms (such as Naïve Bayes and Support Vector Machines). As Zhang et al.

(2011) explain, these algorithms or sentiment classifiers are trained using features such as

unigrams or bigrams (sequences of one or two words, respectively). Machine learning systems

can then deduce rules or patterns based on manually annotated data to predict polarity labels of

unseen data. The model is thus built based on labelled pieces of texts (sentences, paragraphs,

documents), called training data.

During this training phase, the data is processed in the form of structured information or

“features” extracted from the text. On the basis of all this information, called “feature vectors”,

self-learning systems can then predict which combination of information results in which

polarity label. This approach is therefore referred to as “supervised learning, because the

classifier is given direction in terms of which are good or bad examples of the class.” (Taboada,

2016, p.6).

To set up a statistic machine learning system, data has to be pre-processed in advance on

different levels. Large pieces of text are split up in separate sentences (sentence splitting) and

these are then again broken down into words or tokens (tokenization). Another pre-processing

step can be Part-of-Speech tagging (PoS-tagging) which attributes the morphosyntactic

category to the corresponding word (adjectives, adverbs, nouns, verbs). To analyse various

word forms as a single item, lemmatisation can be used to group together inflected forms of a

word. The infinitives of verbs will then, for example, be recognized in their conjugated forms.

After the pre-processing phase, features (namely lexical features and syntactic features) can be

extracted from a certain text. This process is called feature extraction. Lexical features, on the

one hand, such as tokens or lemmas (E.g., the word ‘worse’ is derived from ‘bad’), can offer

insight in which words occur in a text. Syntactic features, on the other hand, give grammatical

information about the text, such as the syntactic categories of words (e.g. bad is an adjective).

2.2.3 Classification: Support Vector Machines

Shoeb and Ahmed (2017) state that data classification aims to classify data into categories in

the most efficient and productive way. The goal is then to predict the correct category for unseen

data. One regularly used supervised machine learning algorithm is a Support Vector Machine

(SVM) which can be applied for both classification and regression. According to Pang and Lee

Page 16 of 76

(2002), SVMs have been rather effective in comparison to other traditional text categorization

algorithms, showing better results than the Naïve Bayes method for example. By means of a

number of features, the classifier categorizes objects into one of two classes, represented by a

vector. In the context of Twitter, each feature could represent a single word found in a tweet.

Another application of SVMs could be to classify a set of documents into two sentiment groups:

positive or negative documents. The classification process would then be based on other

documents which have already received a positive or negative label. The goals of a SVM is to

train a system that classifies new unseen objects into a certain category. Apart from the

application of SVMs in machine learning approaches to sentiment analysis, an SVM is also

used for text classification tasks such as detecting spam. For this, the classification would rely

on a large corpus of e-mails or other documents which have already manually been marked as

spam or non-spam. Moreover, SVMs are used for the recognition of images, where the

algorithm attempts to recognize aspects or colours of an image.

More formally12, SVMs attempt to find a hyperplane that can divide a dataset into two classes.

A hyperplane is a line that linearly separates and classifies a set of data. The data points (or

feature vectors) that are the nearest to this hyperplane are called support vectors. These points

1 http://blog.aylien.com/support-vector-machines-for-dummies-a-simple 2 https://www.quantstart.com/articles/Support-Vector-Machines-A-Guide-for-Beginners

Page 17 of 76

are the hardest to classify. However, the further away data points are positioned from the

hyperplane, the more reliable the classification of these points are.

2.3 ABSA: Aspect Based Sentiment Analysis

Whereas regular SA determines whether a piece of text is positive, negative or neutral, ABSA

or Aspect Based Sentiment Analysis tries to trace back the target of an opinion. According to

Declercq et al. (2017), this comes down to a very fine-grained approach to SA. ABSA systems

attempt to detect all expressions of sentiment within a piece of text. On sentence level, this

means certain entities can be identified and then be paired up with the corresponding attribute.

In a review on the newest Iphone, the entity types ‘battery’ or ‘camera’ can for example be

linked to the right attribute label such as ‘price’ or ‘quality’ (Pontiki & Galanis et al., 2016,

p.20). The system then attempts to detect the main attributes (features) of the entity to make an

estimate of the average sentiment of a certain text per aspect. In other words, an overview is

given of the positivity or negativity of opinions for each single aspect mentioned (Pavlopoulos,

2014, p.2).

Pavlopoulus (2014) distinguishes three subtasks of Aspect Based Sentiment Analysis: aspect

term extraction, aspect term aggregation and aspect term polarity estimation. First of all, aspect

term extraction helps to detect words or phrases that indicate a certain aspect of the entity that

is being discussed (e.g. battery, camera). After this first step, the extracted words or phrases are

called aspect terms. Secondly, the next subtask is described as aspect term aggregation, which

means the system clusters aspect terms which are quite similar (such as ‘camera’ and ‘video

camera’). Lastly, during the aspect term polarity estimation, the system evaluates all aspect

terms and estimates the average sentiment of every aspect term or cluster of aspect terms.

2.4 Sentiment analysis for political tweets

With the help of SA, companies and services can gain insight in the perception and reception

of their product or service. It is a source of valuable customer feedback that can help companies

to fine-tune or make adaptions to their product by taking the results of the SA into account.

Moreover, social organizations might be interested to know people’s opinions on current

controversial or social debates. Intuitively, the domain where opinions differ regularly is

politics. As reported by Pak and Paroubek (2010), it may be profitable for political parties to

gain a perception of whether people support their party programme or not. Whether it involves

a new policy or upcoming elections, social media such as Facebook or Twitter are constantly

Page 18 of 76

booming with posts containing political views. Tumasjan et al. (2010) point out that,

specifically in the weeks leading up to a political event (such as elections), political themes are

clearly on many users’ minds. Even politicians themselves attempt to reach the electorate and

mobilize possible supporters by communicating via Twitter. In chapter 2.4.1 we will give an

overview of the particular characteristics of Twitter and why this could be an interesting social

platform to perform SA on.

2.4.1 Twitter

Social networks offer a wide range of accessible opinions and sentiment. Khan et al. (2015)

indicate that Twitter3 in particular has a great amount of data at their disposal. On this social

platform, users can share their opinions with the rest of the tweeting community in the form of

tweets (posts on Twitter) consisting of maximum 280 characters. Each day an estimated 60

million tweets are sent into the world. Twitter is therefore an easy-accessible and informative

platform for both researchers and marketers to collect sentiment of a large audience. According

to Liu (2002), tweets are easier to analyse thanks to their length, when compared to reviews for

example, because tweeters attempt to come to the point in a concise answer. Pak and Paroubek

(2010) also indicate Twitter users come from different social groups with varying interests and

backgrounds. Even though American users are prevailing, Kulshrestha et al. (2012) indicate

that the twitter audience is represented by users from all over the world (231 countries).

Therefore, it is possible to collect data and build a corpus in different languages.

Besides all the advantages of using Twitter data, the special characteristics of Twitter can cause

some specific problems. The language use is often informal and can contain typical

abbreviations or words which are only used in online messaging (eg., ‘lmao’, ‘lovvve’). In that

regard, an automatic SA system probably will not recognize any opinion words or positive

sentiment in a sentence such as “I loooove McDonald’s new hamburger!”, because ‘loooove’

is not picked up as the word ‘love’. Furthermore, tweets with the hashtag ‘not’ (#not) usually

hold an ironic or sarcastic message, as can be seen in the following sentence “I love it when my

train is delayed #not”. The hashtag makes the previous statement invalid, but when it is not

picked up by the SA system, the tweet will receive a positive label instead of the correct

negative label. From time to time, Twitter users tend to self-annotate their own usage of irony,

3 www.Twitter.com

Page 19 of 76

as is confirmed by Reyes et al. (2012). They then add the hashtag ‘irony’ (#irony) to point out

their ironic use of language.

In addition, emoticons or emoji’s that express a certain emotion can also give an entirely

different meaning to a tweet, such as in the sentence:

“Luckily, my train has arrived right on time as usual ☹”.

On first glance, the tweet seems to contain positive sentiment, however, the sad and unsatisfied

smiley indicates the statement is written in a negative tone. Moreover, retweets (the sharing of

someone’s tweet on one’s own profile), replies to other tweets (marked with an ‘@’-sign

followed by a username) and adding pictures or links can pose a challenge. Retweets and replies

can be misinterpreted when the tweets are read separately, because then there is a risk that

valuable context is lost.

Even though the phenomena mentioned above are challenging, there have already been several

attempts to tackle such problems. A study by Van Hee et al. (2014), for example, has shown

that feature extraction (as discussed in chapter 2.2.2) using a machine learning approach

performs better than SA systems using lexicon-based approaches. The results reveal that after

the extraction of specific Twitter features, the SA system performed very well (with an F1-score

of 86,28) on a Twitter corpus. After programming rules for flooding, for example, the word

‘loooove’ could be picked up in the previously mentioned sentence “I loooove McDonald’s

new hamburger!”.

2.5 Stance detection

Whereas SA aims at detecting the sentiment of an opinion in a piece of text, stance detection

(SD) is used to pick up whether someone is for or against the subject (target) being debated.

This target may be an organisation, product, service, person or policy. To decide whether an

author is for or against an issue, it is important to follow the reasoning rather closely. Mandya

et al. (2016) indicate the problem that posts containing rebuttal arguments are not clear enough

to be classified as ‘for’ or ‘against’ the main issue being debated. Posts are most often

independent or non-dialogic and thus all features for classification have to be derived from the

post itself. To facilitate stance classification, Mandya et al. (2016) state that topic-stance

features or topic terms can be automatically extracted. For the topic ‘gun control’ the terms

would include for instance ‘firearm’, ‘rifle’ or ‘license’. Each topic term is then associated with

the author’s stance towards that topic. In the sentence “Firearms are nothing but trouble.”, for

Page 20 of 76

example, ‘firearms’ would be associated with the topic ‘gun control’. Then, the stance towards

‘gun control’ would be negative.

The author’s stance or outlook towards the target is in favour if we can deduce from a piece of

text that he or she supports the target. Mohammad et al. (2016) observe different expressions

of favourability, such as simply supporting the target, repeating the positive stance of someone

else or opposing someone who is opposed to the target. Opposite expressions generally result

in negative stance and disapproval of the target. If there is no evidence found whether someone

is for or against a certain target, this does not necessarily mean the author is neutral. It can also

be the case that the stance from this piece of text simply could not be detected.

2.6 Irony detection

Barbieri and Saggion (2014) argue that computational creativity or the creative use of language

has been one of the most challenging topics of Artificial Intelligence (AI) and NLP nowadays.

Even though irony has received little attention in computational linguistics, it is considered to

be a vital and relevant aspect in fields of study such as SA. Therefore, irony detection has

become an increasingly discussed task. Irony detection is the task of automatically classifying

pieces of text into the classes ‘ironic’ or ‘non-ironic’. According to Reyes et al. (2012) the

automatic detection of irony could be relevant in various research areas, such as electronic

commerce, product tracking and online marketing. Van Hee (2017) completes the list with other

fields of study such as language psychology, sociolinguistics and cyberbullying detection.

For SA, the presence of irony can affect the outcome drastically. Classic sentiment analysis

tools are generally not sensitive to the use of irony. They will therefore perform less accurate

when applied to ironic utterances. To successfully detect irony, it is important to firstly define

the concept and possible subcategories of irony. Only then, the specific forms and

characteristics of irony that are susceptible to computational analysis can be identified. For that

reason, we will elaborate on what is understood as ironic in chapter 2.5.1. Then, secondly, the

difficulties and challenges of irony detection will be covered in chapter 2.5.2.

2.6.1 What is irony?

Irony is a creative use of language which is omnipresent in human interaction. Van Hee et al.

(2015) define irony as “an evaluative expression whose polarity (i.e., positive, negative) is

changed between the literal and the intended evaluation, resulting in an incongruence between

the literal evaluation and its content.”. Since the language use is figurative, ironic pieces of text

Page 21 of 76

should not be interpreted literally. A more complex approach is thus required to also detect the

correct context or make associations with common knowledge.

As irony is a playful way to express oneself it comes in many different varieties. Kreuz and

Roberts (1993) already made a distinction between three varieties of irony: Socratic or dramatic

irony, irony of fate and verbal irony. Firstly, Socratic irony or dramatic irony describes the

tension between the hearer’s knowledge and what the hearer pretends to know. Here, ignorance

is sometimes feigned in order to reveal the errors in someone’s viewpoint or argument. An

example of Socratic irony could be a situation in which a parent is aware that his/her child has

come home after curfew. Instead of confronting the child with the facts, the parent will ask a

series of seemingly innocent questions that will eventually result in a confession. Secondly,

irony of fate, is explained as an incongruence between two situations. It is also referred to as

situational irony as the situations that are being discussed fail to meet some expectations. An

example of this could be a no-dog sign in an animal shelter. Lastly, if someone uses verbal

irony, the speaker intentionally implies the opposite of what he or she believes. Reyes et al.

(2012) only make a distinction between the two broad categories of verbal irony and situation

irony. Karoui et al. (2017), however, retain eight different categories. The first category covers

analogies, metaphors and comparisons which aligns two things with contrasting or different

concepts or domains. Secondly, the category of hyperboles and exaggeration enlarges a

situation to lay emphasis on a point. Thirdly, Karoui et al. (2017) also distinguish euphemisms,

which are phrases or words that help to soften reality. The fourth category contains rhetorical

questions and the fifth context shifts. The latter covers an abrupt change of the topic or tone of

the conversation. False assertions, the sixth category, are declarations that conflict with

common sense. Then, whereas a false assertion is implicit, an oxymoron or paradox (the seventh

category) explicitly expresses the contradiction. In the last category, all other expressions

containing situational irony are covered.

Furthermore, other figurative uses of language such as sarcasm need to be distinguished. Even

though there is an overlap in usage, they differ in “usage, tone and obviousness” (Reyes et al.,

p.260, 2013). Sarcasm, for example, has a higher level of aggressiveness to it. According to

Van Hee (2017), it is often used with the intention to hurt a target directly and intentionally. In

comparison with irony, the intensity is greater due to the combination of ridicule and negativity.

Irony is considered to be more so subtle and therefore more sophisticated.

2.6.2 Difficulties and challenges

Page 22 of 76

Since irony is already such a complex concept on its own, Reyes et al. (2013) indicate that it

would be unrealistic to set hopes on a single computational silver bullet for irony. More specific,

the lack of facial expression and vocal intonation in ironic tweets makes it a challenging task to

automatically detect irony. Considering that irony touches on almost every aspect of language,

a multidimensional approach for detecting irony in Twitter is desirable.

Furthermore, as already mentioned earlier (cf. chapter 2.4.1), micro-bloggers in Twitter

regularly add the hashtag irony (#irony) to indicate their use of it. Some speakers are aware

their use of language is ironic, however, Wang (2013) found Twitter users make no distinction

between irony and sarcasm. Reyes et al. (2013) affirm that users who add #irony to their tweet

merely have a diffuse and vague idea of what it is understood as an ironic text. Furthermore,

Van Hee (2017) found that one in five tweets carrying the hashtag were not ironic. As a result,

it can be concluded that manual annotations are of help training automatic irony detection

systems.

Current machine learning approaches to irony detection, for example the system of Van Hee

(2017), show that ironic tweets that hold a polarity contrast are more likely to be identified than

other types of irony. Yet, what remains a challenge is the detection of implicit sentiment (E.g.,

situations that have a specific positive or negative connotation, such as ‘going to the dentist’ or

‘hearing your train is delayed’). Experiments in this research, however, reveal “that analysing

tweets about a concept or situation appears to be a viable method to determine implicit

sentiment related to that concept or situation.” (Van Hee, p. 124, 2017).

3. RESEARCH DESIGN

In this study, we attempt to explore how well a machine learning system performs for SA on

Twitter. The following chapters will offer insight into the data collection and annotation of the

corpus. We will then discuss and analyse the results in chapter 4.

3.1 Research questions and hypotheses

This research aims at answering the question of how well a machine learning approach to SA

performs on a Twitter corpus of political tweets. Additionally, we will attempt to provide more

insight into whether the presence of ironic language in tweets influences the predictability of

tweets. To formulate a well-founded answer to these questions, we will firstly try to answer

following sub questions: Can we automatically predict sentiment with the help of sentiment

Page 23 of 76

analysis? Which impact does irony have on the predictability of sentiment and stance? Can a

machine learning system detect implicitly expressed stance?

Based on previous studies (cf. chapter 3) we can formulate the following hypotheses: We expect

our machine learning system to deliver reliable results for SA on a political Twitter corpus.

Presumably, they will not yet be able to detect and interpret ironic language use correctly in

most cases.

3.2 Methodology

To answer our research question(s), a Twitter corpus of 482 tweets was built and the tweets

were manually annotated with the labels positive, negative or neutral. The same tweets were

then annotated by a machine learning system for SA. The results of both the manual labelling

and automatic labelling will be compared and analysed in the results section (cf. chapters 4.1,

4.2, 4.3). In this section, we will discuss in detail how our data were collected and annotated.

3.2.1 Data collection

As one of the first steps in the data collection process, we decided on which topic we would

collect tweets. We chose to gather English tweets with the hashtag Brexit (#Brexit) of the 24th

of June 2016. Since the Brexit referendum was held the day before, on the 23rd of June, we

believed the day after the outcome would provide us with divided opinions. The referendum

decided on whether the UK would leave the European Union or not. 51,9% of the voters

appeared to be in favour of leaving the EU and won, whereas an almost equally large group

(48,1%) was against the Brexit and lost. Ever since the announcement of the Brexit referendum,

it has been a highly discussed topic within the European political landscape.

We included every tweet of the 24th of June 2016 with #Brexit in chronological order, but

decided to ignore tweets that appeared twice or more. Furthermore, we decided to discard tweets

with double opinions. An example of such a tweet would be “Scary stuff! Still #brexit is best!!”.

In the first part of the tweet, ‘scary stuff’ could be interpreted as negative, however the second

sentence clearly contains positive sentiment. The same conclusion can be drawn from the

following tweet “Britons will enjoy their victory today. But tomorrow, the hangover will be

fierce #Brexit #UKReferendum”. The first sentence is positive, whereas the second part warns

Page 24 of 76

for probable negative consequences of the Brexit. Since these contradictory sentiments could

easily confuse the classifier of the machine learning system, we decided to exclude tweets with

double opinions from the corpus. Our aim was to collect circa 500 tweets and originally our

corpus consisted of 512 tweets with #Brexit. After removing tweets that appeared twice or

more, or tweets with double opinions, we eventually retained 482 tweets.

3.1.1 Annotation

For the manual annotation of the Twitter corpus we focussed on four categories: sentiment,

topic/aspect, stance and irony. The purpose of these four categories was to gain insight into the

nature of the tweet, considering various approaches and classifications. In the first category,

sentiment, we looked at the tweet as a whole and decided on whether the content was positive,

negative or neutral, regardless what the actual subject of the tweet was. If the opinion expressed

in the tweet was indefinable, we attached a neutral label to the piece of text.

Furthermore, we manually classified all tweets into various topics or aspects, the second

category. Eventually we narrowed 72 topics (cf. Appendix 1) down to eight categories of topics:

Brexit, celebrities/politicians, economy, EU, Scottish referendum, Trump, USA and other. We

decided to focus on a small number of topics that serve as an umbrella term for several aspects,

to keep an convenient overview of the themes discussed. Under the topic

‘celebrities/politicians’ for example, we classified all tweets that mentioned a specific name of

a public or political figure (e.g., Lindsey Lohan, Boris Johnson, Nigel Farrage). The topic

‘other’ served as an undefined category to classify numerous dissimilar subjects (e.g., personal

information, jokes).

The annotation of the third category concentrated on the stance expressed by the author towards

the Brexit. As already mentioned in chapter 2.4.2, SA aims at detecting the sentiment of an

opinion in a piece of text, whereas stance detection is used to pick up whether someone is for

or against the subject (in this case: the Brexit) being debated.

Lastly, to explore which impact irony has on the predictability of sentiment, we added a fourth

category in the annotation process. Based on the eight different categories of irony Karoui et

al. (2017) distinguished (analogies, metaphors and comparisons; hyperboles and exaggerations;

Page 25 of 76

euphemisms; rhetorical questions; context shifts; false assertions; oxymorons or paradoxes;

situational irony), we marked tweets with ironic messages.

3.1.2 Experimental approach

For the experimental component of this thesis, a machine learning model was created to label

both the sentiment and stance of each tweet automatically. In total, two experiments were thus

carried out: one to predict sentiment and one to predict stance. Firstly, the Twitter corpus was

split up in, on the one hand, the separate tweets and, on the other hand, their manually annotated

stance and sentiment label. The tweets were separated from the labels by tab.

Secondly, all tweets were tokenised with LeTs, a multilingual linguistic pre-processing toolkit

developed by Van de Kauter et al (2013). This toolkit includes Part-of-Speech taggers,

lemmatizers and named entity recognizers. For this study, all tokens (words, punctuation,

numbers, symbols) were separated from each other with the pre-processing tool.

Thirdly, several features were extracted: unigrams, bigrams, trigrams, character n-gram features

with a range of 3-4 tokens, the number of flooded tokens, the number of flooded punctuation

tokens, the number of capitalized tokens and a sentiment lexicon look-up. The lexicon used is

called AFINN4, which is a list of English words for valence with an integer between minus five

(negative) and plus five (positive). With this sentiment lexicon, the number of positive, negative

and neutral tokens as well as the overall value of one tweet can be extracted.

Lastly, the Support Vector Machine was run and tested on a tenfold cross-validation scheme,

which means that 90% of the Twitter corpus was used as a train fold and 10% as a test fold.

This process was repeated ten times: each time with another 10% as test corpus up until the

moment that the entire corpus has served as test fold. The machine learning experiments were

carried out with LIBSVM5, which is an integrated software for support vector classification,

regression and distribution estimation. For this study, the linear kernel was used.

4 http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6010 5 https://www.csie.ntu.edu.tw/~cjlin/libsvm/

Page 26 of 76

17%

53%

30%

FIGURE 1 - MANUAL ANNOTATION SENTIMENT (N=482)

Positive (84) Negative (254) Neutral (144)

Figure 1 - Manual annotation sentiment

4. RESULTS

In this chapter, we will discuss both the results of the SA in the manually annotated Twitter

corpus (cf. chapter 4.1) and the outcome of the automatically annotated Twitter corpus (cf.

chapter 4.2). Then, the two result sections will be compared and thoroughly analysed (cf.

chapter 4.3).

4.1 Results manual annotation

4.1.1 Sentiment and topics

The Twitter corpus consisted of 482 tweets with the hashtag Brexit (#Brexit). The manual

annotation resulted in 84 tweets with a positive sentiment label (17%), 254 tweets with a

negative sentiment label (53%) and 144 (30%) with a neutral sentiment label.

As already mentioned (cf. chapter 3.2.2), we made a distinction between 8 categories of topics

(cf. Figure 2). The greater part of the tweets (57%) addressed the topic of the Brexit itself. This

could be explained by the specific hashtag in every tweet (#Brexit). Then, several users also

mentioned the consequences for the economy after the Brexit outcome (7%). To nearly the

same extent, users tweeted about celebrities or politicians, using the hashtag Brexit (6%). Only

a negligible number of tweets (2 out of 482 tweets) considered a Scottish referendum, however,

the USA (4%) and Trump (4%) were discussed more frequently. Both the USA and Trump

were equally discussed within the context of the Brexit, mostly as a political point of

Page 27 of 76

57%

6%

7%

3%0%4%

4%

19%

FIGURE 2 - TOPICS IN TWEETS (N= 482)

Brexit (273) Celebrities/politicians (29) Economy (34)

EU (13) Scottish referendum (2) Trump (21)

USA (21) Other (89)

Figure 2 - Topics in tweets

comparison. American elections were then to be held in November 2017, which was a highly

debated topic at that time as well. An example of this would be: “Britons voted to strengthen

their borders. Will you do the same in November? #Brexit”. Besides discussing the referendum

or the outcome of the referendum, many users gave personal information or any other remarks

in their tweets. This resulted in a fair number of tweets commenting upon other topics or

considering personal information (19%). Following tweet is an example of such a tweet: “A

very good cereal served at my amazing property in Turnberry! #Brexit of Champions, just like

me! Enjoy!”.

By comparing the previous results (cf. Figure 1 and Figure 2) and joining them together (cf.

Table 1), it is noticeable that, with the exception of the category ‘other’, negative labels are

predominant in nearly every topic. Especially when discussing the Brexit and its outcome itself,

the tweets are notably negative. In the category ‘other’, there are 34 negative and 37 neutral

tweets. An explanation for this could be the presence of personal information, which tends to

be either critical or merely narrative. An example of the latter could be the following tweet: “I

know it is not good for me, but, on days when Britain chooses to #Brexit, I like to drink a Coke

and eat a cookie. #EURefResults”. In this tweet, the actual subject is not the Brexit but the

author’s diet. The sentiment is negative even though the tweet considers ‘Coke and cookies’.

Besides personal information, jokes are also omnipresent in the category ‘other’, as can be

observed in the following tweet: “Is this going to affect my chances of getting into Hogwarts?

Page 28 of 76

#Brexit”. Here, the sentiment is neutral, because the author does not express itself negatively

towards the Brexit, but simply jokes around.

TOPIC NEGATIVE NEUTRAL POSITIVE

Brexit (273) 153 (56%) 74 (27%) 46 (17%)

Other (89) 34 (38%) 37 (42%) 18 (20%)

Economy (34) 17 (50%) 11 (32%) 6 (18%)

Celebrities (29) 19 (66%) 8 (28%) 1 (34%)

USA (21) 7 (33%) 7 (33%) 7 (33%)

Trump (21) 17 (81%) 1 (5%) 3 (14%)

EU (13) 7 (54%) 4 (31%) 2 (15%)

Scottish referendum

(2)

0 (0%) 2 (100%) 0 (0%)

TOTAL 254 144 84 Table 1 - Sentiment per topic (manual annotation)

4.1.2 Stance and irony

To explore whether the author is for or against the subject of the Brexit, we also labelled the

stance of a tweet as positive, negative or neutral. In Figure 3, an overview is given of the stance

expressed in our Twitter corpus. The majority of the authors (288 tweets) expresses itself

negatively towards the Brexit (60%), whereas 26% has a rather neutral stance (124 tweets).

Moreover, 14% takes a positive stance on the referendum (70 tweets).

In some tweets, the stance is rather implicit, for instance in the following tweet: “So, that was

the dress rehearsal. Now that you Leavers have seen the effects of your vote, would you like to

try that again? #Brexit”. Even though the author does not explicitly express itself for or against

the Brexit, it is clear that he/she mocks with the leave-voters and is against Britain leaving the

EU. Tweets can also contain implicit positive stance, such as in the following tweet: “Leftists

are freaking out over the #Brexit. Why, because the people finally rejected tyranny? Just proves:

liberals = tyrants. #tcot”. The author is fairly negative towards the people who voted to remain

part of the EU and is therefore a supporter of the Brexit, which results in a positive stance label.

Page 29 of 76

Figure 3 - Manual annotation stance

If we compare the results of the SA with those of our stance detection (cf. Table 2), we find that

218 tweets (out of 482) hold negative sentiment as well as negative stance. Additionally, 43

tweets contain positive sentiment and stance, whereas 89 tweets have neutral sentiment and

stance. In total, 350 tweets (73%) have the same sentiment and stance label, which means 132

tweets show differences in the annotation of sentiment and stance. An example of the latter

could be the following tweet: “The only good thing to come out of the #Brexit is the dearth of

insults being hurled at @realDonaldTrump by the lovely people of Scotland”. Here the

sentiment of the tweet is positive, because the author explains a positive result of the Brexit.

The stance expressed, however, is negative, for the reason that the author does not see any other

positive consequences of the Brexit apart from new insults about Trump.

It is noticeable, that besides the relatively frequent use of irony in tweets (9%) with negative

sentiment and stance, especially tweets with neutral sentiment holding either negative or neutral

stance also contain ironic language (4%). In chapter 4.3, we will further explore whether a

machine learning system for sentiment analysis and stance detection will be influenced by the

usage of irony. As can be drawn from the overview in Table 3, irony is particularly used in

tweets specifically considering the economic consequences of the Brexit, followed by tweets

about Trump and the Brexit itself.

14%

60%

26%

FIGURE 3 - MANUAL ANNOTATION STANCE (N=482)

Positive (70) Negative (288) Neutral (124)

Page 30 of 76

SENTIMENT STANCE TOTAL

TWEETS PERCENTAGE

IRONIC

TWEETS

IRONY

PERCENTAGE

NEGATIVE NEGATIVE 218 45% 47 22%

NEUTRAL 19 4% 3 16%

POSITIVE 10 2% 0 0%

NEUTRAL NEGATIVE 45 9% 19 42%

NEUTRAL 89 18% 19 21%

POSITIVE 10 2% 0 0%

POSITIVE NEGATIVE 25 5% 8 32%

NEUTRAL 16 3% 4 25%

POSITIVE 43 9% 2 5%

= 482 = 103 Table 2 - Manual annotation sentiment, stance and irony

4.2 Results machine learning system

The second step in the experimental part of this study

consisted of the automatic annotation of the 482 political tweets with the hashtag Brexit. In this

section, we will discuss the results of both the sentiment and stance labels. In chapter 4.3, we

will compare the results of the manual with those of the machine learning approach.

4.2.1 Sentiment and topics

As can be drawn from the pie chart below (cf. Figure 5), the greater part (69%) of the Twitter

corpus consists, according to the SA-tool, of negative tweets. The following tweet, for example,

was picked up by the system and labelled as negative: “Still so sad about #Brexit. What is this

dark, absurd future being carved out for the world?”. Here, the sentiment words ‘sad’ and ‘dark’

were presumably a deciding factor. 117 tweets or 24% of the corpus received a neutral label

and a small 7% of the tweets was labelled as positive. An example of the latter could be:

“Learning some great new swears, thanks Scotland! #Brexit”. In this tweet, the author is

delighted to learn new insulting language phenomena from Scotland.

TOPIC NUMBER OF TWEETS

CONTAINING IRONY

IRONY PERCENTAGE

PER TOPIC

Brexit (274) 60 22%

Celebrities/politicians (29) 6 21%

Scottish referendum (2) 0 0%

Economy (34) 9 26%

EU (14) 2 14%

other (94) 19 20%

Trump (21) 5 24%

USA (22) 2 9%

= 103 Table 3 - Presence of irony per topic

Page 31 of 76

7%

69%

24%

FIGURE 4 - MACHINE LEARNING ANNOTATION

SENTIMENT (N=482)

Positive (34) Negative (331) Neutral (117)

Figure 4 - Machine learning annotation sentiment

As can be drawn from Table 4, the distribution of sentiment in each topic is not equally divided:

in every topic category, the negative labels are predominant, followed by considerably less

neutral and even fewer positive tweets. This can also be concluded from Table 1. Only in the

category ‘USA’, there is one more positive tweet than the neutral ones and in the category

‘Scottish referendum’, there are merely 2 tweets, which are both neutral. The majority of the

Twitter corpus consists of negative tweets on the Brexit referendum itself. An example of this

could be: “Never underestimate the power of stupid people in large groups! #Brexit

#jokeofthecentury”. In this tweet, the author utters itself negatively towards the Leave-voters

of the Brexit. The topic on ‘voters’ was classified under ‘Brexit’, as can be seen in Appendix

1.

TOPIC NEGATIVE NEUTRAL POSITIVE

Brexit (273) 190 (70%) 64 (23%) 19 (7%)

Other (89) 53 (60%) 30 (34%) 6 (6%)

Economy (34) 23 (68%) 9 (26%) 2 (6%)

Celebrities (29) 23 (80%) 4 (14%) 2 (7%)

USA (21) 14 (67%) 3 (14%) 4 (19%)

Trump (21) 17 (81%) 3 (14%) 1 (5%)

EU (13) 11 (85%) 2 (15%) 0 (0%)

Scottish referendum

(2)

0 (0%) 2 (100%) 0 (0%)

TOTAL 331 117 34 Table 4 - Sentiment per topic (machine learning approach)

4.2.2 Stance and irony

Considering the results of the stance detection returned by our classifier, it can be said that the

distribution of negative, neutral and positive stance is fairly similar compared to the distribution

Page 32 of 76

of sentiment (cf. Figure 5). However, the percentage of tweets with negative stance (79%) is

considerably higher than tweets with negative sentiment (69%). In Table 5, we can see that,

according to the automatic system for SA, 302 tweets contain both negative sentiment and

stance, which counts for 63% of all tweets. Of all these tweets, 13% percent contains ironic

language. With regard to the neutral labels, it can be drawn from Table 5 that 10% of the tweets

contain a neutral sentiment label as well as a neutral stance label. In 12% of all cases, tweets

with neutral sentiment received a negative stance label. Furthermore, it is remarkable that in

total there are very little entirely positive tweets (2%).

Figure 5 - Machine learning annotation stance

Concerning the percentage of ironic tweets, it can be stated that 32% of the tweets that received

a positive sentiment label and a negative stance label from the machine learning system contain

ironic language. Moreover, it is notable that a quarter of the tweets that contain positive

sentiment and negative stance are ironic. In absolute numbers, the category with both negative

sentiment and negative stance holds the highest number of ironic tweets, namely 64. In section

4.3.2.2, we will compare the accordance of sentiment with stance to the influence of irony in

further detail.

5%

79%

16%

FIGURE 5 - MACHINE LEARNING ANNOTATION

STANCE (N=482)

Positive (22) Negative (381) Neutral (79)

Page 33 of 76

SENTIMENT STANCE TOTAL

TWEETS

PERCENTAGE IRONIC

TWEETS

IRONY

PERCENTAGE

NEGATIVE NEGATIVE 302 63% 64 21%

NEUTRAL 22 5% 4 18%

POSITIVE 7 1% 1 14%

NEUTRAL NEGATIVE 60 12% 15 25%

NEUTRAL 50 10% 9 18%

POSITIVE 7 1% 2 29%

POSITIVE NEGATIVE 19 4% 6 32%

NEUTRAL 7 1% 1 14%

POSITIVE 8 2% 1 13%

= 482 = 103 Table 5 - Machine learning annotation sentiment, stance and irony

4.3 Analysis

In this chapter, we will compare and interpret the results of both the manual annotation and the

labelling of the machine learning system. We will verify how accurate the machine learning

system performs for sentiment analysis and stance detection on a Twitter corpus with political

tweets. Furthermore, we will analyse whether or not the presence of ironic language affects the

predictability of sentiment or stance.

4.3.1 Sentiment analysis: tenfold cross-validation scheme

In the methodology section (cf. 3.2) of this study, it was already explained that our machine

learning system was run and tested on a tenfold cross-validation scheme6. This means that 90%

of the Twitter corpus was used as a train fold and 10% as a test fold. This process was repeated

10 times to check the quality of the system. The tenfold cross-validation approach is found to

be reliable, for the reason that the entire corpus is used for both training and validation.

Moreover, each set is used for validation exactly once. In Table 10, an overview is given of the

results of our tenfold cross-validation scheme. In chapter 4.3.2, we will discuss the global

results of dataset as a whole.

It is noticeable that the scores can vary from fold to fold, considering that the cross-validation

process uses different train and test data in every fold. In Table 8, the ten accuracy scores are

similar to one another, whereas precision and recall scores differ notably. For example, the

precision scores of the positive labels in the various folds vary from zero to 0.67. The same

goes for the positive recall score: in Fold 01, for instance, the score is zero, whereas a score of

6 https://www.openml.org/a/estimation-procedures/1

Page 34 of 76

0.43 is given Fold 08. The F1 scores of the negative labels are all closely situated next to one

another, with only one peak in Fold 06 of 0.8. All negative F1 scores are well above 0.6, which

makes negative sentiment the best predicted label. The precision and recall of neutral labels

tend to lie far apart from each other in the different folds. In Fold 02, for instance, the neutral

precision score is 0,27, when in Fold 03 the score is remarkably higher (0.65).

Sentiment pos

prec

neg

prec

neutr

prec

pos

recall

neg

recall

neutr

recall

pos

F1

neg

F1

neutr

F1

accuracy

Fold 00 0.20 0.61 0.27 0.11 0.73 0.25 0.14 0.67 0.26 0.49

Fold 01 0 0.58 0.70 0 0.84 0.50 0 0.69 0.58 0.58

Fold 02 0.50 0.60 0.27 0.14 0.75 0.23 0.22 0.67 0.25 0.52

Fold 03 0 0.54 0.65 0 0.78 0.52 0 0.64 0.58 0.54

Fold 04 0.67 0.56 0.38 0.18 0.71 0.42 0.29 0.63 0.40 0.52

Fold 05 0 0.58 0.50 0 0.73 0.33 0 0.64 0.40 0.50

Fold 06 0 0.75 0.40 0 0.86 0.67 0 0.80 0.50 0.63

Fold 07 0 0.62 0.33 0 0.78 0.25 0 0.69 0.29 0.52

Fold 08 0.43 0.70 0.55 0.43 0.78 0.43 0.43 0.74 0.48 0.63

Fold 09 0.75 0.50 0.57 0.30 0.87 0.22 0.43 0.63 0.32 0.53

Table 6 - Tenfold cross-validation scheme for sentiment analysis

4.3.2 Sentiment analysis: general overview

In the first column of Table 7, an overview is given of the number of the manually given labels

per sentiment. The second column shows in yellow how many correct labels the machine

learning system assigned to the tweets. Besides the number of these true positives, an overview

is given of all false positives per sentiment category. The third column presents the number of

ironic tweets per automatically predicted label to provide insight into the effect of irony on the

predictability of sentiment

Page 35 of 76

MANUALLY

ANNOTATED

SENTIMENT

TWEETS THAT ARE

AUTOMATICALLY

PREDICTED AS

NUMBER OF

IRONIC TWEETS

(103)

IRONY

PERCENTAGE

NEGATIVE 254 NEGATIVE 199 (78%) 39 20%

NEUTRAL 40 (16%) 8 20%

POSITIVE 15 (6%) 4 27%

NEUTRAL 144 NEGATIVE 81 (56%) 21 26%

NEUTRAL 54 (38%) 16 30%

POSITIVE 9 (6%) 1 11%

POSITIVE 84 NEGATIVE 51 (61%) 9 18%

NEUTRAL 23 (27%) 2 9%

POSITIVE 10 (12%) 3 30% Table 7 - Comparison manual and automatic sentiment analysis + irony presence

To interpret the results presented in Table 7, we calculated precision, recall, and F1-score as

well as the accuracy of our machine learning system. Firstly, precision is the number of true

positives divided by the total number of true positives and false positives returned by the

classifier and is used to find out how “correct” the predictions per label are. Secondly, the recall

score reveals how many sentiment labels are predicted by dividing true positives by the total

number of true positives and false negatives returned by the machine learning system (per

label). Thirdly, the F1-score is the average of both precision and recall and shows how well the

system performs per sentiment category. The F1 or F-score is the result of the following

division: 𝑓 =2 . (precision.recall)

precision+recall . Lastly, the accuracy of the test set was calculated as a whole,

by dividing the number of correctly predicted labels by the total number of instances.

SENTIMENT Negative Neutral Positive

Precision 𝑝 =199

(199 + 81 + 51)

𝑝 = 0.6

𝑝 =54

(40 + 54 + 23)

𝑝 = 0.46

𝑝 =10

(15 + 9 + 10)

𝑝 = 0.29

Recall

𝑟 =199

254

𝑟 = 0.78

𝑟 =54

144

𝑟 = 0.38

𝑟 =10

84

𝑟 = 0.12

F1-score

f = 0.69 f = 0.42 f = 0.21

Accuracy 𝑎 =199 + 54 + 10

482

𝑎 = 0.55

Table 8 - Precision, recall, F1-score and accuracy of sentiment labels

Page 36 of 76

As can be drawn from Table 8, the accuracy of the machine learning system results in a score

of 0.55. This means that the performance of our SA system can be evaluated as moderate,

because the score is slightly above 0.5. The high precision score of the negative sentiment

shows that the system returned more true positives than false positives. Moreover, the high

recall score indicates that the SA system returned most of the relevant results. Yet, there is still

room for improvement, especially when looking at the precision and recall scores of the neutral

and positive sentiment labels. For example, the recall score of the positive labels shows that the

machine learning failed to return many of the positive tweets.

If we compare Table 8 with Table 6, we notice that our machine learning system for SA is rather

biased towards negative tweets. In other words, our classifier tends to overgeneralize negative

labels and attributes them falsely in 34% of the cases. Neutral labels are assigned falsely for

13% of the tweets and positive labels are only ascribed wrongly for 5% of the cases. This might

explain the high precision, recall and F1 scores within the negative category and the low results

for both positive and neutral labels. As a result of this difference between results for the negative

sentiment category on the one hand, and those for the positive and neutral sentiment category

on the other hand, the accuracy scores are only slightly above average.

From the third column in Table 7, it can be concluded that 45 of all 215 false positive cases

contained ironic language. In other words, irony was used in 21 percent of all falsely labelled

tweets. This could be an explanation for many of the errors, meaning that irony influences the

predictability of sentiment in political tweets. In chapter 4.3.1.3 we will discuss the various

errors in greater detail.

4.3.2.1 Impact of irony on the prediction of sentiment

In Table 7, an overview is given of how many tweets contain irony to analyse the possible effect

of creative language on the predictability of sentiment. In the fourth column, the irony

percentage in each sentiment category is presented. It can be noted, that in the negative

manually annotated category the irony percentages all lie between 20 and 27 percent. As a

result, it is difficult to conclude, whether the false positives (negative tweets that received a

neutral or positive label) are influenced by the presence of the ironic language or not, without

looking at the content of the tweets itself. For instance, in 20% of all negative tweets that

received a positive label, 27% contained irony. The same goes for the neutral and positive

sentiment category, where the true positives itself (e.g. neutral tweets that received a neutral

Page 37 of 76

label by our machine learning system) appear to contain the highest percentages of ironic

tweets. In total, if the amount of ironic false positives is divided by the total number of false

positives, it appears that 21% of the falsely labelled tweets by the machine learning system

contained irony. If we follow the same procedure for all true positives, it shows that 22% of the

correctly automatically labelled tweets also contained irony.

Considering the content of all falsely labelled tweets, it can, however, be concluded that irony

does have a certain effect on the machine learning system. In the following negative tweets, for

instance, the classifier ascribed a positive or neutral label due to irony:

(1) “Thanks British #brexit twats, I'm feeling poorer today.”

(2) “The next James Bond will just be him spending 2 hours in passport control De

Gaulle #Brexit #JamesBond”

(3) “Sebastian was right, can I become a mermaid now pls #Brexit”

(4) "They took back their country and that’s a great thing," Trump said of #Brexit, while

in Scotland IN SCOTLAND!!!!!

In tweet (1), our machine learning for SA does not detect the ironic expression of gratitude and

the negative connotation attributing ‘feeling poor’. In other words, the implicit sentiment in

‘feeling poor’ was not recognized. Moreover, the negative and insulting word ‘twats’ was not

picked up. Then, in tweet (2) the classifier decided on a neutral label, regardless of the fact that

spending 2 hours in passport control is a rather unpleasant activity. In the third tweet (3), a

hyperbole is used to express the author’s disbelief, with regard to the leave-voters in the Brexit

referendum. Here, some specific background knowledge is needed to understand the reference

made to the Little Mermaid. There, the character Sebastian attempts to warn mermaid Ariel for

the foolishness of human beings, by saying ‘The human world is a mess’. In (3), the author

confirms the truth in Sebastian’s reasoning. The system, however, fails to interpret the context

and irony in the tweet and assigns a neutral label. Lastly, tweet (4) is an example of situational

irony. The fact that the American president Trump expresses himself in favour of Britain ‘taking

their country back’ while being in Scotland is ironic, in the sense that Scotland itself has been

struggling in their search for independency for years. Here again, the system lacks the right

Page 38 of 76

contextual information to interpret the tweet correctly. Furthermore, the repetition of ‘in

Scotland’ in capital letters, followed by a punctuation flooding was not detected.

In chapter 4.3.4.1 we will go further into the possible effect of irony on our stance classifier.

4.3.2.2 Error analysis

Besides the effect of irony possibly causing the machine learning system to make errors, we

will consider another factor that may influence the automatic attribution of a wrong sentiment

label in this section. In some tweets, the machine learning system appears to have a notable lack

of common of specific knowledge. Therefore, following tweets were misinterpreted and

therefore falsely labelled:

(5) #Brexit, Monty Python & Silly Walks.

(6) Wales should have been more careful what it wished for. It's going to be given it.

#Brexit

(7) Proud to be #Brexit! Proud to stand alone in a world where most are too scared to

be alone, to have their own opinion. Proud to me! #UKref

In tweet (5), the classifier attributed a negative label, whereas it was originally granted a neutral

label. Presumably, the word ‘silly’ was picked up as a negative word. The tweet, however,

refers to the popular Monty Python sketch ‘Ministry of Silly Walks’, firstly broadcast in 1970,

which does not hold a negative connotation. The sixth tweet (6), contains an allusion to the

common expression ‘be careful what you wish for’. According to the Merriam-Webster7

dictionary, it is usually used ‘to tell people to think before they say that they want something

and to suggest that they may not actually want it’. Nonetheless, this negative tweet received a

neutral label. The last example (7), being alone is valued as a positive situation. The author

explains that he/she sees being alone as having an individual opinion, which is something to be

proud of. Therefore, the tweet is positive, but the classifier hands out a negative label because

it sees being alone as something negative.

7 https://www.merriam-webster.com/

Page 39 of 76

4.3.3 Stance detection: tenfold cross-validation scheme

To gain more detailed information on the accuracy of the stance labels returned by the classifier,

the tenfold cross-validation approach was once again applied on the Twitter corpus. In

comparison with the accuracy scores of the SA, it can be concluded that, in general, the

classifier returned better accuracy results for stance detection than for SA. This will be further

discussed in chapter 4.3.4. Furthermore, it is remarkable that in, for instance, Fold 05 the

accuracy is relatively low (0.45) when compared to the high score of 0.73 in Fold 09. Mainly,

these differences can be declared by focussing on the positive precision and recall scores.

Remarkably, the positive precision scores vary between zero and one. An explanation for this

phenomenon could be the fact that in a tenfold cross-validation scheme, only 10 percent of all

data is used as test data in each fold. Considering that only 10 positive stance labels were

predicted correctly, it is comprehensible that in some folds none of the corresponding tweets

appeared. This then results in a score of zero for both precision and recall.

In general, the positive recall scores are all fairly low, varying between zero and 0.22. In

comparison, however, the precision scores are higher than the recall scores in the positive stance

category. The higher precision score indicates a low false positive rate, but it can be concluded

from the low recall that, in general, very few results were predicted. As regards the neutral

accuracy, the precision scores are again rather varied. They run from a very low 0.22 in Fold

02 to a fairly high score of 0.67 in Fold 03. The neutral recall scores do not reach above average,

with the exception of Fold 08. Moreover, the negative recall scores are all fairly continuous and

all lie between 0.75 and an almost perfect score of 0.93. It is noteworthy that the negative

precision scores are, for example in Fold 03, rather average, whereas Fold 02 shows a peak of

0.76.

Page 40 of 76

Stance pos

prec

neg

prec

neutr

prec

pos

recall

neg

recall

neutr

recall

pos

F1

neg

F1

neutr

F1

accuracy

Fold 00 0.50 0.64 0.42 0.13 0.75 0.45 0.20 0.69 0.43 0.57

Fold 01 1 0.59 0.63 0.13 0.92 0.33 0.22 0.72 0.43 0.60

Fold 02 0.50 0.76 0.22 0.14 0.82 0.29 0.22 0.79 0.25 0.65

Fold 03 0 0.47 0.67 0 0.9 0.29 0 0.62 0.40 0.48

Fold 04 1 0.67 0.38 0.14 0.84 0.30 0.25 0.74 0.33 0.63

Fold 05 0.25 0.45 0.67 0.14 0.81 0.20 0.18 0.58 0.31 0.46

Fold 06 0 0.7 0.43 0 0.88 0.27 0 0.78 0.33 0.65

Fold 07 0.50 0.68 0.25 0.17 0.81 0.2 0.25 0.74 0.22 0.60

Fold 08 1 0.70 0.67 0.22 0.93 0.55 0.37 0.80 0.60 0.71

Fold 09 0.50 0.77 0.33 0.40 0.90 0.13 0.44 0.83 0.18 0.73

Table 9 - Tenfold cross-validation scheme for stance detection

4.3.4 Stance detection: general overview

As already explained in chapter 4.3.2, an overview (cf. Table 7) was made of both the manually

distributed labels, with in the second column the corresponding labels extracted by the machine

learning system. In the third column, the number of ironic tweets was listed per label to analyse

whether ironic language influences the predictability of stance or not. Furthermore, precision,

recall, F1 score and accuracy were once again calculated to provide more insight into the quality

of the stance labels returned by the system. Table 10 and 11 show results of the entire data set,

after 10-fold cross validation.

MANUALLY

ANNOTATED

STANCE

TWEETS THAT ARE

AUTOMATICALLY

PREDICTED AS

NUMBER OF

IRONIC TWEETS

(103)

IRONY

PERCENTAGE

NEGATIVE 288 NEGATIVE 246 (85%) 63 26%

NEUTRAL 33 (12%) 8 24%

POSITIVE 9 (3%) 3 33%

NEUTRAL 124 NEGATIVE 84 (68%) 19 23%

NEUTRAL 37 (30%) 6 16%

POSITIVE 3 (2%) 1 33%

POSITIVE 70 NEGATIVE 51 (73%) 3 6%

NEUTRAL 9 (13%) 0 0%

POSITIVE 10 (14%) 0 0% Table 10 - Comparison manual and automatic stance detection + irony presence

Page 41 of 76

As can be concluded from Table 10 and 11, the most accurate predictions were made for the

negative stance category. Both precision and recall show reasonably good results with

correspondingly a score of 0.65 and 0.85, which results in an F1-score of 0.75. It is remarkable

that the scores in the neutral and positive category are again noticeably lower, with F1-scores

of 0.39 and 0.30. In total, the machine learning system reaches a fairly high accuracy score of

0.61. Furthermore, it is remarkable that the precision score of the positive category is nearly

three times higher than the recall score. This indicates that the machine learning system predicts

on the one hand very few positive stance labels, but that, on the other hand, these labels are in

many cases correct.

STANCE Negative Neutral Positive

Precision 𝑝 =246

(246 + 84 + 51)

𝑝 = 0.65

𝑝 =37

(33 + 37 + 9)

𝑝 = 0.47

𝑝 =10

(9 + 3 + 10)

𝑝 = 0.45

Recall

𝑟 =246

288

𝑟 = 0.85

𝑟 =37

124

𝑟 = 0.30

𝑟 =10

70

𝑟 = 0.14

F1-score

f = 0.75 f = 0.39 f = 0.30

Accuracy 𝑎 =246 + 37 + 10

482

𝑎 = 0,61

Table 11 - Precision, recall, F1-score and accuracy of stance labels

4.3.4.1 Impact of irony on the prediction of stance

To verify the impact of ironic language on the predictability of stance, Table 10 shows the

absolute numbers and percentages of irony in the Twitter corpus. In total 103 of 482 tweets

were manually labelled as ironic. The use of irony is fairly equally distributed between the

negative and neutral stance categories. It is, however, noticeable that the irony use in tweets

with positive stance is rather rare. In 26% and 16% of the corresponding negative and neutral

categories of true positives, the tweets contain irony. In addition, it can be stated that the irony

percentage in the false positives is just as high (26%) for tweets with negative stance and

somewhat higher (23%) for tweets with neutral stance.

Page 42 of 76

To interpret the meaning of these irony percentages, a closer look to the content of the falsely

labelled tweets is required. By considering merely the ironic tweets that were predicted falsely,

the following examples were retained.

(8) Can we solve #Brexit issues using the old "reset" method? Switch off, leave Europe

for 10 secs then plug ourselves bck in? Oh wait... oops?

(9) So, that was the dress rehearsal. Now that you Leavers have seen the effects of your

vote, would you like to try that again? #Brexit

(10) Churchill said, "Heroes fight like Greeks". Like a Greek i have to say that

"Heroes, vote like British!" #Brexit

In tweet (8) the author raises several rhetorical questions, a form of irony as defined by Karoui

et al. (2017). The classifier fails to pick up on this and returns a neutral stance label, whereas

the tweet was manually labelled as negative. The same goes for tweet (9), where the machine

learning system does not detect the ironic use of language and therefore decides to label the

stance as positive. The author, however, is against the outcome of the Brexit and jokingly

proposes to organize another referendum now the leave voters understand the negative

consequences of leaving the European Union. As a result, the tweet was manually annotated

with a negative stance label. In the last example (10), the author uses a metaphor and analogy

to express his positive attitude towards the Brexit. The system could not interpret the complex

nature of this tweet and assigned a negative label to it. Apart from tweet (8), (9) and (10), clear

examples of how irony influenced the detection of stance were rare.

4.3.4.2 Error analysis

As already established in chapter 4.3.2.2, it can be assumed that other factors influence stance

detection apart from irony presence. In this section, several examples of errors will be listed

and discussed.

(11) "protection!? Protection of what!? #TzeGermans!?" #Brexit #funfacts

(12) Proud of #Britain and @David_Cameron for doing the right thing today. #Brexit

#EUref #independence #freedom

Page 43 of 76

(13) Britons voted to strengthen their borders. Will you do the same in November?

#Brexit

(14) Your country is truly inspiring thank you #Brexit

(15) Man, that was so cool. #Brexit

(16) The #Brexit was necessary. The EU was turning into that evil corporation from

the Aliens franchise.

(17) On my room balcony posing with the freshly brexited EU flag. Giving it some

company. #iifa2016 #iifadiaries #brexit

The author’s stance in tweet (11) is negative. The content refers to one of the main arguments

of the Leave-voters in the Brexit referendum: leaving the EU is a form of protectionism and

equals ‘putting the UK first’. The author of tweet (11) uses a reference from the British movie

Snatch to point out that there is nothing to protect the UK from. The machine learning system

is not aware of this specific movie reference and judges that the stance is neutral. In tweets (12),

(13), (14) and (15) the trend that the classifier tends to overgenerate negative labels – just as

specified in the analysis of the SA results (cf. 4.3.2) – can be observed. The four tweets contain

positive stance and positive sentiment words. Nonetheless, the machine learning system decides

on a negative label. In the last example (16), the author appears to be in favour of the Brexit

outcome and against the EU as an institution. Therefore, the author’s stance towards the Brexit

is positive. It can be presumed that the classifier interpreted the stance as negative, as a

consequence of the use of ‘evil’. In the last tweet (17) with positive stance, there are no clear

sentiment words present, which results in a false attribution of a neutral stance label.

4.3.5 Comparison sentiment analysis and stance detection

In chapters 4.3.1, 4.3.2, 4.3.3 and 4.3.4 we analysed the results returned by the machine learning

system. We separately analysed the results of the sentiment analysis and stance detection, by

observing precision, recall, F1 and accuracy scores. In this section, we will focus on differences

Page 44 of 76

in performance and accordance between the prediction of sentiment and stance. Furthermore,

we will again consider the possible impact of irony use.

4.3.5.1 Comparison performance

In Table 12, all results of both the sentiment analysis and stance detection were listed. The first

and third column show the total number of tweets with negative, neutral or positive

sentiment/stance. The second and fourth column give an overview of how the machine learning

system attributed sentiment and stance labels. In addition, to compare the performance of our

system in terms of sentiment analysis and stance detection, Table 13 shows all precision, recall,

F1 and accuracy scores.

MANUALLY

ANNOTATED

SENTIMENT

TWEETS THAT ARE

AUTOMATICALLY

PREDICTED AS

MANUALLY

ANNOTATED

STANCE

TWEETS THAT ARE

AUTOMATICALLY

PREDICTED AS

NEGATIVE 254 NEGATIVE 199 NEGATIVE 288 NEGATIVE 246

NEUTRAL 40 NEUTRAL 33

POSITIVE 15 POSITIVE 9

NEUTRAL 144 NEGATIVE 81 NEUTRAL 124 NEGATIVE 84

NEUTRAL 54 NEUTRAL 37

POSITIVE 9 POSITIVE 3

POSITIVE 84 NEGATIVE 51 POSITIVE 70 NEGATIVE 51

NEUTRAL 23 NEUTRAL 9

POSITIVE 10 POSITIVE 10 Table 12 - Comparison results sentiment analysis and stance detection

SENTIMENT Negative Neutral Positive STANCE Negative Neutral Positive

Precision 𝑝 = 0.6 𝑝 = 0.46 𝑝 = 0.29

Precision 𝑝 = 0.65 𝑝 = 0.47 𝑝 = 0.45

Recall

𝑟 = 0.78 𝑟 = 0.38 𝑟 = 0.12 Recall

𝑟 = 0.85 𝑟 = 0.30 𝑟 = 0.14

F1-score

f = 0.69 f = 0.42 f = 0.21 F1-score

f = 0.75 f = 0.39 f = 0.30

Accuracy 𝑎 = 0.55 Accuracy 𝑎 = 0.61

Table 13 - Precision, recall, F1-score and accuracy of sentiment and stance labels

As can be drawn from Table 12 and 13, the classifier shows very similar results for the

prediction of both sentiment and stance. The negative and neutral precision scores are very

similar to one another. The positive precision score, however, is remarkably higher for the

prediction of stance labels (0.45) than for the prediction of sentiment labels (0.29). Overall, it

Page 45 of 76

is remarkable that the negative labels are predicted best, followed by the neutral category and

then the positive category. A possible explanation for this could be the smaller number of

neutral and positive tweets in our corpus, due to which the machine learning system had more

had more training examples to be trained on for the negative tweets.

In comparison, our classifier performs slightly better on stance detection, considering the

accuracy score of 0.55 for sentiment analysis and 0.61 for stance detection.

4.3.5.2 Comparison accordance and the influence of irony

In this section, we will compare the accordance of the manually assigned sentiment labels with

their stance labels (cf. Table 14) to the accordance of the automatically assigned sentiment

labels with their stance labels (cf. Table 15). In the fourth and fifth column, the number of ironic

tweets per sentiment/stance category was listed with their percentage.

SENTIMENT

PREDICTIONS

(classifier)

STANCE PREDICTIONS

(classifier)

ACCORDANCE IN

PERCENTAGES

NUMBER

OF IRONIC

TWEETS

IRONY

IN

PERCENTAGES

NEGATIVE 331 NEGATIVE 302 91% 64 21%

NEUTRAL 22 7% 4 18%

POSITIVE 7 2% 1 14%

NEUTRAL 117 NEGATIVE 60 51% 15 25%

NEUTRAL 50 43% 9 18%

POSITIVE 7 6% 2 29%

POSITIVE 34 NEGATIVE 19 56% 6 32%

NEUTRAL 7 20% 1 14%

POSITIVE 8 34% 1 13% Table 15 - Accordance of automatically assigned sentiment labels with their stance labels

From Table 14, it can be concluded that 86% of all tweets with negative sentiment also hold

negative stance. Then, for tweets with neutral sentiment, the correspondence with neutral stance

is somewhat lower, with a percentage of 62. Moreover, in 51% of all positive tweets the author’s

stance towards the Brexit is also positive. It is remarkable that negative tweets rarely hold

Table 14 - Accordance of manually assigned sentiment labels with their stance labels

SENTIMENT

PREDICTIONS

(manual annotation)

STANCE PREDICTIONS

(manual annotation) ACCORDANCE IN

PERCENTAGES NUMBER OF

IRONIC

TWEETS

IRONY

IN

PERCENTAGES

NEGATIVE 254 NEGATIVE 218 86% 47 22%

NEUTRAL 19 7% 3 16%

POSITIVE 17 7% 1 6%

NEUTRAL 144 NEGATIVE 45 31% 19 42%

NEUTRAL 89 62% 19 21%

POSITIVE 10 7% 0 0%

POSITIVE 84 NEGATIVE 25 30% 8 32%

NEUTRAL 16 19% 4 25%

POSITIVE 43 51% 2 5%

Page 46 of 76

neutral or positive stance, whereas differences between sentiment and stance are more frequent

in the neutral and positive category. A possible explanation for this could be the presence of

irony, especially for tweets with neutral/positive sentiment and negative stance. This means

that, at first glance, a certain tweet seems positive/neutral, however, after the interpretation of

irony use, it becomes clear that the author’s stance is actually negative. An example of this

could be: “If I had a time machine, I'd happily take a molesting from Farage, just to Yewtree

the cunt out of any political standing. #Brexit”. The sentiment in this tweet is positive because

of his enthusiasm expressed in the word ‘happily’. However, the author presumably does not

literally mean he would violently attack Nigel Farage, if he had a time machine. The author is,

in other words, ironically discussing his negative attitude/stance towards the politician.

In addition, Table 15 shows similar trends in the results of the machine learning system. Yet,

the results returned by the classifier do differ in terms of accordance in percentages, with the

exception of the negative category. In our gold standard, the majority of all tweets with neutral

sentiment hold neutral stance, whereas the machine learning system returns a higher percentage

of neutral tweets holding negative stance. The same goes for the largest manually annotated

category of positive tweets with positive stance, where the classifier finds more positive tweets

holding negative stance. Concerning the effect of irony in Table 15, it can be observed that the

highest irony percentages pop up in the neutral and positive tweets with negative sentiment.

Therefore, this could mean that our machine is able to interpret irony correctly to some extent.

However, from chapters 4.3.2.1 and 4.3.4.1 we learnt that in some cases the system fails to label

sentiment or stance correctly as a result of ironic language use.

5. CONCLUSION

With the emergence of Web 2.0 at the beginning of the 21st century, easy-accessible

microblogging platforms such as Facebook and Twitter have become omnipresent within the

digital landscape. Blogs, forums and social media websites allow users to easily share their

point of view, by means of blogposts, reviews, reactions and ratings. Due to the high amount

of feedback or criticism on, for instance, products, services or political ideas there was a call

for an automatic system that could gather a whole range of opinions on a certain topic. With

the help of SA, a business or organization can find out which sentiment (positive, negative or

neutral) a piece of text contains. As discussed by Pak and Paroubek (2009), political parties and

Page 47 of 76

politicians can gain a perception of how people view their programmes or how people see them

within the political landscape. In addition, SD aims at discovering the point of view expressed

by the author towards the subject being discussed. Nowadays, organisations consequently have

enough feedback at their disposal to examine how people’s views differ on a certain product,

service or policy. As a result, it is easier to detect the reason behind, for example, the success

of a certain campaign or the low sales figures of a certain product.

In this study, we attempted to contribute to the existing findings on automatic sentiment analysis

and stance detection on Twitter. Furthermore, we aimed to explore whether ironic language

influences the performance of machine learning systems. For these purposes, we built a Twitter

corpus consisting of 482 political tweets with the hashtag Brexit. The manually annotated

corpus was then compared to both the predicted sentiment and stance labels. Based on the

collected and analysed data and experimental results, we attempted to draw conclusions

concerning the predictability of sentiment and stance, as well as the impact of irony on the

performance of automatic systems.

Firstly, we observed that our Twitter corpus consisted mostly of negative sentiment (53%),

followed by tweets with neutral (30%) and positive sentiment (17%). Similarly, the machine

learning system also mostly predicted tweets with negative sentiment (69%), then tweets with

positive sentiment (24%) and lastly positive tweets (7%). The topic that was discussed the most

was the Brexit itself, presumably because all tweets already contained “#Brexit”. Furthermore,

negative labels were predominant in nearly every topic (such as ‘Brexit’, ‘USA’, ‘Trump’,

‘other’, …), which was a logical consequence, considering the high percentages of negative

tweets. We noticed that our machine learning system for sentiment analysis was rather biased

towards negative tweets, meaning that the classifier tended to overgenerate negative labels. It

was observed that negative labels were attributed falsely in 34% of the cases, whereas neutral

and positive labels were only assigned falsely in correspondingly 13 and 5 percent of the cases.

This could be a possible explanation for the high precision, recall and F1-scores within the

negative category and the rather low results in both the positive and neutral category. Overall,

the system scored 0.55 on accuracy which is, considering the relatively small amount of tweets

in our corpus, a fairly good score. What remains a challenge is the detection of implicit

sentiment (e.g. ‘feeling poor’ is a negative feeling), as was already stated by Van Hee (2017).

Page 48 of 76

Secondly, the distribution of the stance labels was relatively similar to the percentages of

negative, neutral and positive sentiment labels. The gold standard consisted mostly of tweets

with negative stance (60%), followed by tweets holding neutral (26%) and positive (5%) stance.

Our machine learning system predicted that 79% of all tweets contained negative stance, 16%

neutral stance and 5% positive stance. Regardless of the lower precision and recall scores in the

neutral and positive stance categories, our system scored 0.61 on accuracy, which is slightly

better than the score for sentiment analysis. It was noticeable that the system again tended to

overgenerate negative stance labels, which would explain the rising of the percentage of

negative labels predicted by the machine learning system with nearly a fifth. Furthermore, it

became clear in the analysis of the content of the tweets that the system failed to interpret

specific references to, for instance, movies correctly.

Regarding the accuracy results of our sentiment analysis (0.55) as well as stance detection

(0.61), our first hypothesis can be confirmed: the machine learning system delivers fairly

reliable results on a political Twitter corpus.

Thirdly, we considered the impact of the 103 ironic tweets in our corpus on the predictability

of both sentiment and stance. It appeared that true positives as well as false positives contained

similar irony percentages. As a result, it was rather hard to interpret whether irony influenced

the outcome of the sentiment analysis or stance detection, without evaluating the content of the

falsely predicted labels. Considering the content of all falsely labelled tweets, it could, however,

be concluded that in some cases irony was interpreted literally. In other words, irony did have

a certain negative effect on the performance of the machine learning system, but not in all cases.

As a result, our second hypothesis, which said that our machine learning system would not be

able to detect and interpret ironic language use correctly in most cases, cannot be confirmed

entirely.

In conclusion, it can be stated that a machine learning approach for sentiment analysis and

stance detection is already seemingly reliable. However, there is still room for improvement,

considering both accuracy scores are under 65%. Furthermore, we need to tackle the challenge

and indistinctness concerning ironic language in tweets. In chapter 6, suggestions on further

research will be formulated.

Page 49 of 76

6. LIMITATIONS AND FURTHER RESEARCH

Overall, we found that our system for sentiment analysis and stance detection scores fairly well,

however, there is still room for improvement. Therefore, we will discuss the limitations we

encountered during our research and we will provide some suggestions for further research in

this section.

We chose to build a corpus consisting of more or less 500 political tweets. Due to practical

limitations, the corpus was limited in size compared to other studies using Twitter corpora (cf.

Van Hee, 2017). This resulted in less reliable scores and percentages to interpret or to draw

conclusions on. Moreover, we had no insight into which sentiment words were or were not

detected by the machine learning system. Further research could, however, provide us with

more insight for the error analysis. Furthermore, it would be interesting to compare a machine

learning approach to a lexicon-based approach, to explore which approach generates the best

results.

In further research on sentiment analysis and stance detection on a Twitter corpus, it would also

be interesting to further explore the consequences of irony presence in political tweets. The

reason for this is twofold. Firstly, it remained relatively unclear in this research to which extent

irony influences the predictability of sentiment and/or stance. Secondly, the combination of

sentiment/stance on Twitter and irony has been rarely been investigated, which offers many

possibilities for improvement. Moreover, it would also be useful to perform irony detection on

the Twitter corpus itself, as was done by Van Hee (2017). In this study, we were not able to

automatically extract irony labels and thus, we could not analyse the effect of ironic language

thoroughly. For further research, it would be interesting to compare the results of, for example,

stance detection as well as irony detection.

Page 50 of 76

List of references

Bambrick, N. (24.06.2016). Support Vector Machines for dummies; A Simple Explanation.

Retrieved February 20, 2018 from http://blog.aylien.com

Barbieri, F., & Saggion, H. (2014). Modelling Irony in Twitter. In EACL (pp. 56–64).

Boertjes, E., Jamakovic A. & Raaijmakers S. (2011). Analyse van sociale media. STAtOR,

November, 3-4, 12, (pp. 41-47).

Cambria, E., Schuller, B., Xia, Y., & Havasi, C. (2013). New avenues in opinion mining and

sentiment analysis. IEEE Intelligent Systems, 28(2), 15–21.

Chang, C., Lin, C. (2016). LIBSVM: A Library for Support Vector Machines. Retrieved February

20, 2018 from https://www.csie.ntu.edu.tw/~cjlin/libsvm/

Dave, K., Lawrence S. & Pennock D.M. (2003). Mining the peanut gallery: Opinion extraction and

semantic classification of product reviews. In Proceedings of the 12th international conference

on World Wide Web(WWW), (pp. 519–528).

De Clercq, O., Lefever, E., Jacobs, G., Carpels, T., & Hoste, V. (2017). Towards an integrated

pipeline for aspect-based sentiment analysis in various domains. Proceedings of the 8th

Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media

Analysis (pp. 136–142). Presented at the EMNLP2017, Association for Computational

Linguistics.

Desmet, B., De Clercq, O., Van de Kauter, M., Schulz, S., Van Hee, C., & Hoste, V. (2014).

Taaltechnologie 2.0: sentimentanalyse en normalisatie. In Beschouwingen uit een talenhuis :

opstellen over onderwijs en onderzoek in de vakgroep Vertalen, Tolken en Communicatie

aangeboden aan Rita Godyns, (pp. 157–161).

Ghosh, A., Li, G., Veale, T., Rosso, P., Shutova, E., Barnden, J., & Reyes, A. (2015). Semeval-2015

task 11: Sentiment analysis of figurative language in twitter. In Proceedings of the 9th

International Workshop on Semantic Evaluation (SemEval 2015) (pp. 470–478).

Page 51 of 76

Karoui, J., Farah, B., Moriceau, V., Patti, V., Bosco, C., & Aussenac-Gilles, N. (2017). Exploring

the impact of pragmatic phenomena on irony detection in tweets: A multilingual corpus study.

In Proceedings of the 15th Conference of the European Chapter of the Association for

Computational Linguistics: Volume 1, Long Papers (Vol. 1, pp. 262–272).

Khan, A. Z., Atique, M., & Thakare, V. M. (2015). Combining lexicon-based and learning-based

methods for Twitter sentiment analysis. International Journal of Electronics, Communication

and Soft Computing Science & Engineering (IJECSCSE), 89.

Kolkur, S., Dantal, G., & Mahe, R. (2015). Study of different levels for sentiment analysis.

International Journal of Current Engineering and Technology, 5(2), 768Ű770.

Kreuz, R. J., & Roberts, R. M. (1993). On satire and parody: The importance of being

ironic. Metaphor and Symbol, 8(2), 97-109.

Kulshrestha, J., Kooti, F., Nikravesh, A., & Gummadi, P. K. (2012, June). Geographic Dissection of

the Twitter Network. In ICWSM.

Kumar, A., & Sebastian, T. M. (2012). Sentiment analysis on twitter. IJCSI International Journal of

Computer Science Issues, 9(3), 372–378.

Liu, B. (2012). Sentiment Analysis and Opinion Mining. Synthesis lectures on human language

technologies, 5(1), 1-167.

Mandya, A., Siddharthan, A., & Wyner, A. (2016). Scrutable feature sets for stance classification.

ACL 2016, 60.

Mohammad, S., Kiritchenko, S., Sobhani, P., Zhu, X.-D., & Cherry, C. (2016). SemEval-2016 Task

6: Detecting Stance in Tweets. In SemEval@ NAACL-HLT (pp. 31–41). Retrieved from

http://www.aclweb.org/anthology/S/S16/S16-1003.pdf

Nasukawa, T., & Yi, J. (2003). Sentiment analysis: Capturing favorability using natural language

processing. In Proceedings of the 2nd international conference on Knowledge capture (pp. 70-

77). ACM.

Page 52 of 76

Nielsen, F. (2011). IMM Publications: AFINN. Retrieved February 20, 2018 from

http://www2.imm.dtu.dk/

Pak, A., & Paroubek, P. (2010). Twitter as a corpus for sentiment analysis and opinion mining. In

LREc (Vol. 10).

Pang, B., & Lee, L. (2008). Opinion mining and sentiment analysis. Foundations and Trends. In

Information Retrieval, 2(1–2), 1-135

Pang, B., Lee, L., & Vaithyanathan, S. (2002). Thumbs Up?: Sentiment Classification Using

Machine Learning Techniques. In Proceedings of the ACL-02 Conference on Empirical

Methods in Natural Language Processing - Volume 10 (pp. 79–86). Stroudsburg, PA, USA:

Association for Computational Linguistics. doi: 10.3115/1118693.1118704

Pavlopoulos, I. (2014). Aspect based sentiment analysis. Athens University of Economics and

Business.

Pontiki, M., Galanis, D., Papageorgiou, H., Androutsopoulos, I., Manandhar, S., AL-Smadi, M., …

others. (2016). SemEval-2016 task 5: Aspect based sentiment analysis. In ProWorkshop on

Semantic Evaluation (SemEval-2016) (pp. 19–30). Association for Computational Linguistics.

Retrieved from https://biblio.ugent.be/publication/8131987

QuantStart Team. (12.09.2014). Support Vector Machines: A Guide for Beginners. Retrieved

February 20, 2018 from https://www.quantstart.com/

Reyes, A., Rosso, P., & Buscaldi, D. (2012). From humor recognition to irony detection: The

figurative language of social media. Data & Knowledge Engineering, 74, 1–12.

Reyes, A., Rosso, P., & Veale, T. (2013). A multidimensional approach for detecting irony in twitter.

Language Resources and Evaluation, 47(1), 239–268.

Shoeb, M., & Ahmed, J. (2017). Sentiment Analysis and Classification of Tweets Using Data

Mining. Work, 4(12).

Page 53 of 76

Taboada, M. (2016). Sentiment Analysis: An Overview from Linguistics. Annual Review of

Linguistics, 2. doi:10.1146-011415-040518

Tumasjan, A., Sprenger, T. O., Sandner, P. G., & Welpe, I. M. (2010). Predicting elections with

twitter: What 140 characters reveal about political sentiment. Icwsm, 10(1), 178–185.

Van de Kauter, M. (2013). LeTs preprocess: the multilingual LT3 linguistic preprocessing toolkit.

Van Hee, C. (2017). Can machines sense irony? : exploring automatic irony detection on social

media. Ghent University. Faculty of Arts and Philosophy, Ghent, Belgium.

Van Hee, C., Lefever, E., & Hoste, V. (2015). LT3: sentiment analysis of figurative tweets: piece of

cake #NotReally. In SemEval : 9th International Workshop on Semantic Evaluations at Naacl

2015, Proceedings (pp. 684–688). Association for Computational Linguistics. Retrieved from

http://hdl.handle.net/1854/LU-6827146

Van Hee, C., Van de Kauter, M., De Clercq, O., Lefever, E., & Hoste, V. (2014). LT3: sentiment

classification in user-generated content using a rich feature set. In Proceedings of the 8th

International Workshop on Semantic Evaluation (pp. 406–410). Association for Computational

Linguistics. Retrieved from http://hdl.handle.net/1854/LU-5684543

Wang, P.-Y. A. (2013). # Irony or# Sarcasm—A Quantitative and Qualitative Study Based on

Twitter. In Proceedings of the 27th Pacific Asia Conference on Language, Information, and

Computation (PACLIC 27) (pp. 349–356).

Zhang, L., Ghosh, R., Dekhil, M., Hsu, M., & Lius, B. (2011). Combining Lexicon-based and

Learning-based Methods for Twitter Sentiment Analysis. In Hewlett-Packard

Labs Technical Report HPL-2011-89.

Page 54 of 76

APPENDIX 1

Brexit - Brexit

- Consequence Brexit

- Ignorance about Brexit

- Joke on Brexit

- Link to article on Brexit

- Link to video on Brexit

- Second referendum

- Voters

Celebrities - Angela Merkel

- Boris Johnson

- Boris Johnson + link to article

- Celebrities

- Celebrities & link to article on Brexit

- Celebrities (Sarah Palin)

- David Cameron

- Hilary Clinton

- Lindsey Lohan

- Nicola Sturgeon

- Obama

- Thatcher

Economy - Economy

- Economy

- Voters & economy

EU - EU

- EU & Brexit

- EU & NATO

- EU & UN

Other - Article

- British housing

- Citizenship

- British politics

- Consequences & voters

- Democracy

- Education

Page 55 of 76

- Football

- German Foreign Office

- Greece

- Housing

- Immigration

- International politics

- Japan

- Joke

- Leftists

- Link to an article

- Monarchy

- Other

- Personal information

- Petition

- Press

- Racism

- Refugees

- Science

- Scotland and Ireland

- Sore losers

- Texas

- UN

- US housing

- Work opportunities

Scottish referendum - Economy & Scottish referendum

- Scottish referendum

- Scottish referendum + link to article

Trump - Brexit & Donald Trump

- Brexit & Trump

- Donald Trump

- Donald Trump + Texas

- Insults about Donald Trump

- Trump

- USA & Trump

USA - America

- American elections

Page 56 of 76

- Brexit & US

- US

- USA

Page 57 of 76

APPENDIX 2

Tweet senti

ment

topic/aspec

t

stan

ce

tow

ards

the

Bre

xit

iro

ny

Senti

ment

predic

tions

Stanc

e

predic

tions

$2.7 TRILLION lost on global markets after

@RupertMurdoch has his #Brexit dreams come true. People

will die from impacts of such losses.

negat

ive economy

nega

tive 2 2

@sorrelita "protection!? Protection of what!?

#TzeGermans!?" #Brexit #funfacts

neutr

al other

nega

tive 2 3

Final #Brexit tally is in: 48% Sense and Sensibility, 52%

Pride and Prejudice.

neutr

al other

nega

tive 2 2

Do we even care? – #Brexit aftermath

http://snowcalmth.online/dowecare/ Overview: - Blogposts

delayed. - The Forgotten #Youth - Do we know the #EU?

negat

ive other

nega

tive 3 3

#Brexit, Monty Python & Silly Walks. @NewYorker neutr

al other

nega

tive 2 3

Proud of #Britain and @David_Cameron for doing the

right thing today. #Brexit #EUref #independence #freedom

positi

ve Brexit

posit

ive 2 2

@stuartpstevens @AshleyRParker I didn't know about

#brexit either just some gun control stunt

negat

ive Brexit

neut

ral 2 2

"I told y'all to vote REMAIN tho" #Brexit negat

ive Brexit

posit

ive 3 3

"The World Turned Upside Down." Not necessarily the

song I wanted to be humming this morning. #Brexit

negat

ive Brexit

nega

tive 1 1

They want to put all of US to the back of the bus to their

global masters. We say...piss off! #Brexit #Trump2016

negat

ive USA

posit

ive 2 1

said of #Brexit, while in Scotland IN SCOTLAND!!!!! negat

ive Trump

nega

tive x 1 2

Why are #liberal #democrats all over TV crying about

#Brexit and the markets? Didnt you people want the big banks

to lose money and power?

negat

ive economy

nega

tive x 2 2

Boris Johnson goes from court jester to crown prince after

#Brexit win http://bloom.bg/28RCZLw

neutr

al

celebrities/p

oliticians

neut

ral 3 3

Love being in Sweden, Bruce in every bar #priceyoupay

#sweden #Brexit

positi

ve other

neut

ral 2 2

Breaking!!! #UK votes to ease epcot!! #BrexitVote

#Brexit #BrexitHumor #tcot #RedNationRising

neutr

al Brexit

neut

ral

1 3

Lindsay Lohan fumes over #Brexit, Elizabeth Hurley

sleeps soundly

neutr

al

celebrities/p

oliticians

neut

ral

2 2

@torontodan As the final lines of the old Presbyterian

joke go: "Lord, Lord, we didna ken". "Weel, Weel, ye ken

noo".#Brexit

negat

ive Brexit

nega

tive x 2 2

Why #Brexit is terrible for UK science, in one map

http://www.economist.com/news/britain/21699504-most-

scientists-want-stay-eu-european-experiment …

negat

ive Brexit

nega

tive 2 2

.@piersmorgan - I always thought #Brexit was that good

dump you take after morning coffee. TOTALLY confused. Pls

call me.

negat

ive Brexit

nega

tive x 2 2

Ahead on New York Tonight: Leaders react to #Brexit.

@POTUS designates @TheStonewallNYC as nat'l monument,

and a preview of #NYCPrideMarch

neutr

al Brexit

neut

ral 2 2

Calls for a second Scottish independence referendum

grow louder after #Brexit http://econ.st/28XxXhn

neutr

al

Scottish

referendum

nega

tive 3 3

The nightmare of the #EU & #UN Elites is for these two

nations to rise up and say ENOUGH! #Brexit #America

#Britain

negat

ive EU

nega

tive 2 2

I leave Twitter for a week and #Brexit happens. neutr

al Brexit

neut

ral 3 3

Page 58 of 76

You know what's "bizarre"? Media folks who see the

#Brexit vote & take virtually no relevant lesson from it.

negat

ive Brexit

nega

tive 2 2

Watch Christiane Amanpour Get ANGRY That Britain

ALLOWED The People To Vote On #Brexit

negat

ive Brexit

nega

tive 2 2

.@LizClaman: “The losses on paper are now tallying

$900 billion on the U.S. stock market.” #Greta #Brexit

negat

ive economy

nega

tive 2 2

No: History will show that #brexit Is good for nobody negat

ive Brexit

nega

tive 3 3

Take Note! Here in the States this would be the equivelant

of “I thought it wud be funny to vote for Trump" #Brexit

negat

ive USA

nega

tive 2 2

Shall we start preparing EU shores for an influx of British

refugees? #brexit

negat

ive Brexit

nega

tive x 3 2

I'm going to go with #brexit damage control for $1000

Alex!

positi

ve economy

nega

tive 2 2

SG #NATO Sees Unifying Role as #UK #Brexit Shakes

#Allies:#Britain’s vote to leave #EU leaves Europe more

fragmented.

neutr

al EU

nega

tive 2 2

Lindsay Lohan passionately expressed her stance against

#Brexit in now-deleted tweets http://peoplem.ag/ullfemm

neutr

al

celebrities/p

oliticians

neut

ral 1 3

Celebrating #Brexit with the British Players in beautiful

Kensington, MD.

positi

ve Brexit

posit

ive 3 2

Odd how some on the right are in despair over #Brexit.

Socialist globalization is no way to run a planet, people.

negat

ive Brexit

nega

tive 2 2

Poor folks voting for #Brexit is the equivalent of a Turkey

voting for Thanksgiving. White Nationalists can never be

accused of rationality.

negat

ive Brexit

nega

tive x 2 2

Oh, God, they're giving the keys for the Tridents to BoJo

the Clown... #Brexit

negat

ive

celebrities/p

oliticians

nega

tive 2 2

Britons voted to strengthen their borders. Will you do the

same in November? #Brexit

positi

ve USA

posit

ive 3 2

Wales should have been more careful what it wished for.

It's going to be given it. #Brexit

negat

ive Brexit

nega

tive 3 2

And suddenly the birds are singing.....still glued to the TV

though #Brexit

positi

ve Brexit

posit

ive 3 2

The worst has yet to come #sarahpalin #brexit

#exitstupidity

negat

ive Brexit

nega

tive

2 2

#Blairites using #Brexit to (yet again) try & unseat

#Corbyn proof of their lack of allegiance to Labour.

negat

ive

celebrities/p

oliticians

neut

ral 2 2

Hey United Kingdom imma let you finish but America

had one of the greatest #Brexit's of ALL TIME

positi

ve USA

neut

ral x 1 3

Good branding: #Brexit, how "attractive" is the name! +

Good hype: Thanks to #SocialMedia. + Intense emotion: Fear

= Results = #Marketing101

positi

ve Brexit

posit

ive x 2 2

Really interesting piece on the #Brexit where the fragility

of masculinity surfaces again.

positi

ve Brexit

neut

ral 2 2

.@jasonrileywsj: EU needed Britain more than Britain

needed the EU -OTR #greta #Brexit #PoliticalPanel

@FoxNews

neutr

al EU

posit

ive 2 2

Don’t think #brexit is a big deal? Here’s a chart. negat

ive Brexit

nega

tive 2 3

Everyone should be worried about a Trump #Drumpf

presidency! Now #Brexit! World heading in an interesting

direction

negat

ive Trump

nega

tive 2 2

#Brexit threatens damage to U.S.-UK ties, could

embolden Russia's Putin/@mattspetalnick @yarabayoumy

negat

ive Brexit

nega

tive 2 2

#Academics fear new #Brexit – a brain exit – after

#referendum vote

http://www.independent.co.uk/news/science/brain-drain-brexit-

universities-science-academics-referendum-eu-

a7100266.html … #EURefResults

negat

ive

Brexit

nega

tive 3 3

The #Brexit was necessary. The EU was turning into that

evil corporation from the Aliens franchise.

positi

ve EU

posit

ive 2 2

GBP weakens, world markets fall, housebuilders shares

drop 25%. Scottish referendum 2.0? Youth vote ignored.

Cameron resigns. #Brexit #Day1

negat

ive Brexit

nega

tive 2 2

Page 59 of 76

The @realDonaldTrump right again. #Brexit #VoteLeave

#UKIP #UK #MakeAmericaGreatAgain #TrumpTrain

positi

ve Trump

posit

ive 3 1

With all the hysteria and over the top hyperbole over

Brexit, maybe we should all just take a deep breath and relax

for a few days. #Brexit

neutr

al other

neut

ral 3 2

Lonng #Brexit day done, let's get back to #IREvFRA!!

#Remain #EURO2016

positi

ve other

posit

ive 2 2

Brexit vote: Anger in the bedroom, joy on the streets

http://cnn.it/28V6bCD #Brexit #seeEUlater

neutr

al Brexit

neut

ral x 2 3

Trump/Sanders #Brexit philosophy: Seize corps overseas

ops & ban foreign sales. HUGE govt! @SMShow

@SueinRockville

neutr

al USA

neut

ral 2 2

Can Nicola Sturgeon get a Faroes-style opt out from

#Brexit? And might that persuade Scots we r ready 4

#indyref2?

neutr

al Brexit

neut

ral x 3 2

On my room balcony posing with the freshly brexited EU

flag. Giving it some company. #iifa2016 #iifadiaries #brexit

positi

ve Brexit

posit

ive 2 3

Do those who claim it is "stupid" to propose Esperanto as

official for EU after #Brexit even know anything about the

language?

negat

ive other

neut

ral x 2 2

Still so sad about #Brexit. What is this dark, absurd future

being carved out for the world?

negat

ive Brexit

nega

tive 2 2

If I had a time machine, I'd happily take a molesting from

Farage, just to Yewtree the cunt out of any political standing.

#Brexit

negat

ive Brexit

nega

tive x 1 2

@charlescwcooke and I disagree on #Brexit. But I love

his British understatement.

negat

ive Brexit

nega

tive 1 2

your country is truly inspiring thank you #Brexit positi

ve Brexit

posit

ive 2 2

A lot of the #Glastonbury audience are wearing either

#Hibs tops, #sunglasses or #SantaHats - I am confused in this

post-#brexit world.

negat

ive other

nega

tive 2 2

Briton on FB claims his country was "raped" - you loose a

referendum and your country was raped? #brexit #democracy

negat

ive Brexit

posit

ive

2 2

Great Britain secedes from the European Union. Millions

in the EU now looking for real jobs. #Brexit

neutr

al EU

nega

tive

2 2

Massive props to David Cameron for giving Poms chance

to decide own fate, honourably stepping aside when result

against him. #brexit

positi

ve celebrities/p

oliticians

neut

ral 2 2

#Brexit no, Corbyn has a great deal to answer for as do

many others, its not just about the Tories however unpopular

that view is.

negat

ive celebrities/p

oliticians

nega

tive 2 2

The Working Classes will be the first to be shunted! So

short sighted! #Brexit #shouldhavegonetospecsavers

#universityofJeremyKyle

negat

ive Brexit

nega

tive 2 2

business: Boris Johnson goes from court jester to crown

prince after #Brexit win http://bloom.bg/28RCZLw

neutr

al

celebrities/p

oliticians

neut

ral 3 3

I never thought of Britain as being European anyway.

#Brexit

negat

ive EU

posit

ive 2 2

Instead of posting hilarious gifs maybe the #remain side

should start thinking about the future of this country outside of

the EU #Brexit

negat

ive Brexit

nega

tive 2 2

"World's 400 Richest Lose $127 Billion" It's happening!

#Brexit #LeaveWins #FirstHubrisThenNemesis

http://bloom.bg/28TnoIm

neutr

al economy

neut

ral 3 2

<em>The Atlantic</em> Daily: The Great British Break-

Off #brexit http://brexitwhatnext.com/2016/06/emthe-

atlanticem-daily-the-great-british-break-off/ …

neutr

al Brexit

neut

ral x 3 3

#Trump reaction to #Brexit: 1) See, they like my wall

plan too, and 2) I'll make $$ off the tanking pound. #prick

negat

ive Trump

nega

tive 2 2

Only 72% voter turnout to a decision that literally

changed the entire economy. #WakeUpPeople #Brexit

negat

ive economy

nega

tive 2 2

Do you think Boris Johnson will become the next Prime

Minister of Great Britain?

http://americansdecide.com/topic/do-you-think-boris-johnson-

neutr

al celebrities/p

oliticians

neut

ral 2 2

Page 60 of 76

will-become-the-next-prime-minister-of-great-britain/ …

#Brexit

Is this going to effect my chances of getting into

Hogwarts? #Brexit

neutr

al other

nega

tive x 2 2

This is one factor of the #Brexit vote. Lot more

complications than this.

negat

ive Brexit

nega

tive 2 2

' See EU Later 'was my best headline on #Brexit positi

ve other

nega

tive 2 3

#Calls For #Texas #Independence #Surge In #Wake Of

#Brexit #Vote - http://www.angrysummit.com/calls-for-texas-

independence-surge-in-wake-of-brexit-vote …

neutr

al other

neut

ral 3 3

Even his wife doesn't believe him...just look at her

expression. This man is a pariah #DavidCameron #Brexit

negat

ive

celebrities/p

oliticians

nega

tive 2 2

Assume everything said by politicians lawyers &

corporate moguls is a lie until U see credible proof. #Brexit

negat

ive Brexit

nega

tive 2 2

Not liking this because it's exactly what I feared #Brexit

lot have led everyone into #neverneverland OMG

negat

ive Brexit

nega

tive x 2 2

Interesting how we act on uncertainty when things are

under control. Panic creates crisis.. instead of keeping a cool

head #Brexit aftermath

negat

ive Brexit

nega

tive 2 2

The UK mucked up big time and the UKIP still wants

access to the EU market? Lmao what a joke #Brexit

#EURefResults #UKreferendum

negat

ive economy nega

tive

2 2

Man, that was so cool. #Brexit positi

ve Brexit

posit

ive

2 2

Putting himself 1st, Trump says #Brexit will help HIS

resort: “When the down pound goes down, more people are

coming to Turnberry, frankly" - Why…because now they have

to swim the #EnglishChannel to invade Britain?

negat

ive

Trump

nega

tive x 2 2

Sooo like everyone today, I logged in to check my 401k

because #Brexit...expecting it to be but is..anyone want to

explain?

neutr

al economy

neut

ral 2 2

Ya fucked up #Brexit negat

ive Brexit

nega

tive 2 2

HAHAHA! Everyone's trying to move here (Canada),

now! #Brexit #BrexitOrNot #BelieveItOrNot

positi

ve Brexit

nega

tive 2 2

This. #Brexit neutr

al Brexit

neut

ral 2 2

Well #Brexit happened. Here's a look at the possible

impacts on UK #RealEstate.

http://www.forbes.com/sites/carlapassino/2016/06/24/will-the-

uks-real-estate-sector-survive-brexit/#1427abb22120 …

#pound

neutr

al

Brexit

neut

ral 3 2

#Brexit should be a wake-up call to US #liberals: don’t

assume Drumpf will lose

http://www.vox.com/2016/6/24/12023816/brexit-donald-

trump-

winning?utm_campaign=vox&utm_content=article%3Afixed

&utm_medium=social&utm_source=twitter … via

@voxdotcom #Worried

negat

ive Trump

nega

tive

2 2

Hozier calls #Brexit a "massive betrayal": "My heart

breaks" http://blbrd.cm/iAA1ss pic.twit...

http://bit.ly/296vVcl #ShowTime

negat

ive

celebrities/p

oliticians neut

ral

3 3

What does #Brexit mean to the San Francisco housing

market? Read more from @PacUnion Chief Economist Selma

Hepp:

neutr

al Brexit neut

ral

2 2

Proud to be #Brexit! Proud to stand alone in a world

where most are too scared to be alone, to have their own

opinion. Proud to me! #Ukref

positi

ve Brexit posit

ive 2 2

Repeated refrain re #Brexit -Elections have consequences

-remember that in November folks #NeverTrump

#Election2016

negat

ive USA nega

tive 2 2

If you care about our future join 450,000 people

petitioning parliament for a 2nd referendum

http://www.independent.co.uk/news/uk/brexit-petition-for-

negat

ive Brexit nega

tive 2 2

Page 61 of 76

second-eu-referendum-so-popular-the-government-sites-

crashing-a7099996.html# … #Brexit

"Drunk Shakespeare, probably the only proper activity

after #Brexit https://www.instagram.com/p/BHDnj20jXxO/

negat

ive Brexit

nega

tive x 2 2

Can anyone tell me what's the status of eu member state

citizens now residing in the Uk? Illegal? Visa? #Brexit

#BrexitVote #Lexit #Leave

neutr

al Brexit nega

tive x 3 2

This #Brexit thing has me really worked up. It doesn't

bode well for the U.S. staying in the EU.

negat

ive Brexit

nega

tive 2 2

After #Brexit, another EU is possible with UK, Norway

and Switzerland.

neutr

al Brexit

nega

tive x 3 3

I think is the natural progression of human society to

become more integrated as time advances. #Brexit is the old

world fighting back.

negat

ive Brexit nega

tive 2 2

Why the surprise over #Brexit? This is the same country

that threw Churchill out of office after he pulled their nuts out

of the fire in WW2

neutr

al Brexit nega

tive 2 3

Secede and keep seceding, don't stop until you get to the

individual! #Brexit

negat

ive Brexit

nega

tive x 2 2

If u are rich & white- yup! Here is #Brexit promise

walked back hours after count

negat

ive Brexit

nega

tive 2 2

Brits will be poorer because #brexit but don't have to

weigh bananas with #metric system. A wise people, indeed.

@ilduce2016 @billmaher

negat

ive Brexit nega

tive 2 2

#Brexit: Do you suspect that the new negotiated treaties

will replicate membership in the European Union?

#worldismovingtoofastdepartment

negat

ive Brexit nega

tive 2 2

Traditionalist Catholic blog: The Filioque Clause.

http://www.stuart-filioque.blogspot.com #PopeFrancis #Brexit

neutr

al other

neut

ral 2 2

This is not like Idiocracy. In Idiocracy, once they found a

smart person, they made him fix their problems. #Brexit

negat

ive Brexit

nega

tive x 2 2

Nice attempt at making shit up! #Brexit #abc7chicago

#iteam #millennials ??

negat

ive Brexit

nega

tive 3 2

We won't win Eurovision for 69 years #EUref #Brexit negat

ive Brexit

nega

tive x 2 2

Beneath the cross of Jesus, His family is my own. #Brexit

#EUref

neutr

al other

neut

ral 2 2

Britons seek to 'move to Canada' after #Brexit vote neutr

al Brexit

nega

tive 2 2

A very good cereal served at my amazing property in

Turnberry! #Brexit of Champions, just like me! Enjoy!

positi

ve other

posit

ive 2 2

I'd like bier, croissant and wusrt please. What's the tinned

stuff? spam? #Brexit #EURefResults #WhatHaveWeDone

negat

ive Brexit

nega

tive 3 2

I've never seen Americans talk about Britain on my

Twitter feed before. And they're all taking the piss #Brexit

negat

ive Brexit

nega

tive 2 2

I did speak out on the positives of a sensible #Brexit based on

democratic process. But this is just populist bollocks.

negat

ive Brexit

nega

tive 2 2

Attention fellow Scots!!! It's ok!! I've had an idea!! We

can build a wall... #Brexit #remain

#MakeDonaldDrumpfAgain

negat

ive other nega

tive 2 2

I dare not go to YouTube for #Brexit videos. Can you

imagine the dross?

negat

ive Brexit

nega

tive 2 2

Learning some great new swears, thanks Scotland!

#Brexit

positi

ve other

nega

tive 1 2

#BrExit is exercise of the most important check on elite

mismanagement - the peoples' power to vote for HORRIBLE

IDEAS.

negat

ive Brexit nega

tive 2 2

#Regrexit: Speculation grows on the options for #Brexit

actually NOT happening.

https://waitingfortax.com/2016/06/24/when-i-say-no-i-mean-

maybe/ …

negat

ive Brexit

neut

ral 2 2

The U.K. could really use a more robust system of

excheques & balances #Brexit #BrexitVote

negat

ive Brexit

posit

ive 3 3

#Brexit Is Sending Markets Diving. #Twitter Could Be

Making It Worse http://dlvr.it/Lf6Zcl #Wired

negat

ive economy

nega

tive 2 3

Page 62 of 76

Brought some humor to the situation @camilluddington

#Brexit

positi

ve other

nega

tive 2 2

I've changed my mind. #Brexit is good. positi

ve Brexit

posit

ive 3 3

Brought some humor to the situation #BrexitVote #Brexit

#BrexitHumor #tcot #RedNationRising

positi

ve other

nega

tive 3 2

Okay can someone explain #brexit to me neutr

al Brexit

neut

ral 2 3

The French now want to move the Calais 'jungle' migrant

camp to British soil after #BREXIT

neutr

al Brexit

nega

tive 2 3

Patsy & Edina totally voted Remain. #Brexit negat

ive Brexit

nega

tive x

3 1

Mass referendums at their best Brits don't know what they

voted for #Brexit #EU #EuropeanUnion

#DavidCameronResigns

negat

ive Brexit nega

tive

2 2

#Brexit: the day rational choice theory blew up into

thousand pieces

negat

ive Brexit

nega

tive 3 2

Churchill said, "Heroes fight like Greeks". Like a Greek i

have to say that "Heroes, vote like British!" #Brexit

positi

ve Brexit

posit

ive

2 2

Can we have a redo? Where is the reset button? #Brexit negat

ive Brexit

nega

tive

2 2

#Brexit – The New Modern-Nationalism is #Global

#Governance - http://www.angrysummit.com/brexit-the-new-

modern-nationalism-is-global-governance …

#ModernNationalism

neutr

al Brexit

neut

ral 2 3

WaPo: #Brexit vote sends a message to politicians

everywhere: It can happen here

neutr

al Brexit

nega

tive

2 2

Now keep the promise of £350m a week for our #NHS -

Sign the petition: #EuRef #Leave #Brexit

https://you.38degrees.org.uk/petitions/invest-ps350-million-

saved-from-eu-in-nhs-by-2018?bucket=fb&source=twitter-

share-button … via @38_degrees

neutr

al Brexit

posit

ive

3 1

A big day with #Brexit, but I made history too when I

bent over to tie my shoe at the gym and a guy rushed over to

ask if I was OK. #only42!

negat

ive other nega

tive x 1 2

The pound goes down and so do stocks #Brexit negat

ive economy

nega

tive 3 3

Last chance to vote: Is #Brexit good for

@realDonaldTrump? Tweet YES OR NO using #greta

@FoxNews

neutr

al USA neut

ral 2 2

Leftists are freaking out over the #Brexit. Why, because

the people finally rejected tyranny? Just proves: liberals =

tyrants. #tcot

negat

ive Brexit posit

ive 2 2

#Brexit #Leave Please, welcome a #Britishrefugee neutr

al Brexit

nega

tive 2 2

Congrats to the UK for #Brexit positi

ve Brexit

posit

ive 2 2

Last in first Out #brexit #uk #eu #eng neutr

al Brexit

neut

ral 3 3

@JaneNormanInt Need to make my order now when it is

still possible before #Brexit. Any plans to move your office to

#EU? #onlineordering

neutr

al other neut

ral 3 2

ISIS takes credit for every terrible thing that happens on

earth, but even they're saying today "don't hang that #Brexit

crap on us!"

negat

ive Brexit nega

tive x 2 2

#Brexit Well, that required an active stupidity that rivals negat

ive Brexit

nega

tive x 2 2

#Brexit’s uncertainty in Europe will ripple back to Central

Texas http://atxne.ws/28T4wvL

negat

ive other

nega

tive 3 3

"Migration isn't the underlying cause of the thrust towards

#Brexit. Austerity is." - @yanisvaroufakis

negat

ive Brexit

nega

tive 2 2

UK want to leave the republic...aaahm eu? First thing in

my mind is a clone army #StarWars #Brexit #justkidding #sad

but that's #democracy

neutr

al Brexit neut

ral x 3 1

#Brexit: Up until midnight last night #voteremain was

leading on social media: http://brnw.ch/28Tnrr1

neutr

al Brexit

neut

ral 2 2

Page 63 of 76

Beginning of the end for the European Union: Best précis

of impact of #Brexit I've read to date. #corpgov #strategy

negat

ive Brexit

nega

tive 2 2

Hillary Clinton urges 'experienced leadership' after

#Brexit from #EU http://goo.gl/fb/0eksvp #europe

#europeanunion

neutr

al Brexit neut

ral 3 2

#Brexit could break up #EU and #NATO, prevent World

War III: Paul Craig Roberts http://goo.gl/fb/Ihi2EQ #europe

negat

ive Brexit

nega

tive

3 2

Britain: Let's grab a pint. EU: No thanks. I don't drink

during the day. #Brexit

negat

ive Brexit

nega

tive x

2 2

#Brexit, the political equivalent of cat videos negat

ive Brexit

nega

tive x 2 2

JPMorgan staff memo from Jamie Dimon, others about

Brexit referendum vote #Brexit #JPMorgan #UKref

neutr

al Brexit

neut

ral 2 2

Scotland, Wales, & London voted to #Remain, everyone

else voted for #Brexit. #Texas wants to #Secede. Can we trade

Texas for those first 3?

positi

ve Brexit posit

ive x 3 2

sooooooo i still have over £40 that i never exchanged for

dollars. i feel less guilty about that now. #Brexit

positi

ve other

neut

ral 2 3

Hillary Clinton urges 'experienced leadership' after

#Brexit from #EU http://goo.gl/fb/aoXo3G #europe

#europeanunion

neutr

al Brexit neut

ral 3 2

#Brexit could break up #EU and #NATO, prevent World

War III: Paul Craig Roberts

http://goo.gl/fb/qkMCWm #europe

neutr

al Brexit neut

ral 3 2

MSM treats #Brexit as Europe's demise.History might say

it set in motion needed revisions of Social Contract&economic

machinery of Europe

positi

ve Brexit posit

ive 3 2

President Vladimir Putin says #Russia has 'never

interfered' in #Brexit http://goo.gl/fb/Dq5PGX #eu #europe

neutr

al Brexit

neut

ral 3 3

Top Google search in Britain AFTER the #Brexit vote

was "What is the EU?" After the vote? #Feckin morons need a

monarch.

negat

ive other nega

tive

3 2

Also strenghtened by the renovated and expanded flexible

credit line with the IMF #Brexit #Mexico #PressRelease

neutr

al Brexit

neut

ral

1 2

President Vladimir Putin says #Russia has 'never

interfered' in #Brexit http://goo.gl/fb/tJXPni #eu #europe

neutr

al Brexit

neut

ral 3 3

Boris Johnson goes from court jester to crown prince after

#Brexit win http://bloom.bg/28RCZLw

neutr

al

celebrities/p

oliticians

neut

ral 3 3

I h8 the phrase "take back our country," whether it's used

4 the US or the UK bc it's fear-mongering by spreading hate 4

foreigners #Brexit

negat

ive Brexit nega

tive 2 2

My vote was to be free of unelected EU commissioners

passing laws that our country has no say in. Glad to be out,

they need us more. #Brexit

positi

ve Brexit posit

ive 2 2

@truthout It hasn't even been 24 hours and you're judging

the outcome? Britain hasn't even officially the left EU yet.

neutr

al Brexit

posit

ive 2 2

Hillary Clinton represents the crony capitalist kleptocracy

the author identifies as responsible for #BREXIT. DOA.

negat

ive

celebrities/p

oliticians

nega

tive 2 2

The latest The Sciarra Stefano Daily!

http://paper.li/Colonnasciarra/1334314750?edition_id=998632

10-3a67-11e6-92d0-0cc47a0d1605 … Thanks to @rpinci

@VentagliP @Surfiniae #brexit #business

neutr

al other

neut

ral 3 3

True elites want to run their own lives and own countries

not be told by central government what to do. #Brexit

positi

ve Brexit

neut

ral 2 1

After failure to get into #NSG, MODI JI should start

pushing to get a entry into #EU, Britain's vacant place awaits

for India. #Brexit

neutr

al Brexit neut

ral 2 2

The idea that UKIP will now disband, its mission

accomplished, is delusional. What emboldened reactionary

ever gave up their fight? #Brexit

negat

ive Brexit nega

tive 2 2

“I still have my ice cream. How can #Brexit be a big

deal.” - @yogurtearl, while eating ice cream.

positi

ve Brexit

neut

ral x 2 2

Sacramento: Tune into @kcranews at 5PM to catch our

very own Theo Slater provide our campaign's reaction to

#Brexit #BrexitVote.

neutr

al Brexit neut

ral 1 2

Page 64 of 76

http://Brexit101.com is for sale

http://ht.ly/LS98301BKIq #Brexit #BrexitVote #UK #Trump

#Cameron #Britain #EU #Greece #FX #Zika #Grexit

neutr

al other neut

ral 2 2

#Brexit Get out of the stockmarket now! Go see your

local ResiShare agent http://bit.ly/28IlT3x

negat

ive economy

nega

tive 2 2

ICYMI- Dow plunges 611 Points, British PM David

Cameron to resign, England to free from EU... #brexit lots

going on http://fb.me/7RgbzHxU0

neutr

al Brexit neut

ral 2 2

Amazing, Blair involved in illegal war killing 100s of

thousands no 1 tried 2 remove him,Corbyn in job a wet week,

coup against him #brexit

negat

ive

celebrities/p

oliticians nega

tive x 2 2

So does this mean we get a football team in England?

#Brexit #Nfl

neutr

al other

posit

ive 3 3

I just read that France is planning to send the thousands of

refugees in Calais towards UK. #brexit #immigration

neutr

al Brexit

neut

ral 2 2

What the hell did you do uk? #brexit #banksy

#yearofthemonkey #day137 #stupid #uk #eu #exit…

https://www.instagram.com/p/BHDnfFsj8DM/

negat

ive Brexit nega

tive 2 2

#Brexit is the warning to the EU leadership after Greek

referendum. They did not get it last year I hope they do now!

@EU_Commission

negat

ive other posit

ive 2 2

The frustrated Cold War warriors seeing #Brexit through

the prism of their paranoia about Vladimir Putin appear to be

revving their engines.

negat

ive Brexit nega

tive 2 2

@ABCNews24 LOL give me a 'Dog's Brexit' anyday over

an AbbottTurnbull government mate! #brexit was Rich v's

Poor, Far Right v's Centre Left

negat

ive Brexit nega

tive 1 1

Local Leave supporters 'pleased' & 'happy' with #Brexit

referendum result: http://chattelevision.ca/__news/local-leave-

supporters-pleased-and-happy-with-eu-referendum-result/ …

positi

ve Brexit posit

ive 3 3

Oh no #Brexit , how could you do this to large investors!

Today was mildly annoying, caused by "uncertainty".

#traderphobic #EUref #UK

negat

ive economy nega

tive 2 2

Now keep the promise of £350m a week for our #NHS -

Sign the petition: #EuRef #Leave #Brexit

https://you.38degrees.org.uk/petitions/invest-ps350-million-

saved-from-eu-in-nhs-by-2018?bucket=fb&source=twitter-

share-button … via @38_degrees

neutr

al Brexit

posit

ive 3 1

#Brexit sounds like a breakfast cereal #comedy

https://vine.co/v/5u7ntpa5B3U

neutr

al Brexit

neut

ral 3 2

MUST-READ #Brexit commentary: "But first, we will

have to think, probably more deeply than ever."

neutr

al Brexit

neut

ral 2 2

I wanna find my fellow Jubilee line riders who took down

the drunk Welsh #brexit supporter.

neutr

al other

neut

ral 2 2

Crude oil prices slammed after Britain votes to leave EU

http://klou.tt/104hlokjipjzm #brexit #oilprice #oilandgas

#petroleum

negat

ive economy nega

tive 3 2

Wonderfully succinct statement #Brexit #wtfbritain positi

ve Brexit

nega

tive x 2 2

You know, seriously, it's vapid & arrogant comments

from talking blow-dry heads like this that CAUSED #brexit. :/

negat

ive Brexit

nega

tive 2 2

#brexit today's removal of Paul Day's St Pancras "Meeting

Place" lovers statue. Things are happening fast.

neutr

al other

neut

ral 2 3

Watching Underworld at #Glastonbury2016 tonight in the

aftermath of #Brexit, I miss the 90's more than ever. Such

innocent times. Such hope.

negat

ive Brexit nega

tive 2 2

The club regret to confirm that the transfer of

@Ibra_official has collapsed due to the fall in value of the

pound as a result of #Brexit.

negat

ive other nega

tive 2 2

We owe so much to Nigel Farage for his unwavering

commitment to the #Brexit cause. What an historic day for

Britain #IndependenceDay

positi

ve Brexit posit

ive

2 2

Saying "everyone" online talking about #Brexit is as

smart as their dog inherently is discrediting those who are

talking

negat

ive other nega

tive x 2 2

Page 65 of 76

@realDonaldTrump so busy promoting his biz, his initial

reaction to #Brexit was, "there's nothing to talk about."

negat

ive Trump

nega

tive 2 2

Honestly most elites seem to think everyone needs a

daddy mommy bureaucrat to manage their lives for them. More

effete than elite. #Brexit

negat

ive Brexit nega

tive 2 2

New pictures of referendum result emerge... #brexit

@Oldfirmfacts1

neutr

al Brexit

neut

ral 2 2

This prerequisite of being a sociopath to have power is

getting a bit tedious now. #Brexit

negat

ive Brexit

nega

tive 2 2

.@SketchesbyBoze Perhaps this whole mess could've

been avoided if #Brexit was named Brexity McBrextface

negat

ive Brexit

nega

tive x 3 3

END OF THE EU? #Germany warns FIVE more

countries could leave Europe after #Brexit | World | News |

Daily Empress

neutr

al Brexit neut

ral 3 3

I regret voting for #Brexit under these false pretenses. negat

ive Brexit

nega

tive 2 2

Britain will be better off just like a spun out Corp from a

conglomerate, give 'er a day or two ;) #Brexit

positi

ve Brexit

posit

ive 2 2

there's no perfect society but Britain voted for worst out

of two. However 48% of UK ppl ain't happy with it #Brexit

negat

ive Brexit

nega

tive 2 2

I heard this great line on the @marklevinshow "when

immigrants don't assimilate it's called colonisation."

#immigration #Brexit #vidcon2016

negat

ive Brexit nega

tive 2 2

So, appears I successfully avoided making premature, not

informed enough direct correlation tween #brexit & U.S. 2016.

Mission Accomplished

positi

ve other nega

tive 2 2

With revolution life is so much better #Brexit positi

ve Brexit

posit

ive 3 2

The UK is like a kid that's been threatening to run away

from home, finally do & then immediately regret it #Brexit

negat

ive Brexit

nega

tive 2 2

Yep, him too. Saw a woman on #newsnight crying with

joy because she thought #Brexit had saved the NHS.

positi

ve Brexit

nega

tive 2 2

A day full of sad news. #Brexit then @Yellowcard

announce their end. Last tour tickets go on sale on payday,

seems like fate. Have to go!

negat

ive other nega

tive

2 2

Hey older generations in the UK. Thanks for putting that

final nail in the coffin for the rest of us...You CUNTS.

#BrexitVote #Brexit

negat

ive Brexit nega

tive 1 3

The #majority imposed their will on a very significant

minority in the #Brexit referendum

negat

ive Brexit

nega

tive 2 2

holy shit Nigel Farage has an Alan Partridge voice you

should have picked up on this British people! #brexit

neutr

al

celebrities/p

oliticians

nega

tive x 2 2

@realDonaldTrump You had NO FUCKING IDEA what

#Brexit or #BrexitVote was 2 days ago. LMAO

duhhhhhhhhhhhhhhhhh

negat

ive Trump nega

tive 2 2

Well well well, an appropriate song for the #Brexit result!

Sex Pistols Anarchy in The UK

positi

ve Brexit

nega

tive x 1 2

Sorry to see this happening... #Brexit negat

ive Brexit

nega

tive 3 3

Is this #Brexit going to mess with my UPC/VirginMedia

because my internet has been shit all day

negat

ive other

nega

tive 3 2

Dominar Farage strikes again... #farscape #Brexit #EUref negat

ive Brexit

nega

tive 3 3

#Brexit #GOT #GameofThrones The 'Brexit' referendum

and 'Game of Thrones' aren't all that different

http://mashable.com/2016/06/23/brexit-game-of-

thrones/#MQSrpW_B805A … via @mashable

neutr

al other

nega

tive 2 2

Mark it down, folks who are against #Brexit & calling the

#Leave folks racist, bigots, & xenophobes don't have a clue

about the real issues

negat

ive Brexit posit

ive 2 2

Buy Pounds .... Save Euros ... #Brexit neutr

al economy

neut

ral 3 2

By George, he gets it! Britain voted for #Brexit because it

wants to be #Canada http://pllqt.it/Tn6552

positi

ve Brexit

nega

tive x 2 2

Page 66 of 76

IMO David Cameron & Jeremy Corbyn should've put

politics & campaigned together on the platform of staying in

the EU... #Brexit #BrexitVote

negat

ive Brexit nega

tive

2 2

Dow jones took a MASSIVE blow today due to #Brexit.

This is a sneak peak at what would happen under Trump.

negat

ive economy

nega

tive

3 2

Victoria Nuland, Assistant Secretary of State, supporting

#Brexit in 2014, before it was mainstream.

neutr

al Brexit

neut

ral 2 2

#Brexit has led to some great memes #BrexitVote

#politics

positi

ve Brexit

neut

ral 3 3

This Dude Who Thought His #Brexit Vote Wouldn't

Matter Is A Valuable Lesson For All Of Us

https://www.mhb.io/e/1c6ei/cp

negat

ive other nega

tive 2 2

#Brexit is really complicated. I think the UK will be better

off in the long run, but I'm not certain.

positi

ve Brexit

posit

ive 2 2

Hillary picked to be on the wrong side of history for the

951st time. @realDonaldTrump #Brexit

negat

ive

celebrities/p

oliticians

nega

tive 2 3

Right, off to bed. Pretty tired so it'll be nice to lay my

head down on my soon to be de-regulated pillow

@iamjohnoliver #Brexit

positi

ve other posit

ive 1 2

Going in on @bondibeachradio in a couple of minutes

with #brexit bangers. Some geezer classics, some euro power.

neutr

al other

neut

ral 3 2

Did #vapers sway the #brexit vote? Perhaps everyone has

to deal with #Article50 because of #Artilce20 of the #TPD

negat

ive Brexit

nega

tive 1 2

#Brexit example of democracy in action. You must suck

and shut up

positi

ve Brexit

posit

ive 2 2

#Brexit today, #Libexit in a week. #auspol #ausvotes neutr

al other

neut

ral 3 3

From Rule Britannia to Cool Britannia to Fool Britannia

#brexit

negat

ive Brexit

nega

tive 1 1

Sigh. You had one job; #Britain #brexit negat

ive Brexit

nega

tive

2 2

Surprised #Sutton was one of few #London boroughs to

vote #Leave as was a @LibDems seat until 2015 #Brexit

@scullyp

negat

ive Brexit neut

ral

1 2

The #EU is far more about the New World Order of One

World Government than it is about a Global economy. #NWO

#Brexit @morningmika @brithume

negat

ive EU

posit

ive 2 2

#Brexit #history - future history exam question(s) neutr

al other

neut

ral 2 2

Super serious situation, but I chuckled. #Brexit Peace to

all OUR UK family! #Repost @tcb… http://itsOURshow.net

positi

ve Brexit

nega

tive 2 2

TheEconomist: Calls for a second Scottish independence

referendum grow louder after #Brexit …

neutr

al

Scottish

referendum

neut

ral 3 2

Before and after, my grass lost the referendum too

#grassxit #brexit , see what's left?

negat

ive Brexit

nega

tive 2 2

Damn you #Brexit supporters. I lost almost $10K in my

retirement today because of you and I'm from the U.S. I hope

you feel a similar pain!

negat

ive Brexit nega

tive 2 2

Does spellcheck have any part in Western Civilization?

#Brexit

neutr

al other

neut

ral 2 2

Nice work @CommBank leaving customers stranded

without access to their OWN money in UK cos it costs YOU

$$ #Brexit

negat

ive other nega

tive 1 1

Age old struggle between freedom vs. security. When

security just wasn't all that secure the people chose freedom.

#Brexit

positi

ve Brexit posit

ive 2 2

#Brexit generational gap is unbelievable -

http://www.cbc.ca/1.3650826

negat

ive Brexit

neut

ral 2 2

#Brexit: The Consequences Of Xenophobic Nationalism

https://theobamadiary.com/2016/06/24/brexit-the-

consequences-of-xenophobic-nationalism/ … via

@TheObamaDiary

negat

ive Brexit

nega

tive 2 3

Right behind you Marsha and Robert!!

#MakeAmericaGreatAgain #Brexit

positi

ve USA

posit

ive

3 1

Page 67 of 76

Haven't seen much excitement over the #Brexit vote.

Mostly a lot of despair and anger. I don't know if I should feel

sorry for the UK.

negat

ive Brexit neut

ral 2 2

@EricIdle #Brexit voters reminds me of the Crimson

Permanent Assurance pirates. Have they interrupted our lives

and ruined the world?

negat

ive Brexit nega

tive 2 2

END OF THE EU? Germany warns FIVE more countries

could leave Europe after #Brexit

neutr

al other

neut

ral 3 3

The #brexit discussion on Hardball dismissing Bernie and

both D & R voters for not embracing the "way we do things".

Still don't get it!

negat

ive USA neut

ral 2 2

Education matters, sure, but it's also a privilege. Scary

thing is that it's been becoming more and more of one. So ...

your move? #brexit

negat

ive other

nega

tive 2 2

How do you two on 'team globalist' feel about being on

the wrong side of history today? #Brexit #IndependenceDay

negat

ive Brexit

neut

ral 2 1

#Brexit Britain's biggest mistake since the 1773 Tea Act negat

ive Brexit

nega

tive

2 2

I take this as Target forewarning us of the disastrous

#Brexit fallout. Just the title, not the hackneyed plot.

negat

ive Brexit

nega

tive

2 2

Look out for Ed Sheerans Brexit cash in single "You need

me, I don't need EU" #edsheeran #Brexit #BrexitVote

#EUreferendum #EUref #jokes

neutr

al Brexit neut

ral x 2 2

Up until this weekend I thought #Brexit was some sort of

cracker for tea.

neutr

al Brexit

neut

ral x 2 2

don't even use #Brexit you were for Remain your

continual lying will not get you anything traitor!

negat

ive Brexit

posit

ive 2 2

Jermaine Pennant asks the most important question of the

day http://dailym.ai/28RBRZW via @MailOnline #brexit

#hesaidwhat?

neutr

al other neut

ral 2 2

If the EU savings aren't going to the NHS as promised,

where pray tell are they going? #Brexit

neutr

al Brexit

nega

tive 2 2

#IVotedLeave #England #brexit I'm not British but i hope

the best for England

positi

ve Brexit

posit

ive

2 2

Being united w other countries is something so important

- it provides a sense of safety & solidarity. I am so, so sorry

Britain. #Brexit

negat

ive Brexit nega

tive

2 2

Of all UK tweets responding to #Trump stupidity, this is

my fave: "Scotland hates both #Brexit and you, you mangled,

apricot, hellbeast."

positi

ve Trump nega

tive

2 2

i miss original recipe steven colbert. hed be killing #brexit

watching him on the late show feels like watching jordan play

baseball

negat

ive other nega

tive 2 2

Stay tuned for some amazing Crop Circles. #brexit neutr

al other

neut

ral 1 3

Disaster Ahead! Trump on Brexit: America is next

http://cnn.it/28VPZjy #Brexit #Election2016

negat

ive Trump

neut

ral 3 2

#Brexit Won! What does it mean to #domain investors? neutr

al economy

neut

ral 2 2

Or is it the fear of Changing the Paradigm or is it the Fear

that the Paradigm will NEVER change? #BrexitVote

#BREAKING #Brexit

neutr

al Brexit neut

ral 2 3

EU can’t go on forever without earning consent of the

governed: @InklessPW http://on.thestar.com/28WI4Cr #euref

#Brexit #democracy #consent

negat

ive Brexit posit

ive 2 2

Hats off to the German Foreign Office! #BrexitVote

#Brexit #EURef

positi

ve other

neut

ral 3 2

#Brexit & #Czechout & #Finish, oh my! Denmark too. Is

#EUXit next? #MAGA #Trump2016

neutr

al Brexit

neut

ral x 3 2

#Brexit #DavidCameron I wonder will Cameron be see

as. #nevillechamberlain figure "Europe in our time" figure.

Disaster #PM

negat

ive Brexit nega

tive 2 2

Just kidding One of the key reasons you have to be careful

when trusting populist propaganda. #brexit

negat

ive Brexit

nega

tive 2 2

Page 68 of 76

LOL. I heard someone say, "make England great again."

That sounds awfully familiar... #Brexit

#MakeAmericaGreatAgain

negat

ive USA nega

tive 3 3

1/ A possible historical parallel to London post #brexit is

Montreal

neutr

al Brexit

neut

ral 2 2

At what point does #Leave #Brexit get called out for flat

out lying in campaign commercials? Nope. NHS won't get that

money. #UKreferendum

negat

ive Brexit nega

tive 3 2

British Lose Right to Claim That Americans Are Dumber

http://www.newyorker.com/humor/borowitz-report/british-

lose-right-to-claim-that-americans-are-dumber … via

@BorowitzReport #Brexit

neutr

al Brexit

nega

tive x 3 2

"#Brexit is not gonna play well in Washington."

#BBCWorld

negat

ive Brexit

nega

tive 2 2

Should we start calling this what it is - the rise of fascism

in #Britain #brexit #EURefResults

negat

ive Brexit

nega

tive 2 2

After #Brexit? Departugal. Italeave. Fruckoff. Czechout.

Oustria. Finish. Slovlong. Latervia. Byegium. = Germlonely

neutr

al Brexit

neut

ral x

3 3

Separatists in Scotland and terrorists in Northern Ireland

are cynically calling for new referenda to achieve their raisons

d'être. #Brexit

negat

ive other neut

ral

2 2

Don't like bureaucracy but a lot of #Brexit/Trump votes

are based on fear and hate. That's what I'm against more than

anything. Depressing.

negat

ive Trump nega

tive 2 2

Just something to lighten the mood a little bit.

@naomivowles you can never go wrong w/ Spice Girls #Brexit

http://www.youtube.com/watch?v=SoxxHeBJmz8&sns=tw …

positi

ve other nega

tive 3 2

BostonGlobe: After #Brexit, European property investors

may see better value — and stability — in Mass. …

positi

ve economy

neut

ral 3 2

#Brexit would have been defeated if #EU had accepted

NO to #treaties,listened to ppl & reformed.predict

#dominoeffect

negat

ive EU nega

tive 2 2

People "going with thier gut" instead of their brain has

never lead to anything other than rasict dumbassery. #Brexit

negat

ive Brexit

nega

tive 2 2

I wanted #Brexit hugely. positi

ve Brexit

posit

ive 3 3

@MSNBC @amjoyshow Lindsey Lohan tweeting about

#Brexit is no more “Breaking News” than me tweeting about

it!

negat

ive celebrities/p

oliticians

neut

ral 2 3

@realDonaldTrump Florida is in Scotland? Or is Scotland

in Florida? I'm so confused. #brexit #yousirareanidiot

negat

ive Trump

nega

tive 2 2

Sweet, I just found £8 down the back of the sofa. Thanks

#Brexit

positi

ve other

neut

ral x 2 2

How can we criticize the #brexit when our president flies

on a plane named after a shoe

negat

ive Trump

nega

tive x 2 2

Serious question: does anyone have a time machine?

#Brexit

negat

ive other

nega

tive x 2 3

BostonGlobe: #Brexit was spawned by tensions over

globalization, President Barack Obama says …

negat

ive Brexit

nega

tive 2 2

#Brexit is all shits & giggles to me until I check my BT

European funds

negat

ive Brexit

nega

tive 3 2

lol. Looking forward to the day the english come crawling

back. #Brexit #Leave #brexitfail

positi

ve Brexit

nega

tive x 3 2

Almost incredible that none of the #Eurocrats seem to

have had a contingency plan for #brexit, but should have

expected it.

negat

ive Brexit nega

tive 2 2

I use to think I was proud to be British but not proud of

Britain, but now, im not so sure about the second part #Brexit

negat

ive Brexit

nega

tive 2 1

if someone had said that today’s economic situation

would be the immediate outcome of #brexit how many would

have believed? (vs more FUD)

negat

ive economy nega

tive x 2 2

Out of the #Brexit ashes I feel sorry for the Japanese. The

Japan economy desperately needs a lower Yen but the Brexit

computer says "No"

negat

ive economy

neut

ral x 2 2

Page 69 of 76

This is a pretty perfect response. @dominicnahr #brexit

#montypython

https://www.instagram.com/p/BHDnVGoA3Ve/

positi

ve other neut

ral 3 2

Finally logged off from work. Thanks for the overtime

cash #Brexit

positi

ve other

neut

ral 2 2

There is some satire in this #brexit negat

ive other

nega

tive 3 3

Rise of #Bitcoin and #Gold as #Brexit Turns into Reality neutr

al economy

neut

ral 2 2

Israel and America now should leave the UN #Brexit neutr

al other

posit

ive 3 2

It can't be "take back control" for England and "lose all

control" for Scotland & NI. #UnitedIreland #Indy2 #IndyScot

#Brexit

negat

ive Brexit nega

tive 2 2

considering the breakdown of voting in #Brexit I propose

a new country form and its name should be Scotirelondon

#scotirelondon

neutr

al Brexit nega

tive x 2 2

Have the four horseman appeared yet? #Brexit negat

ive Brexit

nega

tive x 2 2

When we leave EU we won't be protected by the Privacy

Shield agreement with the US. Goodbye privacy, thanks

#Brexit.

negat

ive Brexit nega

tive 1 2

I think if #Texit happens it will start a domino effect just

like #Brexit did

neutr

al other

nega

tive 3 2

The only good thing to come out of the #Brexit is the

dearth of insults being hurled at @realDonaldTrump by the

lovely people of Scotland

positi

ve Trump

nega

tive x 2 2

#Canada pls keep it together so we have somewhere to go

if Drumpf becomes POTUS #Brexit :(

negat

ive Trump

nega

tive 3 2

Come to think of it, #Brexit does have a smoky taste to it negat

ive Brexit

nega

tive 2 2

$2.7 TRILLION lost on global markets after

@RupertMurdoch has his #Brexit dreams come true. People

will die from impacts of such losses.

negat

ive economy

nega

tive 2 2

Stop being afraid of what could go wrong and start being

excited about what could go right . #Brexit #VotedLeave

positi

ve Brexit

posit

ive 3 2

"Democracy is the theory that the common people know

what they want, and deserve to get it good and hard." -H.L.

Mencken #Brexit #BrexitVote

neutr

al other

posit

ive 2 2

Thought of #TREXIT tantalizing, not happening. After

#Brexit the Scots hate him, where shall The Donald go?

Instead #ImWithHer

negat

ive Trump

nega

tive 2 2

Despite the alerts, did not open Robinhood once today.

#Brexit #dontpanic #letthatpoundcomedown #ineverbeentogb

positi

ve other

nega

tive 2 2

..And the Oscar as worst world leader ever goes to..

#Cameron #Brexit #ByeByeUKEP #UKreferendum

negat

ive

celebrities/p

oliticians

nega

tive 2 2

U2 said it best "And I wait without EU / With or without

EU" #Brexit

neutr

al Brexit

neut

ral x 3 2

My point is the racists that have always been there now

seem to think it is acceptable to be openly racist since #Brexit

negat

ive other

nega

tive 2 2

Sloooow down #MariaBartiromo...it's not the end of the

world yet. #Brexit #greta

positi

ve Brexit

posit

ive 2 2

#Brexit' to be followed by Grexit. Departugal. Italeave.

Fruckoff. Czechout. Oustria. Finish. Slovakout. Latervia.

Byegium.

neutr

al Brexit

neut

ral x 3 3

isn't that exactly what voters HATE. Pos afraid for

THEIR political futures..the country's be damned, be it #brexit

or NRA

negat

ive Brexit

nega

tive 2 2

The first task of the new #ToryLeadership will be how to

diminish #UKIP #bestofenemies #Brexit

neutr

al Brexit

nega

tive 3 3

I bet right now almost everybody in Florida is looking at

this whole #Brexit thing and thinking, "Aw, man, did I just shit

my pants again?"

negat

ive Brexit

nega

tive x 2 2

Every time someone says #brexit our cats lift up head

with big eyes as if it means #biscuit ... at this rate they'll never

learn English.

neutr

al other

neut

ral x 3 3

Page 70 of 76

#Brexit was just a massive hoax so we could have a

global meme competition.

negat

ive Brexit

nega

tive x 2 2

Does this mean no more Tim Tams? #Referendumb

#Brexit

negat

ive other

nega

tive x 3 3

Brits were tired of hearing from "outside experts" #brexit

#lohanknew

negat

ive

celebrities/p

oliticians

nega

tive x 2 2

I bet that's not the first time Obama's been mentioned in

the same sentence with an "eight ball." #Brexit

neutr

al

celebrities/p

oliticians

neut

ral x 2 2

I know it is not good for me, but, on days when Britain

chooses to #Brexit, I like to drink a Coke and eat a cookie.

#EURefResults

negat

ive other

nega

tive 2 2

On some level UK after #Brexit must feel a little like US

after GW Bush reelection. Mainly in how the rest of the world

is like "Seriously?"

negat

ive Brexit

nega

tive x 2 2

No one saw #Brexit coming. No one will see FBI

indictment recommendations of Hillary coming. 2016 is a wild

year. Be ready.

negat

ive USA

nega

tive 2 2

The average life experience of those whining about

#Brexit seems to be about 12 years.

negat

ive Brexit

posit

ive x 2 2

#OutMeansOut - Let's leave this mess before it all

collapses all around us. #LeaveEU #Brexit

negat

ive Brexit

posit

ive 3 3

As someone who works closely with highly skilled

migrants, I can say I value your work and you are an asset to

this country #brexit

positi

ve other

nega

tive 3 2

Did that 1% growth from the $50billion business tax cut

after 10 years include modelling in a #brexit scenario?

#ausvotes

neutr

al economy

neut

ral x 2 2

Sebastian was right, can I become a mermaid now pls

#Brexit #EURefResults

negat

ive Brexit

nega

tive x 3 2

If some of you people had your way, the USA would've

never existed. Would've been happy paying taxes to the British

for a lifetime. #Brexit

positi

ve Brexit

nega

tive 2 2

#HiddleSwift or #Brexit, don't make me choose!

(Meanwhile, in America...) #TGIF and the markets are closed,

whew!

neutr

al other

nega

tive x 2 2

Don't brex my heart, Never leave me again #Brexit

#BrexitVote #ToniBraxton #Nailedit #Youaresonotfunny

neutr

al Brexit

neut

ral x 2 2

When you see that Trump endorsed UK leaving EU, you

can realise how stupid that idea it is. #Brexit

negat

ive Trump

nega

tive 2 2

its begun, The Begining of the END #Brexit #Damn negat

ive Brexit

nega

tive 2 2

Now keep the promise of £350m a week for our #NHS -

Sign the petition: #EuRef #Leave #Brexit

https://you.38degrees.org.uk/petitions/invest-ps350-million-

saved-from-eu-in-nhs-by-2018?bucket=blast&source=twitter-

share-button … via @38_degrees

neutr

al

Brexit

posit

ive 3 1

#Mckinney #FreddieGray #GeneleLaird #Brexit

#immigration The news has been exhausting these past few

days

negat

ive other

nega

tive 2 2

All I have to say about #Brexit #love #ignorance #hate

#nofear #peace #tolerance…

https://www.instagram.com/p/BHDnRjwA-_k/

neutr

al Brexit

nega

tive 1 3

So Black & brown ppl should stay in their "native"

countries? I could be ok w/ that, white ppl, if you also stay in

"yours." #Brexit #Trump

negat

ive Trump

nega

tive 2 2

Churchill must be smiling down on #obama as Briton

follows her history & refuses to surrender, once again #brexit

#tcot #p2 #gop #dem

positi

ve Brexit

posit

ive 2 2

#Cameron & #Osborne tried to call the bluff of Tory

troublemakers for party reasons -it's backfired badly #brexit

negat

ive Brexit

nega

tive 2 2

Brexit big blow to UK science, say top British scientists-

https://www.theguardian.com/science/2016/jun/24/brexit-big-

blow-to-uk-science-say-top-british-scientists?CMP=twt_a-

science_b-gdnscience … #Brexit #Science

negat

ive

Brexit

nega

tive 2 2

This outcome was set from the stary, calling it #Brexit

instead of Bremain.

neutr

al Brexit

nega

tive 2 2

Page 71 of 76

#Brexit affirms what's innate about bureaucracy - in spite

of intentions it grows, empowers elites, squelches public and

self determination

negat

ive Brexit

nega

tive 3 3

I guess all those UK Citizens of every color voting for

#Brexit were racists too?

negat

ive other

nega

tive x 2 2

"I'm divided." British expat in Co talks about uncertainty

in the UK after #Brexit. Her story @6.

negat

ive Brexit

neut

ral 2 2

#BREXIT good timing!!! EU nazi caliphate waiting just

on other side of the channel. @FoxNews

positi

ve Brexit

posit

ive 3 2

David #Cameron, you are so outta here! #Brexit #EU

#EUreferendum #notmyvote #Johnson

negat

ive Brexit

nega

tive x 2 2

We may end up w/ @BorisJohnson and

@realDonaldTrump WOW!!! FDR and Churchill r checking

w/ God about reincarnation #Brexit #insanity

negat

ive celebrities/p

oliticians

nega

tive x 2 3

Relatedly I’m going to compile a reading list for myself

on possible EU reforms bc clearly that was a huge

(understandable) #Brexit factor.

neutr

al other

neut

ral 2 2

So....... the Queen kills everybody now, right? #Brexit neutr

al Brexit

nega

tive x 2 2

#Brexit well wishes from the neighbours! positi

ve Brexit

posit

ive 2 2

BRB. Reading all about #Brexit and possibly planning a

trip to England ASAP before things get too crazy.

neutr

al other

neut

ral 3 3

I bet a lot of UK businesses are feeling pretty bummed

about their .eu domain extensions... #brexit #EURefResults

#EUref #firstworldproblems

negat

ive other

nega

tive x 2 2

This bozo found a legal loophole... #Brexit but with the

"benefit" of unfettered immigration into the UK...

negat

ive Brexit

nega

tive 3 2

Nice to see how it takes #brexit to make @guardiannews

see the private sector as more than just an iniquitous force that

needs to be taxed

negat

ive Brexit

nega

tive 2 2

Looks like the Coudenhove-Kalergi plan and his book

Praktischer Idealismus for the European Union are failing

thanks to Britain. #Brexit

negat

ive other

nega

tive 2 2

Whatif?@bernardchickey #Brexit #RemainINEU UK had

heard non political/self-interest view, just facts - same result?

neutr

al Brexit

neut

ral x 2 2

Day 1 of #brexit Still in the EU. neutr

al Brexit

neut

ral x 3 2

Following that course, disintegration of social bonds and

solidarity is the result. That's where we are where we are.

#Brexit

negat

ive Brexit

nega

tive 2 2

Will England remain at Epcot Center? #Brexit neutr

al Brexit

neut

ral 2 2

I am starting to learn #French and using more of my

#german ... #Brexit #tragic

negat

ive other

nega

tive 3 3

@David_Cameron 's pull out game is really weak. You're

supposed to pullout BEFORE you mess up "her" life. RIP

Britain. #BrexitVote #Brexit

negat

ive Brexit

nega

tive 2 2

So when do we start sending ration packs to the UK.

#Brexit

neutr

al Brexit

nega

tive x 1 1

Soooo what if Trump bought Texas and renamed it

Trumpxas. Then we can vote for a #Trexit... #Brexit

neutr

al Trump

nega

tive x 2 2

I just woke up and there is a severed horse head next to

me... What does this mean? Did I piss off the Mafia? #help

#Brexit #Mafia #horse

negat

ive other

nega

tive x 2 2

Two #Brexit fears are hard to reconcile 1. Scotland, N

Ireland leaving the UK to join EU while 2. EU countries like

France move toward exit.

negat

ive Brexit

nega

tive 1 2

His cluelessness of #Brexit neg impact on Sctland as he

talked is n xmple of infantile diplomacy.

negat

ive Trump

nega

tive 2 2

What can the EU do that it never did before against ISIS,

#Brexit changed nothing UK doing all heavy lifting and paying

EU invites them in

negat

ive EU

posit

ive 2 2

I just hope my 401(k) didn't shit the bed today over

#Brexit. It probably did tho.

negat

ive other

nega

tive 2 2

Page 72 of 76

“Anyone else think #brexit sounds like a super healthy

digestive type of biscuit? 'Ooeck owsabout you pass me the...

http://fb.me/5jalEFbdB

neutr

al Brexit

nega

tive x 2 2

France is now the 5th largest economy in the world, took

just 5hrs, thanks to #Brexit

positi

ve economy

nega

tive x 1 2

I want #AMERICA back!! #MakeAmericaGreatAgain

#MakeAmericaSafeAgain #Brexit

neutr

al USA

posit

ive 1 1

.@lsarsour #Brexit is, or should be, a first step to local

autonomy and bio-regional governance, without which

#sustainability is impossible

positi

ve Brexit

posit

ive 3 3

Seems like my boys spent most of their time at HSS for

the last week or so. Wall Street was like being at a wake today.

#Brexit

negat

ive economy

neut

ral x 2 2

If Trump and the French national front celebrate the brexit

vote.... You know you've fucked up royally.... #Brexit

negat

ive

celebrities/p

oliticians

nega

tive 2 2

The latest The Makam News!

http://paper.li/igbariam/1347306962?edition_id=3e09c910-

3a67-11e6-b556-0cc47a0d15fd … #nbadraft #brexit

neutr

al other

neut

ral 3 3

We'll find out the effect of #Brexit on the performance of

British national teams very soon. #EURO2016, #England,

#Wales, #Ireland, #NIR

neutr

al other

neut

ral 3 2

Why are people encouraging the dismantling of the EU?

#Brexit

negat

ive Brexit

nega

tive 2 2

#greta who would you rather making the deals with

#Brexit @HillaryClinton or @realDonaldTrump this kind of

skews the election

neutr

al USA

neut

ral 2 2

If no fed rate hike now, mtg. rates should stay around

historic lows for a while. How long though and is that really

beneficial? #Brexit

negat

ive Brexit

nega

tive x 3 2

Just watched Jim Cramer on this morning's

@TheTodayShow telling an astonished Matt Lauer that the

economic impact of #Brexit is not so bad.

neutr

al economy

nega

tive 2 2

Americans have no respect for Obama why would the

Britts #Brexit #greta #Trump2016

negat

ive

celebrities/p

oliticians

nega

tive 2 2

Just for clarification since I've gotten 10+ messages on

this: No I don't think #Brexit was solely due on racism, clearly.

negat

ive Brexit

nega

tive 2 2

Colonises half the World and complains about

#immigrants. #Brexit

negat

ive other

nega

tive x 3 2

Seriously, this is so retarded my head hurts. #brexit got

the most votes on a referendum. You lost. Cry elsewhere

negat

ive Brexit

posit

ive 2 2

Ahh @JoyAnnReid brings up the “Anti Expert” element

of #Brexit. I look around & see debate against science &

experts in my own state often

negat

ive Brexit

nega

tive 2 2

The fascist, one-world tyrant-lovers cuss a lot when they

venture outside of their safe place to complain about #texit or

#brexit. Haha

negat

ive Brexit

posit

ive 2 2

We should have right to retain our EU citizenship even

after #Brexit.

neutr

al Brexit

nega

tive 2 2

Brits who voted to leave in #Brexit, to be safer, apparently

never heard of #DivideAndConquer

negat

ive Brexit

nega

tive 2 2

#Brexit I'm American, but I am also (by decision) Greek,

French, Italian, Spanish, Dutch, and Belgian. But I will never

be an #Englishwoman.

negat

ive other

neut

ral 1 2

American Media narrative bout #Brexit focuses on

THEIR stupidity while reportin Lohan tweet, views of

presumptive nominee Trump #GlassHouses

negat

ive USA

nega

tive 2 2

Utility stocks, along with Treasury bonds, serve as safe

haven during tumultuous times. @SmithRebecca

http://ow.ly/ldWK301CoUC #brexit

neutr

al economy

neut

ral 3 3

This is Great Britain not a 3rd world country. The elite

selling today will be buying next week, part of their scare

tactics. #Brexit

negat

ive economy

nega

tive 2 2

Can we solve #Brexit issues using the old "reset" method?

Switch off, leave Europe for 10 secs then plug ourselves bck

in? Oh wait... oops?

neutr

al Brexit

nega

tive x 3 3

Page 73 of 76

Shot Across #liberal Bow: Brits decided to ‘Make

England Great Again' →NEXT-UP #MakeAmericaGreatAgain

#Brexit #NEVERhillary @LouDobbs @greta

neutr

al USA

nega

tive 2 3

Hoping for Steve Harvey to come out and say the

#EUreferendum result was wrong lol. #Brexit #EURefResults

#NotMyVote

neutr

al Brexit

nega

tive x 2 2

Am I reading that right? Is the fact that Lindsay Lohan

tweeted about #brexit "Breaking News." Or did I have a

stroke? #msnbc

negat

ive celebrities/p

oliticians

neut

ral 2 3

Salute to British state and people to sticking to will of

majority and democracy, come what may #Brexit #Uk

negat

ive Brexit

nega

tive 2 2

I gotta be honest, it's bit of a relief to get confirmation that

America isn't the only country with stupid citizens. #Brexit

positi

ve USA

nega

tive 2 2

Last in first Out! #brexit neutr

al Brexit

neut

ral 3 3

Hey, United Kingdom, Imma let u finish, but USA had

the greatest #Brexit of ALL TIME!

positi

ve USA

neut

ral 1 3

oh boy, #texas is inspired to secede again based on

#Brexit. Not only is it illegal, but the real question is who

really cares for Texas?

negat

ive other

nega

tive 2 2

#NeverTrump goes to Scotland and talks about

sprinkler.Missed the whole point! #Brexit was crucial and

warranted some explanation from trump

negat

ive Trump

nega

tive 2 2

Thanks British #brexit twats, I'm feeling poorer today negat

ive economy

nega

tive x 1 2

Asked my wife if she heard about #Brexit and she said no.

Started to explain and she doesn’t even know about the EU. I

quit.

negat

ive Brexit

neut

ral 3 3

#Brexit is crushing victory 4 ppl against the

establishment. Get used 2 it. #EngalndLiberation #LeaveWins

positi

ve Brexit

posit

ive 2 2

@StephenNolan @bbc5live this Gerry guy is the

embodiment of smug, arrogant BBC leftwing metropolitan

self-proclaimed elites. Happy 4 #Brexit

negat

ive Brexit

posit

ive 2 2

An electoral college system sounds like a good idea right

abut now dunnit? #Brexit

neutr

al other

nega

tive x 2 2

We have an extraordinarily high level of international

reserves 177 billion dollars #Brexit #Mexico #PressRelease

positi

ve economy

posit

ive 3 2

#Brexit is similar to union-busting but not a union of dock

workers, it's a union of bankers, technocrats, oligarchs, fascists,

etc.

negat

ive Brexit

nega

tive 2 2

Looks like the EU has about 1 GB more free space now.

#Brexit

neutr

al EU

neut

ral x 3 2

Let love win! Let the religion of peace engulf your once

proud nation. Celebrate their tolerance! #onpoli #Brexit

positi

ve Brexit

posit

ive 1 1

@Lizabs68 By time #Brexit is complete, a majority of

those who voted & are still alive will be #Remain supporters.

#BrexitMustBeStopped

negat

ive Brexit

nega

tive 2 2

Talk of a 2nd referendum if EU gives us a new deal which

will appease the people voting #Leave Hopefully that comes to

pass #EUref #Brexit

neutr

al Brexit

nega

tive 2 2

This is shocking. “What is the EU”? asks citizens of the

UK, AFTER #Brexit.

negat

ive EU

nega

tive 2 2

Help my 'Get the fuck out of England' fund with this

#EURefResults design.

http://www.redbubble.com/people/lunarblaze/works/22256458

-dont-blame-me-eu … #EUreferendum #BrexitVote #Brexit

neutr

al

other

nega

tive x 3 3

Bed after a long two days. Three events today with one

topic #Brexit. People are uncertain but rational about what

happens next.

neutr

al Brexit

nega

tive 2 2

Nothing can save this day, but I guess we will always

remember it! #brexit

negat

ive Brexit

nega

tive 2 2

Guess I won’t have to deal with anymore Brits on holidy

who never tip #cheerio #brexit

positi

ve other

neut

ral x 2 2

Hey. England. If you wanna be a wanker go ahead… But

how about cutting Scotland and Ireland loose before ya drag

them down too! #Brexit

negat

ive Brexit

nega

tive 2 2

Page 74 of 76

What would it be called if the US decided to do a #Brexit?

U Sexit?

neutr

al USA

neut

ral x 2 2

#HistoryNotes #Brexit illustrates economic point no

school teaches: Money is Imaginary – Economies are Ideas –

they are what you make them

neutr

al economy

nega

tive x 2 2

There’s a vacancy, we already have a foot in the door via

Eurovision Australia might ass well have a crack! Let’s join

the EU! #Brexit

positi

ve EU

nega

tive x 2 2

Increase Google searches in that region of the wold asking

“what is #Brexit?” and “what is the UK?”

neutr

al Brexit

neut

ral 2 2

I'll bet all these Hollywood stars so knowledgable about

#Brexit are the same ones who were primate experts just a few

weeks ago. #clueless

negat

ive celebrities/p

oliticians

nega

tive x 2 2

#Brexit Angela Merkel's country faces having to pay an

extra £2.44billion a year to the annual EU budget once Britain

has left.

negat

ive celebrities/p

oliticians

nega

tive 2 1

DID YOU HEAR? The UK #Brexit took their country

back. Time for s to take USA back! Donate to Trump

Campaign now at

positi

ve USA

posit

ive 2 2

Both #Brexit and #Trump are disasters. No thanks. I'll

vote for @HillaryClinton and other sane people.

negat

ive

celebrities/p

oliticians

nega

tive 2 2

Will #Brexit disintegrate the EU? The EU needs a new

democratic constitution or it will disintegrate! Join #DiEM25

community, save EU!

negat

ive EU

nega

tive 3 2

To understand #brexit, the immigration issue etc...is to

understand y they are leaving home in the first place. Who is

shaping these events?

neutr

al Brexit

neut

ral x 2 2

The British call their Independence #Brexit... maybe we

should call ours #Mexit Vote #Trump2016

neutr

al USA

posit

ive 2 1

One positive from #Brexit is that it has shown which areas

of the UK need better funding for education.

positi

ve Brexit

nega

tive 1 2

Congratulations on your #Brexit from the globalists, hope

to do the same in Nov #Trump2016 #MakeAmericaGreatAgain

positi

ve USA

posit

ive 1 1

The EU's 'techno party' is hollowing out democracy

http://openermedia.blogspot.com/2015/05/the-eus-techno-

party-is-hollowing-out.html … #Brexit LoiTravail David

Cameron Scotland

negat

ive

other

nega

tive 2 2

If Sarah "Dumbass" Palin is excited about your decision..

you just made a HUGE global boner of a mistake. #Brexit

negat

ive

celebrities/p

oliticians

nega

tive 2 2

My thoughts and prayers go out to all the people affected

by this #Brexit fiasco

neutr

al Brexit

nega

tive 2 2

I'd be more upset about #brexit if it didn't sound so much

like #breakfast . Instead, I'm just hungry.

neutr

al other

nega

tive x 2 3

So, that was the dress rehearsal. Now that you Leavers have

seen the effects of your vote, would you like to try that again?

#Brexit

negat

ive Brexit

nega

tive x 2 1

Above anything else, I genuinely feel sad for our country

today. #EUref #Brexit #notmyvote

negat

ive Brexit

nega

tive 2 2

I never thought I would live to see it broken up. #Brexit negat

ive Brexit

nega

tive 2 2

IMHO #Brexit fallout is totally overblown. People are

making money off the rampant panic & uncertainty. Folks need

to calm the fuck down.

negat

ive Brexit

nega

tive 2 2

So long….and thanks for all the bypasses. #Brexit neutr

al Brexit

nega

tive 1 2

Something is changing........ #EURefResults #Brexit neutr

al Brexit

neut

ral 2 2

It’s failing #Brexit negat

ive Brexit

nega

tive 3 2

Maybe he thought Brexit was the guy's name? Man Who

Voted For #Brexit Is 'A Bit Shocked' His Vote Counted,

'Worried'

negat

ive Brexit

nega

tive x 2 2

#Brexit is the logical result of Thatcher's 'Big Bang.'

State-sanctioned inequality + the creation of ultra-capital &

#financialization.

negat

ive Brexit

nega

tive 2 2

Page 75 of 76

Lotsa peeps loving my fiction on #Brexit -> #Trump ->

World War 3 via Scots, Texit, Huxit, Grexit, Bletchit, Putin

neutr

al other

nega

tive 2 2

#Brexit processing takes years.dont see why #US #Stock

to be affected now?instead it is a good hands to buy in!

positi

ve economy

neut

ral 2 2

#Brexit sounds like a planet in the #StarWars EU which

makes this a very confusing day.

neutr

al Brexit

nega

tive x 3 2

Don't know a lot about #Brexit, but i do enjoy watching

the Super Pundit Class hyperventilating and clutching their

pearls.

positi

ve economy

nega

tive x 2 2

Yes, stock markets will plummet. $$ interests only care

about making more $$ for the sake of itself. Common sense.

#brexit #financialization

neutr

al economy

nega

tive 2 2

I served in the UK at RAF Bentwaters/Woodbridge &

could not be prouder of them taking their country back!

#Brexit

positi

ve Brexit

posit

ive 2 2

Never underestimate the power of stupid people in large

groups! #Brexit #jokeofthecentury

negat

ive Brexit

nega

tive x 2 2

#UK left the European Union...I guess it's time to find

work elsewhere =_= Thanks #Brexit

https://youtu.be/I17j7vzFnN0 via @YouTube

negat

ive Brexit

nega

tive x 2 2

The next James Bond will just be him spending 2 hours in

passport control De Gaulle #Brexit #JamesBond

negat

ive other

nega

tive x 3 2

Will #brexit hurt English Football League? neutr

al other

neut

ral 3 3

I need the #Brexit jokes to stop until I can refill my

prescription

neutr

al other

nega

tive x 2 2

#BrexitAdam didn't think his vote would count? now you

know folks, your vote ALWAYS counts! #Brexit #BrexitVote

#BrexitOrNot #Election2016

negat

ive Brexit

nega

tive 2 2

Can we all just fast forward to 2017 instead? #Brexit

#DonaldTrump #RefugeeCrisis

negat

ive Brexit

nega

tive x 2 2

Thank you, @NicolaSturgeon, for demonstrating what

true, compassionate leadership looks like in the face of

adversity. #Brexit

positi

ve celebrities/p

oliticians

nega

tive 1 2

if #Brexit was Rigged Find out who Invested in Gold

Profited just like insider Trading Before 911 on thr Airline

Stocks SECURITY FIRM COMEX

neutr

al economy

neut

ral 3 2

#BREXIT – Americans! Watch & learn from a VOTE

based on revenge and xenophobia. If you vote unhinged, there

are consequences! #NeverTrump

negat

ive Brexit

nega

tive 2 2

#brexit is so catchy I love it positi

ve Brexit

posit

ive 2 2

Page 76 of 76