Empower Public Health through Social Media
Empower Public Health through Social MediaZhen Wang,
Ph.D.Insight Health Data Science
http://54.191.168.240
TextCleaning, TokenizingConvert to Feature Vectors
I like food!Food is good!I had some good food.
i, like, foodfood, is, goodi, had, some, good, food
e.g., TF-IDF
Im really good with numbers!
ilikefoodisgoodhadsome111000000111001010111
Downweight, NormalizeMachine LearningNumbersNatural Language
Processing
Text ClassificationNormalized Retweet CountsNumber of
TweetsDistribution of Tweets Sample Imbalance Classification (0/1:
Not / Retweeted)Logistic Regression
Threshold: 0.005
Misclassification Error: 22%
0011Train Testdownsampling
0.810.740.260.19Normalized Confusion MatrixCodes:
github.com/zweinstein/SpreadHealth_dev
Zhen (Jen) Wang
Beta TesterSince 2015
Editor since 2015
Traditional MedicineScience FictionPublic Speaking
Online EducationPh.D. in Physical Chemistry
Thank you!
See the App in Action:
Text Preprocessing PipelineText Cleaning:Convert to lower
caseReplace URL, #, and @Remove special characters other than
emoticonsRemove stopwords
Tokenizing:Splitting each documents into individual elements
Bag-of-Words or N-gramsStemming Porter Stemmer was usedSnowball or
Lancaster stemmer faster but more aggressiveLemmatization
computationally more expensive but little impact on the performance
of text classification Term Frequency-Inverse Document Frequency
(tf-idf):Term Frequency--tf(t,d): the number of times a term t
occurs in a document dUsed to downweight frequently occurring words
in the feature vectors tf(t,d)Document Frequency--df(d,f): the
number of documents d that contain a term t.The implementation in
Scikit-learn
Lastly, L2 normalization
Train Dataset: 10000 tweets on diabetes (4782 retweeted);Test
Set Accuracy (Random Chance 0.49 on positive class): KNN: 60%Naive
Bayes: 67%Logistic regression: 75% (chosen and tested on imbalanced
test data)Potential Improvements: Decision Trees with
Bagging/Boosting (e.g., Random Forest, XGBoost)Other
Features:Polarity & SentimentLength Out-of-Core Incremental
Learning with Stochastic Gradient Descent (Advantages of Logistic
Regression)Automatic Update to SQLite Database and to the
ClassifierPrediction Algorithms
