View
553
Download
2
Category
Tags:
Preview:
DESCRIPTION
ECIR 2013 workshop keynote
Citation preview
Recruiters, Job Seekers and Spammers: Innovations in Job Search at LinkedIn
Daria SorokinaSenior Data ScientistLinkedIn
Part I: Recruiters
“Multiple Objective Optimization in Recommendation Systems”, Mario Rodriguez, Christian Posse, Ethan Zhang. RecSys’12
TalentMatch
TalentMatch Job Posting
Member Profiles Ranked
Talent
Job Posting
titlegeocompany
industrydescriptionfunctional area
…
Candidate
Generalexpertisespecialtieseducationheadlinegeoexperience
Current Positiontitlesummarytenure lengthindustryfunctional area…
Text similarityfeatures
TalentMatch Model
The model can be trained on user activity signals like job ad clicks or job applications
TalentMatch Utility = fn(email rate, reply rate)
Email Rate
ReplyRate
Problem! Job seeker?
Recruiter
Job Seeker Intent
Model: time till the job change o How long will this person stay in this job after this date?o Trained on past job positions from our users profileso Accelerated failure time (AFT) modelo
ACTIVE
PASSIVE
NON-JOB-SEEKER
Job-SeekerFeatureExample: Attrition by Industry
Pro
babi
lity
Time
Job-Seeking Intent:16x reply rate on career-related mail
ReplyRate
TalentMatch Utility
fn(email rate, reply rate)
Talent Match rankingMatch Score1, Item X, 0.98, Non-Seeker2, Item Y, 0.91, Non-Seeker---------------------------------------3, Item Z, 0.89, Active
Improved rankingMatch Score, Reranking Score1, Item X, 0.98, 0.98, Non-Seeker2, Item Z, 0.89, 0.93, Active--------------------------------------------3, Item Y, 0.91, 0.91, Non-Seeker
Re-rankingfunction f()
Divergencescore
Objective Score: #Active in top N
How: Controlled Re-ranking Ranking Score
Distributions
optimize for both
Part II: Job Seekers
Learning to Rank. Fast and personalized.
Job Search. Query “Data Scientist LinkedIn”
Regular approach– A data point is a pair: {Query, Document}– Data label: “Is this document relevant for this query?”
Can be done by crowdsourcing
Job Search reality– A data point is a triple: {Query, Job position, User}– Data label: “Is this job relevant for this user who asked
this query?” Depends on the user’s location, industry, seniority… Too much to ask from a random person Have to collect labels from user signals
Learning To Rank
Each pair is flipped with a 50% chance
Choose pairs where only the lower document is clicked
Save 1 positive (lower) and 1 negative (upper) results for the labeled data set
We use simplified version of FairPairs(Radlinski, Joachims AAAI’06)
not flipped
flipped
flipped
not flipped
Clicked!
✔
✗
✔
✔
✔
✗
✗
✗
label 0
label 1
The user clicks or skips only whatever is shown Bad results are not shown So there will be no “really bad” negatives in the training data We need to add them!
For queries with many results, add all results from the last page as “easy negatives”
Fair Pairs data is not enough for training
…
label 0
label 0
label 0
label 0
…
Best models for LTR are complex ensembles of trees– See results of Yahoo Learning to Rank ‘10 competition– LambdaMART, BagBoo, Additive Groves, MatrixNet …
Complex models come at a cost– It takes long to calculate predictions– Requires a lot of optimization, often used with multi-level
ranking
Can we train a simple model that will resemble a complex one?
– Train a complex model– Get insights on what it looks like– Modify a simple model accordingly
Learning To Rank – Training a Model
Base simple model – logistic or linear regression
– Does not handle well features with non-linear effects– Does not handle interactions (e.g., if-then-else rules)
Target complex model – Additive Groves– (Sorokina, Caruana, Riedewald ECML’07)
– Comes with interaction detection and effect visualization tools
+…+(1/N)· + (1/N)· +…+ (1/N)·+…+ +…+
Training a Simple Model using a Complex Model
Additive Groves can model and visualize non-linear effects
Improving LR – Feature Transformations
feature values
aver
age
pred
ictio
n Approximate the effect curve with a polynomial transform T(x)
– anything simple will do
Apply T(x) to the original feature values
Now the feature effect is linear Regression model will love it!
T(x) values
aver
age
pred
ictio
n
Additive Groves’ interaction detection tool produces a list of strong interactions and corresponding joint effect plots
Improving LR – Interaction Splitsav
erag
e pr
edic
tion
values of feature X1
Effect of X1 is stronger when X2 = 0
Simple regression will not capture this
Often such X2 interacts with other features as well
X 2=0
X2=1
Solution: Build separate models
for different values of X2
X2=?
X2=1
X2=0
Both operations (effect transforms and interaction splits) can be applied multiple times in any order
Resulting model – a simple tree with regression model leaves
Gives a significant boost to the performance of the basic LR model
Improving LR – Tree with LR leaves and transforms
X 2=0
X2=?
X2=1
X10< 0.1234 ?
Yes
No
TreeExtra package
A set of machine learning tools– Additive Groves ensemble– Interaction detection– Effect and interaction visualization
http://additivegroves.net– Created by Daria Sorokina while in Cornell, CMU, Yandex, LinkedIn
from 2006 to 2013
Part III: Spammers
Fighting black SEO
Search Spam
Search Spam
Search Spam
Training data for the search spam classifier
Find the queries targeted by spammers.– 10,000 most common non-name queries.– Spammers love optimizing for [marketing] – But not so much for [david smith]
Look at top results for a generic user.– i.e., show unpersonalized search results.
Label data by crowdsourcing.– Definition of spam is non-personalized
Train a model– Spam scores are recalculated offline once in a while– So the model complexity is not an issue– Additive Groves works well. (Could use any ensemble of trees)
ROC curve. Choosing thresholds.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
a
b
Spam score threshold
0 < a < b < 1
Spam model yields a probability between 0 and 1.
Convert spam score into a factor– [0.0 <= score <= a]
not a spammer, factor = 1.0
– [b <= score <= 1.0] Spammer factor = 0.0
– [a <= score <= b] Suspicious linearly scale score from [a, b] to [1, 0]
Multiply relevance score by factor
Integrating the Spam Score into Relevance
We are hiring!
Recommended