Embed Size (px)
Citation preview
DeeplearningforbiomedicineII
15/11/17 1
Source:rdnconsul6ngSeoul,Nov2017
TruyenTranDeakinUniversity @truyenoz
truyentran.github.io
letdataspeak.blogspot.com
goo.gl/3jJ1O0
Diagnosis PrognosisDiscovery Care
Resources
Slidesandreferences: hJps://truyentran.github.io/acml17-tute.html
Keysurveypaper(updatedfrequently): Ching,Travers,etal."Opportuni=esAndObstaclesForDeepLearningInBiologyAndMedicine."bioRxiv(2017):142760
15/11/17 2
Agenda
Topic1:Introduc6on(20mins)
Topic2:Briefreviewofdeeplearning(25mins) Classicarchitectures Capsules Graphs Memory-augmentednets
Topic3:Genomics(25mins) Nanoporesequencing Genomicsmodelling
QA(10mins)
15/11/17 3
Topic4:Biomedicalimaging(15mins) Cellularimaging Diagnos6csimaging EEG/ECG
Topic5:Healthcare(25mins) Time-seriesofphysiomeasures Trajectoriespredic6on
Topic6:Genera6vebiomed(30mins) Few-shotlearning Genera6vemodels Drugdesign Futureoutlook
QA(10mins)Break (20 mins)
Sensingtechnologiesanddata
15/11/17 4
#REF:Ravì,Daniele,etal."Deeplearningforhealthinforma6cs."IEEEjournalofbiomedicalandhealthinforma4cs21.1(2017):4-21.
Rawsignalsareidealcandidatesfordeeplearning Speech&visiontechniquescanbeappliedwithminimalchanges
Thestateofbiomedicalimaging Ases6matedbyIBM,90%dataofhealthcareisimaging.
BiomedicalimagingisperhapsthemostreadyareaforcurrentDLtechniques: IttypicallymeansCNN! Thegameisinthedataacquisi=onandproblemdefini=on/transforma=on
Examplesofapplica6onareas: Cellularimaging TumorDetec6on&tracking BloodFlowQuan6fica6onandVisualiza6on MedicalInterpreta6on Diabe6cRe6nopathy
15/11/17 5 https://www.healthcare-informatics.com/news-item/analytics/ibm-unveils-watson-powered-imaging-solutions-rsna
http://engineering.case.edu
Tasks Classifica6on Segmenta6on Localiza6on
Challenges Lowquality Veryhighresolu6on Tinylocalizedareas
Skincancerdiagnosisusingincep=on-v3
15/11/17 6
#REF:AEstevaetal.Nature1–4(2017)doi:10.1038/nature21056
• 129,450clinicalimages• 2,032differentdiseases• Testagainst21board-cer6fieddermatologists• Usecase1:kera6nocytecarcinomasversusbenignseborrheickeratoses;• Usecase2:malignantmelanomasversusbenignnevi.
Microscopy+mobilephone+CNN
Highlyrelevantfor: Developingcountries Ruralareas
15/11/17 7
#REF:Quinn,JohnA.,etal."Deepconvolu6onalneuralnetworksformicroscopy-basedpointofcarediagnos6cs."MachineLearningforHealthcareConference.2016.
EEGàTensorRBMforalcoholicdiagnosis
control alcoholic
Subject representation
Tensor Restricted Boltzmann Machine 3D Spectrogram
15/11/17 8
#Ref:TuD.Nguyen,TruyenTran,D.Phung,andS.Venkatesh,Tensor-variateRestrictedBoltzmannMachines,AAAI2015.
EEGàMatrixLSTMàClassifica6on
EEGsegmentsasmatrices Temporaldynamicsasrecurrence
15/11/17 9
#REF:KienDo,TruyenTran,SvethaVenkatesh,“LearningRecurrentMatrixRepresenta6on”,ThirdRepresenta4onLearningforGraphsWorkshop(ReLiG2017)
Recurrent dynamics
ECGàCNNforheartaJackdetec6on
15/11/17 10
#REF:Acharya,U.Rajendra,etal."Applica6onofdeepconvolu6onalneuralnetworkforautomateddetec6onofmyocardialinfarc6onusingECGsignals."Informa4onSciences415(2017):190-198.
15/11/17 11
“Theyshouldstoptrainingradiologistsnow.” GeoffHinton(asofApril2017)
https://www.newyorker.com/magazine/2017/04/03/ai-versus-md
Agenda
Topic1:Introduc6on(20mins)
Topic2:Briefreviewofdeeplearning(25mins) Classicarchitectures Capsules Graphs Memory-augmentednets
Topic3:Genomics(25mins) Nanoporesequencing Genomicsmodelling
QA(10mins)
15/11/17 12
Topic4:Biomedicalimaging(15mins) Cellularimaging Diagnos6csimaging EEG/ECG
Topic5:Healthcare(25mins) Time-seriesofphysiomeasures Trajectoriespredic6on
Topic6:Genera6vebiomed(30mins) Few-shotlearning Genera6vemodels Drugdesign Futureoutlook
QA(10mins)Break (20 mins)
Modelingelectronicmedicalrecords(EMR) Needtomodelthehealthcareprocesses,whichareinterac6onsof: Diseaseprogression Interven6ons&careprocesses Recordingprocesses(ElectronicMedical/HealthRecords)
15/11/17 13
Source: medicalbillingcodings.org
EMRConnectsServices:SystemofSystems
Experts
Medical References Hospital
Practice Management
Pharmacy Radiology
Laboratory
Fivemainfunc6ons Integratedviewofpa6ent
data Clinicaldecisionsupport Clinicianorderentry Accesstoknowledge
resources Integratedcommunica6on
andrepor6ngsupport
ClinicalDecisionSupports
Supportprotocol/planningoftreatment/discharge. Suggestcourseofac6ons: E.g.,medica6on/dose/dura6on.
Es6materisk&predictoutcomes. Alert/reminder. Support(semi)automateddiagnosis.
15
riskpredic6on(prognosis)
readmission death toxicity
stress quality-of-life
progression to advanced stages length-of-stay
side effects suicide attempts
mental health
cancers
diabetes
COPD
heart failure
heart attack preterm
Warning:leakage!
Makesurethepa6entsarecountedAFTERfirstdiagnosis Oten,wehavefuturedataaswell Retrospec6venature
Neveruseoutcomestodoanything,exceptfortrainingthemodel OurearlysuicideaJemptclassifica6onfromassessmentswasaformofleakage: AnyaJemptinhistoryisconsideredasanoutcome.BUT: PreviousaJemptswereaccountedincurrentassessmentalready!
Preprocessing:Datanormaliza6on&dic6onarycompression Drugs&tests Drugcompaniesofferdifferentbrandnamesoftheessen6allythesamedrug DDD/ATCisthecentralregisterforthemedica6onclasses,maintainedbyWHO Severaltestnamesmaybethesame
Itmaynotberobusttousetheoriginal“vocabularies” TensofthousandsofICD-codes,thousandsofprocedures,hundredsofDRGs,thousandsofmedica6onclasses Codesareusuallyorganizedinhierarchy Choosingtherighthierarchyissta6s6calissue
AJendtorisksinIntensiveCareUnit(ICU)
15/11/17 18
Source: healthpages.org
Time,Parameter,Value 00:00,RecordID,132539 00:00,Age,54 00:00,Gender,0 00:00,Height,-1 00:00,ICUType,4 00:00,Weight,-1 00:07,GCS,15 00:07,HR,73 00:07,NIDiasABP,65 00:07,NIMAP,92.33 00:07,NISysABP,147 00:07,RespRate,19 00:07,Temp,35.1 00:07,Urine,900 00:37,HR,77 00:37,NIDiasABP,58 00:37,NIMAP,91 00:37,NISysABP,157 00:37,RespRate,19 00:37,Temp,35.6 00:37,Urine,60
Data: Physionet 2012
#REF:PhuocNguyen,TruyenTran,SvethaVenkatesh,“DeepLearningtoAJendtoRiskinICU”,IJCAI'17WorkshoponKnowledgeDiscoveryinHealthcareII:TowardsLearningHealthcareSystems(KDH2017).
Theneeds Accuracy Interpretability Asearlyaspossible
Theprocess: Irregular6me-seriesàRegular6me-stepsàDataimputa6onàBi-LSTMàMul=pleaUen=onsàClassifica6on
AJendtorisksinICU
15/11/17 19
AJen6onprobabili6es
Statetransi6on
Physiologicalmeasures
Time
#REF:PhuocNguyen,TruyenTran,SvethaVenkatesh,“DeepLearningtoAJendtoRiskinICU”,IJCAI'17WorkshoponKnowledgeDiscoveryinHealthcareII:TowardsLearningHealthcareSystems(KDH2017).
DeepPa=ent:Represen6ngmedicalrecordswithStackedDenoisingAutoencoder
15/11/17 20
Auto-encoder Feature detector
Representation
Raw data
Reconstruction
Deep Auto-encoder
Encoder
Decoder
#Ref: Miotto, Riccardo, et al. "Deep patient: An unsupervised representation to predict the future of patients from the electronic health records." Scientific reports 6 (2016): 26094.
DeepPa6ent:Resultsondiseaseclassifica6on
Time Interval = 1 year (76,214 patients)
Patient Representation AUC-ROC
Classification Threshold = 0.6
Accuracy F-Score RawFeat 0.659 0.805 0.084
PCA 0.696 0.879 0.104
GMM 0.632 0.891 0.072
K-Means 0.672 0.887 0.093
ICA 0.695 0.882 0.101
DeepPatient 0.773* 0.929* 0.181*
15/11/17 21
Trajectoriesmodeling:Challenges&opportuni6es Long-termdependencies
Irregular6ming
Mixtureofdiscretecodesandcon6nuousmeasures
Complexinterac6onofdiseasesandcareprocesses
Cohortofinterestcanbesmall(e.g.,<1K)
Richdomainknowledge&ontologies
15/11/17 22
Mul6modali6es:Text,physiologicalsignals(e.g.,EEG/ECG),images(e.g.,MRI,X-ray,re6na),genomics
Newmodali6es:socialmedial,wearabledevices
Explainability!
visits/admissions
time gap ?
prediction point
Visualisa=onandinterpreta=onarekeys!
Deepr:CNNforrepeatedmo6fsandshortsequences
15/11/17 24
output
max-pooling
convolution -- motif detection
embedding
sequencing
medical record
visits/admissions
time gaps/transfer phrase/admission
prediction
1
2
3
4
5
time gap record vector
word vector
?
prediction point
#REF:PhuocNguyenetal.,Deepr:AConvolu6onalNetforMedicalRecords,IEEEJournalofBiomedicalandHealthInforma4cs,vol.21,no.1,pp.22–30,Jan.2017
Deepr:Diseaseembedding&mo6fsdetec6on
15/11/17 25
E11I48I50Type2diabetesmellitusAtrialfibrilla6onandfluJerHeartfailureE11I50N17Type2diabetesmellitusHeartfailureAcutekidneyfailure
DeepCare:intervenedlong-termmemoryofhealth
Illnessstatesareadynamicmemoryprocess→moderatedby6meandinterven6on
Discreteadmission,diagnosisandprocedure→vectorembedding
Timeandpreviousinterven6on→“forgeyng”ofillness
Currentinterven6on→controllingtheriskstates
15/11/17 26
#REF: Trang Pham, et al., Predicting healthcare trajectories from medical records: A deep learning approach, Journal of Biomedical Informatics, April 2017, DOI: 10.1016/j.jbi.2017.04.001.
DeepCare:Dynamics
15/11/17 27
memory
*input gate
forget gate
prev. memory
output gate
*
*
input
aggregation over time → prediction
previous intervention
history states
current data
time gap
current intervention
current state
New in DeepCare
DeepCare:Structure
15/11/17 28
Time gap
LSTM
Admission (disease)
(intervention)
Vector embedding
Multiscale pooling Neural network
Future risks
Long short-term memory
Latent states
Future History
LSTM LSTM LSTM
DeepCare:Twomodesofforgeyngasafunc6onof6me15/11/17 29
→ decreasing illness
→ Increasing illness
DeepCare:predic6onresults
15/11/17 30
Intervention recommendation (precision@3) Unplanned readmission prediction (F-score)
12 months 3 months 12 months 3 months
Trajectoriespredic6onGenera6ngasubsetoftreatments
Genera6nganen6rehealth/caretrajectory
Challenges:globalloss,meaningfulevalua6onmetrics
Asolu6on:Memory-augmentedneuralnets(MANN)
15/11/17 31
Source: deepmind.com
#REF: Graves, Alex, et al. "Hybrid computing using a neural network with dynamic external memory.” Nature 538.7626 (2016): 471-476.
(LeCun, 2015)
DualControllerWrite-protectedMemory-AugmentedNeuralNetworks(DCw-MANN)(Leetal,workinprogress)
LSTMDLSTMELSTMELSTME
M
LSTMD
WE
WDoutput
c,h
input
c,h
c,h
Separatecontrollersforencodinganddecoding Memoryiswrite-protectedduringdecoding Usememoryasareplacementfor: AJen6on Skipconnec6on
DCw-MANN
Modelingmul6pledisease-treatmentinterac6onsover6me Co-morbidityisthenorminmodernmedicine
Eachhospitalvisitcontainsasetofdiseasesandasetoftreatments
Thereareinterac6onsbetweenmul6-diseasesandmul6ple-treatments
Algebraicview:Health=RNN(Illness–Interven=on)
15/11/17 34
#REF: P Nguyen, T Tran, S Venkatesh, “Finding Algebraic Structure of Care in Time: A Deep Learning Approach”, NIPS Workshop on Machine Learning for Health (ML4H), 2017
Results(AUC)
15/11/17 35
Bigroom:Towardspersonalizedhealthcare Medicalprac6ceasrecommendersystems(XavierAmatriain,healthcareRecsysWorkshop,Como,2017)
ClinicalPrac6ceGuidesarenotpersonalized Researchdoneon“homogeneous”,healthysubjects
Itisveryhardfordoctorsto“manually”personalizetheir“recommenda6ons”
15/11/17 36
Otherrefs
Choi,Edward,etal."Doctorai:Predic6ngclinicaleventsviarecurrentneuralnetworks."MachineLearningforHealthcareConference.2016.
Lipton,ZacharyC.,etal."LearningtodiagnosewithLSTMrecurrentneuralnetworks."arXivpreprintarXiv:1511.03677(2015).
Harutyunyan,Hrayr,etal."Mul6taskLearningandBenchmarkingwithClinicalTimeSeriesData."arXivpreprintarXiv:1703.07771(2017).
Choi,Edward,etal."RETAIN:Aninterpretablepredic6vemodelforhealthcareusingreverse6meaJen6onmechanism."AdvancesinNeuralInforma4onProcessingSystems.2016.
15/11/17 37
Agenda
Topic1:Introduc6on(20mins)
Topic2:Briefreviewofdeeplearning(25mins) Classicarchitectures Capsules Graphs Memory-augmentednets
Topic3:Genomics(25mins) Nanoporesequencing Genomicsmodelling
QA(10mins)
15/11/17 38
Topic4:Biomedicalimaging(15mins) Cellularimaging Diagnos6csimaging EEG/ECG
Topic5:Healthcare(25mins) Time-seriesofphysiomeasures Trajectoriespredic6on
Topic6:Genera6vebiomed(30mins) Few-shotlearning Genera6vemodels Drugdesign Futureoutlook
QA(10mins)Break (20 mins)
Few-shotdeeplearning
Lotsofbiomedicalproblemsaredatapoor Raredrugs Rarediseases
Distancemetricslearning(DML)methods Learntofullanypairofthesimilardatapoints,andpushthedissimilar Well-knownmethods:Siamesenetworks
Meta-learningstrategies Tasksarepresentedinsequence Newtaskscanborrowfromsimilarpriortasks
15/11/17 39
#REF:Chopra,Sumit,RaiaHadsell,andYannLeCun."Learningasimilaritymetricdiscrimina6vely,withapplica6ontofaceverifica6on."ComputerVisionandPaZernRecogni4on,2005.CVPR2005.IEEEComputerSocietyConferenceon.Vol.1.IEEE,2005.
(Santoroetal,2016)
Meta-learningstrategies
15/11/17 40
(Hochreiteretal.,2001)
(Mishraetal,2017)
Genera6vemodels
15/11/17 41
Many applications:
• Text to speech
• Simulate data that are hard to obtain/share in real life (e.g., healthcare)
• Generate meaningful sentences conditioned on some input (foreign language, image, video)
• Semi-supervised learning
• Planning
Afamily:RBMàDBNàDBM
15/11/17 42
energy
Restricted Boltzmann Machine (~1994, 2001)
Deep Belief Net (2006)
Deep Boltzmann Machine (2009)
Varia6onalAutoencoder(Kingma&Welling,2014) Twoseparateprocesses:genera6ve(hiddenàvisible)versusrecogni6on(visibleàhidden)
http://kvfrans.com/variational-autoencoders-explained/
Gaussian hidden variables
Data
Generative net
Recognising net
GAN:implicitdensitymodels(AdaptedfromGoodfellow’s,NIPS2014)
15/11/17 44
GAN:GeneratedSamples SomehighqualitypicturesgeneratedbyGAN
15/11/17 45 Real Generated
http://kvfrans.com/generative-adversial-networks-explained/
Genera6vedeeplearningfordrugdiscovery Predic6ngbioac6vi6esfrommolecules Drugrepresenta6on,unsupervisedlearningfromgraphs
Generatefrombioac6vi6estomoleculargraphs #REF: Roses, Allen D. "Pharmacogenetics in drug discovery
and development: a translational perspective." Nature reviews Drug discovery 7.10 (2008): 807-817.
$500M - $2B
Combinatorialchemistry
Generatevaria6onsonatemplate Returnsalistofmoleculesfromthistemplatethat Bindtothepocketwithgoodpharmacodynamics? Havegoodpharmacokine6cs? Aresynthe6callyaccessible?
15/11/17 47
#REF: Talk by Chloé-Agathe Azencott titled “Machine learning for therapeutic research”, 12/10/2017
Firststep:Mapmolecule→drugproper6es(binding/ac6ng) Drugsaresmallbio-molecules Tradi6onaltechniques: Graphkernels(ML) Molecularfingerprints(Chemistry)
Moderntechniques Moleculeasgraph:atomsasnodes,chemicalbondsasedges
15/11/17 48
#REF: Penmatsa, Aravind, Kevin H. Wang, and Eric Gouaux. "X-ray structure of dopamine transporter elucidates antidepressant mechanism." Nature 503.7474 (2013): 85-90.
Molecularfingerprints
15/11/17 49
#REF: Duvenaud, David K., et al. "Convolutional networks on graphs for learning molecular fingerprints." Advances in neural information processing systems. 2015.
Graphmemorynetworks(Phametal,2017,workinprogress)
15/11/17 50
Controller
… Memory
Drug molecule
Task/query Bioactivities 𝒚
query
15/11/17 51
Graphmemorynetworks:Results
One-shotlearningfordrugdiscovery
15/11/17 52
#REF:Altae-Tran,Han,etal."LowDataDrugDiscoverywithOne-ShotLearning."ACScentralscience3.4(2017):283-293.
Drugdesignandgenera6on
Wenowhavemethodsforcomputebioac6v6esofadrugmolecule
Weneedareversemethodtogeneratedrugmoleculesfromdesirablebioac6vi6es
Thespaceofdrugsises6matedtobe1e+23to1e+60 Only1e+8substancessynthesizedthusfar.
Itisimpossibletomodelthisspacefully.
Thecurrenttechnologiesarenotreadyforgraphgenera6ons.
Butapproximatetechniquesdoexist.
15/11/17 53
Source: pharmafactz.com
Source: wikipedia.org
Thetrick:Moleculeàstring
UsingSMILESrepresenta6onofdrug,toconvertamoleculargraphintoastring SMILES=SimplifiedMolecular-InputLine-EntrySystem
Thenusingsequence-to-sequence+VAE/GANtomodelthecon6nuousspacethatencodes/decodesSMILESstrings Alloweasyop6miza6ononthecon6nuousspace Problem:Stringàgraphsisnotunique!
AbeJerwayistoencode/decodegraphdirectly.
15/11/17 54
#REF:Gómez-Bombarelli,Rafael,etal."Automa6cchemicaldesignusingadata-drivencon6nuousrepresenta6onofmolecules."arXivpreprintarXiv:1610.02415(2016).
VAEfordrugspacemodelling
15/11/17 55
UsesVAEforsequence-to-sequence. #REF:Bowman,SamuelR.,etal."Genera6ngsentencesfromacon6nuousspace."arXivpreprintarXiv:1511.06349(2015).
AdversarialAutoencodersKadurinetal.Molecularpharmaceu4cs2017
Inputoftheencoder:thefingerprintofamolecule Thedecoderoutputsthepredictedfingerprint. Thegenera6vemodelgeneratesavectorE,whichisthendiscriminatedfromthelatentvectoroftherealmoleculebythediscriminator.
Morerefs
Segler,MarwinHS,etal."Genera6ngfocussedmoleculelibrariesfordrugdiscoverywithrecurrentneuralnetworks."arXivpreprintarXiv:1701.01329(2017).
Kusner,MaJJ.,BrooksPaige,andJoséMiguelHernández-Lobato."GrammarVaria6onalAutoencoder."arXivpreprintarXiv:1703.01925(2017).
Gupta,Anvita,etal."Genera6veRecurrentNetworksforDeNovoDrugDesign."MolecularInforma4cs(2017).
15/11/17 57
Livinginthefuture:AIforhealthcare Somespecula6ons(byme): hJps://letdataspeak.blogspot.com.au/2017/02/living-in-future-deep-learning-for.html
Bearinmindthatanythingbeyond5yearsarenearlyimpossibletopredict!
Kai-FuLee’svision: Wave1:Internetdata(àPubMed,socialmedia) Wave2:Businessdata(àEMR) Wave3:Digitalizethephysicalworld(àDrugs) Wave4:Fullautoma6on(àRobotsurgeons,GPs)
15/11/17 58
Towardpersonalizemedicine
Willthispa6entresponsetothattreatment? Canwefindthebesttreatmentforapa6ent? Whichbiomarkerspredictthepa6ent’sresponse? SoundfamiliartoRecommenderSystems(pa6ent=user,treatment=item)?
15/11/17 59
#REF: Talk by Chloé-Agathe Azencott titled “Machine learning for therapeutic research”, 12/10/2017
TowardsadialogsystemàReplaceGP?
15/11/17 60
Leveragingexis6ngknowledge Medicalknowledgebases Medicaltexts Probablyneedstobuildknowledgebasesfromtext
PersonalizingthroughEMRs Learnfromhospitalsdata
Askrightques6onsàFindinganswersfromdatabasesàGenera6ngdialog
Neverendinglearning(NEL).
Me and Woebot
Otherroomsfordeeplearning -omics Geneexpression Proteomics
Neuroscience Modelsforspiketrains Deeplearningforconnectomics Neuroscience-inspiredDL
BiomedicalNLP ClassicalNLP Socialmedia Knowledgegraphs
Wearables Trackingthestateofphysicalandmentalhealth Lifestylemanagement&monitoring
15/11/17 61
HealthInsurance Futureillness/spendingpredic6on Proac6vepreven6onprograms WARNING:Workingforinsurancecompaniesdoesraiseethicalconcerns!
Nutri6on Mobilephonevisionàcalories
ExplainableAI Seeingthroughtheblack-box,e.g.,visualiza6on,mo6fs
Explainablearchitecturesthatusebiologicalmechanismsandmedicalontologies
Dualarchitecture:predictor&explainer
