14
Prepared for: BDAM One Size doesn’t fit all Making a case for Federated Data Science Nitin Lamba & Suhas Gogate (Riya Makhijani, Ruta Joshi, Shantanu Tulshibagwale)

One size doesn’t fit all: Federated Data Science using Ampool - by Nitin Lamba, Suhas Gogate, Ampool

Embed Size (px)

Citation preview

Page 1: One size doesn’t fit all: Federated Data Science using Ampool - by Nitin Lamba, Suhas Gogate, Ampool

Prepared for: BDAM

OneSizedoesn’tfitallMakingacaseforFederatedDataScience

NitinLamba&Suhas Gogate

(RiyaMakhijani,Ruta Joshi,ShantanuTulshibagwale)

Page 2: One size doesn’t fit all: Federated Data Science using Ampool - by Nitin Lamba, Suhas Gogate, Ampool

Prepared for: BDAM

DataScienceisamulti-frameworkenvironment

! DataScientists! ! Business

Analysts!! DataEngineers!

language

tool

Page 3: One size doesn’t fit all: Federated Data Science using Ampool - by Nitin Lamba, Suhas Gogate, Ampool

Prepared for: BDAM

DataProcessingisaniterative,multi-step process

…" " "

# #

&

#

ModelExplore ServeFlatten

Page 4: One size doesn’t fit all: Federated Data Science using Ampool - by Nitin Lamba, Suhas Gogate, Ampool

Prepared for: BDAM

ModelExplore ServeFlatten

" " "

# #

&

#

Whatiffastobjectaccess isavailableacrossstages?

Page 5: One size doesn’t fit all: Federated Data Science using Ampool - by Nitin Lamba, Suhas Gogate, Ampool

Prepared for: BDAM

ModelExplore ServeFlatten

" " "

# #

#

…&

Reducetimetoinsights,buildreal-timemodels

✅ Object-baseddataexchange

✅ NoDataCopies

✅ End-to-endspeedup

✅ IncreasedConcurrency

Page 6: One size doesn’t fit all: Federated Data Science using Ampool - by Nitin Lamba, Suhas Gogate, Ampool

Prepared for: BDAM

Ampool isamemory-orientedActiveDataStore...

AprimarystoreforALLdataprocessing

StoreALLactivedata&updateit,asreqd.

Servesdataconcurrentlytomultiplestages&tenants

DataPersistence

Page 7: One size doesn’t fit all: Federated Data Science using Ampool - by Nitin Lamba, Suhas Gogate, Ampool

Prepared for: BDAM

...offerschoicefordifferentanalyticalneeds...

'

( ) ) …

MPPDatabaseLocalFileSystem

Software-DefinedMemory

*Experimental; waiting feedback fromdesignpartners

Page 8: One size doesn’t fit all: Federated Data Science using Ampool - by Nitin Lamba, Suhas Gogate, Ampool

Prepared for: BDAM

…anddeliversvaluetoall typesofdatausers

Nochangeinapplicationlogic

Makeconfig.changesonly

Nochange inexistingusertools

Getmemoryspeeds

NohassledeploymentUsecurrentmgmt.tools

! DataArchitectDataEngineers!

! BusinessAnalystsDataScientists!

! DataAdminsInfra/SysAdmins!

Page 9: One size doesn’t fit all: Federated Data Science using Ampool - by Nitin Lamba, Suhas Gogate, Ampool

Prepared for: BDAM

In-MemoryDistributedSys

Low-latencyComms.

Key-ValueStore

FunctionPushdown

+

HighThroughput

TableStore

NativeInterfacePluggableStoreManager

JavaAPI

MASH(CLIExt)

JavaAPI

Itisbuiltonaproven in-memorytechnology

SmartDataTiering

MatureEventModel

TunableConsistency

Metadata/Catalog

SecurityAuthZ

Page 10: One size doesn’t fit all: Federated Data Science using Ampool - by Nitin Lamba, Suhas Gogate, Ampool

Prepared for: BDAM

CardHolder

Init PurchasePay bill

Merchant

ScansCardProcessesTx

ABCBankOffersCardsServesBills

AnomalyDetection:Anillustrativeuse-case…

TransactionLogs

Tx Tx Tx…

Tx Tx Tx …

Acquirer

IntermediatesTx

Issuer

ProcessrequestMonitorsactivityBilling

Events

Network

*++

AggregatesTx

1 2 34 5 67 8 9

0

Page 11: One size doesn’t fit all: Federated Data Science using Ampool - by Nitin Lamba, Suhas Gogate, Ampool

Prepared for: BDAM

…with Kafka,Spark,R &Ampool…

IngestRepattheIssuingBank Financial

TransactionsStream&BatchProcessing

Monitoring/App

Tx ValidateUserProfile$impact

Explore&Model

Page 12: One size doesn’t fit all: Federated Data Science using Ampool - by Nitin Lamba, Suhas Gogate, Ampool

Prepared for: BDAM

RStudio

…supportingmultipleanalyticaltools

KafkaDataGenerator

Ingest Streaming Explore/Model

CrossValid Tx Log Outliers

User/AppData

FinTx Logs

SparkStreaming

Fin_Txn

Tx Confirm.UserUpdates

Dstream RDD

In_DF

Model

SparkBatch/ML

Batch_DF

Batch

Page 13: One size doesn’t fit all: Federated Data Science using Ampool - by Nitin Lamba, Suhas Gogate, Ampool

Prepared for: BDAM

Demo

Page 14: One size doesn’t fit all: Federated Data Science using Ampool - by Nitin Lamba, Suhas Gogate, Ampool

Prepared for: BDAM

Questions?

, /company/ampool-inc- -. /AmpoolIO@AmpoolIO/ www.ampool.io