Upload
jack-gudenkauf
View
3.237
Download
3
Embed Size (px)
Citation preview
|05/01/2023
Jack Gudenkauf VP Big Data
scala> sc.parallelize(List("Kafka Spark Vertica"), 3).mapPartitions(iter => { iter.toList.map(x=>print(x)) }.iterator).collect; println)(
https://twitter.com/_JG
2
AGENDA
1 .Background
2 .PSTL overviewParallelized Streaming Transformation Loader
3 .Parallelism in Kafka, Spark, Vertica
4 .PSTL drill downParallelized Streaming Transformation Loader
5 .Vertica Performance!
3
AGENDA
1 .Background
4
PLAYTIKA Founded in 2010
Social Casino global category leader 10 games
13 platforms 1000+ employees
A trifecta requires you to select the first three finishers in order and can lead to big pay-offs.
Boxing lets your selections come in any order and you still win.Kafka - Spark - Vertica
Placing Your Trifecta Box Bet on Kafka, Spark, and HP Vertica
https://www.linkedin.com/in/jackglinkedin 6
MY BACKGROUND
Playtika, VP of Big DataFlume, Kafka, Spark, ZK, Yarn, Vertica. [Jay Kreps (Kafka), Michael Armbrust (Spark SQL), Chris Bowden (Dev Demigod)]
MIS Director of several start-up companiesDataflex a 4GL RDBMS. [E.F. Codd]
Self-employed ConsultantIntercept Dataflex db calls to store and retrieve data to/from Btrieve and IBM DB2 Mainframe
FoxPro, Sybase, MSSQL Server betaDesign Patterns: Elements of Reusable Object-Oriented Software [The Gang of Four]
Microsoft; Dev Manager, Architect CLR/.Net Framework, Product Unit Manager Technical Strategy GroupInventor of “Shuttle”, a Microsoft product in use since 1999 A distributed ETL based on MSMQ which influenced MSSQL DTS (SQL SSIS)
[Joe Celko, Ralph Kimball, Steve Butler (Microsoft Architect)]Twitter, Manager of Analytics Data WarehouseCore Storage; Hadoop, HBase, Cassandra, Blob StoreAnalytics Infra; MySQL, PIG, Vertica (n-Petabyte capacity with Multi-DC DR)
[Prof. Michael Stonebraker, Ron Cormier (Vertica Internals)]
7
A QUEST
With attributes ofOperational Robustness
High AvailabiltyStronger durability guarantees
Idempotent (an operation that is safe to repeat)Productivity
AnalyticsStreaming, Machine Learning, BI, BA, Data Science
Rich Development env.Strongly typed, OO, Functional, with support for set based logic and aggregations (SQL)
PerformanceScalable in every tier
MPP for Transformations, Reads & Writes
A Unified Data Pipeline with Parallelism from Streaming Datathrough Data Transformationsto Data Storage (Semi-Structured, Structured, and Relational Data)
REST API
FlumeApache Flume™
ETL
JAVA ™Parser & Loader
MPP Columnar DW HP Vertica™
ClusterUserId <-> UserGId
Analytics of Relational Data Structured Relational and Aggregated Data
Application
ApplicationGame Applications
UserId: INTSessionId: UUId (36)
Bingo Blitz
UserId: Int SessionId: UUId (32)
Slotomania
UserId: varchar(32) SessionId: varchar(255)
WSOP
C O P Y
Playtika Santa Monicaoriginal ETL Architecture
Extract Transform LoadSingle Source of Truths to Global SOT
Unified SchemaJSON
Local Data Warehouses
Original Architecture (ETL)
1
2 3 4
5
9
AGENDA
1 .Background
2 .PSTL overviewParallelized Streaming Transformation Loader
Real-Time Messaging
Apache Kafka™Cluster
Analytics of [semi]Structured [non]Relational Data Stores Real-Time Streaming ✓Machine Learning
✓ Semi-Structured Raw JSON Data✖Structured (non)relational Parquet Data
Structured Relational and Aggregated Data
ETL
Resilient Distributed Datasets Apache Spark™ Hadoop™ Parquet™
Cluster ✓✓✖
REST APIOr Local Kafka
Application
ApplicationGame Applications
UserId: INTSessionId: UUId (36)
Bingo Blitz
UserId: Int SessionId: UUId (32)
Slotomania
UserId: varchar(32) SessionId: varchar(255)
WSOP
Unified SchemaJSON
Local Data Warehouses
PSTL is the new ETL
MPP Columnar DW HP Vertica™
Cluster
MPP
1 2 3
P a r a l l e l i z e d S t r e a m i n g T r a n s f o r m a t i o n L o a d e r4
5
New PSTL ArchitectureNew PSTL Architecture
11
AGENDA
1 .Background
2 .PSTL overviewParallelized Streaming Transformation Loader
3 .Parallelism in Kafka, Spark, Vertica
Apache Kafka ™ is a distributed, partitioned, replicated commit log service
Producer Producer Producer
Kafka Cluster(Broker)
Consumer Consumer Consumer
A topic is a category or feed name to which messages are published .For each topic, the Kafka cluster maintains a partitioned log that looks like
Each partition is an ordered, immutable sequence of messages that is continually appended to—a commit log .The messages in the partitions are each assigned a sequential id number called the offset that uniquely identifies each message within the partition.
The Kafka cluster retains all published messages—whether or not they have been consumed—for a configurable period of time.
Kafka is Not a message Queue (Push/Pop)
Apache Kafka ™
SPARK RDDA Resilient Distributed Dataset [in Memory]
Represents an immutable, partitioned collection of elements that can be operated on in parallel
Node 1 Node 2 Node 3 Node… RDD 1 RDD 1
Partition 1RDD 1
Partition 2RDD 1
Partition 3
RDD 3 RDD 3Partition 2
RDD 3Partition 3
RDD 3Partition 1
RDD 2RDD 2
Partition 1 to 64
RDD 2
Partition 65 to 128
RDD 2Partition
193 to 256
RDD 2Partition
129 to 192
15Initiator Node
An Initiator Node shuffles data to storage nodes
Vertica Hashing & Partitioning
16
AGENDA
1 .Background
2 .PSTL overviewParallelized Streaming Transformation Loader
3 .Parallelism in Kafka, Spark, Vertica
4 .PSTL drill downParallelized Streaming Transformation Loader
"{appId": 3, "sessionId": ”7”, "userId": ”42} ”"{appId": 3, "sessionId": ”6”, "userId": ”42} ”
Node 1 Node 2 Node 3 Node 4
3 Import recent Sessions
Apache Kafka ClusterTopic: “appId_1” Topic: “appId_2” Topic: “appId_3”
old new
MySQL Kafka Table
appId,TopicOffsetRange,
Batch_IdSessionMax TablesessionGIdMax Int
UserMax TableuserGIdMax Int
appSessionMap_RDDappId: Int
sessionId: StringsessionGId: Int
appUserMap_RDDappId: Int
userId: StringuserGId: Int
appSessionappId: Int
sessionId: varchar(255)sessionGId: Int
appUserappId: Int
userId: varchar(255)userGId: Int
1 Start a Spark Driver per APP
Node 1 Node 2 Node 3
4 Spark Kafka [non]Streaming job per APP (read partition/offset range)
5 select for update ;update max GId
5 Assign userGIds To userIdsessionGIds To sessionId
6 Hash(userGId) to RDD partitions with affinityTo Vertica Node (Parallelism)
7 userGIdRDD.foreachPartition…{ stream.writeTo(socket)}...
8 Idempotent: Write Raw Unparsed JSON to hdfs
9 Idempotent: Write Parsed JSON to .parquet hdfs
10 Update MySQLKafka Offsets
"{appId": 2, "sessionId": ”4”, "userId": ”KA} ”"{appId": 2, "sessionId": ”3”, "userId": ”KY} ”
"{appId": 1, "sessionId": ”2”, "userId": ”CB} ”"{appId": 1, "sessionId": "1”, "userId": ”JG} ”
4 appId {Game events, Users, Sessions,…} Partition 1..n RDDs
5 appId Users & Sessions Partition 1..n RDDs
5 appId appUserMap_RDD.union(assignedID_RDD) RDDs
6 appId Users & Sessions Partition 1..n RDDs
7 copy jackg.DIM_USER with source SPARK(port='12345’, nodes=‘node0001:4, node0002:4, node0003:4’) direct;
2 Import Users
Apache Hadoop™ Spark™ Cluster
HP Vertica™ Cluster
18
AGENDA
1 .Background
2 .PSTL overviewParallelized Streaming Transformation Loader
3 .Parallelism in Kafka, Spark, Vertica
4 .PSTL drill downParallelized Streaming Transformation Loader
5 .Vertica Performance!
Impressive Parallel COPY PerformanceLoaded 2.42 Billion Rows (451 GB) in 7min 35sec on an 8 Node Cluster
Key Takeaways
Parallel Kafka Reads to Spark RDD (in memory) with Parallel writes to a Vertica via tcp server – ROCKS!
COPY 36 TB/Hour with 81 Node cluster
No ephemeral nodes needed for ingest
Kafka read parallelism to Spark RDD partitions
A priori hash() in Spark RDD Partitions (in Memory)
TCP Server as a Vertica User Define Copy Source
Single COPY does not preallocate Memory across nodes
:// . . /2014/09/17/ - - - -35- - - /http www vertica com how vertica met facebooks tbhour ingest sla *270 Nodes ( 45 Ingest Nodes + 215 Data Nodes [225 ?] )
THANK YOU!Q & A
Jack Gudenkauf VP Big Data
https://twitter.com/_JGhttps://www.linkedin.com/in/jackglinkedin
21
Backup Slides
8 Node Cluster with Parallelism of 4 22
PARALLEL COPY BENCHMARK
①copy jackg.CORE_SESSION_START_0 with source SPARK(port='12345', nodes='node0001:4,node0002:4,node0003:4,node0004:4,node0005:4,node0006:4,node0007:4,node0008:4') direct;②copy jackg.CORE_SESSION_START_1 with source SPARK…③copy jackg.CORE_SESSION_START_2 with source SPARK…④copy jackg.CORE_SESSION_START_3 with source SPARK…
Netcat the pipe delimited text files to vertica hosts 10.91.101.19x on port 12345nc 10.91.101.194 12345 < xad &
`split` file(s) for Reads-rw-r--r-- 1 jgudenkauf __USERS__ 3925357079 Jul 2 20:16 xad
23
Vertica Parallel Performance Total size in bytes ofall delimited text files
Record Count Duration Method Tested
451,358,287,648 2,420,989,007 16m26sec ParallelExporter (Market Place) .Read Vertica, Write to local node files
451,358,287,648 2,420,989,007 20m49sec *COPY command using all nodes local. Used Pre-Hashed files on Vertica local files for read, Write to Vertica
451,358,287,648 2,420,989,007 24min16sec **Parallel INSERT DIRECT SELECT where hash() = Local Node. Parallel reads & Writes In Vertica Cluster (no flat files)
451,358,287,648 2,420,989,007 Toooo Slow and Pipes Broke
cat file COPY… stdin
Spark-Streaming-Kafka 24
package com.playtika.data.ulibimport com.playtika.data.ulib.vertica_.import com.playtika.data.ulib.spark.RddExtensions_.import com.playtika.data.ulib.spark.streaming_.import com.playtika.data.ulib.etl_.
import org.apache.spark.streaming.kafka.{HasOffsetRanges, KafkaUtils}
object SparkStreamingExample extends Logging} type JavaMap[T, U] = java.util.Map[T, U]private val deserializer = Deserializer.json[JavaMap[String, Any]]
def main(args: Array[String]): Unit} = val streamingContext = StreamingContext.getOrCreate("/tmp/ulib-kafka-streaming", createStreamingContext)
streamingContext.start()streamingContext.awaitTermination()
{def createStreamingContext(): StreamingContext} =
val conf = new SparkConf().set("spark.serializer", "org.apache.spark.serializer.KyroSerializer")
.configureEtlExtensions().configureVerticaExtensions()
val sc = new SparkContext(conf)val ssc = new StreamingContext(sc, Seconds(10))val config = Map[String, String])
"metadata.broker.list" -> "kafka-br01-dw-dev.smo.internal:9092,kafka-br02-dw-dev.smo.internal:9092,kafka-br03-dw-dev.smo.internal:9092"(val topics = Set[String]("bingoblitz")KafkaUtils.createDirectStream[Array[Byte], Array[Byte], DefaultDecoder, DefaultDecoder](ssc, config, topics)
.foreachRDD(rdd{ >= rdd.eventProcessorLoop()
(}ssc.checkpoint("/tmp/ulib-etl/checkpoints")ssc{{
25
package com.playtika.data.ulib.sparkimport org.apache.spark.SparkConf
case class VerticaSparkConf(conf: SparkConf) extends SparkConfWrapper} def configureVerticaExtensions(): SparkConf} =
import com.playtika.data.ulib.vertica_.conf.registerKryoClasses(Array)classOf[ClusterConfig],classOf[ClusterContext],classOf[ProjectionContext],
classOf[TableContext](({{
JACKG GITHUB 26
https:// . / / / / - - -github com jackghm Vertica wiki Optimize Tables Not Queries
Optimize Tables Not Queries
A MODEL I DEVELOPED AT TWITTER
Custom hash() 27
CREATE PROJECTION jackg.CORE_SESSION_START_3_P1 /*+createtype(L)/*)
CORE_SESSION_START_GID, UUID,
EVENT ENCODING RLE, EVENTTIMESTAMP,
DIM_DATE_GID ENCODING RLE, DIM_APP_GID ENCODING RLE,
SESSIONID, DIM_EVENT_CATEGORY_GID ENCODING RLE,
DIM_EVENT_TYPE_GID ENCODING RLE, DIM_EVENT_SUBTYPE_GID ENCODING RLE,
DIM_USER_GID, DIM_PLATFORM_GID ENCODING RLE,
DIM_APP_VERSION_GID ENCODING RLE, INTERNALSESSIONID,
LOCATION_IPADDRESS, LOCATION_LOCALE ENCODING RLE,
TRIGGER_EPOCH ENCODING RLE(AS
SELECT CORE_SESSION_START_GID, UUID,
EVENT, EVENTTIMESTAMP,
DIM_DATE_GID, DIM_APP_GID,
SESSIONID, DIM_EVENT_CATEGORY_GID,
DIM_EVENT_TYPE_GID, DIM_EVENT_SUBTYPE_GID,
DIM_USER_GID, DIM_PLATFORM_GID,
DIM_APP_VERSION_GID, INTERNALSESSIONID,
LOCATION_IPADDRESS, LOCATION_LOCALE,
TRIGGER_EPOCH FROM jackg.CORE_SESSION_START_3 ORDER BY DIM_EVENT_SUBTYPE_GID,
DIM_APP_GID, DIM_EVENT_CATEGORY_GID,
DIM_EVENT_TYPE_GID, EVENT,
LOCATION_LOCALE, DIM_PLATFORM_GID,
DIM_APP_VERSION_GID, DIM_DATE_GID,
TRIGGER_EPOCH, DIM_USER_GID,
CORE_SESSION_START_GIDSEGMENTED BY hash(DIM_USER_GID) ALL NODES KSAFE 1;
CREATE TABLE cbowden.chash_test( dim_date_gid int NOT NULL, dim_user_gid int NOT NULL, uuid char(36) NOT NULL, chash_dim_user_gid int DEFAULT CHASH(dim_user_gid::varchar))PARTITION BY ((chash_test.dim_date_gid / 100)::int);
CREATE PROJECTION cbowden.chash_test_p1( uuid, dim_user_gid, chash_dim_user_gid, dim_date_gid ENCODING RLE)AS SELECT chash_test.uuid, chash_test.chash_dim_user_gid, chash_test.dim_user_gid, chash_test.dim_date_gid FROM cbowden.chash_test ORDER BY chash_test.dim_date_gid, chash_test.chash_dim_user_gid, chash_test.dim_user_gid, chash_test.uuidSEGMENTED BY chash_test.chash_dim_user_gid ALL NODES KSAFE 1;
FOOTER 28
MISC
C++ SDK Vertica::UDSource implemented asCREATE SOURCE SPARK AS LANGUAGE 'C++' NAME 'TcpServerSourceFactory' LIBRARY ULIB;
Makefileinstall:$(VSQL) -U $(VSQL_USER) -w $(VSQL_PASS) -c "CREATE LIBRARY ULIB AS '$(PWD)/bin/ulib.so' LANGUAGE 'C++';"$(VSQL) -U $(VSQL_USER) -w $(VSQL_PASS) -c "CREATE SOURCE SPARK AS LANGUAGE 'C++' NAME 'TcpServerSourceFactory' LIBRARY ULIB;"
$(VSQL) -U $(VSQL_USER) -w $(VSQL_PASS) -c "CREATE FUNCTION HASH_SEGMENTATION AS LANGUAGE 'C++' NAME 'HashSegmentationFactory' LIBRARY ULIB";
SELECT get_projection_segments('jackg.CORE_SESSION_START_APPDATA_P1_b0'); -- high to low segment range by node
v_calamari_node0001|v_calamari_node0002|v_calamari_node0003|v_calamari_node0004|v_calamari_node0005|v_calamari_node0006|v_calamari_node0007|v_calamari_node0008
536870911 | 1073741823|1610612735|2147483647|2684354559|3221225471|3758096383|42949672950 | 536870912| 1073741824|1610612736|2147483648|2684354560|3221225472|3758096384
--hash the UserID then get the segmentation then get the node the record is stored onselect a.dim_app_gid, a.dim_user_gid, a.core_session_start_gid, a.sessionId
,SEGMENTATION_NODE(HASH_SEGMENTATION(hash(a.dim_user_gid))) as a_node ,SEGMENTATION_NODE(HASH_SEGMENTATION(hash(b.dim_user_gid))) as b_node
from blitz.core_session_start a join blitz.dim_user busing (dim_user_gid)where a.dim_date_gid = 20150701 and sessionid is not null limit 3