RAIN RFID - A world of business & consumer connectivity - Real … · 2020-01-04 · RFID Reader Fixed readers / Phased Arrays all over the environment Readers performing a Session

Real-time RAIN Data in the CloudBy: Colin BookmanJune 9th, 2016

Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Proprietary + Confidential

Realtime Inventory


70%Of the items seen in inventory

systems are actually where they

claim to be. We've found this to

hold true for retail and

distribution centers.



Where are my items What do I need to do How do I get this data

Performing quick lookups of what

physical locations have a particular

item, and what items are in a particular

location.

What items need to be moved from

one location to another. How many of

each item needs to be moved, and to

where.

My Current inventory system onlysupports a count, not EPC level data.My inventory system cannot handlemany minor adjustments of inventorydata per second.

Problems Faced



Key GCP Technology



Pubsub



Storage

NoSQLBlob SQL

CloudStorage

CloudBigtable

CloudDatastore Cloud SQL

Good for:Structured and unstructuredbinary or object data

Such as:Images, large media files,backups

Good for:Getting started,App Engine,serve use cases

Such as:User profiles,product catalog

Good for:Web frameworks,existing applications

Such as:User credentials, customerorders

Good for:Heavy read +write, events, andanalytical data

Such as:AdTech, Financialand IoT data



Bigtable

Read/Write Latency at 99% (ms)



BigTable


Dataflow




Dataflow



Dataflow



Dataflow


BigQuery




Kubernetes

kubelet

UI

kubeletCLI

API

users master nodes

etcd

kubelet

scheduler

controllers

apiserver



Overwhelmed,So am I



Realtime Inventory



Low Latency Simple Stupid onsiteequipment

Real Time InventoryIngestion & Aggregation

Ideally takes server less than 100ms

to prepare a response to an api

request

Equipment will be in remote hard to

get places. Keeping it simple stupid

makes support easier.

Inventory data should be ingested bythe system to as close of real-time aspossible. Ideally no cron jobs tocalculate inventory counts.

Technical Requirements



http://www.dailyhaha.com/_pics/failure-dog.htm

Failures



Attempt #1, Firebase

RFIDReader

● Fixed readers all over the environment

● Readers performing a Session 1 Target AB read

Ingestion ● RPI ingests RFID data, and pushes it to Firebase

Application ● Webapp queries Firebase for all the EPC tags and calculates item counts





Attempt #1, Firebase



Attempt #2, MySQL

RFIDReader

● Fixed readers all over the environment

● Readers performing a synchronized Session 1 Target AB Session 3 Target A read

Ingestion ● RPI ingests RFID data, and pushes it to Firebase AppEngine

● AppEngine writes each tag into MySQL

Application ● Webapp queries Firebase AppEngine for EPC tags Gtin14 Counts



Attempt #2, MySQL

EPC Location Gtin14 LastSeen

3018E511C735D2C0000000EA MTV.CYD1.Hanger1 12345678901231 2016-06-09T01:00:00+00:00

3018E511C735D2C0000000EB MTV.CYD1.Hanger1 12345678901231 2016-06-09T01:00:00+00:00

3018E511C735D2C0000000EC MTV.CYD1.Hanger3 12345678901231 2016-06-09T01:00:00+00:00

3018E511C735D2C0000000ED MTV.CYD1.Hanger2 12345678901231 2016-06-09T01:00:00+00:00

MySQL Schema





Attempt #2, MySQL



Attempt #3, Datastore (NoSQL)

RFIDReader

● Fixed readers / Phased Arrays all over the environment

● Readers performing a Session 3 target A read

Ingestion ● RPI ingests RFID data, and pushes it to AppEngine Pubsub

● AppEngine pulls pubsub messages and writes each tag into MySQL Datastore

Application ● Webapp queries AppEngine for Gtin14 Counts




Key Tags LastUpdated

<ReaderId> EPCSGTIN96[] <UTC Time>

Google.MTV.Foo [3018E511C735D2C0000000EB,

3018E511C735D2C0000000EC,

3018E511C735D2C0000000E0]

2016-06-

09T01:00:00+00:00

Raw RFID Tag Documents




Key Count LastUpdated

<Location>.<Gtin14> <Integer> <UTC Time>

Google.MTV.Foo.12345678901231 10 2016-06-

09T01:00:00+00:00









What worked



Attempt #4, Dataflow + Datastore + BigQuery

RFIDReader

● Fixed readers / Phased Arrays all over the environment

● Readers performing a Session 3 target A, followed by a shorter Session 3 Target B read

Ingestion ● RFID Reader ingests RFID data, and pushes it to Pubsub

● A sliding window dataflow job processes tags and outputs to a pubsub topic `inventory-

aggregation` aggregation counts, which app engine ingests and pushes to datastore.

● Streaming inserts to bigquery also process the raw rfid tag stream

Application ● Webapp queries AppEngine for Gtin14 Counts



Dataflow + Datastore + BigQuery

RFID ReaderCapture RFID Data

Physical Location

Cloud Pub/SubStreaming Changes

Messaging

Cloud DataflowAggregation

Processing

Cloud DataStoreInventory Store

Events

Cloud BigQueryData Warehouse

Analytics

App EngineInventory APIs

Services



Attempt #4, Dataflow + Datastore + BigQuery






Future Plans



RFIDReader

● Fixed readers / Phased Arrays / Mobile Readers all over the environment

● Fixed Readers performing a Session 3 target A, followed by a shorter Session 3 Target B read

Ingestion ● RFID Reader ingests RFID data, and pushes it to Pubsub

● A dataflow job pushes the raw RFID data to bigtable and bigquery. It also analyzes the rfid data

on a sliding window and writes the results to bigtable and bigquery.

Application ● Webapp queries AppEngine Managed VMs for Gtin14 Counts

Dataflow + BigTable + BigQuery




RFID ReaderCapture RFID Data

Physical Location

Cloud Pub/SubStreaming Changes

Messaging

Cloud DataflowInventory Streams

Processing

Cloud BigTableInventory Store

Events

Cloud BigQueryData Warehouse

Analytics

App EngineInventory APIs

Services




Row Key EPC: POSCounts

<Location>|<Grouping>.<Gtin14> <EPCSgtin96> 3019C56A40A1288000000002 Square

LastSeen

Latitude

Longitude

ReaderType

lastseen: 2016-06-09T01:00:00+00:00

latitude: 37.422428

longitude: -122.084329

readerType: fixed

25

BigTable Schema








http://cdn-d4d.kxcdn.com/wp-content/uploads/2015/02/4.jpg

Security



Security

PKI Each RFID Reader has its own Public and Private key pair

TLS All of our cloud endpoints use http over TLS

IAM Cloud services use Identity Access Management tolimit access to minimal permission set

Logs Store logs for post mortem analysis




http://treasure.diylol.com/uploads/post/image/222172/resized_the-most-interesting-man-in-the-world-meme-generator-i-have-a-question-for-you-what-do-you-want-to-ask-me-46a76c.jpg

Questions