雲端影音與物聯網平台的軟體工程挑戰:以 Skywatch 為例-陳維超

Preview:

Citation preview

http://www.skywatch24.com

雲端影⾳音與物聯網平台之軟體⼯工程挑戰

陳維超Wei-Chao Chen | Skywatch Inc. | National Taiwan University

WARNINGMay contain shameless self-promotion

YOU HAVE BEEN WARNED

1998 2002 2006 2010 2014

ABOUT MYSELF

Internet Camera x Cloud Powerwww.skywatch24.com

extend����������� ������������������  your����������� ������������������  perception����������� ������������������  ����������� ������������������  giving����������� ������������������  meaning����������� ������������������  to����������� ������������������  the����������� ������������������  places

raw data

there����������� ������������������  are����������� ������������������  3����������� ������������������  people����������� ������������������  in����������� ������������������  your����������� ������������������  yard!����������� ������������������  ����������� ������������������  

meaningfulmessage

Past

RouterNAS

Active

SureConnectcan’t reach

can break

simple����������� ������������������  and����������� ������������������  reliable����������� ������������������  connectivity

in����������� ������������������  over����������� ������������������  10����������� ������������������  countries����������� ������������������  complete����������� ������������������  hardware����������� ������������������  &����������� ������������������  software����������� ������������������  solution

cloud����������� ������������������  video����������� ������������������  analytics����������� ������������������  

make����������� ������������������  regular����������� ������������������  network����������� ������������������  cameras����������� ������������������  smart����������� ������������������  !

?…

318 BROADCASTING

THE RESEARCH

VIDEO ANALYTICSA methodology to reason and understand video

I saw three person

last hour

People Counthttps://www.youtube.com/watch?v=TPjmsha6gWY

Loiteringhttps://www.youtube.com/watch?v=f5DxUe5I8as

Trip wirehttps://www.youtube.com/watch?v=MH2HZi6-4jA

Speed testhttps://www.youtube.com/watch?v=KV6CXdzmhNs

SMART FAST FORWARD

Video activity level

Single button interface

SMART FAST FORWARD

•20x average speedup

•Compressed domain processing

•30x faster processing compared to competition

•3 seconds for every 10 minutes

SMART FAST FORWARD

SMART DAILY•Intelligent video summary

•Daily email on your past day’s activity

- Compressed domain analysis

- Very fast - totally I/O bound

- Detect events based on activity / sensible motion

Click to see video Yes, it is secure

(How?)

SMART CAPTION•Image to text / deep learning

SMART CAPTION•Accuracy not critical for application (?!)

•Deep learning classification

•Berkley Caffe

• ImageNet / Microsoft COCO

• Stanford NeuralTalk

•GPU / CUDA powered

•<1 second per image

NEURALTALK

http://cs.stanford.edu/people/karpathy/deepimagesent/generationdemo/

NEURALTALK

http://cs.stanford.edu/people/karpathy/deepimagesent/generationdemo/

FEATURE MATRIXTr

aini

ng C

ost

Runtime Cost

FaceDetect

LoiteringTrip wire

Area AlertLeft object

Motion

LicensePlate

PeopleDetection

Face

Reco

gnitio

n

Object

Classificat

ion

Age

Gender

PeopleCount

HeatMap

SmartCaption

SmartFast

Forward

SmartDaily

Skywatch

RegularIVA

EDGE COMPUTATION

Compute The Internet

Video Meta-data

Je suis un Edge

CLOUD COMPUTATION(SOMEWHAT POINTLESS)

The Internet

Video

Meta-dataCompute

CLOUD COMPUTATION(SENSIBLE)

Video

Meta-data

The Internet

Compute

Data model Video storage

HYBRID COMPUTATION(OUR VISION)

Early Vision

Partial Meta-data

The Internet

Late Vision

Data model Video storage

Various meta-data

results

CLOUD VS LOCALCloud Local

How? Cheap camera Cloud computer

Special camera Local computer

Best for Search & classify Big data Real-time alerts

AdvantagesEasy to deploy & manage

Cheap camera Rental model

Lower bandwidth Real-time applications

Disadvantages Higher bandwidth Maybe not for alerts

Difficult to deploy & manage Expensive camera Higher upfront cost

THE ENGINEERING

Ivan Sutherland, the Sketchpad

WE HAVE BUILT STUFFS

•Chief shader architect for Playstation 3 (RSX)

•75% of the GPU chip

•Expensive bugs (> $100K USD each)

•Targeted test, random test, monkey test

•From architecture to driver to hardware

•Shipped over 100M systems

WE HAVE INVENTED STUFFS

•OpenCV

•MPEG4 Source / OpenLF

•Fancy publications

•US Patents

HOW HARD CAN IT BE?Get the plumbing done, then add smartness

Prototype

Scalable & Stable

Initial Production

Prototype

SOFTWARE ENGINEERING

Aspects What do we do?

Development Scrum / continuous integration

Testing Unit / regression testing

Review Design review / Design patterns Code review

Operation Push-button code deployment Server monitors / camera diagnostic tools

What about the cameras?

TESTABILITY

Test Subject

Data

Support Units

Interface

Sandbox (Truman’s world)

TESTABILITY

Test Subject

Test Data

Fake / Mock

Interface

Sandbox (Truman’s world)

TESTABILITY

Test Subject

Test Data

Fake Camera

Interface

Really hard to make a good fake device

COMPLEXITY

Firmware version Camera models

Server config Software delivery

Browser type App version

CONTROL THE COMPLEXITY

OTA upgrade Fewer models

VM / container Push-button release

Versioned API Limit OS revision

Frameworksare yourfriends(?!)

High-bandwidthvideo traffic

Low-bandwidthAPI traffic

System Architecture

Overview (Dual Region)

Storage SAN

ComputeServers

StreamingServers

Region AP2P

Video

Storage SAN

ComputeServers

StreamingServers

Region B

UserDB

ApplicationServersApplicationServers

Embedded Linux

Camera Monitors

VPN Interface

P2P Interface

RTSP Streaming

RTMP Streaming

LAN Interface

Camera API

VPN Interface

RESTfulCloud API

User DB

StreamBroadcaster Video Files

Camera Software Stack

Skywatch Service API

Internet

Skywatch Firmware / Software

Camera Firmware

TOPEARLY SCREW-UPS

#1: BOB SAW ALICE(AND MAYBE CHARLES)

Alice

BobBob

Alice

Charles

Very early mistake — this won’t happen again

•How did this happen?

•Not thread-safe / process-safe

•Unique IDs are not unique (after a certain scale)

•Loose DB & network synchronisation

•Old tunnels hijacked by new cameras

•Camera bug — HTTP authentication off in a firmware version

#1: BOB SAW ALICE(AND MAYBE CHARLES)

Alice

•Why it won’t happen again?

#1: BOB SAW ALICE(AND MAYBE CHARLES)

Alice

Bob

HTTP Password

authenticatedCharles

Individually signed & CRC’ed files

Scanner to make sure camera

authentication is on

Channel scanner to clean up dangling

connections

RESTful API check for signature

#2: FIRMWARE UPGRADE EQUALS ANGRY CUSTOMERS

•Auto upgrade?

•Bad idea — people unplug and brick the camera

•User-triggered OTA upgrade

•Camera restore to default & WiFi settings gone

•Continuous upgrade testing is needed

•Monitor customer upgrade events / failures

#3: JAVA - WRITE ONCE,DEBUG EVERYWHERE

•Browser H.264 decoder / multi-channel player

•Originally JAVA-based. Why?

•Because a big company is using it

•Because Flash was killed by Jobs

•Because we can save money (P2P)

•Tons of support issue (!)

Browser Player Flash / HTML5

player

#3: JAVA - WRITE ONCE,DEBUG EVERYWHERE

•Browser H.264 decoder / multi-channel player

•Originally JAVA-based. Why?

•Because a big company is using it

•Because Flash was killed by Jobs

•Because we can save money (P2P)

Browser Player Flash / HTML5

player

Believe in yourself, not the competitor

Choose a popular weapon

Worry about money saving when it becomes a real issue

THE OUTLOOK

Preikestolen (cc) Anna Bialkowska https://www.flickr.com/photos/annabialkowska/117380607

Converse

Control

Connect

Data and services need to talk

to each other

Soonsensors����������� ������������������  that����������� ������������������  talk����������� ������������������  to����������� ������������������  ����������� ������������������  each����������� ������������������  other����������� ������������������  intelligently

record����������� ������������������  kid’s����������� ������������������  study����������� ������������������  time����������� ������������������  and����������� ������������������  compare����������� ������������������  with����������� ������������������  her����������� ������������������  score

08:08AM 07:38PM

4hours 2hours

a����������� ������������������  stranger����������� ������������������  sneaking����������� ������������������  around����������� ������������������  in����������� ������������������  your����������� ������������������  neighborhood

7/2 7/7 7/9

?

retailrescue.com

People Counting v.s.

Stock Market

Power transmission lines (cc) Oran Viriyincy https://www.flickr.com/photos/viriyincy/7004881064

Home Automation v.s.

Grid Management

http://www.skywatch24.com

THANK YOU陳維超

Wei-Chao Chen weichao.chen@skywatch24.comwww.facebook.com/weichao.chen.tw

We are hiring!!