View
6.677
Download
8
Embed Size (px)
Citation preview
http://www.skywatch24.com
雲端影⾳音與物聯網平台之軟體⼯工程挑戰
陳維超Wei-Chao Chen | Skywatch Inc. | National Taiwan University
WARNINGMay contain shameless self-promotion
YOU HAVE BEEN WARNED
1998 2002 2006 2010 2014
ABOUT MYSELF
Internet Camera x Cloud Powerwww.skywatch24.com
extend����������� ������������������ your����������� ������������������ perception����������� ������������������ ����������� ������������������ giving����������� ������������������ meaning����������� ������������������ to����������� ������������������ the����������� ������������������ places
raw data
there����������� ������������������ are����������� ������������������ 3����������� ������������������ people����������� ������������������ in����������� ������������������ your����������� ������������������ yard!����������� ������������������ ����������� ������������������
meaningfulmessage
Past
RouterNAS
Active
SureConnectcan’t reach
can break
simple����������� ������������������ and����������� ������������������ reliable����������� ������������������ connectivity
in����������� ������������������ over����������� ������������������ 10����������� ������������������ countries����������� ������������������ complete����������� ������������������ hardware����������� ������������������ &����������� ������������������ software����������� ������������������ solution
cloud����������� ������������������ video����������� ������������������ analytics����������� ������������������
make����������� ������������������ regular����������� ������������������ network����������� ������������������ cameras����������� ������������������ smart����������� ������������������ !
?…
318 BROADCASTING
THE RESEARCH
VIDEO ANALYTICSA methodology to reason and understand video
I saw three person
last hour
People Counthttps://www.youtube.com/watch?v=TPjmsha6gWY
Loiteringhttps://www.youtube.com/watch?v=f5DxUe5I8as
Trip wirehttps://www.youtube.com/watch?v=MH2HZi6-4jA
Speed testhttps://www.youtube.com/watch?v=KV6CXdzmhNs
SMART FAST FORWARD
Video activity level
Single button interface
SMART FAST FORWARD
•20x average speedup
•Compressed domain processing
•30x faster processing compared to competition
•3 seconds for every 10 minutes
SMART FAST FORWARD
SMART DAILY•Intelligent video summary
•Daily email on your past day’s activity
- Compressed domain analysis
- Very fast - totally I/O bound
- Detect events based on activity / sensible motion
Click to see video Yes, it is secure
(How?)
SMART CAPTION•Image to text / deep learning
SMART CAPTION•Accuracy not critical for application (?!)
•Deep learning classification
•Berkley Caffe
• ImageNet / Microsoft COCO
• Stanford NeuralTalk
•GPU / CUDA powered
•<1 second per image
NEURALTALK
http://cs.stanford.edu/people/karpathy/deepimagesent/generationdemo/
NEURALTALK
http://cs.stanford.edu/people/karpathy/deepimagesent/generationdemo/
FEATURE MATRIXTr
aini
ng C
ost
Runtime Cost
FaceDetect
LoiteringTrip wire
Area AlertLeft object
Motion
LicensePlate
PeopleDetection
Face
Reco
gnitio
n
Object
Classificat
ion
Age
Gender
PeopleCount
HeatMap
SmartCaption
SmartFast
Forward
SmartDaily
Skywatch
RegularIVA
EDGE COMPUTATION
Compute The Internet
Video Meta-data
Je suis un Edge
CLOUD COMPUTATION(SOMEWHAT POINTLESS)
The Internet
Video
Meta-dataCompute
CLOUD COMPUTATION(SENSIBLE)
Video
Meta-data
The Internet
Compute
Data model Video storage
HYBRID COMPUTATION(OUR VISION)
Early Vision
Partial Meta-data
The Internet
Late Vision
Data model Video storage
Various meta-data
results
CLOUD VS LOCALCloud Local
How? Cheap camera Cloud computer
Special camera Local computer
Best for Search & classify Big data Real-time alerts
AdvantagesEasy to deploy & manage
Cheap camera Rental model
Lower bandwidth Real-time applications
Disadvantages Higher bandwidth Maybe not for alerts
Difficult to deploy & manage Expensive camera Higher upfront cost
THE ENGINEERING
Ivan Sutherland, the Sketchpad
WE HAVE BUILT STUFFS
•Chief shader architect for Playstation 3 (RSX)
•75% of the GPU chip
•Expensive bugs (> $100K USD each)
•Targeted test, random test, monkey test
•From architecture to driver to hardware
•Shipped over 100M systems
WE HAVE INVENTED STUFFS
•OpenCV
•MPEG4 Source / OpenLF
•Fancy publications
•US Patents
HOW HARD CAN IT BE?Get the plumbing done, then add smartness
Prototype
Scalable & Stable
Initial Production
Prototype
SOFTWARE ENGINEERING
Aspects What do we do?
Development Scrum / continuous integration
Testing Unit / regression testing
Review Design review / Design patterns Code review
Operation Push-button code deployment Server monitors / camera diagnostic tools
What about the cameras?
TESTABILITY
Test Subject
Data
Support Units
Interface
Sandbox (Truman’s world)
TESTABILITY
Test Subject
Test Data
Fake / Mock
Interface
Sandbox (Truman’s world)
TESTABILITY
Test Subject
Test Data
Fake Camera
Interface
Really hard to make a good fake device
COMPLEXITY
Firmware version Camera models
Server config Software delivery
Browser type App version
CONTROL THE COMPLEXITY
OTA upgrade Fewer models
VM / container Push-button release
Versioned API Limit OS revision
Frameworksare yourfriends(?!)
High-bandwidthvideo traffic
Low-bandwidthAPI traffic
System Architecture
Overview (Dual Region)
Storage SAN
ComputeServers
StreamingServers
Region AP2P
Video
Storage SAN
ComputeServers
StreamingServers
Region B
UserDB
ApplicationServersApplicationServers
Embedded Linux
Camera Monitors
VPN Interface
P2P Interface
RTSP Streaming
RTMP Streaming
LAN Interface
Camera API
VPN Interface
RESTfulCloud API
User DB
StreamBroadcaster Video Files
Camera Software Stack
Skywatch Service API
Internet
Skywatch Firmware / Software
Camera Firmware
TOPEARLY SCREW-UPS
#1: BOB SAW ALICE(AND MAYBE CHARLES)
Alice
BobBob
Alice
Charles
Very early mistake — this won’t happen again
•How did this happen?
•Not thread-safe / process-safe
•Unique IDs are not unique (after a certain scale)
•Loose DB & network synchronisation
•Old tunnels hijacked by new cameras
•Camera bug — HTTP authentication off in a firmware version
#1: BOB SAW ALICE(AND MAYBE CHARLES)
Alice
•Why it won’t happen again?
#1: BOB SAW ALICE(AND MAYBE CHARLES)
Alice
Bob
HTTP Password
authenticatedCharles
Individually signed & CRC’ed files
Scanner to make sure camera
authentication is on
Channel scanner to clean up dangling
connections
RESTful API check for signature
#2: FIRMWARE UPGRADE EQUALS ANGRY CUSTOMERS
•Auto upgrade?
•Bad idea — people unplug and brick the camera
•User-triggered OTA upgrade
•Camera restore to default & WiFi settings gone
•Continuous upgrade testing is needed
•Monitor customer upgrade events / failures
#3: JAVA - WRITE ONCE,DEBUG EVERYWHERE
•Browser H.264 decoder / multi-channel player
•Originally JAVA-based. Why?
•Because a big company is using it
•Because Flash was killed by Jobs
•Because we can save money (P2P)
•Tons of support issue (!)
Browser Player Flash / HTML5
player
#3: JAVA - WRITE ONCE,DEBUG EVERYWHERE
•Browser H.264 decoder / multi-channel player
•Originally JAVA-based. Why?
•Because a big company is using it
•Because Flash was killed by Jobs
•Because we can save money (P2P)
Browser Player Flash / HTML5
player
Believe in yourself, not the competitor
Choose a popular weapon
Worry about money saving when it becomes a real issue
THE OUTLOOK
Preikestolen (cc) Anna Bialkowska https://www.flickr.com/photos/annabialkowska/117380607
Converse
Control
Connect
Data and services need to talk
to each other
Soonsensors����������� ������������������ that����������� ������������������ talk����������� ������������������ to����������� ������������������ ����������� ������������������ each����������� ������������������ other����������� ������������������ intelligently
record����������� ������������������ kid’s����������� ������������������ study����������� ������������������ time����������� ������������������ and����������� ������������������ compare����������� ������������������ with����������� ������������������ her����������� ������������������ score
08:08AM 07:38PM
4hours 2hours
a����������� ������������������ stranger����������� ������������������ sneaking����������� ������������������ around����������� ������������������ in����������� ������������������ your����������� ������������������ neighborhood
7/2 7/7 7/9
?
retailrescue.com
People Counting v.s.
Stock Market
Power transmission lines (cc) Oran Viriyincy https://www.flickr.com/photos/viriyincy/7004881064
Home Automation v.s.
Grid Management
http://www.skywatch24.com
THANK YOU陳維超
Wei-Chao Chen [email protected]/weichao.chen.tw
We are hiring!!