智慧城市生態圈的AI應用技術 - cinsa.org.cn · Deputy Director， State Key Lab of IoT for Smart City ... •Typical knowledge graphs •WordNet, Freebase, Nell ... SpamTracer:

引領創新科技建設智慧澳門

智慧城市生態圈的AI應用技術賈維嘉，講座教授、實驗室副主任

Weijia Jia, Chair Professor

Deputy Director， State Key Lab of IoT for Smart City

University of Macau 1


➢ 智能搜索➢ 語義理解➢ 知識圖譜

➢ 對象關聯➢ 實體識別➢ 霧式管理

➢ 全效感知➢ 時空覆蓋➢ 三元協同

Feed

back

Co

ntro

l

Ap

p D

ata

2

Info/Data & Knowledge Flows

IoT/sensor networksApplications

候選子圖 Top-k結果圖知識圖譜用戶查詢圖

應用層

中間層

感知層


我們要做什麼？智慧城市物聯網生態圈！


1. Background

2. AI at Bottom: Sensors/IoT

3. AI at Middle: Fog/Edge Networks

4. AI at Upper: Knowledge Graph

5. Discussions： AI S&P

Contents

3


o 2018年7月獲國家科技

部批准

o 2018年10月8日揭牌

背景

4引領創新科技建設智慧澳門

引領創新科技建設智慧澳門 5

智慧城市物聯網國家重點實驗室


1. 全球已啟動或在建的智慧城市達1000多個

• 正在以20%以上的速度增加

• 歐盟在2009年提出《歐洲2020戰略》

• 日本提出 “智慧日本”

• 新加坡推動政府服務的電子化

• 荷蘭提出 Amsmarterdam 城市能源地圖

2. 國內截止2015年底

• 85%以上的城市都在進行智慧城市建設

• 智慧城市試點已接近300個

• 規劃對智慧城市的投資總規模將逾5000億元

智慧城市國內外研究現狀



澳門的挑戰與機遇


安全防災

能源配置生態影響

交通擁塞


State Key Lab@University of Macau

Fundamental Research & Key

Technology 基础研究/关键技术

为支撑

Talent Training人才培养为平台

City Development 城市发展为主线

IOTSC 智慧城市物联网国家重点实验室

Applications应用支持为特征

8


Research focuses

Smart Sensing 感

知技术

Data Analysis and Control

数据分析与控制技术

Advance Networking &

Fog Computing网络存储,传输与雾

计算

Platformof AI/Big

Data

Smart Grid

能源物联网

Smart

Transportation

基于物联网的智能交通

Public Safety Monitoring

and Disaster Prevention

城市公共安全监控和灾害防治

Big Data &

Intelligent Tech

大数据与智能技术

Intelligent Sensing

and Net Comm.

智能傳感與網路通信

9


1. Background

2. Bottom: Sensors/IoT

3. Middle: Fog/Edge Networks

4. Upper: Knowledge Graph

5. Discussions： AI S&P

Contents

10





Feed

back

Co

ntro

l

Ap

p D

ata

11

Info/Data & Knowledge Flows

IoT/sensor networksApplications


應用層

中間層

感知層


智慧城市物聯網生態圈！

Our IoT devices

1. High frequency

radio, low power

consumption, wall

penetration

2. Real-time video

push and alarm to

user's mobile

phone

3. WeChat alarm and

personal service

inquiry and report

12

Optimal patterns achieving 2D-coverage and k-connectivity: k = [1,…, 6]

Rs – Sensing RadiusRc – Connection Radius

2D Optimal Connectivity/Coverage

13


1. Background




5. Discussions: AI S&P

Contents

14


Key Tech: Fog/Edge Computing

PublicCloud A

Cloud/Servers

Fog/edge networks

App Environments

IoT & SensorNetworks

Feed

back

Co

ntro

l

EnterpriseCloud B

PrivateCloud C

Ap

p D

ata

IoT/sensors

Info/Data Flows

Bottom layer

Middle layer

Upper layer

How the data/tasks are handled by the

fog nodes?15


Issues with Fog/Edge

• Resource allocations

• Power/energy awareness

• Real-time QoS

• Mobility service support

• Offloading

• Edge-to-edge data transfer

• Edge-to-edge AI services …

Container Migrations

Migrating data/tasks➔Allocating resources from fog node 𝐹1to 𝐹3 through containers ➔ QoS ↑ costs↓ .

Migration Modeling and Learning Algorithms for Containers in Fog Computing, IEEE Transactions on Service

Computing (TSC), 2018 17

• Containers are much lighter than VMs

• Compared with VM, application data and runtime libs are migrated without migrating the guest OS.

Containers: Why not VM?

18


Delay

• Network delay:

• Computation delay:

• Total delay:


Parameter Attentions

• Resource utilization:

• Total power consumption:

• Container migration cost:


Nightmare: Dimensional Curse

• Delay of Ci at Fj

• Matrix of delays:

• Overall state space:

• To minimize• Delay 𝑑𝑡𝑜𝑡𝑎𝑙• Power 𝑝𝑡𝑜𝑡𝑎𝑙• Migration cost 𝑚𝑡𝑜𝑡𝑎𝑙 subject to resource

requirements:min𝐶 = 𝜔1𝑑𝑡𝑜𝑡𝑎𝑙 +𝜔2𝑝𝑡𝑜𝑡𝑎𝑙 +𝑚𝑡𝑜𝑡𝑎𝑙

𝑠. 𝑡. 0 ≤ σ1 𝐶𝑗.𝑙 𝑡 =𝐹𝑖

𝐶𝑗.𝑎 𝑡

𝐹𝑖.𝑐≤ 1 𝑖 = 1,2, … ,𝑚

• Difficulties: • Bin-pack (NP-hard) • High dimensional state

Objectives

22

• RL Loop:

• State 𝑆𝜏, Action 𝐴𝜏, and Reward 𝑅𝜏:• 𝑆𝜏 = 𝑋 𝜏 , 𝐶. 𝑙 𝑡0 + 𝜏 𝑇 , 𝐶. 𝑎 𝑡0 + 𝜏 𝑇 ∈ 𝑺.• 𝐴𝜏 = 𝐶. 𝑙 𝑡0 + 𝜏 𝑇 ∈ 𝑨.

• 𝑅𝜏 = − 𝜔1𝑑𝑡𝑜𝑡𝑎𝑙 𝜏 + 𝜔2𝑝𝑡𝑜𝑡𝑎𝑙 𝜏 + 𝑚𝑡𝑜𝑡𝑎𝑙 𝜏

• 𝑄 𝑣𝑎𝑙𝑢𝑒: 𝑄(𝑆𝜏−1, 𝐴𝜏−1)

Reinforcement Learning (RL)

23


1. Background



4. Upper: Knowledge Graph – The key


Contents

24


AI framework

PublicCloud A

Cloud/Servers

Fog/edge networks

App Environments

IoT & SensorNetworks

Feed

back

Co

ntro

l

EnterpriseCloud B

PrivateCloud C

Ap

p D

ata

IoT/sensors

Info/Data Flows

Bottom layer

Middle layer

Upper layerAI Framework

25


• 重點研究知識圖譜與數據相結合的學習與表示技術算法

• 探討城市大數據的多源獲取，提出多源數據的融合技術，利用遷移學習和聯邦學習對城市大數據進行挖掘和分析

• 提出將結構化信息和數據結合的聯合模型，以刻畫知識圖譜內外實體的有效聯繫

26

城市大數據與智能技術


27

研發具國際領先的智慧城市物聯網共性基礎科學及平臺方案；人-物&物-物之間的多關係提取與智能鏈接。

城市大數據與物聯網智能

代表性成果:

• 物聯網邊緣動態傳算：IEEE TPDS 2019，TSC2018

- 長尾任務動態調度

- 容器智能遷移

- 強化學習以降低解空間維度

• 語意實體多關係提取：AAAI 2019, ACL2019， EMNLP 2018, NAACL2019

- 實體多關係並行提取

- 優化膠囊網絡及设计新型對抗網路進行關係精煉提取

• 多反饋網絡實體信息抽象集成, IJCAI 2019，ACM Trans Data Mining/Discovery 2019

- 多反饋網絡建模- 關係網絡對演模型


1. Extracting Entities (E)

2. Extracting Relations (R)

3. Predicating E-R in KG

Knowledge Graph (KG) --Fundamental Challenges

28


What is Knowledge Graph?• Freebase, Concept Graph, NELL, WordNet…

• Triplet: (head entity--A, relation--r, tail entity--B)

• ”The height of Liu is 174cm” ➔ (Liu, height, 174cm)


Entity Triple Extractions

• Entity linking forming KG• entity = node• relation = edge• fact = triple (A, r, B)

• Typical knowledge graphs• WordNet, Freebase, Nell• DBpedia, Wikidata• …

App Example: Time-Sync CommentsVideo-Tagging

• Time-sync comments are crowdsourced & interactive

• Content involving current video, feelings of users or replies to other comments

• Obtain more accurate and specific tags

(1) Semantic relevance

(2) Real-time

(3) Inter-dependence

(4) Noise

31

Mining Tags as Relations

• 1. Graph Construction

• 2. Topic Partitioning & Radius

• 3. Weight Distribution and Tag Extraction

ICME, 2017, ICME 2019

Outcomes: {User A, “Messi”, User B} { Video V, “Messi”, [0:18, 0:24]}

32





Fundamental challenges

33


Two recent works on extracting relations from

Natural Language:

• Single Relation Extraction in Sentence

• – EMNLP-18

• Multiple Overlapping Relation Extractions

• – AAAI-19

2. Extracting Relations

34


Problem and Challenges

• Multi-labeled Relation Extraction

ID Instances Relations

S1[Arthur Lee], the leader of Love, died on

Thursday in [Memphis].person/place_death

person/place_lived

S2[Arthur Lee] was born in [Memphis],

Tennessee, and lived there until 1952.person/place_birth

person/place_lived

S3

[Abraham Lincoln] was an American

statesman and lawyer who served as the

16th President of the [United States].

person/president_of

person/nationality

Overlapping

Relations

Discrete

Features


Extracting Overlapping Features

• Capsule Network

Capsule Network inImage Recognition

Capsule Network inRelation Extraction (RE)

Examples for Capsule Network in image recognition come from Aurelien Geron’s lecture.https://youtu.be/pPN8d0E3900

[Arthur Lee] was born in [Memphis], Tennessee, and lived there until 1952.

place_birth

place_lived


Attentive Capsule Network

• Attention-based Routing Algorithm

• Sliding-margin loss


Experiments• Experiments on two previous benchmarks (NYT-10 and

SemEval-2010 Task 8)





Fundamental challenges

39


KGC– Predict Entity/Relations

• Triples in KGs.• (Isaac Newton, _birth_in, Lincolnshire)

• How to predict the missing part: (A,r, ?), (A, ?, B), (?,r, B)

John was bornin ?

John was born in Miami

KnowledgeGraph

VectorSpace


Translation-based Models

• Representation learning: Similar entities arerepresented as similar vectors

• Typical: TransE (Translation Embedding):

𝐴, 𝑟, 𝐵 ➔ Ԧ𝐴 + Ԧ𝑟 = 𝐵

• Optimization: minimizing Ԧ𝐴 + Ԧ𝑟 − 𝐵• E.g.: TransH, TransR, TransG, DKRL, TKRL,

SSP, …

TransT: Type-based Multiple Embedding Representations for Knowledge Graph Completion, Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2017.


Contributions: Multiple Embeddings

• Assign one vector for each semantics

𝒑𝒕𝒓𝒖𝒆 𝑨, 𝒓, 𝑩 = σ𝑖=1𝑛𝐴 σ𝑗=1

𝑛𝐵 𝑤𝐴,𝑖𝑤𝐵,𝑗𝑝𝑡𝑟𝑢𝑒 𝑣𝐴,𝑖 , 𝑣𝑟 , 𝑣𝐵,𝑗

𝑝𝑡𝑟𝑢𝑒is defined as

𝑝𝑡𝑟𝑢𝑒 𝑣𝐴,𝑖 , 𝑣𝑟 , 𝑣𝐵,𝑗 = 𝑒− 𝑣𝐴,𝑖+𝑣𝑟−𝑣𝐵,𝑗


Relation Prediction Classifications

• Predicting the missing relation when given two entities. FB15K is the benchmark dataset.

• Triple classifications predict if a given triple is correct. FB15K is the benchmark dataset of this task.


1. Introduction


3. Middle : Fog computing



Contents

44

From attacker’s view: What can attackers do in AI?

From AI’s view: How can we prevent attacks from AI?

ATTACKER

AI

privacysecurity

DATAMODEL

Security & Privacy

45

DeepFakes

46

SpamTracer: Manual Fake Review Detection for O2O Commercial Platforms by Using Geolocation Features, Inscrypt 2018.

Machine learning for detectingfake reviewers

47

Bae H, Jang J, Jung D, et al. Security and Privacy Issues in Deep Learning, ACM Computing Surveys 2018

Potential Threats in AI

48





S&

P F

eed

back

Co

ntro

l

S&

P A

pp

Data

49

S&P Info/Data & KnowledgeFlows

S&P 5G & IoT/sensor networksApplications


應用層

中間層

感知層


智慧城市物聯網生態圈


MOST China; FDCT Macau, SAR; NSFC

University of Macau

Shanghai Jiaotong University

Acknowledgements

50


謝謝！Thanks！

51

Documents

智慧城市生態圈的AI應用技術 - cinsa.org.cn · Deputy Director， State Key Lab of IoT for Smart City ... •Typical knowledge graphs •WordNet, Freebase, Nell ... SpamTracer: