亚太示范电子口岸网络建设汇报

Dr. Xu Kai (Kernel)

8-21-2015

Shipping Big Data Applications Solutions

Research

Contents

New Requirements

How To Deal With

Some Initial Results

Not Only For Shipping

BACKGROUND

Today, international trade and logistics is undergoing profound changes. On the one hand,

e-commerce is changing the traditional business model, a large number of transaction

information has been generated through the network; on the other hand, with the

IOT(Internet of things), sensors distributed in all aspects of logistics and transportation are

also keep generating large amounts of data. Therefore, government departments and

enterprises have access to more data than in the past, people urgently need new methods

to maximize the value of these data.

Shanghai International Shipping Institute, the place where I work, provides consulting and

information services for the government and enterprises. We know the value and

significance of the data in the strategic decision of the government and enterprises, so we

have done a lot of basic work in these areas.

Part 1

China Shipping Database

More than 400 statistical indicators;

More than 2.3 million statistics;

Five sections：

Port &Terminal, Shipping Market, Shipping Service, Economic Indicator, Prosperity Index

February 2012:

Start construction

September 2013:

Go live

CSDB is very good. But the data it provides are from the traditional way.

When we search for "Shanghai", the 62

kinds of statistical indicators associated

with Shanghai will focus displayed on a

page, such as : Cargo throughput of China’s

coastal ports, Oil natuaral gas and their

products Traffic, etc.

From the traditional point of view, the CSDB has reached a advanced level.

When we search for "iron ore", the 8

kinds of Statistical indicators about it

will be displayed in one webpage. such

as : Iron Ore Production Traffic, China

Iron Ore Shipping Volume for Import,

etc.

Acquisition Integration Storage Processing Analysis

Past Investigation/

Manual

Paper/

Digital info Isolate

Low speed/

Low capacity/ Localization

Processing batch

Statistics/

Causal

analysis/ Sampling analysis

Now Internet of

things/ Automatic

Enterprise Service bus(ESB)

Real time/

Distributed/

Cloud

computing

Parallel Computing

Correlation

analysis/ Full sample analysis

new problems

Information Explosion

Standardization Data Center Super computing

Data Mining

What is changing in Big Data Era？

Big data research is going to solve these new problems.

The

Internet of

Things

Computer

Interaction

Big Data

How Big Data is generated?

Some data collection techniques we have.

OBD汽车数据总线

a11

a22

3a3

4a4

b1

b2

b3

b4

5

6

7

8

Vcc1

0

GND

0

MCU控制器SPI接口

尾气传感器

Vin

GND

Vref

B1

B8

Sign

ENB

A/D模数转换器

北斗定位模块/通信模块

I2C总线

串口通信

北斗网络

Human Animal

Cargo Container

Ship

Car Truck Machine

Wearable Device

Smart bracelet Logistics black box Terminal system for ship Car network suite

Geographic

Data

Electronic chart

See floor map

Shoreline berth

resources

Monitoring

Data

AIS

Beidou

satellite

positioning

Shipping lane

RFID

identification

VDR

Statistics

Transport

volume

Transport

capacity

Port throughput

Shipping route

congestion

Dock operation

efficiency

Freight rate

Business

Data

Ship visa

Ship registration

data

Rescue request

B/L data

Customs

clearance data

Shipping order

Textual

Data

Shipping

Regulations

Maritime case

Shipping

contract

Media Data

Port surveillance

video

Ship photos

What shipping data do we have today?

"4V" Features of Big Data

•Freight has strong

timeliness, it changes

every day

•20 million AIS data can

be compressed to 7

million

•Format heterogeneity,

including video,

images, other

unstructured data

•More than 20 million

AIS data being

produced every day

throughout the world

Volume Variety

Velocity Value

Part 2

Data is the specific form of information that can be used to save, copy and

spread.

So the "dead" data is useless, only to analyze the valuable part of the

information can be called "big data“; otherwise, it’s just "a pile of data".

And the purpose of data analysis is to help company make the right decision

in the business.

Data analysis serves business decision

Big Data Driven Model

Demand Transaction

DataService

Demands orientation Data driven

Quote from the book "Big Data Driven Shipping Informatization“

Shannon(C.E.Sannon), the founder of information theory proposed the

concept of "information entropy“ in 1948 to measure the number of

“information“ a data contains.

A large amount of information is has been lost from the first hand data

to the data we see.

Data and information

Data

entropy

multidimensional structure

Change an angle

Focus on detail and structure

The conclusion has changed.

We can not see the whole picture without big data

226cm

157cm

192cm(AVG)

Distributed and Parallel Computing

Information

System

DataBase

Information

Platform

Big Data

Develop

Integrate

Cloud

Computing

Online Data

Realize

Use

Stage of Shipping Informatization

Evolution of Data Utilization

Quote from the book "Big Data Driven Shipping Informatization“

Data utilization is developed with information technology

Shipping

Big Data

Storage

Processing

Visualization

Analysis

What Shipping Big Data Research?

Storage

A distributed database can store hundreds of billions of mobile target

monitoring data.

There are 350 billion AIS data in the database.

Processing

Key algorithms for data cleaning and compression.

Analysis

Multiple analytical models and key algorithms.

Concurrent computation environment based on Hadoop technology.

Experimental environment of data analysis based on R and Python.

Visualization

Tracks and Scene playback technology.

Various graphics report technology.

Part 3

Typical domestic route analysis results

Linyun River412275000 Qingyun River412148000

Tiansheng River412658000

"Shipping & Port Big Data Laboratory" Research Results

By analyzing AIS data, we can know the real voyage condition of

each liner routes and calculate actual voyage and fuel consumption

of each route. So shipping company can account cost accurately

and tell owner the punctuality rate of every route.

Global shipping route fitting results

"Shipping & Port Big Data Laboratory" Research Results

We can calculate the number of ships and the total capacity of each route, which provides

data support for market analysis and helps to predict the trend of market price.

Statistics of port production index

We have calculated the number of arrived ship and time of berthing to have an

objective understanding of port loading/unloading efficiency and berth utilization

rate, which provides data support for optimization of port management.

Data Sources AIS Analytical Data Statistical Data

Statistics range Solid Cargo Ship Liquid Cargo Ship Total Total

Statistics Berthing

Time

Berthing

Frequency

Berthing

Time

Berthing

Frequency

Berthing

Time

Berthing

Frequency Annual Throughput

Unit Hour Times Hour Times Hour Times Thousand Tons

Busan Port 28203 800 36600.47 420 64803.47 1220 346100

Tianjin Port 54416.9 1356 25788.3 454 80205.2 1810 540000

Singapore Port 37472.6 1012 29562.3 524 67034.9 1536 581270

"Shipping & Port Big Data Laboratory" Research Results 2014-09

Regional emissions and energy consumption analysis

We can measure the concentration spatial distribution of pollutant emissions and

carbon emissions precisely , and the results are being applied to the research of

establishing gas emission control area in Yangtze River Delta and Pearl River Delta.

Singapore 1 Day CO2 Emissions Shanghai 1 Day CO2 Emissions

"Shipping & Port Big Data Laboratory" Research Results

Ships Behavior Recognition Model

Quay

BerthBerth BerthBerth BerthBerth BerthBerth

AnchorageAnchorageAnchorageAnchorage

Port Area

Event A

Event B

Event C

P1

Pt

Pk

P2

PN

Pm

Pn

"Shipping & Port Big Data Laboratory" Research Results

Previous studies are based on the identification of ship behavior.

Big Data technologies help us find all moored events

Within 15 days, on a global scale, all the ship mooring points.

"Shipping & Port Big Data Laboratory" Research Results

According to location we distinguish between anchoring and berthing Events

"Shipping & Port Big Data Laboratory" Research Results

An example from the Xiamen Port

Ship track data will be converted ship into a sailing log.

Part 4

For Multimodal Transport

Generic Target Monitoring System, GTMS

Unified data format: Multi-target Packet, MTP

Our research is scalable

Shipping

Data

Cargo

Data

Truck

Data

Common Database Structure

Common Processing Platform

In the course of our research,

we reserve the room for

future expansion.

Future data analysis system

SQL

Not only for shipping industry, it will face to global trade and logistics.

Thank You!

Aug, 2015

Any comments, suggestions or feedbacks

will be greatly appreciated！

Dr. Xu Kai

E-mail：kernxu@gmail.com

Phone: +86 13816950360