DEVELOPER

MEETUP IN TAIWAN

LINE

#02

自己紹介

謝良奇

Data Engineer

2018年1月入社

LINE Fukuoka 株式会社

データラボチーム所属

LIANG-CHI HSIEH

Agenda

Data Engineering in LINE Fukuoka DataLab: Newcomer’s View

What is Data Engineering

The Status of Data Engineering in LINE and LINE

Fukuoka

Daily Work in LINE Fukuoka

What is Data Engineering

Data Engineering is…

From Quora*:Data engineering is inherently a multi-

disciplinary field, because of the number of

technologies involved: visualization, data

analysis, knowledge engineering, perhaps

data bases, and of course the subject matter of

the application.

*https://www.quora.com/What-is-data-engineering-What-

does-a-data-engineer-do-What-are-common-

responsibilities-of-a-data-engineer

What is Data Engineering

Data Engineer

・Data architecture

・Data management

・Data quality

…

What does a data engineer do?

・Transform data into useful format for analysis

Data engineer as a member in Data Science Team

What is Data Engineering

Data Engineer’s Skills

・Platform/DB/Programming Languages expertise

・Parallel and distributed computing, VMs/Cloud

・Hadoop, Spark, Hive, SQL/NoSQL…

・(Better to have) Machine Learning, statistics…

Data Engineering in LINE Fukuoka

yanagishima

・Web UI for Presto and Hive

・Easy to install and use

・Query history/bookmark

・TSV/CSV download…

https://github.com/yanagishima/yanagishima

Data Engineering in LINE Fukuoka

Azkaban

・Opensource workflow manager

・Manage batch jobs like ETL jobs

https://github.com/azkaban/azkaban

Daily Work in LINE Fukuoka

Develop/Maintain ETL jobs

・Data migration/transformation between data sources

(kafka/DBs/ElasticSearch) and various Hadoop

clusters

・Most are Python scripts

Example

・Extract and transform services data from ElasticSearch

for BI analysis

Daily Work in LINE Fukuoka

Machine Learning Engineering/Application・Not only data engineering but also possibly machine

learning related tasks

・So it’s better to have some machine learning skills too

Example

・Automatic sticker tagging using deep learning

Daily Work in LINE Fukuoka

Japanese Language Study

・It can be considered as part of this job

・Developers can speak English mostly, but it still needs

Japanese in daily life

Know more about LINE Fukuoka・LINE Fukuoka Corporation:

https://linefukuoka.co.jp/ja/

・How LINE Works – LINE Fukuoka Engineers:

https://www.youtube.com/watch?v=1TDUsMnIL6I