1から学ぶCaliperimsjapan.org/file/1_Caliper_20190916.pdf2019/09/16 · AWS Lambda 接続 SQS Pros & Cons 5 Pros S3あるいはDBよりも速書き込みができる。 Cons 最

1から学ぶCaliper

2019年9⽉ 24⽇

常盤祐司⽇本IMS協会

法政⼤学情報メディア教育研究センター[email protected]

IMS JapanConference 2019

プログラム

1

Part1: 概要およびCaliperによるデータ⽣成 (60分)李在範（ IMS Certified T&I Manager、ネットラーニング） Caliper概要既存システムへのCaliper Sensor埋め込み

Part2: Caliperデータの取得と分析 (40分)常盤祐司（技術委員⻑、法政⼤学） AWSを⽤いた Caliper Recordの取得 AWSからPCへのデータダウンロード Tableau による分析

QA (20分)

Part2 Caliperデータの取得と分析

2

AWSを⽤いた Caliper Recordの取得

AWSからPCへのデータダウンロード

Tableau による分析

3




学習履歴データ分析概要

4

AmazonSQS

AmazonS3

AWSLambda

接続

SQS Pros & Cons

5

ProsS3 あるいは DB よりも⾼速書き込みができる。

Cons最⼤14⽇間しか保持されない。

Canvas設定

6

AWSのregionはCanvasと同様のus-east-1とした。

The name of the queue must begin with canvas-live-events.

取得する Caliper Profile 設定

7

AWS Simple Queue Service

8

SQS 設定

9

InstructureAccount Number

参考：https://community.canvaslms.com/docs/DOC-14163-how-do-i-create-an-sqs-queue-to-receive-live-events-data-from-canvas

SQSにおける蓄積

10

lambda

11

参考︓DevelopersIO、AWS LambdaがSQSをイベントソースとしてサポートしました︕https://dev.classmethod.jp/etc/aws-lambda-support-sqs-event-source/

IAM ロール

12

chaliceを⽤いてポリシーを設定する⽅法もある。

sqs-poll

13

lamda python

14

from __future__ import print_function# coding: utf-8import boto3 from datetime import datetime, timedelta, timezoneimport json

def lambda_handler(event, context):s3 = boto3.resource('s3')bucket = s3.Bucket('hosei-media-lambda')

JST = timezone(timedelta(hours=+9), 'JST')now = datetime.now(JST).strftime("%Y%m%d-%H%M%S-%f")

for i,record in enumerate(event['Records']):file_contents = record["body"]fn = 'sqs-poll_' + str(now) + '_' + str(i)ret = bucket.put_object(ACL='private',Body=file_contents,Key = fn,ContentType='text/plain')

return

s3コンソール

15

レコード概要

16

s3レコード表⽰

17

18




s3からのダウンロード

19

aws s3 cp --region us-east-1 s3://hosei-media-lambda/ . --recursive

AWS CLI によるCurrent Directoryへのレコードダウンロード

20

{"sensor": "http://hosei.instructure.com/","sendTime": "2019-09-07T07:09:40.074Z","dataVersion": "http://purl.imsglobal.org/ctx/caliper/v1p1","data": [{

"@context": "http://purl.imsglobal.org/ctx/caliper/v1p1","id": "urn:uuid:ae1587d5-493e-4c8c-9435-3c8c22cb024e","type": "SessionEvent","actor": {

"id": "urn:instructure:canvas:user:125310000000000006","type": "Person","extensions": {

"com.instructure.canvas": {"user_login": "[email protected]","root_account_id": "125310000000000001","root_account_lti_guid": "tkMMgk9o7kftOu2oN50lbvwXxSPJM9yjcYsRIE0p:canvas-lms","root_account_uuid": "tkMMgk9o7kftOu2oN50lbvwXxSPJM9yjcYsRIE0p","entity_id": "125310000000000006"

}}

},"action": "LoggedIn","object": {

"id": "http://hosei.instructure.com/","type": "SoftwareApplication"

},"eventTime": "2019-09-07T07:09:39.764Z","edApp": {


},"session": {

"id": "urn:instructure:canvas:session:f66e564864e641476657696e0734915f","type": "Session"

},"extensions": {

"com.instructure.canvas": {"hostname": "hosei.instructure.com","request_id": "68ce4f5d-615e-408f-bbfb-46ec08008eef","user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36","client_ip": "133.25.246.231","request_url": "https://hosei.instructure.com/login/canvas","version": "1.0.0"

}}

}]}

Caliper Data

配列

JSON全データ読込⽤Python事例

21

全データ読込⽤Pythonプログラム事例# -*- coding: utf-8 -*-# python r_20.py caliper.json# 2018-8-16 Y.Tokiwa

import jsonimport sys

args = sys.argv

def keyGet(k,v): if isinstance(v,dict):

print("Dict-> ",k)for k,v1 in v.items():

keyGet(k,v1)return

# "data"keyに対して配列(=list)を想定しているための処理elif isinstance(v,list):

print("List-> ",k)

# listの形式"data"からjsonをv1に取り出すfor num,v1 in enumerate(v):

keyGet(k,v1)return

# ここでキーとバリューを書き出す。else:

print(k,":",v)return

if __name__ == "__main__":with open(args[1], encoding='utf-8') as f:

record = json.load(f)keyGet("json",record)

{"sensor": "multiverse-course-1003032","sendTime": "2018-06-25T05:00:01.6857809Z","data": [

{"@context": "http://purl.imsglobal.org/ctx/caliper/v1/Context","@type": "http://purl.imsglobal.org/caliper/v1/AssessmentEvent","actor": {

"@context": "http://purl.imsglobal.org/ctx/caliper/v1/Context","@id": "https://nlp.netlearning.co.jp/user/14383282","@type": "http://purl.imsglobal.org/caliper/v1/lis/Person","name": "LTI00000031","description": null,"extensions": {},"dateCreated": null,"dateModified": "2018-06-18T09:26:11Z"

},"action": "http://purl.imsglobal.org/vocab/caliper/v1/action#Started","object": {

"@context": "http://purl.imsglobal.org/ctx/caliper/v1/Context",…

$ winpty python r_20.py nobom.jsonDict-> jsonsensor : multiverse-course-1003032sendTime : 2018-06-25T05:00:01.6857809ZList-> dataDict-> data@context : http://purl.imsglobal.org/ctx/caliper/v1/Context@type : http://purl.imsglobal.org/caliper/v1/AssessmentEventDict-> actor@context : http://purl.imsglobal.org/ctx/caliper/v1/Context@id : https://nlp.netlearning.co.jp/user/14383282@type : http://purl.imsglobal.org/caliper/v1/lis/Personname : LTI00000031description : NoneDict-> extensionsdateCreated : NonedateModified : 2018-06-18T09:26:11Zaction : http://purl.imsglobal.org/vocab/caliper/v1/action#StartedDict-> object@context : http://purl.imsglobal.org/ctx/caliper/v1/Context…

Caliper v1.0 JSON

Output

Tableau⽤データ⽣成

22

Tableauデータ出⼒⽤Pythonプログラム事例user_loginがあるデータのみ抽出

# -*- coding: utf-8 -*-# 2018-8-16 Y.Tokiwa

import jsonimport sys

import globimport os

from datetime import datetimeimport pytzimport sys

args = sys.argv

def keyGet(dict):for k,v in dict.items():

keyGet1(k,v)return

def keyGet1(k,v):global ind_user_login, user_login, eventTime, action, asset_type, name

if isinstance(v,str):if k == "user_login":

user_login = vind_user_login = 1

if ind_user_login == 1:if k == "eventTime": eventTime = v if k == "action": action = v if k == "asset_type" :asset_type = v if k == "name": name = v

if isinstance(v,dict):# print("Dict-> ",k)return keyGet(v)

def iso_to_jstdt(iso_str):dt = Nonetry:

dt = datetime.strptime(iso_str, '%Y-%m-%dT%H:%M:%S.%fZ')dt = pytz.utc.localize(dt).astimezone(pytz.timezone("Asia/Tokyo"))

except ValueError:try:

dt = datetime.strptime(iso_str, '%Y-%m-%dT%H:%M:%S.%f%z')dt = dt.astimezone(pytz.timezone("Asia/Tokyo"))

except ValueError:pass

return dt

if __name__ == "__main__":

s3_directory = "./"+args[1]+"/*"s3_files = glob.glob(s3_directory)

for s3file in s3_files:with open(s3file, encoding='utf-8') as f:

ind_user_login = 0user_login = ""eventTime = ""action = ""asset_type = ""name = ""data = json.load(f)keyGet(data["data"][0])if ind_user_login == 1:

print(iso_to_jstdt(eventTime), user_login, action, asset_type, name,sep=",")続く

Tableau⽤データ事例

23

2019-08-06 13:19:29.156000+09:00,[email protected],Modified,,2019-08-06 13:19:29.470000+09:00,[email protected],Modified,,Report32019-08-06 13:20:38.110000+09:00,[email protected],NavigatedTo,course,home2019-08-06 13:20:42.358000+09:00,[email protected],NavigatedTo,course,announcements2019-08-06 13:25:57.934000+09:00,[email protected],NavigatedTo,discussion_topic,2019-08-06 13:25:58.018000+09:00,[email protected],Created,,Report5 返却および期末テストを除いた成績のお知らせ2019-08-06 13:25:58.285000+09:00,[email protected],NavigatedTo,discussion_topic,2019-08-06 13:26:08.521000+09:00,[email protected],NavigatedTo,course,roster2019-08-06 13:27:28.553000+09:00,[email protected],NavigatedTo,course,outcomes2019-08-06 13:27:34.110000+09:00,[email protected],LoggedOut,,2019-08-06 13:52:51.928000+09:00,15X0001,NavigatedTo,course,home2019-08-06 15:07:32.991000+09:00,17X0001,NavigatedTo,course,home2019-08-06 15:07:41.194000+09:00,17X0001,NavigatedTo,course,home2019-08-06 17:06:05.952000+09:00,17X0001,NavigatedTo,course,home2019-08-06 17:06:16.572000+09:00,17X0001,NavigatedTo,course,home2019-08-06 18:04:15.548000+09:00,16X0001,NavigatedTo,course,home2019-08-06 18:04:24.205000+09:00,16X0001,NavigatedTo,course,home2019-08-06 19:41:41.927000+09:00,17X0002,NavigatedTo,course,home2019-08-06 19:41:52.515000+09:00,17X0002,NavigatedTo,course,home2019-08-06 20:06:26.529000+09:00,17X0002,LoggedIn,,2019-08-06 20:06:26.848000+09:00,17X0002,NavigatedTo,course,home2019-08-06 20:06:27.556000+09:00,17X0002,LoggedIn,,2019-08-06 20:06:31.622000+09:00,17X0002,NavigatedTo,course,home2019-08-06 20:06:38.494000+09:00,17X0002,NavigatedTo,course,grades2019-08-07 13:49:01.311000+09:00,[email protected],LoggedIn,,2019-08-07 13:49:06.044000+09:00,[email protected],NavigatedTo,course,home2019-08-07 13:49:22.107000+09:00,[email protected],NavigatedTo,course,roster

次のスライドにこのデータを⽣成した元のCaliper

データを⽰す

24

{"sensor": "http://hosei.instructure.com/","sendTime": "2019-08-06T04:25:59.834Z","dataVersion": "http://purl.imsglobal.org/ctx/caliper/v1p1","data": [{

"@context": "http://purl.imsglobal.org/ctx/caliper/v1p1","id": "urn:uuid:3d5cf58b-1ef1-46df-bd83-13af7483be65","type": "ThreadEvent","actor": {

"id": "urn:instructure:canvas:user:125310000000000006","type": "Person","extensions": {

"com.instructure.canvas": {"user_login": "[email protected]","root_account_id": "125310000000000001","root_account_lti_guid": "tkMMgk9o7kftOu2 …YsRIE0p:canvas-lms","root_account_uuid": "tkMMgk9o7kftOu2 …. YsRIE0p","entity_id": "125310000000000006"

}}

},"action": "Created","object": {

"id": "urn:instructure:canvas:discussion:125310000000000099","type": "Thread","name": "Report5 返却および期末テストを除いた成績のお知らせ","extensions": {

"com.instructure.canvas": {"is_announcement": true,"entity_id": "125310000000000099"

}}

},"eventTime": "2019-08-06T04:25:58.018Z","edApp": {


},

"group": {"id": "urn:instructure:canvas:course:46","type": "CourseOffering","extensions": {

"com.instructure.canvas": {"context_type": "Course","entity_id": "46"

}}

},"membership": {

"id": "urn:instructure:canvas:course:46:Instructor:125310000000000006","type": "Membership","member": {

"id": "urn:instructure:canvas:user:125310000000000006","type": "Person"

},"organization": {

"id": "urn:instructure:canvas:course:46","type": "CourseOffering"

},"roles": ["Instructor"]

},"session": {

"id": "urn:instructure:canvas:session:93c6587b79128ee5cd6dd0ef9accb136","type": "Session"

},"extensions": {

"com.instructure.canvas": {"hostname": "hosei.instructure.com","request_id": "3aee1472-6dc1-4b56-aedb-85e655e3bea5","user_agent": "Mozilla/5.0 …. Chrome/75.0.3770.142 Safari/537.36","client_ip": "133.25.246.231","request_url": "https://hosei.instructure.com/api/v1/courses/46/discussion_topics","version": "1.0.0"

}}

}]}

Tableauデータを⽣成したCaliper v1.1 Data 事例

続く

25




Why Tableau

26

Amazon Redshift

アカデミックプログラム

27

LMSで記録された学習履歴サマリー

28

Tableauデータ読み込み

29

Tableauイベントの表⽰

30

Tableau分析結果まとめ

31

データソース置換

32

33

Documents

1から学ぶCaliperimsjapan.org/file/1_Caliper_20190916.pdf2019/09/16 · AWS Lambda 接続 SQS Pros & Cons 5 Pros S3あるいはDBよりも 速書き込みができる。 Cons 最

1から学ぶCaliperimsjapan.org/file/1_Caliper_20190916.pdf2019/09/16 · AWS Lambda 接続 SQS Pros & Cons 5 Pros S3あるいはDBよりも速書き込みができる。 Cons 最