110701 asakusaè¬è³‡–™

  • View
    1.369

  • Download
    0

Embed Size (px)

DESCRIPTION

Japanese presentation for Asakusa Framework

Text of 110701 asakusaè¬è³‡–™

  • 1. HadoopAsakusa Rev. 1.02011/7/1 OSS110701Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved

2. Asakusa HadoopHadoop 13 100 Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 2 3. Hadoop BI 4-520-30 IF Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 3 4. Hadoop BI MapReduce Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 4 5. Asakusa Hadoop BI/ MonkeyMagic Oozie Pig HiveMapReduceJavaCore HDFS MapReduce Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 5 6. HadoopAsakusa Hadoop Asakusa DAGDSL Ashigel ModelGenarator MapReduce Ashigel MonkeMagic ThunderGateMySQL Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 6 7. AsakusaDSL I/F DBAshigel ThunderGateImporter/AsakusaExporter/ MapReduceMonkeyMagicRecovererJOB JOB ModelGenerator Hadoop HadoopMonkeyMagicHDFS MapReduceMySQL TemplateGeneratorTest Driver Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 7 8. Asakusa ModelGenerator WritableDSL ThunderGate Ashigel Import/Export API DSLMapReduce DAGDSL DSL Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 8 9. Asakusa JavaEclipse/MavenMapReduce Model Generator DSL Domain Specific Language Ashigel CompilerRumtime LibraryThunderGate Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 9 10. ModelGeneratorMapReduce Java writable ModelGenerator MySQL DDL SQL) Table View HadoopIOEclipseDSL TableHadoop Hadoop Table Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 10 11. AshigelHadoop DSL BatchDSL FlowDSLDSL OperatorDSLMap/Reduce Map/Reduce MapReduce Map Reduce Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 11 12. DAG DSL DSL 3DSL BatchDSL FlowDSL OperatorDSL Replace TX DSLDB MR Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 12 13. BatchDSL BatchDSLDSL job Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 13 14. FlowDSL Flowpart FlowDSL@Overrideprotected void describe() { // Join join = op.join(itemIn, orderIn); // SetStatus missing = op.setStatus(join.missed, " "); orderOut.add(missing.out);Java DAG //Sum sum = op.sum(join.joined);Operator Obj //ToAmount result = op.toAmount(sum.out); resultOut.add(result.out); //core.stop(result.original);} Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 14 15. FlowDSL Flowpart FlowDSL 2@OverrideOperatorprotected void describe() { ReNewTxApMachingOperatorFactory f = new ReNewTxApMachingOperatorFactory(); UpdateTransactionsOperatorFactory f1 = new UpdateTransactionsOperatorFactory(); CoreOperatorFactory core = new CoreOperatorFactory(); // No OpeedgeBranchSlipNoWithWithout bra11 = f.branchSlipNoWithWithout(inApMaching);edge // No BranchRecGapAndUnmatch bra12 = f.branchRecGapAndUnmatch(bra11.out2); //edgeInitSlipInfo upd11 = f.initSlipInfo(bra12.out1); // InitBillInfo upd12 = f.initBillInfo(bra12.out1); //NoConfluent cfl11 = core.confluent(bra12.out2, upd12.out);edge// GroupSortBranchDeficitSurplusDiv grs11 = f1.groupSortBranchDeficitSurplusDiv(cfl11.out); Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 15 16. OperatorDSL DSL DAG OperatorOperatorMapReduce /** * @param info * @param order @return /@MasterJoinpublic abstract JoinOrder join(ItemInfo info, OrderDetail order); Asakusa/**@param each@return /@Summarizepublic abstract SumOrder sum(JoinOrder each);/** * @param total * @return /@Convertpublic OrderAmount toAmount(SumOrder total) { amount.setAmount(total.getAmount()); amount.setOrderId(total.getOrderId()); return amount; } 16 Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 17. OperatorDSL public abstract class ExampleOperator { /** @param masters@param tx @return null / @MasterSelection public ItemMst selectItemMst(List masters, HogeTrn tx) { for (ItemMst mst : masters) { if (mst.getStart() "asakusa") doboot_jobs("order") # jobflow order jobnet("order", :to => []) doauto_sequencejobnet("__IMPORT__") do boot_jobs("bulkloader") jobnet("bulkloader") doauto_sequencejob("bulkloader.asakusa", ""$ASAKUSA_HOME/"bulkloader/bin/importer.sh primary asakusa tutorial order "$($ASAKUSA_HOME/monkeymagic/bin/execution_id.sh)"20380101000000 "#{BATCH_ARGUMENTS}"", :instance_name => "asakusa", :credential_name => "asakusa") endendjobnet("__STAGE_GRAPH__") do boot_jobs("stage0001") # stage - tutorial / order / stage0001 hadoop_job_run("stage0001", "$ASAKUSA_HOME/monkeymagic/bin/hadoop_job_run.sh com.example.tutorial.batch.tutorial.order.stage0001.StageClient"$ASAKUSA_HOME/batchapps/tutorial/lib/jobflow-order.jar" -D "com.asakusafw.executionId=$($ASAKUSA_HOME/monkeymagic/bin/execution_id.sh)" -D"com.asakusafw.user=$USER" -D "com.asakusafw.batchArgs=#{BATCH_ARGUMENTS}"", :to => ["stage0002"]) dohadoop_job("tutorial.order.stage0001") end # stage - tutorial / order / stage0002 hadoop_job_run(stage0002, $ASAKUSA_HOME/monkeymagic/bin/hadoop_job_run.sh Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 28