Agile BI/DW
Daniel [email protected]
Experience
Used agile methods 10 years
Developed BI/DW 5 years
Led large BI/DW project during last year using Scrum
Database lead at alaTest.com
Why agile?
Deliver early
Reduce risk
Make the project steerable and transparent
Make development efficient and fun
How?
Just follow the book?
Yes, but
What are the user stories?
Agile DB practices?
Agile operations?
Overview
Before starting
Project cycles
Agile planning
Agile DB practices
Agile operations
Before the project
Introduce agile methods while selling
Agile architecture, dimension models
Agile estimation based offering
Sprint 0
Collocation (set up team room)
Appropriate HW, SW, test server
Involve operations
Contact the data source owners
Cycles
SprintDaily ScrumSprint PlanningSprint DemoSprint
Retrospective
Release cycleRelease cycleEnvision & fund
Plan
Build
Deploy & review
SprintPlan
Develop
Demo
Review
Daily Scrum
Planning
DW for loyalty card company
EpicAs marketing manager I want to use the DW to understand
customer behavior
ThemeAs a campaign manager I want to follow sale over time for a
certain store
User storyI want the customer dimension to be updated daily
Planning (decomposition)
Architecture driven story decompositionSee Agile Data
Warehousing, Ralph Huges
Use course architecture as common language for Product owner and
developers
Architectural dimensionsSubject area, Target layer,
Transformation, type, Refresh type, Refresh frequency, Automation,
User friendliness
Each user story must enable feedback
Dimension model
CustomernamecityStorenameregion
ProductnameclassDateyearmonthReceiptLineamount
Planning (decomposition)
Start with the most importantStar (subject area)
Dimension
Attributes in the dimension
Architecture
Customer registryProduct registryOrg. registryTransactions
Raw dataStagingDimension modelMartsOLAP Cube
ReportingApplications
ETL
Planning (decomposition)
ETL Start with simpler/earlier tasksTransformation typeDirect
data aggregations business rules
Refresh frequencyOne time load monthly weekly daily
Target layerRaw staging presentation end user view
Refresh typeLink snapshot incremental
Planning (decomposition)
BITransformation type
Refresh frequency
AutomationOn demand on work station on demand posted on server
scheduled
User friendlinessSingle table linked table navigation pick list
dash board
Using expert analysis tools reports
Breadth or depth?
BreadthSimplified load of the most important dimension
models
Early feedback, earlier build of dependent systems
DepthComplete load of one dimension at a time
Early deployment of usable sub systems
Agile DB Practices
Agile DB refactoringScott Ambler
Change scripts for everything
Common code ownershipEverybody can can change the models
DB models in visual modeling tool
Don't become a bottleneckBroad contact area between team members
and users, operations and source data specialists
Agile DB Practices
Course architecture
Agile modeling
Just in time design
Agile documentation
Agile DB Practices
DB Schema in repo
Subset data
Your own DB and development sandbox
Staged deployment: dev test prod
Continuous deployment
TDD?
Agile Operations
Involve operations from day one
Operations part of the project
Deploy at demo/production server often
Start source data deliveries ASAP
Agile Operations
Operations, support and maintenance during the projectKanban for
that?
Project time
#Features in prod.
Conclusions
Some mental barriers for DB/DW/BI-developers
Transparent
Steerable
Works great!
Happy customers (and developers)!
Literature
www.danielfagerstrom.com
Click to edit the title text formatClick to edit Master title
style
Click to edit the outline text formatSecond Outline LevelThird
Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline
LevelSeventh Outline LevelEighth Outline Level
Ninth Outline LevelClick to edit Master text styles
Second level
Third level
Fourth level
Fifth level
9/17/09
