2007-10-16
MonetDB spatial
1
Onderzoeksinstituut OTB
MonetDB spatial
Een open-source Spatial DBMS
Media Plaza/Het gebruik van OS software in GEO-ICT
Wilko Quak
2007-10-16 2MonetDB spatial
Onderzoeksinstituut OTB
Overzicht
• TUDelft / OTB / Sectie GISt / GeoInfoNed Project• De spatial DBMS markt.• Categorie-en van DBMS gebruikers• Intermezzo: spatial DBMS.• MonetDB en MonetDB (spatial)• De toekomst van MonetDB (spatial)
2007-10-16 3MonetDB spatial
Onderzoeksinstituut OTB
GIS technology
2007-10-16 4MonetDB spatial
Onderzoeksinstituut OTB
GISt visie
Spatial Information Infrastructure (SII), a vision…• Heading towards an information society with
varying impact on sectors of our economy• Impact on the geo-information sector is huge: geo-
info and processing can be transported via (wireless) networks and can be combined with positioning
• Engines of the SII nodes: geo-DBMSs (Data Base Management Systems) filled with geo-data
• Fair pricing of data and services, no more endless copying of data (and loss of quality/up-to-dateness)
2007-10-16 5MonetDB spatial
Onderzoeksinstituut OTB
GISt: scientific/technological goal
• Central research theme geo-DBMS as ‘glue’ between:
• 3D spatio-temporal modeling• Computational geometry• Distributed GI processing• Mobile GIS (LBS)• Knowledge engineering
Ambition: top 1 (3) of geo-DBMS Universities in the world
2007-10-16 6MonetDB spatial
Onderzoeksinstituut OTB
Geo Database Management Center• TU Delft open R&D center ‘geo-DBMS’• Partnerships with ICT industry: Oracle, ESRI,
Bentley• ‘Database lab’ and ‘3D GIS/VR lab’• Website http://www.gdmc.nl for
• Communication/documentation
• Internet GIS
2007-10-16 7MonetDB spatial
Onderzoeksinstituut OTB
Met welk onderzoek kan de Sectie GISt bijdragen?
• Onderzoek naar bruikbaarheid bestaande producten (benchmarken)
• Wat mist er nog• Meedenken met ontwikkelingen• Eigen DBMS ontwikkelen als research vehikel:
GeoInfoNed Project
2007-10-16 8MonetDB spatial
Onderzoeksinstituut OTB
GeoInfoNed – RGI-232
2007-10-16 9MonetDB spatial
Onderzoeksinstituut OTB
wie
• CWI – Leading DBMS experts with MonetDB
• TUDelft/OTB – Knowledge of spatial processes
• CycloMedia – Huge dataset and interesting problems
• RWS/AGI – Large and diverse datasets and interesting problems
2007-10-16 10MonetDB spatial
Onderzoeksinstituut OTB
Wat
• Is er nog wel ruimte op de markt? Een inventarisatie:• Aanbieders• Gebruikers
We gaan een spatial DBMS bouwen en daar onderzoek mee doen.
2007-10-16 11MonetDB spatial
Onderzoeksinstituut OTB
De spatial DBMS markt (aanbieders)
• Oracle – Marktleider commerciele segment• Informix/DB2 – Prima implemenatie. Niet erg
actief• PostgreSQL/PostGIS – ‘Marktleider’ open-source• SQLServer – komt binnenkort• MySQL – open-source, beperkte functionaliteit
2007-10-16 12MonetDB spatial
Onderzoeksinstituut OTB
Spatial DBMS markt (gebuikers)
4 classes:1. Server Builders: publish spatial data via web
server2. GIS User: Load various datasets and perform
complex analyses3. Data Maintainer: Maintain one core dataset4. Power Users: All of the above and more
2007-10-16 13MonetDB spatial
Onderzoeksinstituut OTB
Class 1: Web Server Builders
• Runs under GeoServer• You do not really need a DBMS for this (You
use a fraction of DBMS functionality)• Only one query counts: Find everything within
BBOX.• Multi-User• Small Queries
2007-10-16 14MonetDB spatial
Onderzoeksinstituut OTB
Class 2: GIS users
• Full fledges GIS is much more powerful but SQL experts might be tempted
• Main interest is functionality• Spend more time on loading data• Need a good query optimizer• Data mining
2007-10-16 15MonetDB spatial
Onderzoeksinstituut OTB
Class 3: Dataset Maintainers
• Limited number of queries.• 24/7 stability• Transactions.• Clustering of data after updates is interesting.
2007-10-16 16MonetDB spatial
Onderzoeksinstituut OTB
Class 4: Power users
• All of the above and more• Take time to understand software• Run into research problems:
• Point clouds from laser-scanning• Support 3d-volumes• Realtime dynamic data
2007-10-16 17MonetDB spatial
Onderzoeksinstituut OTB
Intermezzo: Spatial DBMS
• Spatial data types• Operaties op types• Indexeren op types
2007-10-16 18MonetDB spatial
Onderzoeksinstituut OTB
Spatial Data Types
• Basic types: Point Line Polygon
• More Complex: 3D, Geodetic Coordinates
2007-10-16 19MonetDB spatial
Onderzoeksinstituut OTB
Spatial Data types volgens OpenGIS
PointLineStringLinearRingPolygonMultiPointMultiLineStringMultiPolygon
2007-10-16 20MonetDB spatial
Onderzoeksinstituut OTB
Functions op spatial types
2007-10-16 21MonetDB spatial
Onderzoeksinstituut OTB
• R-tree puts MBR on object- MBR : minimum bounding rectangle- Object can be of type point, line or region- At lowest (leaf) level objects are grouped
to larger MBR’s, which can be grouped (and so on .....)
Indexeren spatial types
D
F
E
G
H
J
K
A
C
L
N
MB
I
A B C
D E F G H I J K L M N
Branching factor 4
Einde Intermezzo
2007-10-16 22MonetDB spatial
Onderzoeksinstituut OTB
MonetDB Introduction*
MonetDB is opens• Hardware trends• MonetDB design considerations• MonetDB architecture
*Slides borrowed from CWI
2007-10-16 MonetDB spatial
Onderzoeksinstituut OTB
Hardware Trends50% p/year:
- cpu speed
- mem size
- mem bandwidth
- disk bandwidth1% p/year:
- mem latency
10% p/year:
- disk latency
2007-10-16 MonetDB spatial
Onderzoeksinstituut OTB
Latency is the enemy!
• Commercial DBMS products (oracle, DB2, SQLserver) stem from OLTP roots
• focus on minimizing random I/Os => depend on latency!
• MonetDB: built for bulk access• optimize CPU and memory performance
2007-10-16 25MonetDB spatial
Onderzoeksinstituut OTB
MonetDB design considerations
• Multi-model database kernel support• Extensible data types, operators, accelerators• Database hot-set is memory resident• Simple data structures are better• Index management should be automatic• Do not replicate the operating system• Optimize when you know the situation• Cooperative transaction management
2007-10-16 26MonetDB spatial
Onderzoeksinstituut OTB
Monetkernels
MAPI protocol
JDBC
C-mapi lib
Perl
End-user application
ODBC PHP Python
SQL XQuery
MonetDB product family
2007-10-16 27MonetDB spatial
Onderzoeksinstituut OTB
SWOT
State-of-the-art SQL & XQuery DBMS; multi-model database support; decomposed storage model; query processing for decision support systems; extensible kernel; multiple APIs;…..
strengthTriggers and SQL procs missing; XQuery; some overhead in simple SQL queries; MIL scripting; missing GIS module
weakness
Commercial systems requirements; competition established market; floating interest; development skills and quality awareness
threats
Unique selling points for advanced applications in multimedia, streaming, covering both small and the large world
opportunities
GET INSPIRED BY ADVANCED APPLICATIONS
2007-10-16 28MonetDB spatial
Onderzoeksinstituut OTB
MonetDB - Physical data organization• Binary Association Tables
ID Day Discount10 4/4/98 0.19511 9/4/98 0.06512 1/2/98 0.17513 7/2/98 0
OID ID100 10101 11102 12103 13104 14
OID Day100 4/4/98101 9/4/98102 1/2/98103 7/2/98104 1/2/99
OID Discount100 0.195101 0.065102 0.175103 0104 0.065
2007-10-16 29MonetDB spatial
Onderzoeksinstituut OTB
Spatial MonetDB
2007-10-16 30MonetDB spatial
Onderzoeksinstituut OTB
Spatial MonetDB -- nu
• Eerste implementatie in volgende release (Q4).• OpenGIS implementatie gebaseerd op PostGIS
• OpenJUMP plugin• Shape2MonetDB converter • GeoTools Plugin (in ontwikkeling)
2007-10-16 31MonetDB spatial
Onderzoeksinstituut OTB
Spatial MonetDB – de toekomst
• MonetDB moet het hebben van zijn speciale architectuur.
• Uitdagingen:• Puntenwolken van Laserscanner.• Data-minen op grote spatio-temporele
datasets.• XML-Data.
• Saaier werk:• Testen huidige implementatie• debuggen
2007-10-16 32MonetDB spatial
Onderzoeksinstituut OTB
Vragen
Discussie