Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
<Insert Picture Here>
Berkeley DB: Data Storage for the Edge and Beyond
Ashok Joshi
Senior Director, Oracle Berkeley DB and Database Mobile Server
development, Oracle NoSQL Database development
2Copyright © 2012, Oracle Corporation Japan. All rights reserved.
以下の事項は、弊社の一般的な製品の方向性に関する概要を説明するもので
す。また、情報提供を唯一の目的とするものであり、いかなる契約にも組み
込むことはできません。以下の事項は、マテリアルやコード、機能を提供す
ることをコミットメント(確約)するものではないため、購買決定を行う際
の判断材料になさらないで下さい。オラクル製品に関して記載されている機
能の開発、リリースおよび時期については、弊社の裁量により決定されます
。
OracleとJavaは、Oracle Corporation 及びその子会社、関連会社の米国及びその他の国における登録商標です。
文中の社名、商品名等は各社の商標または登録商標である場合があります。
3Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Embeddable database libraries
• Berkeley DB is NOT a server
• Embed Berkeley DB within your application to
get
• Transactions
• Recovery
• Scalability
• High availability
• Indexing
• Concurrency and high performance
• Every application needs to manage data
4Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Build vs. Buy
• Consider rebuilding following components
• Graphic Engine
• Network Protocol Stack
• Encryption Library
5Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Build vs. Buy
• Consider rebuilding following components
• Graphics Engine
• Network Protocol Stack
• Encryption Library
• Data Storage?
CoreFunctions
GraphicStorage
Network Encrypt
6Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Benefits of Berkeley DB vs. building
your own solution
• Time to market
• Design, development time and cost
• Resources
• Focus on company‟s core competencies
• Quality
• Proven reliability with various situations
• Lifecycle
• „Software maintenance‟ is defined as the following (IEEE‟91)
• Perfective maintenance
• Adaptive maintenance
• Corrective maintenance
• Preventative maintenance
7Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Benefits of Berkeley DB …
• Commercial solution from leading vendor
• Leverages success of Relational DBMS, SQL, XML/XQuery DB
• Addresses the need for customizable data management
• Transaction
• Recovery
• Replication
• Scalability
• Character Set
• LRU data caching
• Concurrency/Locking
• Indexed access
• SQL
• XML schema and query
8Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Berkeley DB Value Proposition
• Small footprint, efficient and lightweight
• Industrial grade performance, reliability and availability
• Highly flexible, configurable, easy to use
• Maintenance-free deployment
9Copyright © 2012, Oracle Corporation Japan. All rights reserved.
History of Berkeley DB
• ‟91 Started from BSD Unix
• AT&T‟s „dbm‟ replaced by „bdb‟
• Single user (no locking), no transactions, no recovery, few tools
• ‟96 Netscape‟s requirements
• Locking, transactions, ACID compliance, tools, …
• Without installation and no DBA!
• ‟96 Sleepycat founded
• For commercial support
• ‟06 Acquired by Oracle
10Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Did you know Berkeley DB...?
• Is everywhere
• runs on 200 million machines across the world
• Scales up, scales down
• powers mobile phones to stock exchanges
• Is proven
• runs inside every copy of Linux (and MacOS X and Solaris)
• Powers the Internet
• used extensively by Akamai, Yahoo, AOL, Amazon, eBay and many others
• Is highly available
• 70% of the global daily email traffic goes through BDB
11Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Berkeley DB Product Overview
• Embeddable database technology
• Small footprint, efficient and lightweight
• Fast, scalable, transactional, recoverable
• Industrial grade performance, reliability and availability
• Highly flexible, portable, configurable, easy to use
• Easily and seamlessly integrated into any application (C, C++, C#,
Java, XML, scripting languages, ...)
• Drop-in replacement/upgrade for SQLite
• Manage and synchronize mobile and remote data with the Oracle DB
• Proven in 200+ million deployments
• Used by all versions of Linux, BSD Unix, OpenLDAP, OpenOffice and
hundreds of other open source and commercial software packages
BDB in Devices and Appliances
Key Benefits• Small Footprint
• Synchronization
• Performance
• Scalability
• Reliability
• Embeddable
Storage
Smart Metering,
Smart Grid, M2M
Medical &
Life Science
Devices
Mobile Applications
Email &
Messaging
Entertainment &
Gaming Devices
BDB in Infrastructure
Key Benefits• Performance
• Scalability
• Reliability
• High Availability
• Embeddable
NetworkingEquipment
Authentication /Identity Management
Gateways
SOA / WebServices /
EAI
Caching /Logging
Search /Content Management
Resource /
Systems Mgmt
14Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Understanding
BerkeleyDB APIs
15Copyright © 2012, Oracle Corporation Japan. All rights reserved.
RDBMS vs. Berkeley DB
Your Application Code
ODBC, JDBC, OCI, etc.
TCP/IP
TCP/IP
SQL Parser
Query OptimizerYour Application Code
Berkeley DBStorage Manager
Conventional RDBMS Berkeley DB
16Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Key/value pairs
000001 “slkdjflskdjflskdfj”000001 “slkdjflskdjflskdfj”000001 “slkdjflskdjflskdfj”“000001” “가나다라”
Key Value
17Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Key/value vs. Relational
ID First name, Last name, Phone number, City, State, Zip
key value
First name Last name Phone Number City State ZipID First name Last name Phone Number City State ZipID + First name Last name Phone Number City State ZipID
ID First name, Last name, Phone number, City, State, ZipID First name, Last name, Phone number, City, State, Zip
Primary Key
Columns in Relational Model
Key/value of Berkeley DB
18Copyright © 2012, Oracle Corporation Japan. All rights reserved.
ID First name, Last name, Phone number, City, State, Zip
key value
Index
First name Last name Phone Number City State ZipID First name Last name Phone Number City State ZipID + First name Last name Phone Number City State ZipID
ID First name, Last name, Phone number, City, State, ZipID First name, Last name, Phone number, City, State, Zip
Primary Key
Columns in Relational Model
Key/value of Berkeley DB
IndexPhone Number
Phone NumberPhone Number
IDID
ID
Phone number IDSecondary DB
Callback function
19Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Cursor
005 abcdedf
Key Value
002 abcdef…
001 abcd…
013 dkdkw
005 dbaaa
093 dkdkw
……
20Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Btree
Access Methods
Hash
Queue & Recno
Heap
21Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Compression
• Btree supports compression
• Prefix compression is default behavior
• Custom compression algorithms
ABD010320DBCEF1
ABD010320DBCEF2
ABD010320DBCEF3
ABD010320DBC3A5
ABD010320DBCEF1
ABD010320DBCEF2
ABD010320DBCEF3
ABD010320DBC3A5
22Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Encryption
• Rijndael/AES for encryption/decryption
• Configured using 128bit Key
• 16-byte initialization vector generated using Mersenne Twister
• Checksum using SHA1, using 160-bit message digest
• Encrypts DB files and Log files
23Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Replication
Master Replica #1
Replica #2…
App App
readwrite read
App
read
24Copyright © 2012, Oracle Corporation Japan. All rights reserved.
In-Memory Configuration
• Trade off between “Performance” and “Durability”
• Memory only configuration
• DB files
• Log files
• Environment files
25Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Performance Example
• Linux : Suse Linux 9.1, AMD Athlon 64 3200+ at 1GHz, 1GB RAM
• Solaris : Sun Solaris 10(x86), SunFire v20z with Dual AMD Opteron at 2GHz, 16GB RAM
• Windows : XP Professional SP2, AMD Athlon 2600+ 2.0GHz, 768MB RAM
Oper/Sec
1,000,000
800,000
600,000
400,000
200,000
1,200,000
Linux Solaris Windows
1 Record Read (DS)
1 Record Write (DS)
1 Record Read (TDS)
1 Record Write (TDS)
1 Record Write (TDS, Write, No-Sync)
26Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Berkeley DB Product Family
1. The original transactional storage engine for applications developed with C, C++, Java, PHP, Perl, etc.
3. Transactional storage engine written in Java and optimized for Java environments
2. Berkeley DB XML provides XQuery-based access to XML documents
27Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Berkeley DB Java Edition
• New product, not a port of the C product
• Same philosophy
• New language for modern runtime
• 100% pure Java
• Single JAR file for ease of distribution, installation
• Runs in same JVM as the application code
• Additional APIs based on Java Collections standard for
transactional collections
• Single process, multi-threaded data access
• Footprint as low as 435KB
28Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Berkeley DB XML
• Built on Berkeley DB, inherits all those features
• Embeddable XML support via XPATH 2.0, XQuery 1.0
• Combine XML and non-XML data in the same database/transaction
• Flexible indexing, cost-based query optimization
• But no full-text search in the current product
• Flexible storage control
• Node
• Whole document
• Optimize for update or retrieval
• Schema support, including validation
• Document streaming from any URI
• DOM-like navigation of documents and result
29
SQL Support
30
SQL SupportHigh Level Overview
• Oracle Berkeley DB (BDB)• The industry-leading open source, embeddable, key-value
database that provides developers with fast, reliable, local persistence with zero administration
• SQLite• A transactional embedded relational database management
system contained in a small C programming library
• Berkeley DB 11gR2 provides the best of both: a small footprint, embeddable, fast, reliable, scalable SQL library with zero administration.
31
SQL SupportA peek inside the box
• Includes sqlite3, JDBC and ODBC
• Implements SQLite 3.6.21 interface - “drop in" replacement for SQLite
• Better concurrency, performance & scalability
• Scalable to terabytes of data, billions of records,
• Berkeley DB reliability - proven in millions of deployments
Yo
ur
Ap
pli
cati
on
Storage Adapter
sqlite3(), ODBC,
JDBC APIs
Tokenizer Parser Generator
Oracle Database
Virtual Machine
Oracle Berkeley DB
SQLite Library
Concurrent B-TreePage Cache Transaction Logs
Oracle Database Mobile Servers
BDB Library
32
SQL Layer Performance
33Copyright © 2012, Oracle Corporation Japan. All rights reserved.
References
34Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Berkeley DB in Open Source
Berkeley DB
All versions
of Linux
All versions
of BSD UNIX
Apache Web Server,
Directory, et al
LDAP Directory
Productivity suite
Chandler email/PIM
Website traffic
analysis
Kerberos Network
Authentication
Database
Subversion version
control system
Red Hat Package
Manager
Internet search
Text editorMail server (MTA)
Mail server (MTA)
SquidGuard
spam blocker
Spam blocker
Perl
Python
GNU C library
PHP
EmailApplicationsInfrastructure Programming
LanguagesToolsOperating
Systems
35Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Berkeley DB Customers
Financial
Services
Service
Providers
Content
Mgmt
Storage &
Sys Mgmt
Enterprise
Infra/EAISecurityNetworking
Telecom
Applications
infrastructure
Devices/
Appliances
Berkeley DB
36Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Wibro ACR
37Copyright © 2012, Oracle Corporation Japan. All rights reserved.
• 8 Channel Digital Video Recorder
• Features
• 8 Channel video & audio realtime recording capability
• High picture quality : MPEG4
• Remote monitoring and control through network
• Convenient search function with JOG/Shuttle
• Recording during replay
Samsung DVR “SHR 4080”
38Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Motorola A760 A768 A780 E680 Smart Phones
• Requirement: Motorola‟s Linux-based smart phones
needed a data management component to be fast,
flexible, reliable and designed to operate within highly
constrained environments.
• Solution: Motorola selected best-of-breed open
source components including Sleepycat‟s Berkeley DB,
Montavista Linux and Trolltech Qt. Over 3 million of
these devices have been shipped.Motorola A760
Motorola A768
Motorola A780
Motorola E680
39Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Motorola Wireless Network Gateway
Runs on Berkeley DB
• Requirement: Motorola‟s Wireless Network Gateway (WNG) needed carrier-class performance, scalability and reliability. In order to meet 99.999% uptime requirements, the data manager needed to have the ability to automatically restart after a failure with very fast startup and shutdown speed.
• Solution: Motorola selected Berkeley DB to store configuration data such as radio ID and user information for the WNG. Berkeley DB met Motorola‟s requirements for high performance and ability to operate unattended and recover quickly from system crashes.
Ray van Tassle, Senior Staff Engineer, Motorola
“Berkeley DB was 20 times faster than other databases. It has the operational
speed of a main memory database, the startup and shut down speed of a disk-
resident database, and does not have the overhead of a client-server inter-
process communication.”
40Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Cisco Systems Broadband Provisioning
Register Runs on Berkeley DB
• Requirement: Cisco‟s Broadband Provisioning Registrar (BPR) needed to manage up to 5 million networked devices and 150 configuration change transactions/second. BPR needed an embedded data manager which was fast, scalable, reliable and cost-effective.
• Solution: Berkeley DB was used to replace an object-oriented database in BPR. The result was faster, more reliable and saved Cisco $50,000/CPU.
Anton Okmianski, Senior Software Engineer, Cisco Systems
“When we switched to Sleepycat we never looked back. It gave us high
performance, small footprint and a set of well-targeted features that let us
take our solution to a new level.”
Cisco BPR
41Copyright © 2012, Oracle Corporation Japan. All rights reserved.
Openwave Mobile Messaging
Runs on Berkeley DB
• Requirement: Openwave customers demanded a high performance, low TCO solution
• Solution: Openwave uses Berkeley DB for the carrier-class message store and LDAP directory that are the basis for all of its messaging products, including Email Mx, MMSC and IP Voicemail. By eliminating the operator‟s database licensing costs, extra hardware costs and DBA costs, the Berkeley DB based solution provided lower acquisition and operating costs for Openwave‟s customers.
Rich Wong, SVP Products & Solutions Group, Openwave Systems
“Speed, reliability and ease-of-use are important considerations for
telecommunication infrastructure equipment. We’re pleased that Sleepycat
continues to improve Berkeley DB for the telecommunications industry.
Openwave relies on Berkeley DB as the core message store in Email Mx, a
messaging platform that handles more than 1.5 billion messages a day.”
42Copyright © 2012, Oracle Corporation Japan. All rights reserved.
• Requirement: AT&T‟s provisioning software for broadband networks originally used an object oriented database, which was too slow, complex and expensive. Key data manager requirements were:
• Speed to handle the flood of requests following restoration of power after an electrical outage
• Strict transactional integrity to ensure the accuracy of communication with billing and customer care systems
• Solution: Berkeley DB was chosen for use as the backing store for all provisioning operations. Without sacrificing transactional integrity, Berkeley DB provided very fast performance, resulting in significant hardware savings for AT&T‟s customers.
Berkeley DB Powers AT&T Network
Management Systems
43Copyright © 2012, Oracle Corporation Japan. All rights reserved.
• Products Powered by Berkeley DB
• MMSC (Multi-Media Messaging System)
• Highest ranked in Quality of Service
• Highest ranked in Response times
• End-to-end message delivery
• MMSC latency
• WSB (Wireless Service Broker)
• WAP-based services
• Video/Audio streaming
• UOne (Unified communications/messaging)
• Single message store for
• Voice
• Fax
• MMS
• Video
LogicaCMG Mobile Messaging
44Copyright © 2012, Oracle Corporation Japan. All rights reserved.
EMC Storage
Centera content-aware storage:• Conventional file access• Share duplicate blocks and files to reduce cost
What we store:• Device configuration and logs• File metadata• Hashes and reference counts• ACLs for administration
Why we win:• Scalability• Performance and reliability• Embeddability