Webinar: Scaling MongoDB

  • View
    5.645

  • Download
    1

Embed Size (px)

Transcript

PowerPoint Presentation

Achieving Scale with MongoDB

David LutzSenior Solutions Architect

1

AgendaOptimize for ScaleDesign the SchemaBuild IndexesUse Monitoring ToolsUse WiredTiger StorageVertical ScalingHorizontal Scaling

Dont Substitute Scaling for OptimizationMake sure you are solving the right problem

Remedy schema and index problems first

Well discuss both

Optimization Tips: Schema Design

Documents are Rich Data Structures{customer_id : 123,first_name : John,last_name : "Smith,address : { location : [45.123,47.232] street : "123 Main Street", city : "Houston", state : "TX", zip_code : "77027" }profession : [banking, trader]policies: [{policy_number : 13,description : short term,deductible : 500},{policy_number : 14,description : dental,visits : []} ] }

Fields can contain an array of sub-documentsFieldsTyped field valuesFields can contain arraysStringNumberGeo-Location

5

The Document Data ModelMatches Application ObjectsEases development

FlexibleEvolves with application

High performanceDesigned for access pattern

{customer_id : 123,first_name : John",last_name : "Smith",address : { street: "123 Main Street", city: "Houston", state: "TX", zip_code: "77027" }policies: [{policy_number : 13,description: short term,deductible: 500},{policy_number : 14,description: dental,visits: []} ] }

The Importance of Schema Design

Very different from RDBMS schema design

With MongoDB Schema:denormalize the datacreate a schema with prior knowledge of your actual query patterns, then write simple queries

Real World ExampleProduct catalog for retailer selling in 20 countries

{_id: 375,en_US: { name: , description: , },en_GB: { name: , description: , },fr_FR: { name: , description: , },fr_CA: { name: , description: , },de_DE: ,}

What's good about this schema?Each document contains all the data about the product across all possible locales.It is the most efficient way to retrieve all translations of a product in a single query (English, French, German, etc).

Real World Example

But thats not how the data was accessed!

db.catalog.find( { _id: 375 }, { en_US: true } );db.catalog.find( { _id: 375 }, { fr_FR: true } );db.catalog.find( { _id: 375 }, { de_DE: true } ); and so forth for other locales

The data model did not fit the access pattern.Real World Example

Inefficient use of resourcesData in BLUE are being used. Data in RED take up memory but are not in demand.

{_id: 375,en_US: { name: , description: , },en_GB: { name: , description: , },fr_FR: { name: , description: , },fr_CA: { name: , description: , },de_DE: ,}

{_id: 42,en_US: { name: , description: , },en_GB: { name: , description: , },fr_FR: { name: , description: , },fr_CA: { name: , description: , },de_DE: ,}

Consequences of Schema RedesignQueries induced minimal memory overhead20x as many products fit in RAM at onceDisk IO utilization reducedApplication latency reduced{_id: "375-en_GB",name: ,description: , }

Schema Design PatternsPattern: pre-computing interesting quantities, ideally with each write operationPattern: putting unrelated items in different collections to take advantage of indexing

Anti-pattern: appending to arrays ad infinitumAnti-pattern: importing relational schemas (3NF) directly into MongoDB

Schema Design ResourcesThe docs! Data Model Design, Patterns & Exampleshttps://docs.mongodb.org/manual/core/data-model-design/https://docs.mongodb.org/manual/applications/data-models/https://docs.mongodb.org/manual/MongoDB-data-models-guide.pdf

The blogs! http://blog.mongodb.org "6 Rules of Thumb for Schema Design"Part 1: http://goo.gl/TFJ3drPart 2: http://goo.gl/qTdGhPPart 3: http://goo.gl/JFO1pI

Webinars, training, consulting, etc

Optimization Tips:Indexing

IndexesSingle biggest tunable performance factor Tree-structured references to your documentsIndexing and schema design go hand in hand

))))))))))))))

Indexing Mistakes and Their FixesFailing to build necessary indexesRun .explain(), examine slow query log and system.profile collection, download mtools

Building unnecessary indexesTalk to your application developers about usage

Running ad-hoc queries in productionUse a staging environment, use secondaries

Indexing StrategiesCreate indexes that support your queries!Create highly selective indexesEliminate duplicate indexes with compound indexesdb.collection.ensureIndex({A:1, B:1, C:1})allows queries using leftmost prefixOrder index columns to support scans & sortsCreate indexes that support covered queriesPrevent collection scans in pre-production environmentsdb.runCommand( { setParameter: 1, notablescan: 1 } )

Indexing Example Before an Indexdb.tweets.explain("executionStats").find( {"user.lang":ja"} ){"winningPlan" : { "inputStage" : { "stage" : COLLSCAN", "keyPattern" : { "user.lang" : 1 } } },"executionStats" : { "nReturned" : 3560, "executionTimeMillis" : 56, "totalKeysExamined" : 0, "totalDocsExamined" : 51428 } }

Indexing Example After an Indexdb.tweets.explain("executionStats").find( {"user.lang":ja"} ){"winningPlan" : { "inputStage" : { "stage" : "IXSCAN", "keyPattern" : { "user.lang" : 1 } } },"executionStats" : { "nReturned" : 3560, "executionTimeMillis" : 8, "totalKeysExamined" : 3560, "totalDocsExamined" : 3560 } }

Optimization Tips:Monitoring

The Best Way to Run MongoDB

Ops Manager allows you leverage and automate the best practices weve learned from thousands of deployments in a comprehensive application that helps you run MongoDB safely and reliably.

Benefits include:

10x-20x more efficient operationsComplete performance visibilityAssisted performance optimization

MongoDB Ops Manager can do a lot for [ops teams].

Best Practices, Automated. Ops Manager takes best practices for running MongoDB and automates them. So you run ops the way MongoDB engineers would do it. This not only makes it more fool-proof, but it also helps youCut Management Overhead. No custom scripting or special setup needed. You can spend less time running and managing manual tasks because Ops Manager takes care of a lot of the work for you, letting you focus on other tasks.Meet SLAs. Automating critical management tasks makes it easier to meet uptime SLAs. This includes managing failover as well as doing rolling upgrades with no downtime.Scale Easily. Provision new nodes and systems with a single click.

22

Ops Manager Provides:for DevelopersVisual Query Profiler

for AdministratorsIndex SuggestionsAutomated Index BuildsMonitoring and Alerting

for OperationsAPM IntegrationDatabase AutomationBackup with Point-In-Time Recovery

MongoDB Ops Manager can do a lot for [ops teams].

Best Practices, Automated. Ops Manager takes best practices for running MongoDB and automates them. So you run ops the way MongoDB engineers would do it. This not only makes it more fool-proof, but it also helps youCut Management Overhead. No custom scripting or special setup needed. You can spend less time running and managing manual tasks because Ops Manager takes care of a lot of the work for you, letting you focus on other tasks.Meet SLAs. Automating critical management tasks makes it easier to meet uptime SLAs. This includes managing failover as well as doing rolling upgrades with no downtime.Scale Easily. Provision new nodes and systems with a single click.

23

Fast and simple query optimization with the Visual Query ProfilerQuery Visualization and Optimization

Key Takeaway: Save time, effort, reduce risk with Fast and simple query optimization via the new Visual Query Profiler

Talking points: Query and write latency are consolidated and displayed visually; your ops teams can easily identify slower queries and latency spikes Visual query profiler analyzes the data it displays and provides recommendations for new indexes that can be created to improve query performanceOps Manager and Cloud Manager can automate the rollout of new indexes, reducing risk and your teams operational overhead

24

Example Deployment 12 Servers

Install, Configure150+ stepsError handling, throttling, alertsScale out, move servers, resize oplog, etc.10-180+ stepsUpgrades, downgrades100+ steps

Without Ops Manager

It is, of course, possible to do these things without MMS.

But it takes work. Typically manual work, or custom scripting.

In either case, these things take time, require you to check for mistakes and are more prone to having things go wrong.25

With Ops Manager

Also Available in the CloudCloud Manager allows you to leverage and automate the best practices weve learned from thousands of deployments in a comprehensive application that helps you run MongoDB safely and reliably

in the cloud!

http://cloud.mongodb.com

MongoDB Ops Manager can do a lot for [ops teams].

Best Practices, Automated. Ops Manager takes best practices for running MongoDB and automates them. So you run ops the way MongoDB engineers would do it. This not only makes it more fool-proof, but it also helps youCut Management Overhead. No custom scripting or special setup needed. You can spend less time running and managing manual tasks because Ops Manager takes care of a lot of the work for you, letting you focu