NYC Alt 2010 Nosql

Embed Size (px)

Citation preview

  • 8/6/2019 NYC Alt 2010 Nosql

    1/52

  • 8/6/2019 NYC Alt 2010 Nosql

    2/52

    Agenda

    What is NoSQL?

    NoSQL Databases

    Introducing MongoDB .NET Drivers for MongoDB

    Introducing NoRM

    Case Study: RateMySnippet.com

    Questions?

  • 8/6/2019 NYC Alt 2010 Nosql

    3/52

    NoSQL

  • 8/6/2019 NYC Alt 2010 Nosql

    4/52

    What is NoSQL?What is NoSQL?

    Coined in 1998 by Carlos Strozzi to describe aCoined in 1998 by Carlos Strozzi to describe adatabase that did not expose a SQL interfacedatabase that did not expose a SQL interface

    In 2008, Eric Evans reintroduced the term toIn 2008, Eric Evans reintroduced the term todescribe the growing non-RDBMS movementdescribe the growing non-RDBMS movement

    Broadly refers to a set of data stores that do notBroadly refers to a set of data stores that do notuse SQL or a relational model to store datause SQL or a relational model to store data

    Popularized by large web sites such as Google,Popularized by large web sites such as Google,Facebook and DiggFacebook and Digg

  • 8/6/2019 NYC Alt 2010 Nosql

    5/52

    NoSQL Databases

    Wide Column,Wide Column,Key/ValueKey/Value

    and Document Store...and Document Store...Oh My!Oh My!

  • 8/6/2019 NYC Alt 2010 Nosql

    6/52

    NoSQL DatabasesNoSQL Databases

    NoSQL databases come in a variety of flavorsNoSQL databases come in a variety of flavors

    XML (myXMLDB, Tamino, Sedna)XML (myXMLDB, Tamino, Sedna)

    Wide Column (Cassandra, Hbase, Big Table)Wide Column (Cassandra, Hbase, Big Table) Key/Value (Redis, Memcached with BerkleyDB)Key/Value (Redis, Memcached with BerkleyDB)

    Object (db4o, JADE)Object (db4o, JADE)

    Graph (neo4j, InfoGrid)Graph (neo4j, InfoGrid) Document store (CouchDB, MongoDB)Document store (CouchDB, MongoDB)

  • 8/6/2019 NYC Alt 2010 Nosql

    7/52

    Why NoSQL?

    67

    29

    4

    Sources of Greenhouse Gas Emissions*

    *ZEPA Estimate

    Livestock/Agriculture

    Automobiles

    Relational Databases

  • 8/6/2019 NYC Alt 2010 Nosql

    8/52

    RDBMSs and the Environment

    RDBMS Administrators are highly paid

    Highly paid individuals often buy larger than

    average homes or cars Larger than average homes and cars require

    more energy than smaller home and cars

    Therefore RDMBSs contribute to globalwarming more than NoSQL databases whichtypically do not require the addition of a DBA

  • 8/6/2019 NYC Alt 2010 Nosql

    9/52

    RDBMSs and the Environment

    RDBMSs often require high end servers andthat are taxing on disks

    High end servers consume more electricity thanmid-range servers

    Taxed disks fail more often than untaxed disks

    Therefore RDBMSs require more energy andproduce more waste (lots of hard drives inlandfills) than NoSQL DBs, which run on mid-range servers.

  • 8/6/2019 NYC Alt 2010 Nosql

    10/52

    Why NoSQL?

    55

    30

    15

    Leading Causes of Illnesses*

    *ZIH Estimate

    Contaminants in Food Supply

    Overuse of Antibiotics

    Relational Databases

  • 8/6/2019 NYC Alt 2010 Nosql

    11/52

    NoSQL and Healthcare

    The current healthcare crisis requires talentedsoftware engineers to fix the outdated or non-existent IT systems of the hospital system

    Talented software engineers spend a great dealof time mapping objects to tables in RDBMSs

    Talented software engineers are unable to fixhealthcare because they are mapping objectsto tables

    Therefore RDBMSs are causing illnessnes

  • 8/6/2019 NYC Alt 2010 Nosql

    12/52

    Dropping ACID

  • 8/6/2019 NYC Alt 2010 Nosql

    13/52

    Dropping ACID

    CAP Theorem and Eventual Consistency

    A distributed computer system cannot provideConsistency, Availability and PartitionTolerance at the same time

    Immediate consistency is expensive

    Some systems can tolerate dirty reads

    Removing relational storage structure helps toreduce transactional requirements

    Inserts/Updates written to fewer places

  • 8/6/2019 NYC Alt 2010 Nosql

    14/52

    Dropping ACIDDropping ACID

    ACID support is not required for blogs, statusACID support is not required for blogs, statusupdates, product listings, etc.updates, product listings, etc.

    Would a 2 second delay in finding out who theWould a 2 second delay in finding out who the

    mayor of Turnpike Spirits is matter?mayor of Turnpike Spirits is matter? Or that your friend is embarrassed to admit thatOr that your friend is embarrassed to admit that

    she loves Jersey Shore?she loves Jersey Shore?

    Modern relational databases simply do notModern relational databases simply do notscale to Internet proportionsscale to Internet proportions

    Sites like Digg, Facebook and Ebay have dataSites like Digg, Facebook and Ebay have datasets 10s or 100s of TB or even PB largesets 10s or 100s of TB or even PB large

  • 8/6/2019 NYC Alt 2010 Nosql

    15/52

    NoSchema DatabasesNoSchema Databases

    NoSQL databases generally do not impose aNoSQL databases generally do not impose aformal schemaformal schema

    Optimized storage for unstructured dataOptimized storage for unstructured data

  • 8/6/2019 NYC Alt 2010 Nosql

    16/52

    NoSchema Documents

    Related data is stored in a single document

    The ubiquitous Blog example

    Each post is a document

    Post might have reference to Blog or Userdocument or redundantly store blog/user data

    East post document has a nested collection oftags and comments

    Application layer must be smarter aboutreferential integrity

  • 8/6/2019 NYC Alt 2010 Nosql

    17/52

    NoSchema Documents

    var post = { Author : { Name : " J ohn Zablock i " , _i d : " 497ce96f. . . " }, Title : " On Running NerdDinner on MongoDB wi th NoRM" , Tags : [" mvc" , " mongodb" , " norm" ] , CreatedDate : new Date(' 6-14-2010' ), Text : " At the Hartford Code Camp. . . "}

  • 8/6/2019 NYC Alt 2010 Nosql

    18/52

    NoSchema Documents

    J ohn Zabl ock i 497ce96f395f2f052a494fd4 On Runni ng NerdDi nner on MongoDB - Part 1 mvc mongodb norm 6-14-2010

    At the Hartford Code Camp. . .

  • 8/6/2019 NYC Alt 2010 Nosql

    19/52

    Something Completely Different

  • 8/6/2019 NYC Alt 2010 Nosql

    20/52

    Introducing MongoDBIntroducing MongoDB

    Open source, document-oriented databaseOpen source, document-oriented database

    10gen corporate entity behind development10gen corporate entity behind development

    10gen supports official drivers for many10gen supports official drivers for manyplatforms,platforms, but not .NETbut not .NET!!

  • 8/6/2019 NYC Alt 2010 Nosql

    21/52

    MongoDB The BasicsMongoDB The Basics

    Schema-less documents stored in collectionsSchema-less documents stored in collections

    Documents are stored as BSON (Binary JSON)Documents are stored as BSON (Binary JSON)

    JavaScript used to query and manipulateJavaScript used to query and manipulatedocuments and collectionsdocuments and collections

    Each document in a collection has a uniqueEach document in a collection has a unique

    BSON ObjectId field named _idBSON ObjectId field named _id Collections belong to a databaseCollections belong to a database

  • 8/6/2019 NYC Alt 2010 Nosql

    22/52

    MongoDB Server FeaturesMongoDB Server Features

    IndexesIndexes

    Secondary, compound and uniqueSecondary, compound and unique

    ReplicationReplication Master/slave and replica setsMaster/slave and replica sets

    Auto-sharding (1.6)Auto-sharding (1.6)

    MapReduceMapReduce GridFSGridFS

    Large file storageLarge file storage

  • 8/6/2019 NYC Alt 2010 Nosql

    23/52

    MongoDB Hidden Features

    MongoDB will help you:

    Make more friends

    Be more confident in social situations Be more attractive to members of the opposite

    (or same) sex

  • 8/6/2019 NYC Alt 2010 Nosql

    24/52

    MongoDB on Windows

  • 8/6/2019 NYC Alt 2010 Nosql

    25/52

    Installing MongoDB on WindowsInstalling MongoDB on Windows

    Download the binaries from mongodb.orgDownload the binaries from mongodb.org

    Extract to Program Files directory (or wherever)Extract to Program Files directory (or wherever)

    Create a directory c:\data\dbCreate a directory c:\data\db Run mongod.exe from the command line with theRun mongod.exe from the command line with the

    --install switch--install switch

    SeeSee http://bit.ly/aed1RWhttp://bit.ly/aed1RW for some gotchasfor some gotchas

    To run the daemon without installing, simply runTo run the daemon without installing, simply runmongod.exe without argumentsmongod.exe without arguments

    Run mongo.exe to verify the daemon is runningRun mongo.exe to verify the daemon is running

    http://bit.ly/aed1RWhttp://bit.ly/aed1RWhttp://bit.ly/aed1RW
  • 8/6/2019 NYC Alt 2010 Nosql

    26/52

    MongoDB - ShellMongoDB - Shell

    The MongoDB interactive JavaScript shellThe MongoDB interactive JavaScript shell(mongo.exe) is a command line utility for(mongo.exe) is a command line utility forworking with MongoDB serversworking with MongoDB servers

    Allows for CRUD operations on collectionsAllows for CRUD operations on collections

    May be used for basic administrationMay be used for basic administration

    Creating indexesCreating indexes

    Cloning databasesCloning databases

    Also useful as a test-bed while building appsAlso useful as a test-bed while building apps

  • 8/6/2019 NYC Alt 2010 Nosql

    27/52

    MongoDB - Shell

    /*Connect to a server: port/database (defaults are localhost: 27017/test ): */mongo. exe localhost: 27017/AltNetGroup//Switch database:use CodeCamp//View collections in a database:show collections

    //create an i ndex on Name fi el ddb. Artists. ensureIndex({ Name : 1 });//copy one database to anotherdb. copyDatabase( " CodeCamp" , " AltNetGroup" )

  • 8/6/2019 NYC Alt 2010 Nosql

    28/52

    MongoDB Shell CRUDMongoDB Shell CRUD

    //Insert an i tem i nto a col lectiondb. Artists. insert({ Name : The Shins });//Fi nd an i tem i n a col lecti on:db. Artists. findOne({ Name: Radiohead});//Find i tems i n a col lection:db. Artists. find({ Name : /The/i});//Count i tems i n a col lectiondb. Artists. count();

  • 8/6/2019 NYC Alt 2010 Nosql

    29/52

    MongoDB Shell CRUD

    //Update an i tem i n a col lectiondb. Artists. update({ Name : J ack J ohnson }, $set : { Name : Jac k Johnson } });/*Update i tems i n a col lectionWithout $set i n thi s exampl e, whol edocument would be replaced! */db. Artists. update({ Name : { $ne : null } }, { $set : { Category : Rock } }, false, true);/*$ denotes special operators and operations$push, $pop, $pul l, etc. */

  • 8/6/2019 NYC Alt 2010 Nosql

    30/52

    MongoDB MapReduceMongoDB MapReduce

    MapReduce is used for batch manipulation andaggregation of data

    A Map function transforms a collection into aseries of key/value pairs

    A Reduce function operates on that key/valuepair series to produce output based on those

    input values An optional Finalize function is called after the

    reduce function is invoked

  • 8/6/2019 NYC Alt 2010 Nosql

    31/52

    MongoDB Shell MapReduce

    //Create some datadb. Posts. insert({

    Name : " On Instal li ng MongoDB as a Servi ce On Wi ndows" ,Tags : [" mongodb" ]});

    db. Posts. insert({ Name : " On Running NerdDinner on MongoDB wi th NoRM" ,

    Tags : [" mongodb" , " norm" , " mvc" ]});

    db. Posts. insert({ Name : " On A Si mpl e IronPython Route Mapper for ASP. NET MVC" ,Tags : [" mvc" , " ironpthon" ]

    });

  • 8/6/2019 NYC Alt 2010 Nosql

    32/52

    MongoDB Shell MapReduceMongoDB Shell MapReduce

    /*Create the map functionThi s functi on creates a vi ew that look s li k e{ mvc, [1, 1] },{ norm, [1] }*/var map = function() {

    if (! this. Tags) { return; } for (var index in this. Tags) {

    emit(this. Tags[index] , 1); } } ;

  • 8/6/2019 NYC Alt 2010 Nosql

    33/52

    MongoDB Shell MapReduceMongoDB Shell MapReduce

    /*Create the reduce functionconceptual ly, reduce gets cal led l i k e:reduce( " mvc" , [1, 1] );reduce( " norm" , [1]/*var reduce = function( ey, vals) { var count = 0; for (var index in vals) { count += vals[index] ; } return count; } ;

  • 8/6/2019 NYC Alt 2010 Nosql

    34/52

    MongoDB Shell MapReduceMongoDB Shell MapReduce

    /*Run the mapreduce command on the Posts col lectionusing the map and reduce functions defined abovestore the resul ts i n a col lecti on named Tags*/var result = db. runCommand( { mapreduce : " Posts " , map : map, reduce : reduce, out : " Tags " });db. Tags. find()

  • 8/6/2019 NYC Alt 2010 Nosql

    35/52

    .NET and MongoDB

  • 8/6/2019 NYC Alt 2010 Nosql

    36/52

    .NET and MongoDB

    No officially supported 10gen driver

    mongo-csharp

    Actively being developed and supported Supports typed collections

    Simple-mongodb

    Json-centric Actively developed, but less so than other

    drivers like mongo-csharp and NoRM

  • 8/6/2019 NYC Alt 2010 Nosql

    37/52

    NoRM

  • 8/6/2019 NYC Alt 2010 Nosql

    38/52

    NoRMNoRM

    OK, it is Pronounced No R M!OK, it is Pronounced No R M!

    Actively developed by Andrew Theken andActively developed by Andrew Theken and

    James Avery, among othersJames Avery, among others Active community with reliable supportActive community with reliable support

    I received help even as I prepared this slide!I received help even as I prepared this slide!

    Support for typed and untyped collections,Support for typed and untyped collections,MapReduce, Property Mapping, LINQ,MapReduce, Property Mapping, LINQ,Expando, nearly all CRUD operationsExpando, nearly all CRUD operations

  • 8/6/2019 NYC Alt 2010 Nosql

    39/52

    NoRM The BasicsNoRM The Basics

    //Connections managed with IDisposable pattern//Connect to test database on l ocalhostusing (Mongo mongo = Mongo. Create( " mongodb: //localhost/test" )){

    Console. WriteLine(mongo. Database. DatabaseName);}//Mongo instance has MongoDatabase property/*MongoDatabase has GetCollection methods for accessing MongoCollection instances*///CRUD operations performed on collections

    N RM CRUD

  • 8/6/2019 NYC Alt 2010 Nosql

    40/52

    NoRM - CRUDNoRM - CRUD

    //Class properties map to document structure public class Artist { //ObjectId property mapped to _id field i n document public ObjectId Id { get; set; } public string Name { get; set; } //IList property wi l l be mapped to J avaScript Array private IList _albums = new List(0); public IList Albums { get { return _albums ; } set { _albums = value; } } }

    CN RM CRUD

  • 8/6/2019 NYC Alt 2010 Nosql

    41/52

    NoRM - CRUDNoRM - CRUD

    var artist = new Artist() { Name = " BadLands" };//Inserting a document i nto a typed col lection mongo. Database. GetCollection( " Artists" ). Insert(artist);//Updating (replacing) a document i n a typed col lectionartist. Name = " Badlands" ; mongo. Database. GetCollection( " Artists" ). Save(artist);//Updating a nested collectionmongo. Database. GetCollection( " Artists" ). UpdateOne(new { Name = " Badlands" },

    new { Albums = M. Push( " Voodoo Highway" ) } );

    N RM CRUDN RM CRUD

  • 8/6/2019 NYC Alt 2010 Nosql

    42/52

    NoRM - CRUDNoRM - CRUD

    //Find al l documents i n a typed col lectionvar artists = mongo. GetCollection( " Artists" ). Find();Console. WriteLine(artists. FirstOrDefault(). Name);//Query with a document specvar artist = mongo. GetCollection( " Artists" ). FindOne(new { Name = " Badlands" });Console. WriteLine(artist. Albums. Count);//Count the documents i n a col lectionlong count = mongo. GetCollection( " Artists" ). Count();Console. WriteLine(count);

    N RM M R dN RM M R d

  • 8/6/2019 NYC Alt 2010 Nosql

    43/52

    NoRM - MapReduceNoRM - MapReduce

    //Add a Tags col lection added to Artist cl assprivate IList _tags; public IList Tags {

    get { return _tags; }set { _tags = value; }

    }//Add some tags use Set not PushAll, si nce Tags was null mongo. Database. GetCollection( " Artists" ). UpdateOne(

    new { Name = " Badlands" },new { Tags = M. Set(new List(){ " Hard Rock " , " 80s " }) });

    N RM M R dN RM M R d

  • 8/6/2019 NYC Alt 2010 Nosql

    44/52

    NoRM - MapReduceNoRM - MapReduce

    //Create map and reduce functonsstring map = @" function() {if (! this. Tags ) { return; } for (index in this. Tags) {

    emit(this. Tags[index] , 1); } } " ;

    string reduce = @" function(previous, current) {var count = 0;

    for (index in current) { count += current[index];

    } return count; } " ;

    N RM M R dN RM M R d

  • 8/6/2019 NYC Alt 2010 Nosql

    45/52

    NoRM - MapReduceNoRM - MapReduce

    //MapReduce class is responsible for calling mapreduce commandMapReduce mr = mongo. Database. CreateMapReduce(); //Represents the document passed to the//db. runCommand in the shell exampleMapReduceOptions options = new MapReduceOptions( " Artists" ){

    Map = map,Reduce = reduce,OutputCollectionName = " Tags "

    }; MapReduceResponse response = mr. Execute(options);var collection = mongo. Database. GetCollection( " Tags " );Console. WriteLine(collection. Count());

    N RM LINQNoRM LINQ

  • 8/6/2019 NYC Alt 2010 Nosql

    46/52

    NoRM - LINQNoRM - LINQ

    //LINQ provider exposed via AsQueryable method of MongoCollectionvar artists = mongo. Database. GetCollection( " Artists" ). AsQueryable(); //Find i tems i n typed col lectionvar artistsStartingWithB =

    from a in mongo. Database. GetCollection(). AsQueryable()where a. Name. StartsWith( B )select a;

    Console. WriteLine( " Fi rst arti st starti ng wi th B: " +artistsStartingWithB. First(). Name);

    NoRM LINQNoRM LINQ

  • 8/6/2019 NYC Alt 2010 Nosql

    47/52

    NoRM - LINQNoRM - LINQ

    //Find artists without pul ling back nested col lections//Note use of Regex name searchvar artistsWithoutAlbums =

    from a in mongo. Database. GetCollection(). AsQueryable()where Regex. IsMatch(a. Name, " land" , RegexOptions. IgnoreCase)select new { Name = a. Name };Console. WriteLine(artistsWithoutAlbums . First(). Name);

    //Fi nd arti sts wi th a gi ven tagvar artistsWithHardRock Tag =mongo. Database. GetCollection( " Artists" ). AsQueryable(). Where(a => a. Tags. Any(s => s == " Hard Rock " ));Console. WriteLine(artistsWithHardRock Tag. First(). Name);

    Case Study: RateMySnippet com

  • 8/6/2019 NYC Alt 2010 Nosql

    48/52

    Case Study: RateMySnippet.com

    Final Thoughts

  • 8/6/2019 NYC Alt 2010 Nosql

    49/52

    Final Thoughts

    Eat food. Not too much. Mostly plants.- Michael Pollan

    Final Thoughts

  • 8/6/2019 NYC Alt 2010 Nosql

    50/52

    Final Thoughts

    Write Code. Not too much. Mostly C#.- John Zablocki

    Final Thoughts The Stack

  • 8/6/2019 NYC Alt 2010 Nosql

    51/52

    Final Thoughts The Stack

    ASP.NET MVC

    MVCContrib

    MongoDB

    NoRM SQL Server

    NHibernate FluentNHibernate

    Spring.NET Recoil

    AutoMapper

    Q & A

  • 8/6/2019 NYC Alt 2010 Nosql

    52/52

    Q & A

    http://dllHell.net - my blog

    http://www.CodeVoyeur.com - my code

    http://www.linkedin.com/in/johnzablocki http://twitter.com/codevoyeur

    http://mongodb.org - Official MongoDB site

    http://groups.google.com/group/norm-mongodb http://www.summerofnorm.com - Coming Soon!

    http://dllhell.net/http://www.codevoyeur.com/http://www.linkedin.com/in/johnzablockihttp://twitter.com/codevoyeurhttp://mongodb.org/http://groups.google.com/group/norm-mongodbhttp://www.summerofnorm.com/http://www.summerofnorm.com/http://groups.google.com/group/norm-mongodbhttp://mongodb.org/http://twitter.com/codevoyeurhttp://www.linkedin.com/in/johnzablockihttp://www.codevoyeur.com/http://dllhell.net/