06 integrate elasticsearch

AngularJS + Asp.Net Web Api, Signalr, EF6, Redis + Elasticsearch(6/6) - Web : (erhwenkuo@gmail.com)1. AngularJS 2. AngularJS + HighChart3. AngularJS + Asp.Net WebApi 2 +SignalR4. AngularJS + Asp.Net WebApi 2+ SignalR + Entity Framework 6 (ORM)5. AngularJS + Asp.Net WebApi 2+ SignalR +Entity Framework 6 (ORM) + Redis (Nosql)6. AngularJS + Asp.Net WebApi 2+ SignalR +Entity Framework 6 (ORM) + Redis (Nosql) + Elasticsearch ()1Document, Source code & Training Video (6/6)https://github.com/erhwenkuo/PracticalCoding

2Previous Training Session Document, Source code & Training Video (5/6)https://www.youtube.com/watch?v=Xfu4EVBdBKo

http://www.slideshare.net/erhwenkuo/05-integrate-redisSession, , YoubeSlideshare3AgendaElasticsearch IntroductionElasticsearch WorkshopElasticsearch in action using Stackoverflow DatadumpDeveloping Angularjs with ElasticsearchHighchart , AngularJS ,Web API2 , SignalR2 , EF6 , Redis + Elasticsearch Integration

4Elasticsearch Introduction

5Elasticsearch WebsiteHttp://www.elasticsearch.com/

6Whos using elasticsearch?GitHubGitHub uses Elasticsearchs robust sharding and advanced queries to serve up search across data in 4 million users code repositories.GitHub uses Elasticsearchs routing parameter and flexible sharding schemes to perform searches within a single repository on a single shard, doubling the speed at which results are served.GitHub uses Elasticsearchs histogram facet queries, as well as other Elasticsearch analytic queries, to monitor their internal infrastructure for abuse, bugs and more.

7Whos using elasticsearch?WikipediaElasticsearchs reference manual and contribution documentation promised an easy start and pleasant time getting changes upstream when needed to.Elasticsearchs super expressive search API lets Wikimedia search any way needed and gives the company confidence that it can be expanded, including via expressive ad-hoc queries.Elasticsearchs index maintenance API lets Wikimedia maintain the index right from its MediaWiki extension

8Whos using elasticsearch?

3 machines doing search with ElasticSearch

stackoverflow

9ElasticWho?ElasticSearch is a flexible and powerful open source, distributed real-time search and analytics engineFeatures:Real time analyticsDistributedHigh availabilityMulti tenant architectureFull text indexDocument orientedSchema freeRESTful APIPer-operation persistence

10Elasticsearch Workshop

11Download & StartDownload Elasticsearch (Current version: 1.4.2)http://www.elasticsearch.com/downloadsUnzip & Modify elasticsearch.ymlcluster.name: {your_searchcluster_name}node.name: {your_cluster_node_name}http.cors.enabled: trueUse command console to run:bin/elasticsearch on *nixbin/elasticsearch.bat on Windows

123

12Install Elasticsearch Plugins (elasticsearch-head)A web front end for an Elasticsearch clusterhttps://github.com/mobz/elasticsearch-head

You need to have internet access, otherwise it would fail!!1Restart elasticsearch and key in below url in browser:http://localhost:9200/_plugin/head/

213RESTful interfaceElasticsearch default use port 9200 for Http Restful interfaceLets check if Elasticsearch is alive!!

14Elasticsearch TERMs

15

Create IndexElasticsearch Index is similar like Database in relational DBFor example, create a index named stackoverflow

Default setting in Elasticsearch:Each Index will split to 5 shards and has 1 replication

16Create Another Elasticsearch NodeCopy elasticsearch-1.4.2 folder to elasticsearch-1.4.2-Node2Modify elasticsearch-1.4.2-Node2/config/elasticsearch.ymlcluster.name: {your_searchcluster_name}node.name: {your_cluster_node_name-Node2}http.cors.enabled: truetransport.tcp.port: 9301http.port: 9201If we set different Elasticsearch Node on the sameMachine, then we need to assisgn different communication ports.

17

Start All Elasticsearch NodesUse command console to start two Elasticsearch NodesNow we a Elasticsearch Cluster!So so so easy~!18Delete Index

The Index is deleted!!19

Index a Document under a specific TypeType is similar to Table in RelationDB

Document.Id is the unique Id to identify each documentThe content of a Document20

Index a DocumentUnber Browser tab, we can search document21

Get a DocumentType is similar to Table in RelationDB

Document.Id is the unique Id to identify each documentThe content of a Document return by Elasticsearch

22

Update a DocumentType is similar to Table in RelationDB

Document.Id is the unique Id to identify each documentThe version no. of document is incremental if it got updated!!23Searching Document

Control Searching Scope base on URLElasticsearch has very rich Search/Query DSL for document searching. Check below URL for details:http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html24

Searching ResultThe documents which hits the searching query will be located under hits/hitsHow long it takes to calculate the result (in milli-seconds)25

Delete a DocumentType is similar to Table in RelationDB

Document.Id is the unique Id to identify each document26Elasticsearch in action using Stackoverflow Datadump

27

Environment Setup Elasticsearch C# ClientUse NuGet to search NEST (C# elasticsearch client)Click Install

28NEST (Elasticsearch C# client)NEST provides very friendly C# interfaces to interact with Elasticseach, also there are many sample codes in its web sitehttp://nest.azurewebsites.net/

29Stackoverflow DatadumpStackoverflow periodically dump their data for public study usehttps://archive.org/details/stackexchangeFor demonstration purpose, we pick a smaller datasetapple.stackexchange.com.7z (73.7MB)Badges.xmlCommenets.xmlPostHistory.xmlPostLinks.xmlPosts.xml (116,071 records)Tags.xmlUsers.xmlVotes.xml

30Session_06_DataloadToElasticsearchA new C# Console program project (Session_06_DataloadToElasticsearch) is created to parsing data dump xml file and import into ElasticsarchOpen Program.cs file and modify below:

Change the Uri of your elasticsearch cluster IP & portChange the xmlDataFile path to the location of data dump fileThis is the Index need to be created before running this program31Execute Session_06_DataloadToElasticsearchProgram.cs

123It spends 67 seconds to index 116071 records32Exploer Data via Brower

33The _search endpoingTo search with ElasticSearch we use the _search endpointWe make http requests to an URL following this pattern: (index & type are both optional)//_searchFor example:Search across all indexes and all typeshttp://localhost:9200/_searchSearch across all types in the stackoverflow indexhttp://localhost:9200/stackoverflow/_searchSearch explicitly for documents of type post within the stackoverflow indexhttp://localhost:9200/stackoverflow/post/_search

34Search request body and ElasticSearch's query DSL elasticsearch provides a full Query DSL based on JSON to define querieshttp://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.htmlThink of it like ElasticSearch's equivalent of SQL for a relational databaseQuery DSL contains:QueryFilter*** Filter are very handy since they perform an order of magnitude better than plain query since no scoring is performed and they are automatically cached

35Basic free text search The query DSL features a long list of different types of queries that we can useFor "ordinary" free text search we'll most likely want to use one called "query string query".

36Elasticsearch Query DSL - FilterAs a general rule, filters should be used instead of queryies:for binary yes/no searchesfor queries on exact valuesFilters can be caching. Caching the result of a filter does not require a lot of memory, and will cause other queries executing against the same filter (same parameters) to be blazingly fast

37Filter without Query

38Developing Angularjs with Elasticsearch

39Elasticsearch Javascript Client LibraryGo to Elasticsearch web site & get javascript clientelasticsearch-js-2.4.3.ziphttp://www.elasticsearch.org/guide/en/elasticsearch/client/javascript-api/current/browser-builds.htmlUnzip and import to PracticalCoding.Web Project under /Scripts/elasticsearch folder

HightchartHighsoft AS. 2014, 14, 8.

Hightcharts, javascriptsinclude40Setup Fulltext-Search SPA SkeltonCreate 11_AngularWithElasticsearch folder under MyAppCreate files and subfolder according to the diagram

41index.html

angular-sanitize.js is used to handle html content showing on UIui-bootstrap-tpls-.js is used to show & control paginationelasticsearch.angular.js is used to connect Elasticsearch & submit query command

42

factories.jsdefine our elasticsearch cluster host IPs

use esFactory to connect elasticsearch clusters43

app.jsdefine our routing, UI template & controller

44fulltext-search.html (1)

45fulltext-search.html (2)

46fulltext-search.html (2)

47controllers.js

48Fulltext-Search DemoSelect 11_AngularWithElasticsearch/index.html and Hit F5 to runDemo Page

49Highchart , AngularJS ,Web API2 , SignalR2, Redis + Elasticsearch IntegrationIntegration with Entity FrameworkCopy 10_IntegrationWithRedis to 12_IntegrationWithElasticsearch 51Create New ElasticDashboardRepo.cs

Control Unique ID generation and management via Redis52

Switch RedisDashboardRepo to ElasticsearchDashboardRepoCopy RedisDashboardController.cs to ElasticsearchDashboardController.csSwitch our Repository from Redis to Elasticsearch53

Modify Our Angular ChartDataFactorySwitch angular $http communication end point to our new WebAPI urlBeforeAfter

54Integration with ElasticsearchSelect 12_Int