Building specialized industry applications using Solr, and migration from FAST ESP

©2011 Uchida Spectrum, Inc. All rights reserved.

Building specialized applica/ons using Solr; Migra/on from FAST ESP

Rahul Agarwalla Head of Interna/onal Business

Uchida Spectrum Inc.

©2011 Uchida Spectrum, Inc. All rights reserved. Page-2 Page-2

1995 ~ • 　So)ware License Sales • 　License Management Repor:ng • 　License Procurement System • 　License Adjustment Consul:ng

1997 ~ • Network System Consul:ng Services

― Ac:ve Directory Network ― Exchange Messaging Network

• License Management System Consul:ng ― So)ware Management Server

• Portal System Consul:ng ― Share Point Portal Server ― Websphere Portal Server

2002 ~ • Enterprise Intelligence Applica:on

― SMART InSight G2 Enterprise ― SMART InSight G2 Professional

• Search PlaRorm Consul:ng & Support ― FAST ESP ― Lucene/Solr ― Lucid Works Enterprise

Enterprise Search Business

SoDware License Business

Network Technology Services

Uchida Spectrum Overview


Some of Uchida Spectrum’s customers


SMART/InSight History

2003: FAST Alliance

2004: PlaRorm for custom solu:ons

2005: SMART InSight 1.1

Customers in Japan, China & India: • 2 of top 3 Japanese car manufacturers • Top consumer electronics company • Large financial ins8tu8ons • China’s biggest eCommerce firm


What is today’s buzz word?

Smart Phone

• Extreme scalability • Flexibility & Extensibility • Feature rich search


What I learnt from the Japan catastrophe


The power of community

Japanese People

[Open community]

• Quick response

• Disclose / Share • Prac:cal Impact

Japanese Government

[Closed/big brother]

• Slow, behind the curve

• Legacy/CYA • Confusion

Power shift Driver of innova/on

©2011 Uchida Spectrum, Inc. All rights reserved. Page-8

Lessons from FAST ESP Migra/on: advantage LWE/Solr

• Key Issues: 1. Smaller record and index size enable faster index maintenance

2. # of records per node: rule of thumb 10m vs. 2m 3. Licensing & Maintenance Cost: less than ½

• Scalability: 5x

• Cost Performance: 10x • High Flexibility

• Lower Opera/ons Cost • Faster Innova/on


Enterprise Search expecta/ons

• Big data scale • Security is important • Disparate data: geography, systems, languages, format, structures

• KM is good to have, databases are cri:cal

• Support different users & usage: department, role, tasks

• High recall


Lessons from FAST ESP Migra/on: Filling the gaps

• Security • ACL security: complex requirements

• File System: file & folder level control • CRM/ERP… : Keeping ACLs up-‐to-‐date

• Content aggrega/on • Connectors • Normaliza:on

• Open source op:ons for ESP pipeline • Openpipeline

• Pypes


Building specialized applica/ons: Content fusion

• Content fusion from disparate data:

• Single index ≠ integra:on • Modeling of content rela:onships is essen:al


Virtual integra/on based on search

Search Index

Content store Big table, flat index

Applica/on layer Content sets and inter-‐rela/onships

Search Index Search Index


Virtual integra/on based on search…2

Search Index LWE Solr

LWE Adapter SolrAdapter

Result Pipeline

Que

ry Pipeline

Transform

Boos:ng

. . . . . .

App

end Pipe

line

Other ……

Tagging Pipe

line

Content Security

……

Search Service • Data transforma:on: - key:key, key:value, field names

• Query & Result transforma:on • Boos:ng / Relevancy algorithm • Security • Mul:-‐Language support • Federa:on & mashups


Building specialized applica/ons: Personaliza/on

• Applica/on flow depends on the task • Data Personaliza/on increases produc/vity

• SMART InSight approach: Task based UI • Schema independent widgets for analy:cs &

visualiza:on

• Portalized

• Personalized: widgets, func:ons, content, fields


Knowledge Center: made possible by Solr

Scalability and low TCO gives us ability to build new features • Knowledge Centre has logs of all user ac:vity in SMART InSight

• This would be too costly with a commercial Search Engine and would not be feasible in a database

Using this rich data we can: • Profile users, groups and networks • Personalize Recommenda:ons

• Create social ranking algorithms

• Usage analy:cs


Widgets Personaliza/on Page Ajax Portal

Convergent Knowledge Framework

Virtual Integra/on Framework

Management & Security

Data Chain

SA

SA

SA

SA

Recommend

Profiling

Metadata Analysis

Knowledge Centre

Knowledge Log

Contents Set

Design

Engineering

Claims

Internal

Internet

NHTSA

Specs

PLM

Repair

CAD

EDR

Dealers

Claims

Overview of SMART InSight for Automo/ve

Task based UIs

Content Model

Early Defect Warning

Parts Catalog

Claim Analysis

Benchmarking


: Interac/ve Click Log Analysis System

Hadoop

Access Log

Solr

• > $50 Billion sales / year • > 800 Million Items • > 370 Million Users • Billions of clicks per day

xxxxxxxx

Xxxx Xxxx Xxxx Xxxx xxxx

Solr, Hadoop + SMART/InSight G2


: Global Research Community

Solr

• > 270 content sources: Socie/es, Associa/ons, Publishers & Open • IEEE, ACM… • Elsevier, Wiley, Springer…

Broadcast Search

Real /me indexing Dynamic Result Merging

• Top Academic Ins/tutes: • Faculty, Research Fellows & Post graduate students

• Govt. Departments & Corporate R&D • Scien:sts and researchers

Research Discovery & Collec/ve Intelligence (Knowledge Centre)


Demonstra/on


Contact Details

Rahul Agarwalla Head – Interna/onal Business

[email protected]

www.spectrum.co.jp