1. Big DataAnalytics: Profiling the Use of Analytical Platforms in User OrganizationsBY WAYNE ECKERSON Director of Research, Business Applications and Architecture Group, TechTarget, September 2011 BIG DATA ANALYTICS: PROFILING THE USE OF ANALYTICAL PLATFORMS IN USER ORGANIZATIONS 1

2. FROM OUR SPONSORS 3. EXECUTIVE SUMMARYExecutive Summary EXECUTIVETHIS REPORT EXAMINESthe rise of big data and the use of analytics to mine SUMMARYthat data. Companies have been storing and analyzing large volumes of datasince the advent of the data warehousing movement in the early 1990s. While RESEARCH terabytes used to be synonymous with big data warehouses, now its peta-BACKGROUNDbytes, and the rate of growth in data volumes continues to escalate as organi-zations seek to store and analyze greater levels of transaction details, as wellas Web- and machine-generated data, to gain a better understanding of cus- WHY BIG DATA?tomer behavior and drivers.BIG DATAI Analytical platforms. To keep pace with the desire to store and analyze ever ANALYTICS: larger volumes of structured data, relational database vendors have delivered DERIVING VALUE FROM BIG DATAspecialized analytical platforms that pro-vide dramatically higher levels of price-per-formance compared with general-purpose ARCHITECTURE relational database management systemsCompanies have beenFOR BIG DATA ANALYTICS(RDBMSs). These analytical platformsstoring and analyzingcome in many shapes and sizes, from soft-ware-only databases and analytical appli- large volumes of data PLATFORMS FORances to analytical services that run in asince the advent ofRUNNING BIG DATA ANALYTICSthird-party hosted environment. Almostthe data warehousingthree-quarters (72%) of our survey respon-movement in thedents said they have implemented an ana-PROFILING THE USE lytical platform that fits this description.early 1990s. OF ANALYTICAL PLATFORMS In addition, new technologies haveemerged to address exploding volumes ofcomplex structured data, including Web RECOMMENDA-traffic, social media content and machine-generated data, such as sensor and TIONSGlobal Positioning System (GPS) data. New nonrelational database vendorscombine text indexing and natural language processing techniques with tradi-tional database technology to optimize ad hoc queries against semi-struc-tured data. And many Internet and media companies use new open sourceBIG DATA ANALYTICS: PROFILING THE USE OF ANALYTICAL PLATFORMS IN USER ORGANIZATIONS 3 4. EXECUTIVE SUMMARYframeworks such as Hadoop and MapReduce to store and process large vol-umes of structured and unstructured data in batch jobs that run on clusters ofcommodity servers.I Business users. In the midst of these platform innovations, business usersawait tools geared to their information requirements. Casual usersexecu- EXECUTIVEtives, managers, front-line workersprimarily use reports and dashboards SUMMARYthat deliver answers to predefined ques-tions. Power usersbusiness analysts, RESEARCH analytical modelers and data scientistsBACKGROUNDperform ad hoc queries against a variety Most business intelligenceof sources. Most business intelligence (BI) environments have(BI) environments have done a poor job WHY BIG DATA?meeting these diverse needs within a done a poor job meetingsingle, unified architecture. But this isthese diverse needsBIG DATAchanging.within a single, unified ANALYTICS: DERIVING VALUEarchitecture. But this is FROM BIG DATAI Unified architecture. This report por-trays a unified reporting and analysis changing.environment that finally turns power ARCHITECTURE users into first-class corporate citizensFOR BIG DATA ANALYTICSand makes unstructured data a legiti-mate target for ad hoc and batch queries. The new architecture leverages newanalytical technology to stage, store and process large volumes of structured PLATFORMS FORand unstructured data, turbo-charge sluggish data warehouses and offloadRUNNING BIG DATA ANALYTICScomplex analytical queries to dedicated data marts. Besides supporting stan-dard reports and dashboards, it creates a series of analytical sandboxes thatenable power users to mix personal and corporate data and run complex ana-PROFILING THE USE lytical queries that fuel the modern-day corporation. I OF ANALYTICAL PLATFORMS RECOMMENDA- TIONSBIG DATA ANALYTICS: PROFILING THE USE OF ANALYTICAL PLATFORMS IN USER ORGANIZATIONS 4 5. RESEARCH BACKGROUNDResearch Background EXECUTIVETHE PURPOSE OF this report is to profile the use of analytical platforms in user SUMMARYorganizations. It is based on a survey of 302 BI professionals as well as inter-views with BI practitioners at user organizations and BI experts at consultan- RESEARCH cies and software companies.BACKGROUNDI Survey. The survey consists of 25 pages of questions (approximately 50questions) with four branches, one for each analytical platform deployment WHY BIG DATA?option: analytical database (software-only),analytical appliance (hardware-softwareBIG DATAcombo), analytical service and file-based ANALYTICS: analytical system (e.g., Hadoop and[This report] is based DERIVING VALUE FROM BIG DATANoSQL). Respondents who didnt select anoption were passed to a fifth branch where on a survey of 302they were asked why they hadnt purchasedBI professionals as ARCHITECTURE an analytical platform and whether theywell as interviews withFOR BIG DATA ANALYTICSplanned to do so.The survey ran from June 22 to August 2, BI practitioners and2011, and was publicized through several BI experts. PLATFORMS FORchannels. The BI Leadership Forum andRUNNING BIG DATA ANALYTICSBeyeNetwork sent several email broadcaststo their lists. I tweeted about the survey andasked followers to retweet the announcement. Several sponsors, includingPROFILING THE USE Teradata, Infobright, and ParAccel, notified their customers about the survey OF ANALYTICAL PLATFORMSthrough email broadcasts and newsletters.I Respondent profile. Survey respondents are generally IT managers based in RECOMMENDA-North America who work at large companies in a variety of industries (see TIONSFigures 1-4, page 6). I BIG DATA ANALYTICS: PROFILING THE USE OF ANALYTICAL PLATFORMS IN USER ORGANIZATIONS 5 6. RESEARCH BACKGROUNDFigure 1: Which best describes your position in BI?VP/Director ArchitectManager Consultant EXECUTIVE SUMMARY AnalystAdministrator Developer RESEARCHBACKGROUND Other0510 1520 25 30 WHY BIG DATA?Figure 2: Where are you located?Figure 3: What size is yourBIG DATAorganization by revenues? ANALYTICS: DERIVING VALUE FROM BIG DATANorth America. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .66.7% Large ($1B + revenues) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52.4%Europe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16.5% Medium ($50M to $1B). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24.8%Other. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16.9% Small (