Embed Size (px)
DESCRIPTION
These are the slides for my September API Craft SF talk on how we built / design Splunk's API. You can find the video here: https://www.youtube.com/watch?v=vHXcDKK4eGY. My talk starts at about 60 mins. The other two are on Uber and Sift Science and you should watch them as well!
Citation preview
Copyright © 2014 Splunk, Inc. Splunk’s API How we built it!
Agenda
Why we built Splunk’s API
How we built Splunk’s API
What is Splunk
A product for handling REALLY large and varied sets of evented data
What is Splunk
From megabytes to hundreds of terabytes daily
What is Splunk
It is highly scalable and distributed
What is Splunk
Useful in many domains: IT/Ops/DevOps, security, healthcare, financial,IOT/Devices
7
Splunk storage Other Big Data stores
Developer Pla6orm
Data collecUon and indexing
Report and analyze
Custom dashboards
Monitor and alert
Ad hoc search
Why – Product need
Need to build a Splunk UI that surfaces all of Splunk’s capabiliUes
Why – Customer need
Need to allow customers to integrate their applicaUons and scripts with Splunk
Why – Cost
Less code to maintain, a single source of truth
Why – Reach
HTTP is ubiquitous, every plaZorm has a client
The arch
12
Server Tech Stack
13
C/C++
Python – Cherry PI Python – Django
Javascript Node
Client Tech Stack
14
Javascript Node Python Ruby PHP C#
The API
15
Log directly to Splunk via HTTP
Run historical and real-‐Ume searches
What can you do with Splunk’s API?
16
Search Manage
Add/Delete Users
ReporUng/Alerts
Manage Inputs
ConfiguraUon
Index
Login to a Splunk instance and get a session token
Auth
The API design
17
Service
Categories
Endpoints
Endpoints
The API design -‐ Categories
18
The API design – Endpoints
19
The API design – Endpoints
20
Responses -‐ Feeds and Hypermedia <entry xmlns="hjp://www.w3.org/2005/Atom" xmlns:s="hjp://dev.splunk.com/ns/rest" xmlns:opensearch="hjp://a9.com/-‐/spec/opensearch/1.1/"> <Utle>search index</Utle> <id>hjps://localhost:8089/services/search/jobs/mysearch_02151949</id> <updated>2011-‐07-‐07T20:49:58.000-‐07:00</updated> <link href="/services/search/jobs/mysearch_02151949" rel="alternate"/> <published>2011-‐07-‐07T20:49:57.000-‐07:00</published> <link href="/services/search/jobs/mysearch_02151949/search.log" rel="search.log"/> <link href="/services/search/jobs/mysearch_02151949/events" rel="events"/> <link href="/services/search/jobs/mysearch_02151949/results" rel="results"/> <link href="/services/search/jobs/mysearch_02151949/results_preview" rel="results_preview"/> <link href="/services/search/jobs/mysearch_02151949/Umeline" rel="Umeline"/> <link href="/services/search/jobs/mysearch_02151949/summary" rel="summary"/> <link href="/services/search/jobs/mysearch_02151949/control" rel="control"/> </entry>
21
Auth
22
HTTP Basic Token based LDAP/AD
Cookie based
Auth – HTTP Basic
23
curl -‐k -‐u admin:changeme hjps://localhost:8089/services/auth/login -‐d username="admin" -‐d password="changeme"
Auth – Splunk Token
24
curl -‐k -‐H "AuthorizaUon: Splunk SfH2D^zvPyLu^mO61C9kWtB7TOuQs0i9oSzh4lD7ho7Gvw26I61VYRjXkgjLQlJDJ0hER^q^A6v0BHYiKNba^CMbOmC63frGCrDqr2Zt" hjps://localhost:8089/services/search/jobs -‐d output_mode="json" -‐-‐get
Search – Oneshot – Get me results!
25
curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐d search="search sourcetype=sysmon | head 5" -‐d exec_mode="oneshot"
Search – Oneshot – Get me results in json
26
curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐d search="search sourcetype=sysmon | head 5" -‐d exec_mode="oneshot" –d output_mode="json"
Search – Oneshot – Get me json columns
27
curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐d search="search sourcetype=sysmon | head 5" -‐d exec_mode="oneshot" –d output_mode="json_cols"
Search – Oneshot – Get me json rows
28
curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐d search="search sourcetype=sysmon | head 5" -‐d exec_mode="oneshot" –d output_mode="json_cols"
Search – Blocking – Wait Ull done!
29
curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐d search="search sourcetype=sysmon | head 5” -‐d exec_mode=”blocking” output_mode="json” | python -‐mjson.tool curl -‐u admin:changeme /services/search/jobs/{sid}/results -‐d output_mode="json" –get | python -‐mjson.tool
Search – List search jobs
30
curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐d output_mode="json" -‐-‐get | python -‐mjson.tool
Search – Normal– Run in the background
31
curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐d search="search sourcetype=sysmon | head 10000" -‐d exec_mode="normal" output_mode="json"| python -‐mjson.tool curl -‐u admin:changeme /services/search/jobs/{sid}/results -‐d output_mode="json" –get | python -‐mjson.tool
Search -‐ Export
32
curl -‐k -‐u admin:changeme hjps://localhost:8089/servicesNS/admin/search/search/jobs/export -‐d search="search index%3D_internal | head 100000" -‐d output_mode="raw"
Search – Export REALTIME
33
curl -‐k -‐u admin:changeme hjps://localhost:8089/servicesNS/admin/search/search/jobs/export -‐d search="search index%3D_internal" -‐d output_mode="raw" earliest_Ume="rt-‐1m" latest_Ume="rt"
Copyright © 2014 Splunk, Inc. Splunk’s API How we built it!
dev.splunk.com splunk.com/jobs
