Embed Size (px)
Citation preview
Prometheus 알아보기
Agenda
1
2 OKE에서Prometheus 와 Grafana로모니터링
© 2019 Oracle and/or its affiliates. All rights reserved.2
Prometheus 알아보기
Confidential – © 2020 Oracle Internal/Restricted/Highly Restricted3
Prometheus란
Prometheus는오픈소스모니터링
Confidential – © 2020 Oracle Internal/Restricted/Highly Restricted4
Metric기반지표를수집하여저장,검색
쉬운운영, 강력한쿼리기능, Grafana를통한시각화, 많은 Integration
CNCF Graduate, 폭넓은 CNCF 생태계
모니터링의범주
Confidential – © 2020 Oracle Internal/Restricted/Highly Restricted5
Profiling :제한된기간의일부리소스분석 (예: tcpdump)
Tracing : 특정이벤트를추적, HTTP요청->Database -> Cache Server (타이밍)분산트레이싱 : Zipkin, Jaeger
Logging : Transaction, Request, Application
Metric : 다양한유형의이벤트를시간별로집계
예) HTTP요청횟수,처리시간
Prometheus 아키텍처
Confidential – © 2020 Oracle Internal/Restricted/Highly Restricted6
Metric 수집
Confidential – © 2020 Oracle Internal/Restricted/Highly Restricted7
클라이언트라이브러리
공식 : Go, Java, Python, Ruby
UnofficialBash, C, C++, Common Lisp, Dart, Elixir, Erlang,
Haskell, .NET / C#, Node.js, Perl, PHP, R, Rust
Application
Exporter
3r’d Party SW
Prometheus
데이터수집
서비스검색
클라이언트라이브러리
저장장치
Direct Instrument
pull
local on-disk time series database
Prometheus 실행
Confidential – © 2020 Oracle Internal/Restricted/Highly Restricted8
https://prometheus.io/download/ 다운로드
tar -xzf prometheus-*.linux-amd64.tar.gz
cd prometheus-*.linux-amd64
./prometheus
설정파일 : prometheus.yml
global:scrape_interval: 15sscrape_timeout: 10s
scrape_configs:- job_name: 'prometheus'metrics_path: /metricsscheme: httpstatic_configs:- targets: ['localhost:9090']
http://localhost:9090/metrics
Prometheus GUI
Confidential – © 2020 Oracle Internal/Restricted/Highly Restricted9
Prometheus 자체프로세스에대해서 metric을계속수집중
Value는 float64
라벨(Label) : 동일한데이터에서종류를구분하기위한식별자
조회할메트릭선택
PromQL
Prometheus 데이터
Metric 이름 Label 이름=Label Value, …
PromQL
PromQL
[1m] : 현재시간기준으로이전 1분동안의데이터조회prometheus_http_requests_total[1m]s, m, h, d, w, y
정규표현식{job=~".*",method="get"}
가장많이사용하는함수 : sum, avg, ratesum(prometheus_http_requests_total)
Prometheus GUI
Confidential – © 2020 Oracle Internal/Restricted/Highly Restricted13
Status à Target 클릭
현재모니터링대상으로Prometheus 자체
Prometheus GUI – 측정 metric 확인
Confidential – © 2020 Oracle Internal/Restricted/Highly Restricted14
Prometheus자체도metrics으로계측이되고있음.
scrape_configs:- job_name: 'prometheus'metrics_path: /metricsscheme: httpstatic_configs:- targets: ['localhost:9090']
Prometheus GUI – Graph
Gauge메모리사용량과같이현재값을표시
Prometheus GUI – Graph
Counter총요청수처럼지속적으로증가하는값
다양한 Exporter제공
Databases•Aerospike exporter•ClickHouse exporter•Consul exporter (official)•Couchbase exporter•CouchDB exporter•ElasticSearch exporter•EventStore exporter•Memcached exporter (official)•MongoDB exporter•MSSQL server exporter•MySQL router exporter•MySQL server exporter (official)•OpenTSDB Exporter•Oracle DB Exporter•PgBouncer exporter•PostgreSQL exporter•Presto exporter•ProxySQL exporter•RavenDB exporter•Redis exporter•RethinkDB exporter•SQL exporter•Tarantool metric library•Twemproxy
Hardware related•apcupsd exporter•BIG-IP exporter•Collins exporter•Dell Hardware OMSA exporter•IBM Z HMC exporter•IoT Edison exporter•IPMI exporter•knxd exporter•Modbus exporter•Netgear Cable Modem Exporter•Netgear Router exporter•Node/system metrics exporter (official)•NVIDIA GPU exporter•ProSAFE exporter•Ubiquiti UniFi exporter
Storage•Ceph exporter•Ceph RADOSGW exporter•Gluster exporter•Hadoop HDFS FSImage exporter•Lustre exporter•ScaleIO exporter
Issue trackers and continuous integration•Bamboo exporter•Bitbucket exporter•Confluence exporter•Jenkins exporter•JIRA exporter
Messaging systems•Beanstalkd exporter•EMQ exporter•Gearman exporter•IBM MQ exporter•Kafka exporter•NATS exporter•NSQ exporter•Mirth Connect exporter•MQTT blackbox exporter•RabbitMQ exporter•RabbitMQ Management Plugin exporter•RocketMQ exporter•Solace exporter
HTTP•Apache exporter•HAProxy exporter (official)•Nginx metric library•Nginx VTS exporter•Passenger exporter•Squid exporter•Tinyproxy exporter•Varnish exporter•WebDriver exporter
APIs•AWS ECS exporter•AWS Health exporter•AWS SQS exporter•Azure Health exporter•BigBlueButton•Cloudflare exporter•DigitalOcean exporter•Docker Cloud exporter•Docker Hub exporter•GitHub exporter•InstaClustr exporter•Mozilla Observatory exporter•OpenWeatherMap exporter•Pagespeed exporter•Rancher exporter•Speedtest exporter•Tankerkönig API ExporterLogging•Fluentd exporter•Google's mtail log data extractor•Grok exporter
Node Exporter
https://prometheus.io/download/ 에서다운로드
tar -xzf node_exporter*.linux-amd64.tar.gz
cd node_exporter-*.linux-amd64
./node_exporter
노드의 Metric을수집
CPU,메모리,디스크공간,I/O, 네트워크대역폭
설정파일(prometheus.yml)에 아래 추가
global:scrape_interval: 15sscrape_timeout: 10s
scrape_configs:- job_name: 'prometheus'metrics_path: /metricsscheme: 'http'static_configs:- targets: ['localhost:9090']
- job_name: 'node'static_configs:- targets: ['localhost:9100']
Node ExporterNode 1개가모니터링Target으로새롭게추가됨
Node Exporter의 metric 정보
PromQL 을사용한 Graph
Node_exporter에의해추가된값
network interface별로수신된데
이터를표시하는 Counter
PromQL의 rate() 함수: 증가율
클라이언트 Library를이용한 Direct Instrument (Python)
import http.server
class MyHandler(http.server.BaseHTTPRequestHandler):def do_GET(self):
self.send_response(200)self.end_headers()self.wfile.write(b"Hello World")
if __name__ == "__main__":server = http.server.HTTPServer(('localhost', 8001), MyHandler)
server.serve_forever()
클라이언트 Library를이용한 Direct Instrument (Python)
Application
클라이언트라이브러리
Direct Instrument라이브러리설치 : pip install prometheus_client
import http.serverfrom prometheus_client import start_http_server
class MyHandler(http.server.BaseHTTPRequestHandler):def do_GET(self):
self.send_response(200)self.end_headers()self.wfile.write(b"Hello World")
if __name__ == "__main__":start_http_server(8000)server = http.server.HTTPServer(('localhost', 8001), MyHandler)server.serve_forever()
클라이언트 Library를이용한 Direct Instrument (Python)
prometheus.yml 에 job 추가
global:scrape_interval: 15sscrape_timeout: 10s
scrape_configs:- job_name: 'prometheus'metrics_path: /metricsscheme: httpstatic_configs:- targets:
- localhost:9090- job_name: 'example'static_configs:- targets: ['localhost:8000'] python_info 조회가능
클라이언트 Library를이용한 Direct Instrument(Python) – Counter 추가
import http.serverfrom prometheus_client import start_http_serverfrom prometheus_client import Counter
REQUESTS = Counter('hello_worlds_total’,'Hello Worlds requested.')
class MyHandler(http.server.BaseHTTPRequestHandler):def do_GET(self):
REQUESTS.inc()self.send_response(200)self.end_headers()self.wfile.write(b"Hello World")
if __name__ == "__main__":start_http_server(8000)server = http.server.HTTPServer(('localhost', 8001), MyHandler)server.serve_forever()
Metric 정의. Counter설명
Instrument 부분: Metric이사용(호출) 됨Inc()는카운터값 1 증가. Thread Safe 보장URL 호출시마다값이 1씩증가됨
Metric 종류
CounterGauge : 현재상태표시.예) 현재대기열수, 캐시메모리사용량, 활성 Thread수
주요메소드 : inc, dec, set
Summary : 측정된값을전부합한값.주로대기시간을측정하는데사용Histogram : 분포도값반환
클라이언트 Library를이용한 Direct Instrument(Java)
import io.prometheus.client.Counter;import io.prometheus.client.hotspot.DefaultExports;import io.prometheus.client.exporter.HTTPServer;
public class Example {private static final Counter myCounter = Counter.build()
.name("my_counter_total")
.help("An example counter.").register();
public static void main(String[] args) throws Exception {DefaultExports.initialize();HTTPServer server = new HTTPServer(8000);while (true) {
myCounter.inc();Thread.sleep(1000);
}}
}
DEMO
Node exporter + Custom exporter
OKE에서Prometheus 와 Grafana로모니터링
Confidential – © 2020 Oracle Internal/Restricted/Highly Restricted28
Service Discovery를통한 Kubernetes monitoring
Confidential – © 2020 Oracle Internal/Restricted/Highly Restricted29
Target Auto Discovery Pull Datasource
Pull
Kubernetes Nodes
Kubernetes Master API Server
Grafana Dashboard
kubernetes_sd_config
Prometheus Operator
Confidential – © 2020 Oracle Internal/Restricted/Highly Restricted30 Source: prometheus-operator
Kubernetes Operator를사용하여Prometheus 용모니터링인스턴스를자동으로생성/관리
Kubernetes 에 Prometheus Operator 설치
Prometheus Pod를생성
대상 scrape config 를자동으로생성
어떤그룹의서비스를모니터링할지
PrometheusServiceMonitorPodMonitorPrometheusRuleAlertmanager
OKE(Container Engine for Kubernetes) 에설치 – Demo 구성도
Confidential – © 2020 Oracle Internal/Restricted/Highly Restricted31
Oracle Kubernetes Engine
Monitoring namespace
Private Subnet
Public Subnet
Prometheus Service(Type=LoadBalancer)
Grafana Service(Type=LoadBalancer)
Prometheus Pod
Grafana Pod
Load Balancer
VCN
prometheus-kube-state-metrics
prometheus-node-exporter
alertmanager
OKE(Oracle Kubernetes Engine) 사용하기
Confidential – © 2020 Oracle Internal/Restricted/Highly Restricted32
1. OCI CLI 설치
2. Local에 Kube config 설정 (Cluster서버정보및인증서, region)
3. oke-admin 계정생성및 RBAC 권한부여
4. kubectl proxy 로 Local에서 Kubernetes dashboard 실행à Local에 8001
포트로 Listen하고 Remote로 forwarding
kubectl get nodesNAME STATUS ROLES AGE VERSION10.0.10.2 Ready node 127m v1.15.7
OCI CLI
./kube/config
kubectl생성
OKEOracle Cloud
Kubernetes Cluster
Helm 이란?
Confidential – © 2020 Oracle Internal/Restricted/Highly Restricted33
Helm Client
Helm ServerTiller
HelmRepository
Chart
ChartRelease
Tiller는 v3에서제거됨.Oracle OCI는 Helm v2를지원함.
• Helm Repository 관리• Chart 개발• Tiller와통신하여
Kubernetes의 Chart 관리
gRPCDocker Image
Repository
Github / ChartMuseum
OKE 에 Prometheus 설치순서
Confidential – © 2020 Oracle Internal/Restricted/Highly Restricted34
1. Helm chart repository updatehelm repo add stable https://kubernetes-charts.storage.googleapis.comhelm repo update
2. Helm으로 Prometheus + Grafana 설치helm install --name kube-prometheus --namespace monitoring stable/prometheus-operator
3. 설치된 Pod 확인kubectl --namespace monitoring get pods -l "release=kube-prometheus"
https://github.com/helm/charts/tree/master/stable/prometheus-operator
OKE 에설치 – Load Balancer에연결하기
Confidential – © 2020 Oracle Internal/Restricted/Highly Restricted35
> kubectl edit svc kube-prometheus-prometheus-prometheus -n monitoring
apiVersion: v1kind: Servicemetadata:name: kube-prometheus-prometheus-prometheusnamespace: monitoringresourceVersion: "48861"
spec:clusterIP: 10.96.160.249ports:- name: webport: 9090protocol: TCPtargetPort: 9090
selector:app: prometheusprometheus: kube-prometheus-prometheus-prometheus
sessionAffinity: Nonetype: ClusterIP
status:loadBalancer: {}
annotations:service.beta.kubernetes.io/oci-load-balancer-shape: "100Mbps"service.beta.kubernetes.io/oci-load-balancer-backend-protocol: "HTTP"
spec:type: LoadBalancer
추가
변경
> kubectl edit svc kube-prometheus-grafana -n monitoring
1. Prometheus 서비스 OCI Load Balancer 로연결하기 2. Grafana 서비스 OCI Load balancer 로연결하기
서비스확인
Confidential – © 2020 Oracle Internal/Restricted/Highly Restricted36
kubectl get svc -n monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEalertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 167mkube-prometheus-grafana LoadBalancer 10.96.120.86 129.213.68.145 80:32149/TCP 167mkube-prometheus-kube-state-metrics ClusterIP 10.96.18.114 <none> 8080/TCP 167mkube-prometheus-prometheus-alertmanager ClusterIP 10.96.93.30 <none> 9093/TCP 167mkube-prometheus-prometheus-node-exporter ClusterIP 10.96.64.239 <none> 9100/TCP 167mkube-prometheus-prometheus-operator ClusterIP 10.96.185.165 <none> 8080/TCP,443/TCP 167mkube-prometheus-prometheus-prometheus LoadBalancer 10.96.86.59 150.136.188.146 9090:31284/TCP 167mprometheus-operated ClusterIP None <none> 9090/TCP 166m
서비스확인
Confidential – © 2020 Oracle Internal/Restricted/Highly Restricted37
DEMO
OKE + Prometheus + Grafana
Thank you
Confidential – © 2019 Oracle Internal/Restricted/Highly Restricted39
