Kubernetes Scaling SIG (K8Scale)

  • View
    1.408

  • Download
    0

Embed Size (px)

Transcript

  • Kubernetes Scaling SIG (K8Scale)

    Bob Wise Samsung SDS Research America

    Copyright 2015 Samsung SDS Co., Ltd. All rights reserved

    2

  • Copyright 2015 Samsung SDS Co., Ltd. All rights reserved

    This presentation is intended to provide information concerning Samsungs efforts around containers and container orchestration. We do our best to make sure that information presented is accurate and fully up-to-date. However, the presentation may be subject to technical inaccuracies, information that is not up-to-date or typographical errors. As a consequence, Samsung does not in any way guarantee the accuracy or completeness of information provided on this presentation. Samsung reserves the right to make improvements, corrections and/or changes to this presentation at any time.

    The information in this presentation or accompanying oral statements may include forward-looking statements. These forward-looking statements include all matters that are not historical facts, statements regarding the Samsung Data System' intentions, beliefs or current expectations concerning, among other things, market prospects, growth, strategies, and the industry in which Samsung operates. By their nature, forward-looking statements involve risks and uncertainties, because they relate to events and depend on circumstances that may or may not occur in the future. Samsung cautions you that forward looking statements are not guarantees of future performance and that the actual developments of Samsung, the market, or industry in which Samsung operates may differ materially from those made or suggested by the forward-looking statements contained in this presentation or in the accompanying oral statements. In addition, even if the information contained herein or the oral statements are shown to be accurate, those developments may not be indicative developments in future periods.

    Logos remain the property of their respective owners. So there.

    3

  • Presenta@on Goals

    Con@nue to make a posi@ve contribu@on. K8scale is a way for Samsung to contribute to a project that is important to us.

    Transparency enhancement. Encourage involvement in K8scale if you are interested in this area.

    Share any learnings with the rest of the community to help other SIGs.

    Give a perspec@ve on kubernetes scalability. Copyright 2015 Samsung SDS Co., Ltd. All rights reserved

    4

  • A Bit of History July Kubernetes 1.0 Launch Post launch interest in evolving the community by

    breaking into SIGs Auto-scaling Federa@on (recently ac@ve) Network Scalability Storage Configura@on Tes@ng UI (just started mee@ng?) Node Big Data (just kicked off)

    Aug 5 first K8Scale mee@ng

    Copyright 2015 Samsung SDS Co., Ltd. All rights reserved 5

  • K8Scale Goals 1000 nodes 100 pods/sec scheduling rate

    Dense microservices, IOT 99% of API calls to apiserver return in less than 1 second 99%-ile of end-to-end pod startup @me with prepulled

    images less than 5 seconds (up to 30-pods per node) Configura@ons that are fieldable

    HA HTTPS/tokens

    AWS, GCE, and bare metal The more the merrier, please join in!

    Use and improve standard conformance tests Data sharing back to the community

    Copyright 2015 Samsung SDS Co., Ltd. All rights reserved 6

  • K8Scale Info

    Regular Weekly Mee@ngs since Aug 5, with only one cancella@on.

    Co-chairs Bob Wise Joe Beda

    Ac@ve Slack channel Consistent ac@ve par@cipa@on by Google, Redhat, CoreOS, Samsung, and others

    Copyright 2015 Samsung SDS Co., Ltd. All rights reserved 7

  • Context Scaling Dimensions

    Copyright 2015 Samsung SDS Co., Ltd. All rights reserved 8

    Node Count

    Pods/Node (Node Size)

    Pod Rate

    Latency

  • Group Priori@es Focus has been on kubernetes ability to manage

    overall high pod crea@on/destruc@on rates Logging and metrics collec@on is cri@cal Not yet much work yet on very high pod-per-node

    numbers (over 100/node) Very large nodes Dense (very micro) micro-service deployments

    Not yet concentra@on on Docker daemon performance Possibly an issue at higher pod density? Using docker version appropriate to kubernetes release in test

    Not yet much work on pod/pod networking performance

    Copyright 2015 Samsung SDS Co., Ltd. All rights reserved 9

  • Test Info

    K8scale has standardized on the density conformance test.

    Density conformance test runs at: 3 pods/node 30 pods/node 100 pods/node

    Copyright 2015 Samsung SDS Co., Ltd. All rights reserved 10

  • Fun Facts Kubemark is a stubbed-out kubelet, very useful for performance tes@ng.

    Kubernetes CI includes e2e at 500 nodes with kubemark.

    Samsung running 1000 node tests on AWS regularly, 100 node CI

    Redhat CI runs high density on a variety of configura@ons

    Please join k8scale and let us know what _you_ are doing!

    Copyright 2015 Samsung SDS Co., Ltd. All rights reserved 11

  • Challenges Sharing dashboard data and performance runs is not

    really sorted out yet Performance issues occur in some environments but

    not others Federated CI

    There are a lot of tuning (50+?) knobs, e.g. QPS and Burst limits X kubelet, API Server, etc Resync @mers Garbage collec@on @mers (docker images, etc) Really need a beoer way to share full cluster config sepngs

    h8ps://github.com/kubernetes/kubernetes/issues/14916

    Copyright 2015 Samsung SDS Co., Ltd. All rights reserved 12

  • Hypothesis for Inves@ga@on Implement watch in apiserver Offload events from apiserver not used by controllers or the scheduler Interim: Separate etcd server

    Auto-tuning via backpressure and backoff Etcd v3 w/gRPC Use other backing stores Op@mize node status messaging Scheduler throughput op@miza@on

    Copyright 2015 Samsung SDS Co., Ltd. All rights reserved 13

  • Ongoing Ques@ons

    TLS overhead JSON overhead Performance effect of moving to the 2.2 etcd client

    API server load balancing prac@ces At what point is etcd a booleneck? At what point is the scheduler a booleneck? Whats the effect of RKT

    Copyright 2015 Samsung SDS Co., Ltd. All rights reserved 14

  • Observa@ons Having the same people in the mee@ng every week is really

    important this really makes it work: Tim St. Clair (Redhat) Wojtek Tyczynski, Quinton Hoole, Daniel Smith (Google)

    Engagement from the CI team has been extremely helpful Direct support from CoreOS on etcd has been great -

    thanks to Xiang Li and Yicheng Qin We moved to slack for the group before the main

    community really has worked well The email list exists but is effec@vely completely unused

    Great mee@ng notes (esp thanks to Joe) Substan@ve technical discussion has migrated out of the

    main community to the SIG community this is a good thing.

    Copyright 2015 Samsung SDS Co., Ltd. All rights reserved 15

  • Tales from the trenches

    We have to be data driven Intui@ons can easily be wrong Its a bit too hard to replicate test setups from one team to another

    Heres an example

    hops://github.com/kubernetes/kubernetes/issues/14216 Stair-stepping in pods going from Pending to Running

    Copyright 2015 Samsung SDS Co., Ltd. All rights reserved 16

  • Cluster Performance, Expected Result

  • Cluster Performance, Stairstepping

  • Stairstepping Scale Zoom

  • The guessing Scheduler has a bug Etcd is misbehaving Go garbage collec@on is firing QPS rate limits are causing backpressure Some performance difference between AWS and GCE instances is triggering bug (gepng desperate now J ) Only showing on AWS

    We banged our heads on this one for weeks Copyright 2015 Samsung SDS Co., Ltd. All rights reserved

    20

  • Cause

    Scheduler logging was verbose Scheduler hits a buffer dump sync pause Nothing gets scheduled un@l the log buffer gets dumped

    Samsung/AWS setup was just enough bigger to hit the threshold

    Copyright 2015 Samsung SDS Co., Ltd. All rights reserved 21

  • Cause

    Scheduler logging was verbose Scheduler hits a buffer dump sync point Nothing gets scheduled un@l the log buffer gets dumped

    Samsung/AWS setup was just enough bigger to hit the threshold

    Copyright 2015 Samsung SDS Co., Ltd. All rights reserved 22

  • Snapshot of Wojteks API Server Opt 1.0.X

    All watch requests to apiserver are in etcd Apiserver watches every object that matches E.g., if there are 1000 kublets watching for pods with host assigned for their machine 1000 watches in etcd

    1.1 Watch implemented in apiserver Apiserver has one watch open to etcd h8ps://github.com/kubernetes/kubernetes/pull/10679

    Copyright 2015 Samsung SDS Co., Ltd. All rights reserved 23

  • Bobs Final Thoughts on SIGs SIGs should be organic and formed by those who have specific goals and interests.

    Mul@ple organizers is a very good idea. Maybe even more than two.

    Too many SIGs might diffuse energy. Refactor? Would really like to see a release SIG. The SIG format works really well for deeper/longer technical discussions and planning.

    K8Scale is working really well, we are con@nuing our long term involvement here.

    We really need full chat history Nonprofit version of slack? CNCF help!

    Copyright 2015 Samsung SDS Co., Ltd. All rights reserved 24

  • Samsung Perspec@ve

    Copyright 2015 Samsung SDS Co., Ltd. All rights reserved 25