MidoNet - Troubleshooting

1MidoNet Troubleshooting, Copyright (c) 2015 Midokura

Troubleshooting

● General Approach

● Typical Issues

● MidoNet Tools

● Getting Help

General Approach

3MidoNet Troubleshooting, Copyright (c) 2015 Midokura

Network Stack● Multiple layers

Underlay Network

Overlay Network

Topology Simulation

Virtual Network Topology

Network Stack● Multiple layers

Underlay Network → Physical Environment

Overlay Network → Datapath / Tunnel

Topology Simulation → MidoNet Agent

Virtual Network Topology → Neutron, NSDB

Physical Environment

6MidoNet Troubleshooting, Copyright (c) 2015 Midokura

Physical Network● Cabling

o Cannot connect to physical nodeso ethtool → link detected?o ip link → device status?

● Routingo Cannot access VMs on

particular nodeo netstat -nro ip route Overlay Network

Topology Simulation

Virtual Network Topology

Underlay Network

Security Configuration● Firewall

o Cannot connect to serviceso iptables -L

● ACL / SELinuxo Cannot access serviceso getenforceo setenforce

Overlay Network

Topology Simulation

Virtual Network Topology

Underlay Network

Linux Kernel ● Open vSwitch module

o MidoNet Agent does not starto VMs are not reachableo modinfo openvswitcho lsmod | grep openvswitch

● Load manuallyo modprobe openvswitcho insmod /lib/modules/…

Overlay Network

Topology Simulation

Virtual Network Topology

Underlay Network

Clock Synchronization● Sync clock on all nodes

o Same time zone? JTC != UTCo ntpdo ntpdate

● Clocks out of sync?o NSDB troubleso 50% packet loss

Overlay Network

Topology Simulation

Virtual Network Topology

Underlay Network

Datapath / Tunnel

11MidoNet Troubleshooting, Copyright (c) 2015 Midokura

Datapath● VMs unreachable, physical network ok

● Check status of MidoNet datapatho mm-dpctl --show-dp midoneto mm-dpctl --dump-dp midoneto mm-ctl --list-hosts

Overlay Network

Topology Simulation

Virtual Network Topology

Underlay Network

Tunnel● Tunnel Zone

o Check tunnel-zone with midonet-clio Check GRE/VXLAN interfaces

→ mm-dpctl --show-dp midonet

● MTUo Overhead of tunnel protocols

GRE (MTU 1454) VxLAN (MTU 1450)

Overlay Network

Topology Simulation

Virtual Network Topology

Underlay Network

MidoNet Agent

14MidoNet Troubleshooting, Copyright (c) 2015 Midokura

MidoNet Agent● MidoNet configuration issues

o e.g. cannot start MidoNet Agento Errors show up in the log fileo Check NSDB connectionso ping [NSDB_IP]o telnet [NSDB_IP] 2181o telnet [NSDB_IP] 9160

● Otherwise MidoNet bugs...

Overlay Network

Topology Simulation

Virtual Network Topology

Underlay Network

Neutron / NSDB

16MidoNet Troubleshooting, Copyright (c) 2015 Midokura

Neutron / NSDB setup● Security Groups

o Proper rules added?● ICMP, SSH, HTTP

● Gateway Configurationo neutron router-gateway-seto neutron router-interface-add

Overlay Network

Virtual NW Topology Simulation

Virtual NW Topology

Underlay Network

Neutron / NSDB consistency● DO NOT delete components via MidoNet

o neutron port-list returns error if you delete a port!o Neutron server error if you delete a bridge!

● Manual rebuildo Neutron → NSDB

● Use Neutron / Horizon

Overlay Network

Virtual NW Topology Simulation

Virtual NW Topology

Underlay Network

MidoNet Tools

19MidoNet Troubleshooting, Copyright (c) 2015 Midokura

General CLI● midonet-cli

o Interactive or script-controlledo Client to the REST API

o Browse and manipulate topologyo Maintain tunnel zoneo Upgrade MidoNet

o man midonet-cli

Control● mm-ctl

o Show Agent hosts

o Show VM tap interface mappings

o Bind / unbind ports

Datapath● mm-dpctl

o Show flowso Delete flows

o grep by IP / port

Packet Tracing● mm-trace

o Filters● Protocol

● IP / MAC address

● Port number

● …

o mm-trace –help

Statistics● mm-stat

o Statistics

● Packets processed

● Packets dropped

● Packets on-hold● Flows created

● Latency● ...

Getting Help

25MidoNet Troubleshooting, Copyright (c) 2015 Midokura

Getting Help

● Chat– https://slack.midonet.org

● Mailing Lists– https://lists.midonet.org

● Troubleshooting Guide

– http://docs.midonet.org

Getting Help

● Chat– https://slack.midonet.org

● Mailing Lists– https://lists.midonet.org

● Troubleshooting Guide

– http://docs.midonet.org

Thank You!

28MidoNet Troubleshooting, Copyright (c) 2015 Midokura