Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 5. Debugging

5.1. Locate the logs
Copy link

5.1.1. Access OpenDaylight logs
Copy link

OpenDaylight stores logs in containers in the /var/log/containers/opendaylight directory. The most recent log is named karaf.log. OpenDaylight appends incremental numbering to previous logs, for example, karaf.log.1, karaf.log.2.

5.1.2. Access OpenStack Networking logs
Copy link

When OpenStack networking commands fail, first examine the neutron logs. You can find the neutron logs in the server.log file on each neutron node in the /var/log/containers/neutron directory.

The server.log file also includes errors about the communication with OpenDaylight. If the neutron error originates from interacting with OpenDaylight, you must also examine the OpenDaylight logs to locate the cause of the failure.

5.2. Debug networking errors
Copy link

If you experience a network error such as loss of instance connectivity, but no errors are reported when issuing OpenStack commands or in the neutron logs, then it might be useful to inspect the OVS nodes for network traffic and OpenFlow flows:

  1. Log in as superuser to the node where the network error occurs.

  2. Display the information about the br-int switch. 

    # ovs-ofctl -O openflow13 show br-int
    Copy to Clipboard Toggle word wrap

  3. Examine the output. It must be similar to this example: 

    OFPT_FEATURES_REPLY (OF1.3) (xid=0x2): dpid:0000e4c153bdb306
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS GROUP_STATS QUEUE_STATS
OFPST_PORT_DESC reply (OF1.3) (xid=0x3):
 1(br-ex-patch): addr:ae:38:01:09:66:5b
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 2(tap1f0f610c-8e): addr:00:00:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 3(tun1147c81b59c): addr:66:e3:d2:b3:b8:e3
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:e4:c1:53:bd:b3:06
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (OF1.3) (xid=0x5): frags=normal miss_send_len=0
    Copy to Clipboard Toggle word wrap

  4. List the statistics for the br-int switch. 

    # ovs-ofctl -O openflow13 dump-ports br-int
    Copy to Clipboard Toggle word wrap

  5. Examine the output. It must be similar to this example: 

    OFPST_PORT reply (OF1.3) (xid=0x2): 4 ports
  port LOCAL: rx pkts=101215, bytes=6680190, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=0, bytes=0, drop=0, errs=0, coll=0
           duration=90117.708s
  port  1: rx pkts=126887, bytes=8970074, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=18764, bytes=2067792, drop=0, errs=0, coll=0
           duration=90117.418s
  port  2: rx pkts=1171, bytes=70800, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=473, bytes=44448, drop=0, errs=0, coll=0
           duration=88644.819s
  port  3: rx pkts=120197, bytes=8776126, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=119408, bytes=8727254, drop=0, errs=0, coll=0
           duration=88632.426s
    Copy to Clipboard Toggle word wrap

More information

  • In Step 3, note that there are three ports on this OVS node. The first port is a patch port going to the bridge br-ex, which is an External network connectivity port in this scenario. The second port is a tap port, which connects to a DHCP agent instance. We know this because the host is a controller, otherwise on a Compute role it would be an instance. The third port is a VXLAN tunnel port created for the tenant traffic.
  • When you understand what each port is, you can examine the port statistics to verify that the port is receiving/sending traffic (see Step 4).
  • From the output in Step 5, note that each port is receiving (rx pkts) and sending packets (tx pkts).

5.2.1. Advanced debugging using OpenFlow flows
Copy link

For advanced users who are familiar with OpenFlow, you can examine the flows on the switch to detect where the traffic drops.

  1. To list the flows, and to see how many packets have hit them, enter the following command: 

    # ovs-ofctl -O openflow13 dump-flows br-int
    Copy to Clipboard Toggle word wrap

  2. Examine the output of the command to get the necessary information: 

    OFPST_FLOW reply (OF1.3) (xid=0x2):
 cookie=0x8000000, duration=90071.665s, table=0, n_packets=126816, n_bytes=8964820, priority=1,in_port=1 actions=write_metadata:0x20000000001/0xffffff0000000001,goto_table:17
 cookie=0x8000000, duration=88967.292s, table=0, n_packets=473, n_bytes=44448, priority=4,in_port=2 actions=write_metadata:0x40000000000/0xffffff0000000001,goto_table:17
 cookie=0x8000001, duration=88954.901s, table=0, n_packets=120636, n_bytes=8807869, priority=5,in_port=3 actions=write_metadata:0x70000000001/0x1fffff0000000001,goto_table:36
 cookie=0x8000001, duration=90069.534s, table=17, n_packets=126814, n_bytes=8964712, priority=5,metadata=0x20000000000/0xffffff0000000000 actions=write_metadata:0xc0000200000222e0/0xfffffffffffff
ffe,goto_table:19
 cookie=0x8040000, duration=90069.533s, table=17, n_packets=126813, n_bytes=8964658, priority=6,metadata=0xc000020000000000/0xffffff0000000000 actions=write_metadata:0xe00002138a000000/0xffffffff
fffffffe,goto_table:48
 cookie=0x8040000, duration=88932.689s, table=17, n_packets=396, n_bytes=36425, priority=6,metadata=0xc000040000000000/0xffffff0000000000 actions=write_metadata:0xe00004138b000000/0xfffffffffffff
ffe,goto_table:48
    Copy to Clipboard Toggle word wrap
Note

This output has been edited for length.

5.2.2. Packet traverse in OpenFlow
Copy link

The important things to understand are that the network functions performed on a packet are broken into different OpenFlow tables, and packets traverse those tables in order, starting from zero. An incoming packet lands in table 0, and then progresses through the OpenFlow Pipeline until it is sent out of a port, to the OpenDaylight Controller, or dropped. A packet may skip one or more tables depending on which network function it may need to go to. The full diagram of tables and how they correspond to network functions is shown below:

Figure 5.1. OpenDaylight NetVirt OpenFlow Pipeline

OpenDaylight NetVirt OpenFlow Pipeline
View larger image
OpenDaylight NetVirt OpenFlow Pipeline
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2026 Red HatBack to top