Occasionally, when working with an OpenStack installation that uses legacy Nova networking with FlatDHCPManager or VlanManager, you may encounter an issue where an instance does not correctly take the private IP it was assigned. When this happens, obviously you won’t be able to ping the system on the network, but you will also likely see cloud-init hang because it cannot contact the metadata server. This issue is often due to a condition where nova-network’s dnsmasq stops updating its internal list of virtual MAC address to IP address mappings.
Verify nova-network receives the IP
When an OpenStack instance is created, nova-network receives a message that tells it to update dnsmasq’s configuration with the new mapping. This mapping is used to assign the IP address to the instance via DHCP when it boots. nova-network handles this message by updating the contents of /var/lib/nova/networks/nova-<bridgename>.conf. If you open that file in your favorite text editor, you should see contents like this:
|
1 2 3 4 |
fa:16:3e:06:6b:a1,instance1.novalocal,10.130.66.145 fa:16:3e:25:81:e2,instance2.novalocal,10.130.66.194 fa:16:3e:a5:c7:37,instance3.novalocal,10.130.66.196 fa:16:3e:08:f1:92,instance4.novalocal,10.130.64.4 |
If you see your IP address listed, continue.
Restart dnsmasq
dnsmasq is managed and started by nova-network. However, stopping nova-network typically doesn’t stop the dnsmasq service. First, stop the nova-network service:
|
1 |
$ stop nova-network |
At this point, you should still see dnsmasq processes in the process table. Kill them.
|
1 2 3 4 5 |
$ ps aux | grep dnsmasq | grep -v grep nobody 20296 0.0 0.0 28200 836 ? S 18:35 0:00 /usr/sbin/dnsmasq --strict-order --bind-interfaces --conf-file= --pid-file=/var/lib/nova/networks/nova-br100.pid --dhcp-optsfile=/var/lib/nova/networks/nova-br100.opts --listen-address=10.3.0.45 --except-interface=lo --dhcp-range=set:public,10.3.0.2,static,255.255.128.0,86400s --dhcp-lease-max=32768 --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --no-hosts --leasefile-ro --domain=novalocal --addn-hosts=/var/lib/nova/networks/nova-br100.hosts root 20297 0.0 0.0 28200 244 ? S 18:35 0:00 /usr/sbin/dnsmasq --strict-order --bind-interfaces --conf-file= --pid-file=/var/lib/nova/networks/nova-br100.pid --dhcp-optsfile=/var/lib/nova/networks/nova-br100.opts --listen-address=10.3.0.45 --except-interface=lo --dhcp-range=set:public,10.3.0.2,static,255.255.128.0,86400s --dhcp-lease-max=32768 --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --no-hosts --leasefile-ro --domain=novalocal --addn-hosts=/var/lib/nova/networks/nova-br100.hosts $ ps aux | grep dnsmasq | grep -v grep $ |
Then, start nova-network, and after a few seconds you should see the dnsmasq instances again:
|
1 2 3 4 |
$ start nova-network && sleep 5 $ ps aux | grep dnsmasq nobody 23146 0.0 0.0 28200 836 ? S 18:48 0:00 /usr/sbin/dnsmasq --strict-order --bind-interfaces --conf-file= --pid-file=/var/lib/nova/networks/nova-br100.pid --dhcp-optsfile=/var/lib/nova/networks/nova-br100.opts --listen-address=10.3.0.45 --except-interface=lo --dhcp-range=set:public,10.3.0.2,static,255.255.128.0,86400s --dhcp-lease-max=32768 --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --no-hosts --leasefile-ro --domain=novalocal --addn-hosts=/var/lib/nova/networks/nova-br100.hosts root 23147 0.0 0.0 28200 244 ? S 18:48 0:00 /usr/sbin/dnsmasq --strict-order --bind-interfaces --conf-file= --pid-file=/var/lib/nova/networks/nova-br100.pid --dhcp-optsfile=/var/lib/nova/networks/nova-br100.opts --listen-address=10.3.0.45 --except-interface=lo --dhcp-range=set:public,10.3.0.2,static,255.255.128.0,86400s --dhcp-lease-max=32768 --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --no-hosts --leasefile-ro --domain=novalocal --addn-hosts=/var/lib/nova/networks/nova-br100.hosts |
Then, reboot your instance and you should see it grab its IP address from DHCP.
Leave a Reply