If you are using the Foreman within a traditional data center, with libvirt hypervisors and virtual machines managed by Foreman, you should pay attention at the 1.11.2 Foreman release.
We have updated our lab to that release last week, and we have been debugging strange behaviours regarding the provisioning of the virtual machines, and in particular the DHCP provisioning. It turned out that we were affected by two different bugs.
The problem
We noticed the problem when we tried to create a new host in the Foreman. It does not happen daily, as we are mainly rebuilding existing hosts again and again.
The hosts boots correctly but gets no DHCP lease. It then does not get its PXE image and is not provisioned. A check in the DHCP leases file indicates that indeed, we do not have any entry for the newly created host.
DHCP feature of the Smart Proxy
We have several smart proxies, with different features enabled. However, in the long list of features we have enabled (TFTP, DNS, templates…), DHCP is simply missing.
The DHCP feature, while enabled in the smart proxies configuration, is not detected properly by the Foreman.
In the logs, the following entry is found:
ERROR -- : Couldn't enable plugin dhcp: Parameter 'subnets' is expected to have
a non-empty value
This is enough to do some research and find #15240: Couldn’t enable plugin dhcp: Parameter ‘subnets’ is expected to have a non-empty value, which points out to a workaround (explicitely define the subnet parameter) and a patch. It does only affect EL6, which is consistent with our lab.
Intermezzo
Patch applied, feature visible in the Foreman. It is now time to “rebuild config”. Nothing happens, DHCP entry is not yet there.
At that moment, we start restarting every services, putting the proxies in DEBUG log level, without any luck.
A look a the commits in the 1.11-stable git branch provides us the final answer.
DHCP reservation not added when boot mode is static
The commit list sends us to #14905 DHCP reservation for PXE boot not added when using static boot mode. It turns out that DHCP reservation was not done when DHCP was not enabled in the host. On the paper, it is something that makes sense: why should we do something we do not plan to use? But in practice, PXE provisioning obviously requires DHCP reservation.
Once again a patch is available.
Conclusion
If you are using DHCP/PXE boot in the Foreman, please take extra care when updating to 1.11.2, you can also maybe wait for 1.11.3 if you have not updated yet.
Those two independent bugs were very unfortunate and hard to debug, since when you fix one, you don’t see the second one coming. Thanks for the Foreman developers and community to spot and fix those bugs.
If you can have a development environment with the same Foreman setup as production, please do, and test the new releases there. Mimic the same things you do on your production system. That is the best way to increase confidence in Foreman updates.
Last but not least: having two independent bugs on the same feature is exceptional (even more in a .2 release) and that should remind us how important beta-testing and field testing is important. Thanks to everyone contributing to the Foreman and let’s make awesome 1.11.3/1.12.0 releases!