Juniper SRX Old and New DHCP, with Problems

At some point I tried to configure Juniper SRX100 with DHCPv6 server, with no luck. The configurations didn’t just work. I gave up at that time, but returned to the matter some days ago. I realized something in Junos DHCP configurations: people are talking about “old” and “new” ways to configure DHCP server and client in SRX. What is that all about?

The DHCP Server

I have been configuring the DHCP server in my SRX in “system services dhcp” hierarchy, and tried to configure the DHCPv6 feature in “system services dhcp-local-server” and “access address-assignment” hierarchies, just like the guides and hints told me. It appears that “system services dhcp” is the “old” way to do things, and “system services dhcp-local-server” and “access address-assignment” is the “new” way. If you can find some specific and clear documentation that describes the situation and guidelines for the SRX family, please comment below. But this is my understanding.

In fact, if trying to configure DHCP (v4) server in both “system services dhcp” and “system services dhcp-local-server” Junos gives me errors:

admin@srx100# show system services
...
dhcp-local-server {
    ##
    ## Warning: Incompatible with the dhcp server configured under 'system services dhcp'
    ##
    group DHCP {
        interface vlan.10;
    }
}
##
## Warning: Incompatible with 'system services dhcp-local-server group'
##
dhcp {
    pool 192.168.100.0/24;
...

The new way of configuring the DHCP server consists of configuring a DHCP group in the “system services dhcp-local-server” and then configuring the pool settings in “access address-assignment”, instead of having all the configurations in the “system services dhcp” hierarchy.

Here is an example of an old-style configuration:

admin@srx100# show system services
...
dhcp {
    pool 192.168.100.0/24 {
        address-range low 192.168.100.10 high 192.168.100.49;
        router {
            192.168.100.1;
        }
    }
    static-binding 00:11:22:33:44:55 {
        fixed-address {
            192.168.100.49;
        }
    }
    propagate-settings fe-0/0/0.0;
}

Here is the corresponding new-style configuration:

admin@srx100# show system services
...
dhcp-local-server {
    group DHCP {
        interface vlan.10;
    }
}

admin@srx100# show access
address-assignment {
    pool VLAN10POOL {
        family inet {
            network 192.168.100.0/24;
            range RANGE {
                low 192.168.100.10;
                high 192.168.100.49;
            }
            dhcp-attributes {
                router {
                    192.168.100.1;
                }
                propagate-settings fe-0/0/0.0;
            }
            host MYHOST {
                hardware-address 00:11:22:33:44:55;
                ip-address 192.168.100.49;
            }
        }
    }
}

All this sounds fair. The “access address-assignment” approach is apparently a bit more modular and more flexible. It is needed for the DHCPv6 configurations anyway. Note that you cannot just move the configuration from the old place to the new place as there are syntax differences.

The old and new ways are also run by different processes in the device. The old way uses dhcp subsystem while the new way uses dhcp-service subsystem. This gives also differences to the “show” commands: In the old way you used “show system services dhcp” commands but in the new way you use “show dhcp” (or even “show dhcpv6“) commands. If you try the wrong commands you will get this kind of not-really-informative error message:

admin@srx100> show system services dhcp binding
warning: dhcp subsystem not running - not needed by configuration.

In this case the device was running the new-style configuration, thus the dhcp subsystem was not in use. Confusing? Yes, DHCP is still working fine but in the dhcp-service subsystem.

If you need to restart the DHCP process, you will need to use the “restart dhcp” or “restart dhcp-service” command accordingly.

The DHCP Client

Now, that above was only the DHCP server part. My SRX also uses the DHCP client to receive its WAN IP address and the default route. The DHCP server and client are related to each other as they are run by the same subsystem. Here is the old way to configure the Internet interface:

admin@srx100# show interfaces
fe-0/0/0 {
    description Internet;
    unit 0 {
        family inet {
            dhcp {
                update-server;
            }
        }
    }
}

Here is the new way:

admin@srx100# show interfaces
fe-0/0/0 {
    description Internet;
    unit 0 {
        family inet {
            dhcp-client {
                update-server;
            }
        }
    }
}

Not so much different. And the CLI is as helpful as usual with this:

admin@srx100# set interfaces fe-0/0/0.0 family inet ?
Possible completions:
  <[Enter]>            Execute this command
> address              Interface address/destination prefix
+ apply-groups         Groups from which to inherit configuration data
+ apply-groups-except  Don't inherit configuration data from these groups
> dhcp                 Configure DHCP Client
> dhcp-client          Dynamic Host Configuration Protocol client configuration
...

If you try to use the incorrect style it will give the errors when showing or committing the configuration, so you know to use the other style.

The Specific Problem

All this above makes sense, things tend to change. In my case, however, when I had completed the reconfigurations, I realized that my Internet connection was not working anymore. The reason was quickly spotted: the default route was missing in the SRX:

admin@srx100> show route 0.0.0.0

admin@srx100>

There was no special indication in the log:

admin@srx100> show log messages
...
Nov 30 13:56:13  srx100 init: dhcp-service (PID 56658) exited with status=1
Nov 30 13:56:13  srx100 init: dhcp-service (PID 56809) started

These messages are normal, the dhcp-service subsystem was restarted due to reconfigurations.

The problem could be reproduced 100%: whenever a configuration change was committed for the DHCP features, the DHCP client lost the default route within about one minute. The route was not removed right after applying the configuration but some moments later.

The quick fix? I just disconnected and reconnected the Internet interface (fe-0/0/0). The DHCP client then did its job and the route was returned:

admin@srx100> show route 0.0.0.0

inet.0: 14 destinations, 14 routes (14 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0          *[Access-internal/12] 00:00:12
                    > to 88.xx.xx.xx via fe-0/0/0.0

Obviously this was a software problem since nothing similar happened with the old-style configuration in these over two years I’ve used the device, with different Junos versions. The current software was 12.1X44-D40.2, the currently JTAC-recommended version.

My SRX100 is the original low-memory model SRX100B. The device has already received the end-of-life announcement, and the latest Junos 12.1X47 for SRX is not even compatible with it anymore. In 12.1X46 there is no statement about the SRX100B status so I downloaded and installed 12.1X46-D25.7 and tried with it.

The start was not good. After the reboot the device came up but the default route was missing right away. The DHCP client still had all the necessary information:

admin@srx100> show dhcp client binding detail

Client Interface: fe-0/0/0.0
     Hardware Address:             88:e0:f3:xx:xx:xx
     State:                        BOUND(LOCAL_CLIENT_STATE_BOUND)
     Lease Expires:                2014-11-30 18:37:40 EET
     Lease Expires in:             14330 seconds
     Lease Start:                  2014-11-30 14:37:40 EET
     Server Identifier:            193.229.28.26
     Client IP Address:            88.xx.xx.xx
     Update Server                 Yes

DHCP options:
    Name: dhcp-lease-time, Value: 4 hours
    Name: server-identifier, Value: 193.229.28.26
    Name: router, Value: [ 88.yy.yy.yy ]
    Name: name-server, Value: [ 212.54.0.3, 193.229.0.42 ]
    Name: subnet-mask, Value: 255.255.240.0
    Name: domain-name, Value: elisa-laajakaista.fi

I then restarted DHCP:

admin@srx100> restart dhcp-service gracefully
Dynamic Host Configuration Protocol process started, pid 1583

admin@srx100> show log messages
...
Nov 30 14:43:51  srx100 init: dhcp-service (PID 1563) exited with status=0 Normal Exit
Nov 30 14:43:51  srx100 init: dhcp-service (PID 1583) started
Nov 30 14:43:51  srx100 init: l2cpd-service (PID 1585) started
Nov 30 14:43:51  srx100 init: can not access /usr/sbin/hostname-cached: No such file or directory
Nov 30 14:43:51  srx100 init: hostname-caching-process (PID 0) started
Nov 30 14:43:51  srx100 init: security-intelligence (PID 1587) started
Nov 30 14:43:51  srx100 init: can not access /usr/sbin/ipmid: No such file or directory
Nov 30 14:43:51  srx100 init: ipmi (PID 0) started
Nov 30 14:43:52  srx100 init: security-intelligence (PID 1587) exited with status=0 Normal Exit
Nov 30 14:43:52  srx100 init: security-intelligence (PID 1589) started
Nov 30 14:43:53  srx100 init: l2cpd-service (PID 1585) exited with status=0 Normal Exit
Nov 30 14:43:53  srx100 init: l2cpd-service (PID 1590) started
Nov 30 14:43:57  srx100 init: security-intelligence (PID 1589) exited with status=0 Normal Exit
Nov 30 14:43:57  srx100 init: security-intelligence (PID 1593) started
Nov 30 14:43:59  srx100 init: l2cpd-service (PID 1590) exited with status=0 Normal Exit
Nov 30 14:43:59  srx100 init: l2cpd-service (PID 1594) started

admin@srx100> show route 0.0.0.0

inet.0: 10 destinations, 11 routes (10 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0          *[Access-internal/12] 00:00:12
                    > to 88.115.160.1 via fe-0/0/0.0

However, when committing a configuration change the default route was not disappearing anymore:

admin@srx100> show log messages | last 10
...
Nov 30 15:10:12  srx100 init: dhcp-service (PID 1643) exited with status=1
Nov 30 15:10:12  srx100 init: dhcp-service (PID 1671) started

admin@srx100> show route 0.0.0.0

inet.0: 12 destinations, 12 routes (12 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0          *[Access-internal/12] 00:00:10
                    > to 88.115.160.1 via fe-0/0/0.0

The DHCP process was restarted and the default route was added again automatically, just like it should.

I even rebooted the device once more to see how it goes, and it went well: the default route was installed automatically after the second reboot. For me it looks like a sporadic issue right now with 12.1X46-D25.7, when in 12.1X44-D40.2 it was 100% repeatable.

Other Interesting Things

I still don’t like the l2cpd-service errors in the log:

Nov 30 15:19:49  srx100 init: l2cpd-service (PID 1515) exited with status=0 Normal Exit
Nov 30 15:19:49  srx100 init: l2cpd-service is thrashing, not restarted

These are happening every now and then, even before the software upgrade, and they happen around the same time there are some changes. If you know what they are, let me know.

After the reboot the SRX WAN IP was found in the routing table like this:

88.xx.xx.xx/32     *[Local/0] 00:06:22
                      Local via fe-0/0/0.0
                    [Access-internal/12] 00:06:20
                    > to 88.xx.xx.xx via fe-0/0/0.0

However, now that I check the routing table there is no Access-internal route anymore:

88.xx.xx.xx/32     *[Local/0] 00:37:33
                      Local via fe-0/0/0.0

Why is that? And why was that Access-internal route present in the routing table (even though not used) anyway?

And one more thing: After the Junos upgrade my Cacti monitoring says that in the SRX the /cf/var is 100% full even though there is 145 MB available, meaning that it is only 54% full. Maybe there was some MIB changed related to the storage status.

The DHCPv6 Part

Oh, the DHCPv6. It does not work at the moment. I was able to get forward with it yesterday, but then it stopped working at some point. Beats me right now. I will have another post about it later.