View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0009865Kali LinuxKali Package Bugpublic2026-09-19 00:582026-09-24 04:51
ReporterLynx Assigned To 
PrioritynormalSeverityminorReproducibilityhave not tried
Status newResolutionopen 
Summary0009865: systemd-resolved resolve hook (io.systemd.Network) times out for 0000063:0000090-400s during cloud-init init-local on AWS EC2, delaying ever
Description

Environment

  • Kali GNU/Linux Rolling (AWS Marketplace / community AMI), amd64
  • systemd 259 (259.1-1)
  • cloud-init 25.3
  • Platform: AWS EC2 (VPC subnet, IPv4-only; IMDS IPv6 endpoint not available)
  • Reproduced on multiple fresh instances with zero customization.

Affected Package

systemd (259.1-1) — specifically the systemd-resolved resolve-hook mechanism (io.systemd.Network / systemd-networkd-resolve-hook.socket) introduced in systemd 259, as integrated in the Kali cloud image. This is NOT a cloud-init
bug: cloud-init merely triggers a normal name resolution during init-local, and the ~30s-per-query stall happens inside systemd-resolved waiting on the hook.
Restricting cloud-init's metadata_urls to IPv4 does not help, which confirms the problem is in the resolver hook path, not in cloud-init.

Summary

On every reboot / stop-start, cloud-init-local.service hangs for roughly 90–400 seconds. systemd-resolved repeatedly logs:

    Query on hook '/run/systemd/resolve.hook/io.systemd.Network' failed with
    error 'io.systemd.TimedOut', ignoring.

each timing out after ~30s, which dominates the init-local stage duration.

Steps to Reproduce

  1. Launch a fresh Kali Rolling AMI on AWS EC2 in an IPv4-only VPC subnet.
  2. Reboot (or stop/start) the instance a few times.
  3. Watch the serial console / cloud-init analyze show.

Reproduction note (important)

This bug only manifests on boots where cloud-init detects the primary NIC in the init-local stage, i.e. the log shows:

    DataSourceEc2.py[DEBUG]: Looking for the primary NIC in: ['eth0']

Only such boots trigger the name resolution that hits the resolve hook and stalls. Boots that log an EMPTY NIC list:

    DataSourceEc2.py[DEBUG]: Looking for the primary NIC in: []

skip init-local's resolution path entirely and finish in ~0.1s — these do NOT reproduce the problem and must not be used to conclude the bug is absent.

Whether a given boot takes the ['eth0'] path depends on cloud-init's instance cache state; a stop/start (or certain reboots) reliably triggers the ['eth0'] slow path. When verifying a fix, confirm the boot actually took the ['eth0'] path (grep the cloud-init log) before judging the result.

Expected

cloud-init-local completes in well under a few seconds (as it does on a "fast" boot where cloud-init reports Looking for the primary NIC in: []).
Name resolution during init-local should not block on a network manager that has not started yet.

Actual

On boots where cloud-init detects the primary NIC (DataSourceEc2.py: Looking for the primary NIC in: ['eth0']), init-local performs name resolution (Resolving URL took 94s + 34s). Because /etc/nsswitch.conf uses hosts: files myhostname resolve [!UNAVAIL=return] dns, the lookup goes to systemd-resolved. On systemd 259, resolved queries the per-link io.systemd.Network resolve hook (/run/systemd/resolve.hook/io.systemd.Network, socket-activating systemd-networkd.service). During init-local, systemd-networkd has not yet started and cannot answer, so resolved times out ~30s per query, 3–4 times, producing the 90–400s init-local stall. systemd-networkd only starts after cloud-init-local finishes.

cloud-init analyze show (representative slow boot):

    found local data from DataSourceEc2Local @00.015s +131.261s
    Finished stage: (init-local) 131.932 seconds

Ruled out (not the cause)

  • cloud-init metadata_urls / IMDS IPv6 (fd00:ec2::254): restricting metadata_urls to IPv4 only does NOT remove the stall.
  • Attached EBS volumes / fstab (nofail): removing them does NOT change it.
  • Any user-installed packages/services: reproduced on a completely fresh AMI.
  • Adding After=/Wants=systemd-networkd to systemd-resolved: made it WORSE (observed up to 6m50s).

Workaround (verified)

Masking the resolve-hook socket eliminates the timeouts; resolved falls back to DHCP-provided global DNS and name resolution keeps working:

    sudo systemctl mask systemd-networkd-resolve-hook.socket

Results on a fresh instance, worst-case slow path (primary NIC in: ['eth0']), including stop/start:

    init-local: ~152s -> ~16s
    io.systemd.Network TimedOut count: 4 -> 0
    DNS still resolves (getent hosts amazon.com OK)

Recommendation

The resolve-hook query issued by systemd-resolved during early boot should not block for the full timeout when systemd-networkd is not yet running.
Options:

  • resolved should treat the hook as unavailable immediately (fail fast) when the backing service is not started, instead of waiting ~30s per query; or
  • ordering should guarantee the hook provider is answerable before resolved is asked to resolve during the init-local (pre-network) stage; or
  • ship the mask/skip of systemd-networkd-resolve-hook.socket behavior in the Kali cloud image until the upstream ordering is fixed.

This appears to be an ordering/timeout interaction introduced with the systemd 259 resolve-hook mechanism as integrated in the Kali cloud image.

Appendix.

Evidence 1 — cloud-init timing of the slow boot

# sudo cloud-init analyze show
Starting stage: init-local
|`->found local data from DataSourceEc2Local @00.014s +131.272s
Finished stage: (init-local) 131.964 seconds

# /var/log/cloud-init.log (same boot)
2026-09-19 00:30:38,677 log_util.py:     Cloud-init v. 25.3 running 'init-local' ... Up 16.56 seconds.
2026-09-19 00:30:38,711 DataSourceEc2.py: Looking for the primary NIC in: ['eth0']
2026-09-19 00:32:15,553 performance.py:   Resolving URL took 94.495 seconds
2026-09-19 00:32:49,805 performance.py:   Resolving URL took 34.252 seconds
2026-09-19 00:32:54,798 log_util.py:      Cloud-init v. 25.3 finished ... Up 152.10 seconds.

Evidence 2 — systemd-resolved resolve-hook timeouts (30s interval × 4)

# journalctl -b | grep io.systemd.Network
Sep 19 00:31:10 systemd-resolved[688]: Query on hook '/run/systemd/resolve.hook/io.systemd.Network' failed with error 'io.systemd.TimedOut', ignoring.
Sep 19 00:31:41 systemd-resolved[688]: Query on hook '/run/systemd/resolve.hook/io.systemd.Network' failed with error 'io.systemd.TimedOut', ignoring.
Sep 19 00:32:11 systemd-resolved[688]: Query on hook '/run/systemd/resolve.hook/io.systemd.Network' failed with error 'io.systemd.TimedOut', ignoring.
Sep 19 00:32:45 systemd-resolved[688]: Query on hook '/run/systemd/resolve.hook/io.systemd.Network' failed with error 'io.systemd.TimedOut', ignoring.

Evidence 3 — resolve-hook socket unit (socket activation)

# /usr/lib/systemd/system/systemd-networkd-resolve-hook.socket
[Unit]
Description=Network Management Resolve Hook Socket
ConditionCapability=CAP_NET_ADMIN
DefaultDependencies=no
Before=sockets.target shutdown.target
Conflicts=shutdown.target

[Socket]
ListenStream=/run/systemd/resolve.hook/io.systemd.Network
FileDescriptorName=resolve-hook
SocketMode=0666
Service=systemd-networkd.service      # socket-activates networkd on connect
RemoveOnStop=yes

[Install]
WantedBy=sockets.target

Versions

systemd 259 (259.1-1)
cloud-init 25.3-2
Kali GNU/Linux Rolling
nsswitch hosts: files myhostname resolve [!UNAVAIL=return] dns

Activities

Lynx

Lynx

2026-09-19 02:29

reporter   ~0022010

Follow-up comment

Additional investigation: a separate problem exists on systemd 261 / cloud-init 26.2

After the original report (systemd 259.1, resolved io.systemd.Network hook timeout), I continued testing on instances that had been updated to a newer package set. On those, the systemctl mask systemd-networkd-resolve-hook.socket workaround removes the hook timeouts but does NOT fix the boot delay — because a different root cause dominates there.

Environment (where the new problem appears)

Kali GNU/Linux Rolling (AWS EC2, IPv4-only VPC subnet)
systemd      261 (261.2-1)
cloud-init   26.2-1
dhcpcd-base  1:10.5.2-1
(The original 259.1 environment used cloud-init 25.3 / dhcpcd 10.3.0, where the hook mask alone was sufficient.)

Symptom

With the resolve-hook socket already masked (confirmed: io.systemd.Network TimedOut count = 0 for the boot), cloud-init-local.service still hangs ~300s on boots that take the primary-NIC path.

Root cause (confirmed by logs)

cloud-init 26.2 brings up its ephemeral network in the init-local stage by running dhcpcd and waiting for the process to exit with a 300s timeout (
dhcp.py
, Dhcpcd.timeout = 300):

dhcpcd --ipv4only --waitip --persistent --noarp --debug --script=/bin/true eth0

With dhcpcd 10.5.2, on a cold start the client acquires the lease and configures the IP successfully, then forks to background, but the parent invocation never returns, so cloud-init hits the full 300s timeout and then declares the ephemeral setup failed — even though the interface was actually configured.

Evidence (fresh boot, hook already masked, ['eth0'] slow path)

DataSourceEc2.py: Looking for the primary NIC in: ['eth0']
performance.py:   Running ['dhcpcd','--ipv4only','--waitip','--persistent','--noarp','--debug','--script=/bin/true','eth0'] took 300.055 seconds
dhcp.py:          dhcpcd timed out after 300 seconds: stderr: b'dhcpcd-10.5.2 starting ...
                    eth0: leased 172.41.9.169 from 172.41.0.1
                    eth0: adding IP address 172.41.9.169/20 broadcast 172.41.15.255
                    eth0: adding default route via 172.41.0.1
                    forked to background\n'
ephemeral.py:     Failed to bring up eth0 for ephemeral ipv4 networking.
performance.py:   Getting metadata took 300.606 seconds

Note the dhcpcd stderr clearly shows the lease was obtained and the address/route were added; the only problem is the parent process not returning, which cloud-init interprets as a timeout/failure.

Same boot, confirming the hook issue is NOT the cause here:

# journalctl -b | grep -c "io.systemd.Network' failed with error 'io.systemd.TimedOut'"
0

Isolation (single-variable test)

  • 259.1 / cloud-init 25.3 / dhcpcd 10.3.0: mask systemd-networkd-resolve-hook.socket → init-local drops from ~153s to ~16s (slow ['eth0'] path). Fixed.
  • 261.2 / cloud-init 26.2 / dhcpcd 10.5.2: same mask applied, hook timeouts = 0, but init-local still ~300s on the ['eth0'] path because of the dhcpcd --waitip/--persistent hang above. Only the mask was changed between boots.

The cloud-init source comment itself references known dhcpcd non-daemon lease-parsing issues (NetworkConfiguration/dhcpcd 0000285–0000287) and deliberately allows dhcpcd to daemonize via --persistent; with dhcpcd 10.5.2 the parent no longer returns promptly under --waitip, which appears to be the regression.

Recommendation

This is a distinct issue from the original resolved hook timeout. It looks like a cloud-init 26.2 + dhcpcd 10.5.2 interaction:

  • either the dhcpcd parent should exit once --waitip is satisfied (dhcpcd side), or
  • cloud-init should not block on the parent process when --persistent daemonization is expected (cloud-init side), or
  • ship a shorter ephemeral-DHCP timeout / prefer a different DHCP client in the Kali image until fixed.

Reproducible on freshly-launched instances with no user customization other than masking the resolve-hook socket.
kali-bugreport

kali-bugreport

2026-09-24 04:51

reporter   ~0022021

Why are you not talking with the systemd people about this. What should the Kali team do about issues in software of some else?

Issue History

Date Modified Username Field Change
2026-09-19 00:58 Lynx New Issue
2026-09-19 02:29 Lynx Note Added: 0022010
2026-09-24 04:51 kali-bugreport Note Added: 0022021