View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0009865Kali LinuxKali Package Bugpublic2026-09-19 00:582026-09-19 02:29
ReporterLynx Assigned To 
PrioritynormalSeverityminorReproducibilityhave not tried
Status newResolutionopen 
Summary0009865: systemd-resolved resolve hook (io.systemd.Network) times out for 0000063:0000090-400s during cloud-init init-local on AWS EC2, delaying ever
Description

Environment

  • Kali GNU/Linux Rolling (AWS Marketplace / community AMI), amd64
  • systemd 259 (259.1-1)
  • cloud-init 25.3
  • Platform: AWS EC2 (VPC subnet, IPv4-only; IMDS IPv6 endpoint not available)
  • Reproduced on multiple fresh instances with zero customization.

Affected Package

systemd (259.1-1) — specifically the systemd-resolved resolve-hook mechanism (io.systemd.Network / systemd-networkd-resolve-hook.socket) introduced in systemd 259, as integrated in the Kali cloud image. This is NOT a cloud-init
bug: cloud-init merely triggers a normal name resolution during init-local, and the ~30s-per-query stall happens inside systemd-resolved waiting on the hook.
Restricting cloud-init's metadata_urls to IPv4 does not help, which confirms the problem is in the resolver hook path, not in cloud-init.

Summary

On every reboot / stop-start, cloud-init-local.service hangs for roughly 90–400 seconds. systemd-resolved repeatedly logs:

    Query on hook '/run/systemd/resolve.hook/io.systemd.Network' failed with
    error 'io.systemd.TimedOut', ignoring.

each timing out after ~30s, which dominates the init-local stage duration.

Steps to Reproduce

  1. Launch a fresh Kali Rolling AMI on AWS EC2 in an IPv4-only VPC subnet.
  2. Reboot (or stop/start) the instance a few times.
  3. Watch the serial console / cloud-init analyze show.

Reproduction note (important)

This bug only manifests on boots where cloud-init detects the primary NIC in the init-local stage, i.e. the log shows:

    DataSourceEc2.py[DEBUG]: Looking for the primary NIC in: ['eth0']

Only such boots trigger the name resolution that hits the resolve hook and stalls. Boots that log an EMPTY NIC list:

    DataSourceEc2.py[DEBUG]: Looking for the primary NIC in: []

skip init-local's resolution path entirely and finish in ~0.1s — these do NOT reproduce the problem and must not be used to conclude the bug is absent.

Whether a given boot takes the ['eth0'] path depends on cloud-init's instance cache state; a stop/start (or certain reboots) reliably triggers the ['eth0'] slow path. When verifying a fix, confirm the boot actually took the ['eth0'] path (grep the cloud-init log) before judging the result.

Expected

cloud-init-local completes in well under a few seconds (as it does on a "fast" boot where cloud-init reports Looking for the primary NIC in: []).
Name resolution during init-local should not block on a network manager that has not started yet.

Actual

On boots where cloud-init detects the primary NIC (DataSourceEc2.py: Looking for the primary NIC in: ['eth0']), init-local performs name resolution (Resolving URL took 94s + 34s). Because /etc/nsswitch.conf uses hosts: files myhostname resolve [!UNAVAIL=return] dns, the lookup goes to systemd-resolved. On systemd 259, resolved queries the per-link io.systemd.Network resolve hook (/run/systemd/resolve.hook/io.systemd.Network, socket-activating systemd-networkd.service). During init-local, systemd-networkd has not yet started and cannot answer, so resolved times out ~30s per query, 3–4 times, producing the 90–400s init-local stall. systemd-networkd only starts after cloud-init-local finishes.

cloud-init analyze show (representative slow boot):

    found local data from DataSourceEc2Local @00.015s +131.261s
    Finished stage: (init-local) 131.932 seconds

Ruled out (not the cause)

  • cloud-init metadata_urls / IMDS IPv6 (fd00:ec2::254): restricting metadata_urls to IPv4 only does NOT remove the stall.
  • Attached EBS volumes / fstab (nofail): removing them does NOT change it.
  • Any user-installed packages/services: reproduced on a completely fresh AMI.
  • Adding After=/Wants=systemd-networkd to systemd-resolved: made it WORSE (observed up to 6m50s).

Workaround (verified)

Masking the resolve-hook socket eliminates the timeouts; resolved falls back to DHCP-provided global DNS and name resolution keeps working:

    sudo systemctl mask systemd-networkd-resolve-hook.socket

Results on a fresh instance, worst-case slow path (primary NIC in: ['eth0']), including stop/start:

    init-local: ~152s -> ~16s
    io.systemd.Network TimedOut count: 4 -> 0
    DNS still resolves (getent hosts amazon.com OK)

Recommendation

The resolve-hook query issued by systemd-resolved during early boot should not block for the full timeout when systemd-networkd is not yet running.
Options:

  • resolved should treat the hook as unavailable immediately (fail fast) when the backing service is not started, instead of waiting ~30s per query; or
  • ordering should guarantee the hook provider is answerable before resolved is asked to resolve during the init-local (pre-network) stage; or
  • ship the mask/skip of systemd-networkd-resolve-hook.socket behavior in the Kali cloud image until the upstream ordering is fixed.

This appears to be an ordering/timeout interaction introduced with the systemd 259 resolve-hook mechanism as integrated in the Kali cloud image.

Appendix.

Evidence 1 — cloud-init timing of the slow boot

# sudo cloud-init analyze show
Starting stage: init-local
|`->found local data from DataSourceEc2Local @00.014s +131.272s
Finished stage: (init-local) 131.964 seconds

# /var/log/cloud-init.log (same boot)
2026-09-19 00:30:38,677 log_util.py:     Cloud-init v. 25.3 running 'init-local' ... Up 16.56 seconds.
2026-09-19 00:30:38,711 DataSourceEc2.py: Looking for the primary NIC in: ['eth0']
2026-09-19 00:32:15,553 performance.py:   Resolving URL took 94.495 seconds
2026-09-19 00:32:49,805 performance.py:   Resolving URL took 34.252 seconds
2026-09-19 00:32:54,798 log_util.py:      Cloud-init v. 25.3 finished ... Up 152.10 seconds.

Evidence 2 — systemd-resolved resolve-hook timeouts (30s interval × 4)

# journalctl -b | grep io.systemd.Network
Sep 19 00:31:10 systemd-resolved[688]: Query on hook '/run/systemd/resolve.hook/io.systemd.Network' failed with error 'io.systemd.TimedOut', ignoring.
Sep 19 00:31:41 systemd-resolved[688]: Query on hook '/run/systemd/resolve.hook/io.systemd.Network' failed with error 'io.systemd.TimedOut', ignoring.
Sep 19 00:32:11 systemd-resolved[688]: Query on hook '/run/systemd/resolve.hook/io.systemd.Network' failed with error 'io.systemd.TimedOut', ignoring.
Sep 19 00:32:45 systemd-resolved[688]: Query on hook '/run/systemd/resolve.hook/io.systemd.Network' failed with error 'io.systemd.TimedOut', ignoring.

Evidence 3 — resolve-hook socket unit (socket activation)

# /usr/lib/systemd/system/systemd-networkd-resolve-hook.socket
[Unit]
Description=Network Management Resolve Hook Socket
ConditionCapability=CAP_NET_ADMIN
DefaultDependencies=no
Before=sockets.target shutdown.target
Conflicts=shutdown.target

[Socket]
ListenStream=/run/systemd/resolve.hook/io.systemd.Network
FileDescriptorName=resolve-hook
SocketMode=0666
Service=systemd-networkd.service      # socket-activates networkd on connect
RemoveOnStop=yes

[Install]
WantedBy=sockets.target

Versions

systemd 259 (259.1-1)
cloud-init 25.3-2
Kali GNU/Linux Rolling
nsswitch hosts: files myhostname resolve [!UNAVAIL=return] dns

Activities

Lynx

Lynx

2026-09-19 02:29

reporter   ~0022010

Follow-up comment

Additional investigation: a separate problem exists on systemd 261 / cloud-init 26.2

After the original report (systemd 259.1, resolved io.systemd.Network hook timeout), I continued testing on instances that had been updated to a newer package set. On those, the systemctl mask systemd-networkd-resolve-hook.socket workaround removes the hook timeouts but does NOT fix the boot delay — because a different root cause dominates there.

Environment (where the new problem appears)

Kali GNU/Linux Rolling (AWS EC2, IPv4-only VPC subnet)
systemd      261 (261.2-1)
cloud-init   26.2-1
dhcpcd-base  1:10.5.2-1
(The original 259.1 environment used cloud-init 25.3 / dhcpcd 10.3.0, where the hook mask alone was sufficient.)

Symptom

With the resolve-hook socket already masked (confirmed: io.systemd.Network TimedOut count = 0 for the boot), cloud-init-local.service still hangs ~300s on boots that take the primary-NIC path.

Root cause (confirmed by logs)

cloud-init 26.2 brings up its ephemeral network in the init-local stage by running dhcpcd and waiting for the process to exit with a 300s timeout (
dhcp.py
, Dhcpcd.timeout = 300):

dhcpcd --ipv4only --waitip --persistent --noarp --debug --script=/bin/true eth0

With dhcpcd 10.5.2, on a cold start the client acquires the lease and configures the IP successfully, then forks to background, but the parent invocation never returns, so cloud-init hits the full 300s timeout and then declares the ephemeral setup failed — even though the interface was actually configured.

Evidence (fresh boot, hook already masked, ['eth0'] slow path)

DataSourceEc2.py: Looking for the primary NIC in: ['eth0']
performance.py:   Running ['dhcpcd','--ipv4only','--waitip','--persistent','--noarp','--debug','--script=/bin/true','eth0'] took 300.055 seconds
dhcp.py:          dhcpcd timed out after 300 seconds: stderr: b'dhcpcd-10.5.2 starting ...
                    eth0: leased 172.41.9.169 from 172.41.0.1
                    eth0: adding IP address 172.41.9.169/20 broadcast 172.41.15.255
                    eth0: adding default route via 172.41.0.1
                    forked to background\n'
ephemeral.py:     Failed to bring up eth0 for ephemeral ipv4 networking.
performance.py:   Getting metadata took 300.606 seconds

Note the dhcpcd stderr clearly shows the lease was obtained and the address/route were added; the only problem is the parent process not returning, which cloud-init interprets as a timeout/failure.

Same boot, confirming the hook issue is NOT the cause here:

# journalctl -b | grep -c "io.systemd.Network' failed with error 'io.systemd.TimedOut'"
0

Isolation (single-variable test)

  • 259.1 / cloud-init 25.3 / dhcpcd 10.3.0: mask systemd-networkd-resolve-hook.socket → init-local drops from ~153s to ~16s (slow ['eth0'] path). Fixed.
  • 261.2 / cloud-init 26.2 / dhcpcd 10.5.2: same mask applied, hook timeouts = 0, but init-local still ~300s on the ['eth0'] path because of the dhcpcd --waitip/--persistent hang above. Only the mask was changed between boots.

The cloud-init source comment itself references known dhcpcd non-daemon lease-parsing issues (NetworkConfiguration/dhcpcd 0000285–0000287) and deliberately allows dhcpcd to daemonize via --persistent; with dhcpcd 10.5.2 the parent no longer returns promptly under --waitip, which appears to be the regression.

Recommendation

This is a distinct issue from the original resolved hook timeout. It looks like a cloud-init 26.2 + dhcpcd 10.5.2 interaction:

  • either the dhcpcd parent should exit once --waitip is satisfied (dhcpcd side), or
  • cloud-init should not block on the parent process when --persistent daemonization is expected (cloud-init side), or
  • ship a shorter ephemeral-DHCP timeout / prefer a different DHCP client in the Kali image until fixed.

Reproducible on freshly-launched instances with no user customization other than masking the resolve-hook socket.

Issue History

Date Modified Username Field Change
2026-09-19 00:58 Lynx New Issue
2026-09-19 02:29 Lynx Note Added: 0022010