<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Blinking Lights Club]]></title><description><![CDATA[Blinking Lights Club]]></description><link>https://blinkinglightsclub.hashnode.dev</link><image><url>https://cdn.hashnode.com/uploads/logos/6ac7f71976c498d4de0ad142/eafe56e3-c4f0-4d93-9e42-9272438acff8.png</url><title>Blinking Lights Club</title><link>https://blinkinglightsclub.hashnode.dev</link></image><generator>RSS for Node</generator><lastBuildDate>Fri, 16 Oct 2026 17:46:03 GMT</lastBuildDate><atom:link href="https://blinkinglightsclub.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Proxmox LXC: Why apt's network wait times out three times a day]]></title><description><![CDATA[If you run Debian containers on Proxmox, open one and look at its journal:
journalctl --since -2d | grep -i wait-online

There's a good chance you'll see this pair, two or three times a day:
systemd-n]]></description><link>https://blinkinglightsclub.hashnode.dev/proxmox-lxc-apt-wait-online-timeout</link><guid isPermaLink="true">https://blinkinglightsclub.hashnode.dev/proxmox-lxc-apt-wait-online-timeout</guid><category><![CDATA[proxmox]]></category><category><![CDATA[Homelab]]></category><category><![CDATA[Linux]]></category><category><![CDATA[systemd]]></category><category><![CDATA[debian]]></category><dc:creator><![CDATA[Blinking Lights Club]]></dc:creator><pubDate>Thu, 08 Oct 2026 20:26:39 GMT</pubDate><content:encoded><![CDATA[<p>If you run Debian containers on Proxmox, open one and look at its journal:</p>
<pre><code class="language-shell">journalctl --since -2d | grep -i wait-online
</code></pre>
<p>There's a good chance you'll see this pair, two or three times a day:</p>
<pre><code class="language-plaintext">systemd-networkd-wait-online[...]: Timeout occurred while waiting for network connectivity.
apt-helper[...]: E: Sub-process /lib/systemd/systemd-networkd-wait-online returned an error code (1)
</code></pre>
<p>Nothing is actually broken. The container's network is fine the whole time. But it's noise in every log review, and it makes apt's scheduled jobs sit for the full timeout before giving up on the "is the network up?" check.</p>
<h2>What's going on</h2>
<p>Two network stacks are installed in the container, and only one of them is doing anything.</p>
<ol>
<li><p><strong>ifupdown actually configures the network.</strong> The Debian container template writes <code>eth0</code> into the classic <code>/etc/network/interfaces</code>, and that's what brings the interface up.</p>
</li>
<li><p><strong>systemd-networkd is also running, but it manages nothing.</strong> Check it:</p>
<pre><code class="language-shell">$ networkctl list
IDX LINK  TYPE     OPERATIONAL SETUP
  1 lo    loopback carrier     unmanaged
  2 eth0  ether    routable    unmanaged
</code></pre>
<p>Every link says <code>unmanaged</code>.</p>
</li>
<li><p><strong>apt asks the idle one.</strong> apt's daily timers run <code>apt-helper wait-online</code>. When systemd-networkd is active, that hands off to <code>systemd-networkd-wait-online</code>, which waits for networkd to report a managed link as online. networkd manages no links, so the answer never comes, and the wait times out every time.</p>
</li>
</ol>
<h2>The fix</h2>
<p>Only do this if <code>networkctl list</code> shows <strong>every</strong> link as <code>unmanaged</code>. If networkd configures any interface on your container, this is not your problem, so stop here.</p>
<pre><code class="language-shell">systemctl mask --now systemd-networkd.service systemd-networkd.socket systemd-networkd-wait-online.service
</code></pre>
<p>Then check that nothing you care about moved:</p>
<pre><code class="language-shell">ip -4 addr show eth0          # same address as before
ip route show default         # same gateway as before
/usr/lib/apt/apt-helper wait-online; echo "exit $?"    # now returns immediately with exit 0
</code></pre>
<p><code>mask --now</code> stops the units and stops anything from starting them again. Your interface keeps running, because ifupdown owns it. (On systemd 257 you may notice <code>systemd-networkd-persistent-storage.service</code> go inactive too. It's a helper that only exists for networkd, and it stops along with it.)</p>
<p><strong>To undo it:</strong></p>
<pre><code class="language-shell">systemctl unmask systemd-networkd.service systemd-networkd.socket systemd-networkd-wait-online.service
systemctl start systemd-networkd
</code></pre>
<h2>Doing it across many containers safely</h2>
<p>If every container came from the same template, they probably all have this. I fixed them one at a time from the host with a guard: record the address and default route, mask, compare, ping, and roll back and stop at the first difference:</p>
<pre><code class="language-shell">U="systemd-networkd.service systemd-networkd.socket systemd-networkd-wait-online.service"
for id in $(pct list | awk 'NR&gt;1 &amp;&amp; $2=="running"{print $1}'); do
  managed=$(pct exec $id -- sh -c 'networkctl list --no-legend | grep -vc unmanaged')
  [ "$managed" = "0" ] || { echo "$id: networkd manages links - skipped"; continue; }
  before=$(pct exec $id -- sh -c 'ip -4 -o addr show eth0; ip route show default')
  pct exec $id -- systemctl mask --now $U &gt;/dev/null
  after=$(pct exec $id -- sh -c 'ip -4 -o addr show eth0; ip route show default')
  if [ "$before" != "$after" ]; then
    pct exec $id -- systemctl unmask $U; pct exec $id -- systemctl start systemd-networkd
    echo "$id: network changed - rolled back, stopping"; exit 1
  fi
  echo "$id: done"
done
</code></pre>
<p>Across eleven containers, none tripped the guard, and the timeouts stopped.</p>
<p>For new containers, add the same mask to your post-create checklist.</p>
<h2>Scope</h2>
<p>I verified this on Debian 13.7 containers (systemd 257) on Proxmox VE 9.2. Other template versions may differ, so run the <code>networkctl</code> check first. It's the whole safety test.</p>
]]></content:encoded></item></channel></rss>