Skip to content

The metrics say which module is armed for USB boot

Pins bmcd 2.36.3. BMC-UI stays at 3.27.0 and tpi is unchanged.

Firmware v2.30.0, 12 September 2026 — the first image to carry bmcd 2.36.3. Every item below is the repository's own changelog entry, taken apart; the whole entry is a click away under each one.

New

/metrics says which module is armed for USB boot

Two families: bmcd_node_usb_boot_armed, one sample per module, and bmcd_usb_config, which names the node, mode, route and bus type of the persisted configuration.

The whole entry

tpi flash and tpi advanced msd leave that configuration at Flashing(NodeN, …), and the daemon re-applies it on every start, which asserts the module's USB-boot pin and stops it booting from its own eMMC. The module keeps running, because it is already booted — so the fault appears at its next reboot, possibly weeks later.

Until now nothing exported that. A module in flash mode is indistinguishable from dead hardware from every remote angle: nothing at all on the serial console, not even a bootloader banner, because the loader does not use the console; nothing on the network; and the BMC reporting the rail on. tpi usb status cannot separate the two either — it prints the same route for UsbA and Flashing. Measured on hive-6 on 2026-09-12: twenty minutes spent on a module that looked dead, with the answer sitting unexported in the daemon's own database.

/metrics now reports the USB multiplexer, including which module is

armed for USB boot bmcd 2.36.3

Two families: bmcd_node_usb_boot_armed, one sample per module, and bmcd_usb_config, which names the node, mode, route and bus type of the persisted configuration.

The whole entry

tpi flash and tpi advanced msd leave the configuration at Flashing(NodeN, …). The daemon persists that and re-applies it on every start, which asserts that module's USB-boot pin and stops it booting from its own eMMC. The module keeps running, because it is already booted — so the fault appears at its next reboot, possibly weeks later, and then it comes up in the USB loader instead of its operating system.

From outside, that module is indistinguishable from dead hardware: nothing at all on the serial console, not even a bootloader banner, because the loader does not use the console; nothing on the network; and the BMC reporting its rail on. tpi usb status could not separate the two either, because it prints the same route for UsbA and Flashing. The only readout that named the mode was a line in the daemon's own log.

Measured on hive-6, 2026-09-12: twenty minutes spent on a module that looked dead, with the fact that explained it sitting unexported in the daemon's database. bmcd_node_usb_boot_armed{node="node2"} 1 would have answered it in one query, and a rule on fifteen minutes of it pages the operator who armed it while they still remember doing so.

The armed family is four samples rather than one because a single "which node" gauge cannot express none, and none is the normal state it exists to assert.

Changed

  • UsbConfig::parts() is now the single mapping from a stored configuration to (node, mode, route). The type=usb response and the metrics exposition share it, so a board cannot report one thing through the API and another on a dashboard. bmcd 2.36.3

Fixed

  • The board refused every attempt to resume a TLS session, fatally, and that was the intermittent console failure. A serial console or an API call through the fleet gateway would occasionally fail outright, on both boards, with nothing in the daemon's log and the board healthy either side of it.

    The whole entry

    Envoy keeps one TLS session per upstream cluster and offers it on the next connection it opens. On the gateway, a third of all new connections to the two boards died that way — 8 of 28 to one, 19 of 45 to the other — while every connection taken from the pool was fine, which is what made it look random from a browser. Reproduced from a pod on the gateway's own node: 30 fresh connections all succeeded, and 30 offering back a saved session all failed with tlsv1 alert internal error.

    OpenSSL will not resume a session on a server that asks for client certificates unless the context carries a session id context, and the refusal is not a quiet cache miss — it is internal_error, fatal, sent before a byte of HTTP. The error is raised on the server, which never logged it. So the fault arrived with client certificates in 2.30.0 and was invisible for six releases, and it was never specific to the gateway: any client that resumes hit the same wall.

  • The board refused every attempt to resume a TLS session, fatally. This is the intermittent fleet console failure: a serial console or an API call through the gateway would occasionally fail outright, on both boards, with nothing in the daemon's log and the board perfectly healthy either side of it. bmcd 2.36.2

    The whole entry

    Envoy keeps one TLS session per upstream cluster and offers it on the next connection it opens. Measured on the gateway, a third of all new upstream connections to the two boards died this way — 8 of 28 to one, 19 of 45 to the other — while every connection taken from the pool was fine, which is what made it look random from a browser. Reproduced deterministically from a pod on the gateway's own node: 30 fresh connections all succeeded, and 30 that offered back a saved session all failed with tlsv1 alert internal error.

    OpenSSL will not resume a session on a server that asks for client certificates unless the context carries a session id context, and the refusal is not a quiet cache miss — it is internal_error, fatal, sent before a byte of HTTP is exchanged. The error, ssl_get_prev_session: session id context uninitialized, is raised on the server, which never logged it; the client sees only an alert it cannot explain.

    So the fault arrived with client certificates in 2.30.0 and was invisible from the daemon's side for six releases.

    The acceptor now sets a session id context, derived from the client CA so that a board whose trust anchor is replaced will not resume a session authenticated under the old one. There is a test that performs two real handshakes against the real acceptor and offers the first session back; it fails on 2.36.1 with the OpenSSL error above.

Every release of the firmware · the roadmap · follow by feed