Skip to content

cli/libnvme: add --idempotent and --devid-file to nvme connect#3573

Open
martin-belanger wants to merge 3 commits into
linux-nvme:masterfrom
martin-belanger:nvme-connect-idempotent-devid
Open

cli/libnvme: add --idempotent and --devid-file to nvme connect#3573
martin-belanger wants to merge 3 commits into
linux-nvme:masterfrom
martin-belanger:nvme-connect-idempotent-devid

Conversation

@martin-belanger

@martin-belanger martin-belanger commented Jul 10, 2026

Copy link
Copy Markdown

Prep patch for nvme-discoverd (not yet upstreamed). discoverd manages each fabrics connection as a systemd transient unit whose ExecStart= is nvme connect:

  • --idempotent exits 0 when the target is already connected, instead of 1 — otherwise a harmless race with an existing connection fails the unit.
  • --devid-file FILE names a file libnvmf_connect() writes the kernel-assigned device name (e.g. nvme0) to on success, so the unit's ExecStop= can find it again to disconnect.

Three commits:

  1. libnvme/fabrics: write the connected device name to a file on request.
  2. cli/fabrics: add --idempotent and --devid-file to nvme connect — the CLI options themselves.
  3. meson: stop deriving RUNDIR from --prefix — found while testing: RUNDIR was join_paths(prefix, get_option('rundir')), so any build silently got <prefix>/run instead of the real /run tmpfs. /run isn't an install path, so it's now hardcoded, the same way systemd does it.

Martin Belanger added 2 commits July 10, 2026 15:39
Give a caller that manages a fabrics connection as a supervised
process (e.g. a systemd transient unit) a way to recover the
kernel-assigned device name later, e.g. to disconnect. Writing it
from inside libnvmf_connect() means every path out with a live
controller -- freshly connected, found already connected in the
topology already scanned, or found already connected only after a
kernel-race rescan -- reuses the same controller-matching logic
libnvme already relies on, rather than a caller re-deriving it.

Signed-off-by: Martin Belanger <martin.belanger@dell.com>
nvme-discoverd manages each fabrics connection as a systemd transient
unit whose ExecStart is "nvme connect". Without --idempotent, a
connect that races with an existing connection exits 1 and fails the
unit even though the target is reachable. --devid-file names the file
libnvmf_connect() writes the kernel-assigned device name to, so the
unit's ExecStop can find it again to disconnect.

Signed-off-by: Martin Belanger <martin.belanger@dell.com>
@martin-belanger martin-belanger force-pushed the nvme-connect-idempotent-devid branch from 6c87db5 to 2d9774a Compare July 10, 2026 19:55
Comment thread meson.build Outdated
RUNDIR was join_paths(prefix, get_option('rundir')), so any build
without an explicit -Drundir=/run override (this project's own
scripts/build.sh included) silently got <prefix>/run instead of the
real /run tmpfs -- e.g. /usr/local/run on a stock local build. /run is
not an install path: its location is fixed system-wide by the kernel
and the FHS on most distributions, the same reason systemd hardcodes
it rather than deriving it from --prefix. Some distributions still
only provide the legacy /var/run path, though, so rundir stays a
configurable option -- it just must never scale with --prefix. Test
isolation is a separate, already-solved runtime concern
(libnvme_set_test_base_dir()/LIBNVME_TEST_BASE_DIR), so joining it to
--prefix bought nothing but a second way to get this wrong.

Signed-off-by: Martin Belanger <martin.belanger@dell.com>
@martin-belanger martin-belanger force-pushed the nvme-connect-idempotent-devid branch from 2d9774a to 125c84d Compare July 13, 2026 13:04
@igaw

igaw commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Can take a quick timeout and figure out if we really need --devid-file? As far I understand it is used to handle the nvme disconnect case in a systemd service, right?

Couldn't we first try to make nvme disconnect to accept the same arguments as nvme connect and use this info to lookup the correct device? If this doesn't work we can go down this route if needed.

#3474

@igaw

igaw commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

On --idempotent, I haven't really thought thru what we currently do and why. Maybe we should make the idempotent case default? On the other hand tools like 'mkdir' do:

$ mkdir /tmp/test
$ mkdir /tmp/test
mkdir: cannot create directory ‘/tmp/test’: File exists
$ mkdir -p /tmp/test

Hmm, with this in mind --idempotent seems the right way.

@martin-belanger

Copy link
Copy Markdown
Author

I looked into this before adding --devid-file, and I don't think it fully replaces it, for two reasons:

  1. Duplicate connections are allowed (--duplicate-connect/-D), and the matcher we'd use to find "the" controller for a given set of connect args (libnvme_ctrl_find()) just returns the first match it finds — no ambiguity detection. With two identical connections up, disconnecting by args would silently tear down a matching controller, not necessarily the one the caller meant.

  2. traddr/host_traddr get sanitized before connecting, not just resolved. A hostname goes through getaddrinfo() (nvmf_resolve_addr()); a numeric address still gets canonicalized (canon_ip() in tid.c, an inet_pton()/inet_ntop() round-trip) — IPv6 always ends up in its compact form, and IPv4 loses any zero-padding (010.000.000.001 becomes 10.0.0.1). So even a numeric address that never touches DNS isn't necessarily what ends up in sysfs byte-for-byte. disconnect would need to reproduce both steps to match reliably, and if either drifts (DNS answers differently, or a script has the address in a non-canonical form), it'd either match nothing or match the wrong controller.

--devid-file sidesteps both: it's the literal device name from the actual connect call, so there's no ambiguity and nothing to re-resolve at disconnect time.

I still think #3474 is worth doing for the common case (no duplicates, static addressing) — it just doesn't cover the general case, so I'd rather keep --devid-file for that and treat #3474 as a separate, later improvement rather than a replacement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants