Skip to content

Use logging in the decommutator, with configure_logging for Python and the CLI - #87

Open
namurphy wants to merge 13 commits into
PlasmaPy:mainfrom
namurphy:logger
Open

namurphy wants to merge 13 commits into
PlasmaPy:mainfrom
namurphy:logger

Conversation

@namurphy

Copy link
Copy Markdown
Member

Description

Replace the decommutator's hand-rolled log file, print calls, and sys.stdout.write progress display with the logging module, and add pyfaradaycup.configure_logging so that logging works the same way from Python and from pfc_decommutator.

  • Each module has its own logger, and the pyfaradaycup logger has a NullHandler, so the package is silent from Python unless logging is configured.
  • pyfaradaycup.configure_logging(level=..., log_file=..., file_level=..., file_mode=..., console=..., capture_warnings=...) adds a console handler and a log file handler to the pyfaradaycup logger only. It returns a LoggingConfiguration, which can be used in a with block or closed with close(). Log file timestamps are ISO 8601 in UTC.
  • swp_spc_l0_to_l05.main no longer takes logdir or verbose. It logs provenance (software versions, host, working directory, the L0 file's size and SHA-256 checksum, and the kernels and skeletons used) and a summary, and returns a DecommutationResult with the CDF files written, the APIDs that failed and why, packet counts, and the same provenance.
  • pfc_decommutator configures logging:
    • -v prints progress, -vv prints details, and -q prints only errors. Warnings are printed by default.
    • The log file always has every message, and is named like swp_spc_l0_to_l05_0523462910_4_EA_20261029T190014Z.log. It is opened with mode "x", so runs cannot overwrite each other's logs.
    • Python warnings and the warnings about options with no effect go to the log file too.
    • An unexpected error is logged with its traceback, and the exit code is 1. The exit code is also 1 when any APID could not be converted.
  • Messages that were printed for every rejected packet are logged at the DEBUG level with the byte offset, and summarized once.
  • Ruff's banned-api rule prevents adding logging handlers anywhere other than pyfaradaycup/_logging.py.

Motivation

The decommutator will have both a Python API and a CLI, and will run in a data pipeline. Logging needs to:

  • leave the caller in control of where messages go,
  • record enough provenance to trace each L0.5 file back to its inputs and software, and
  • report failures through return values and exit codes rather than only in log text.

This addresses part of "Make it callable from Python" and "Give each run its own log file name" in the decommutator README.

Details

Behavior changes for reviewers to check:

  1. The exit code is now 1 when an APID is skipped, including APID 0x07B, which is not implemented and only appears with --spacecraft. Previously these runs exited with 0.
  2. Log file timestamps are in UTC rather than local time, and the log file name format has changed.
  3. The Python API has changed: main no longer accepts logdir or verbose, and the private reader functions no longer accept verbose.
  4. Errors for a failed APID are logged at ERROR (one was previously a warning). Per-packet IT+ST and APID 0x352 problems are now DEBUG messages, plus one WARNING summary per APID.
  5. The progress display (% Complete) has been removed.

Not done here:

  • The warnings.warn in _parse_ccsds_head is unchanged.
  • Provenance is not yet written to CDF global attributes.
  • configure_logging and LoggingConfiguration are not yet in the API docs, because sphinx.ext.apidoc skips private modules and pyfaradaycup/__init__.py has no __all__.
  • The spacecraft housekeeping (--spacecraft) path still has no end-to-end test.

Tests cover configure_logging (levels, the file format and UTC timestamps, file modes, closing, replacement, propagation, and capturing warnings), packet rejection logging, the CLI's log file, its exit codes and -v/-q, and a Python API run that checks the result and that nothing is printed. They pass on Python 3.9 and 3.14.

🤖 Generated with Claude Code

namurphy and others added 7 commits October 29, 2026 14:37
Prepare to replace print calls and the hand-rolled log file in the
decommutator with the logging module. Each module gets a logger named
after it, and the package logger gets a NullHandler so that library
use emits nothing unless the application configures logging.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Provide one public way to send pyfaradaycup log messages to the console
and a log file, usable both from Python and from the command line tool.
Handlers are added only to the pyfaradaycup logger, a later call
replaces the handlers from an earlier one, and the returned
LoggingConfiguration can be closed or used in a with block. Log file
timestamps are in UTC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Library code should only create loggers, so that applications using
pyfaradaycup keep control of where log messages go. Use ruff's
banned-api rule so that handlers are configured only in
pyfaradaycup._logging.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Warnings captured with logging.captureWarnings go to the py.warnings
logger rather than the pyfaradaycup logger, so the command line tool
needs configure_logging to attach its handlers there too for warnings
to reach the console and log file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace print and sys.stdout.write in ccsds_reader_pipeline with
logging, so that the reader is silent when used from Python unless
logging is configured. Each rejected packet is logged at the DEBUG
level with its byte offset, and the number of packets found for each
APID and rejected for each reason is logged once at the end, rather
than printing one line per packet. The progress display is removed,
and the verbose arguments are no longer needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace _statusmsg and the module-level log file in swp_spc_l0_to_l05
with logging, so that main can be called from Python without writing
files or printing, and remove its logdir and verbose arguments.

The command line tool now configures logging. Its log file name
includes the name of the L0 file and a UTC timestamp, and is opened
with mode "x" so that runs cannot overwrite each other's logs. The
-v/--verbose option can be repeated to print progress (-v) or details
(-vv), -q/--quiet prints only errors, and warnings are printed by
default. Warnings about options with no effect and captured Python
warnings go to the log file too. An unexpected error is logged with its
traceback, and the exit code is 1.

Messages that were printed for every rejected packet are logged at the
DEBUG level and summarized with one warning per APID.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Record where each conversion's results came from, so that L0.5 CDF
files can be traced back to the inputs and software that made them.
main now logs the software versions, host, working directory, and the
size and SHA-256 checksum of the L0 file, along with the kernels and
skeletons it uses, and logs a summary at the end.

main returns a DecommutationResult with the CDF files written, the
APIDs that could not be converted and why, the packet counts, and the
same provenance, so that Python callers such as a Dagster asset do not
need to read log output. pfc_decommutator exits with a code of 1 when
any APID could not be converted, including APID 0x07B, which is not
implemented.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment on lines +235 to +237
data directory. A summary of the number of packets found for each
APID is logged, and the reason each packet was rejected is logged
at the ``DEBUG`` level.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if this level of detail is necessary or would actually be helpful. 🤔

namurphy and others added 3 commits October 29, 2026 18:38
Record follow-up work in the decommutator README: writing provenance
into the CDF global attributes, comparing CDF attributes in the
end-to-end test, and improvements to the log messages, including
tagging records with the L0 file and APID, a JSON Lines format, and
logging from more than one process.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Napoleon turned the Attributes section into attribute descriptions,
and autodoc documented the same dataclass fields again, so a clean
documentation build failed with duplicate object warnings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
namurphy and others added 3 commits November 2, 2026 17:14
Take the logging implementation and its tests from upstream/main, where
configure_logging was merged in PlasmaPy#89, in place of the earlier version on
this branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
configure_logging from upstream/main accepts "exclusive" rather than
"x" for file_mode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The merge took tests/test_logging.py from upstream/main, which does not
have the module loggers that this branch adds to the decommutator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant