PIPE2D-1843: Update H4 NIR linearity and ISR#171
Open
CraigLoomis wants to merge 3 commits into
Open
Conversation
CraigLoomis
force-pushed
the
tickets/PIPE2D-1843
branch
from
f0da30a to
583c0c6
Compare
…ch detector
New ``python/lsst/obs/pfs/h4Linearity/`` package — the H4-specific NIR
ISR machinery. Cleanly separated from the rest of obs_pfs so the ISR
caller (``PfsIsrTask``) is the only thing that ties it into the
broader pipeline.
Subpackages:
- ``types`` — bit-flag constants for the (H, W) uint16 internal
mask (BORDER_PIX, MASKED_BY_INPUT, INSUFFICIENT_
POINTS, FIT_FAILED, NON_MONOTONIC, ABOVE/BELOW_
VALID_RANGE, UNCLASSIFIED, UNSTABLE, ASIC_GLITCH,
HIGH_FIT_RESIDUAL, DEAD composite) + Ramp /
LinearizedRamp / Diagnostics / LinearityCorrection
dataclasses.
- ``fit`` — per-pixel polynomial fit (default Chebyshev)
producing a LinearityCorrection plus
Diagnostics. Emits the fit-time bits above:
MASKED_BY_INPUT for caller defects, DEAD-group
for INSUFFICIENT_POINTS / FIT_FAILED /
NON_MONOTONIC, BORDER_PIX (alone) for the outer
4-pixel ring, HIGH_FIT_RESIDUAL when
residualRms > ``badLinearityMedianMultiplier``
× median(still-good residualRms). Per-pixel
``saturationKnee`` detector uses a leave-one-out
median of the refDelta window so a cluster of
saturating early deltas cannot mask its own
neighbours.
- ``apply`` — apply the fitted correction to a Ramp. ``apply``
takes the caller's internal mask as
``Ramp.validMask``, preserves all incoming bits,
and skips the runtime > fitMax / < fitMin range
check on pixels already flagged (the
"first-reason-wins" rule). Returns a
LinearizedRamp with a uint16 badPixelMask
carrying calib + caller + range bits.
``applyFrame`` handles single-frame CDS endpoints.
- ``io`` — ``saveFits`` / ``loadFits`` (HDU layout: COEFFS,
LIMITS, BPM) + ``isH4LinearityFile`` predicate.
- ``models`` — polynomial model classes (``PolynomialModel``,
chebyshev / monomial evaluators).
- ``cr`` — iterative UTR-rate-based CR + ASIC-glitch
detector. Runs on a (H, W, N-1) delta cube,
does per-pixel IQR-σ + cumulative-drop checks,
classifies flagged deltas as CR / glitch pair /
boundary glitch / unclassified, and adds a
4σ-IQR outlier-count BAD-pixel gate (RTS /
telegraph-noise pixels). ``result.rate`` is the
UTR-weighted slope of the corrected ramp
(closed-form delta weights u[j] = 6(j+1)(N-1-j)
/ (N(N-1)(N+1)), self-consistent fixed point).
ASIC-glitch repair is off by default — interior
pairs are detected so they don't masquerade as
CRs but left in place; end glitches and CRs are
repaired.
- ``sim`` — programmable defect simulator (CRs, ASIC
glitches, ground-truth labels) used by the
test suite.
- ``isrPlots`` — diagnostic plotting (per-pixel ramps,
half-vs-half rate comparisons, glitch histograms).
- ``validate`` — top-level orchestration helpers (``processRamp``,
halves-vs-halves UTR-rate validator, summary
formatters). ``processRamp`` is a thin caller
of ``PfsIsrTask.makeNirExposure`` — single source
of CR detection + mask projection.
- ``saturation`` — pixel-level saturation helpers.
- ``loaders`` — butler-side loader plumbing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Changes are all either to config knobs or to makeNirExposure and its callees. The linearization package has been entirely replaced, and now runs on the full ramp *before* artifact detection/repair/masking and the UTR weighting. CR/glitch/artifact processing is now done together, to give a better chance to distinguish CRs from the rest. CRs are now detected per-pixel, corrected and flagged by default. All the old ASIC glitch and CR machinery has been removed. Bad pixel flagging has been cleaned up: all pixels declared bad by either the defect or linearization products are marked BAD in the postISRCCD mask, and are also marked with DARK_DEFECT or LINEARITY_DEFECT (requires a datamodel change). Pixels which fail various internal tests (e.g. too many outliers from RTS/glitches) get marked BAD and UNSTABLE (also requires the datamodel change). Detected ASIC-glitch pixels are also masked BAD when at least one pair height clears ``h4.asicGlitchHeightMaskADU`` (default 0 → mask all). Internally, after linearization, we convert from (N,H,W) flux to (H,W,N) deltas once, and no longer have unnecessary flux<->delta conversions in the normal code path. The dark cube is now subtracted in place read-by-read, rather than loading the entire ramp into memory first. The logic for how to process standard inputs is codified in rampParams(). Arcs and flats are processed as CDS, darks as UTR, and science ramps as UTR but with the shutter-closed reads trimmed off. The trimming is currently hard-coded (and correct for the current iic sequencing), but once we have the header cards we expect to do that correctly. isrTask and makeNirExposure now support processing sub-ramps. isrTask is controlled by isr:h4.firstRead and isr:h4.lastRead, defaulting to 0 and -1. Dark cube and linearity corrections are done to the correct read and flux ranges. New header cards have been added to describe the span. No use is made of that in this ticket, but PIPE2D-1844 does require the feature. Hoisted in the UTR variance update from PIPE2D-1778, adapted to the new CDS-v-UTR and sub-ramp logic. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Test layer for the H4 NIR ISR refactor. Spread across the natural
boundaries:
- test_h4LinearityFit.py — fit() end-to-end: target recovery,
deterministic tiling, input-mask propagation, multi-ramp,
saturationKnee, HIGH_FIT_RESIDUAL, disabled path.
- test_h4LinearityFitThreading.py — _resolveWorkerCount heuristics
and worker-count-independent determinism for fit().
- test_h4Linearity_apply.py — apply() / applyFrame() correctness
including the "first-reason-wins" gating on internal mask bits.
- test_h4LinearityIo.py — round-trip saveFits / loadFits +
isH4LinearityFile predicate.
- test_polynomial.py — per-pixel polynomial fit / evaluator.
- test_h4CR.py — iterative CR/glitch detector: CR vs
glitch independence, cumulative-drop demotion, BAD-pixel outlier
gate, RTS-flagged-vs-noise discrimination, repair semantics.
- test_sim.py — ramp defect simulator (CRs, ASIC
glitches, ground-truth verification) used by test_h4CR.
- test_h4_cube_semantics.py — pins the (H, W, N) layout end-to-end.
- test_getDarkCube.py — getDarkCube + subtractDarkCube
indexing, gain handling, partial-ramp r0 offset.
- test_calcUTRrates.py — calcUTRWeights / calcUTRrates +
delta-space calcUTRrateFromDeltas equivalence + a three-way
cross-check pinning calcUTRrates, calcUTRrateFromDeltas, and the
CR detector's IterativeRepairResult.rate as numerically tied on
a clean ramp.
- test_isrHeaderKeys.py — _stampRampMetadata writes the
expected H4READ* / H4NREAD / H4NTOT / H4UTRWT keys; plus
coverage for the new _makeInternalMask + _projectInternalMask
helpers (border → BAD only, DARK_DEFECT, LINEARITY_DEFECT, SAT,
UNSTABLE projection rule, ASIC_GLITCH internal bit → BAD only).
- test_validate_processRamp.py — processRamp wires the requested
knobs into makeNirExposure and reads crResult back via
intermediates.
- conftest.py — smallSyntheticRamp / tinyLinearRamp fixtures used
by the fit + threading tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CraigLoomis
force-pushed
the
tickets/PIPE2D-1843
branch
from
583c0c6 to
d3865d4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three commits replacing most of the H4 NIR ISR path. See per-commit messages, especially the one covering isrTask.py
Test plan
sconsclean (131 tests, lint clean)reduceExposure.yaml#isron 141370, 141526, 142109, 144220 × n1/n3/n4 inu/cpl/tickets/PIPE2D-1843_2026-06-03_03Still need to following before this can be merged to master:
DEFECT,LINEARITY_DEFECT,UNSTABLEmask planes. These might get renamed.