2.6.4

Summary

This PR continues the flatbuffer_direct refactoring after the fb-refactor4 work merged in #949. It moves a large remaining set of layout, shape, quantization, fusion, and graph-repair rules out of the central lowerer and into focused, indexed ModelIR passes while preserving the existing CLI, Python API, default backend, artifact names, report contracts, and optional-exporter behavior.

The branch contains 168 commits affecting 230 files. Its largest structural change is the addition of 71 focused pass modules and 83 focused test modules. The large line count is primarily the explicit extraction of production rules and their characterization tests, rather than a new parallel conversion path.

Why this change is needed

The direct TFLite path had accumulated many order-sensitive graph rewrites in a very large lowering function. Those rewrites repeatedly scanned the whole ModelIR, inferred layout locally, and mutated shared state through implicit conventions. This made individual fixes expensive to reason about and increased the probability that a change for one model family would disturb another.

This PR makes rule ownership and state boundaries explicit. The intention is not to remove supported behavior: it is to retain that behavior behind smaller semantic passes, indexed graph access, typed request/artifact policy, and focused regression contracts.

Main improvements

1. A single normalized request and artifact boundary

• The direct conversion entry point now creates ConversionRequest once and stops propagating raw kwargs into lower layers.
• Requested outputs are represented by one immutable ArtifactPlan.
• Quantization, split, SavedModel, PyTorch, calibration, report, and progress options are resolved only when the corresponding artifact is requested.
• Unrequested exporters and their option parsing are not entered. For example, an unused PyTorch timeout cannot break a TFLite-only conversion.
• Direct TFLite evaluation selects files from the builder's returned artifact mapping instead of reconstructing paths from naming conventions.
• Direct-result completion validation and compatibility logging have one owner, and unreachable fallback routes through the legacy TensorFlow conversion path were removed.

These changes retain the legacy public keys, defaults, artifact order, filenames, failure messages, and return dictionary.

2. Shared indexed pass state

• Adjacent registered pass groups now share a lazy ModelIRPassStateScope when there is no intervening raw ModelIR mutation.
• ModelIRGraphIndex owns producer/consumer and operator-type lookups; LayoutState is kept synchronized through differential mutation helpers.
• Scope identity checks prevent graph or layout state from leaking across models or fallback ModelIR instances.
• Prepared pass data is session-local, consumed once, and cleared on rollback.
• Transactional guards, stable pass ordering, bounded iteration, and fingerprint-based convergence remain the safety boundary for risky rewrites.

This removes repeated index construction across many mean/attention, gate, concat, shuffle, boundary, slice, singleton-reshape, QDQ, SPP, and terminal-cleanup clusters without changing their historical execution order.

3. Focused semantic pass extraction

Seventy-one new production pass modules replace central-lowerer rule blocks. The extracted families include:

• Conv/Conv1D input, output, unary, BatchMatMul, InstanceNorm, affine, and deconvolution layout repair;
• concat/global-pool, concat/transpose-conv, split/concat, mixed-concat, input-adapter, and binary-bridge recovery;
• InstanceNormalization residual, dual-statistics, pre/post, bias, and affine-chain handling;
• quantized activation, HardSigmoid, Logistic, Softmax, MaxPool, PReLU, Swish, and TransposeConv cleanup;
• attention/QKV reshape, window partition/reverse, channel shuffle, singleton gate, passthrough, and terminal layout repair;
• dynamic Reshape resolution, static shape reconciliation, HARD_SWISH shape sanitization, and activation fusion;
• specialized historical SiNet chains decomposed into smaller pattern-owned passes with explicit guards.

The central dispatch remains responsible for orchestration, while match/guard/rewrite behavior and its invariants live beside focused tests. Existing rule order was preserved during extraction.

4. PyTorch exporter policy separation

Six fast-precanonicalization policies were moved out of the large exporter into pytorch_fast_precanonicalize_policy.py:

• downstream binary layout evidence;
• Resize/BatchNorm evidence;
• aligned BatchNorm constants;
• local-response-normalization layout propagation;
• rewritten static-shape recording;
• NHWC AveragePool bridge state.

The managed bulk runner also gained an opt-in native-only PyTorch mode. It emits -fdopt without implicitly enabling TorchScript, Dynamo ONNX, or ExportedProgram, while the runner's existing default behavior remains unchanged.

5. Sequential regression and process isolation hardening

• Tier 0-4 validation is fixed to one model at a time with a 600-second per-model ceiling.
• Every converter starts in a new POSIX session. Timeout, SWAP detection, and parent exit now terminate and reap the complete process group before the next model starts.
• Process-tree SWAP is monitored from /proc; any model that generates SWAP is excluded from future managed validation according to the documented policy.
• Resume metadata includes artifact mode and rejects incompatible resumes.
• The managed profile now records vit_b_encoder.onnx and superpoint_lightglue_end2end_fused_cpu.onnx as timeout_after_600s exclusions.
• All run state, normalized failure signatures, timing, accuracy, and exclusion rationale are documented in the checked-in baseline and regression reports. Large generated artifacts remain outside Git.

Regressions found and corrected

The full current-branch run initially exposed four genuine TFLite regressions. Each was recorded, reproduced against detached main, bisected to its first bad extraction commit, and corrected with a generic guard rather than a model-name exception:

1. Nighttime dehaze / InstanceNorm residual concat: accept only the exact valid case where declared TensorIR shape (1,) has a scalar, one-element NumPy backing, then normalize it to the declared shape.
2. RTMPose L and M / concat global-pool fan-out: permit an extra self-gating MUL only when the concat is one operand and the other operand is proven to depend on the matched global-pool Conv output. Arbitrary fan-out remains rejected.
3. GridSample / binary bridge recovery: validate consumer ordering against the binary operator that becomes the producer after rewrite, while retaining the stricter order guard for the preserved adapter.

Final sequential validation passed all four repaired models with zero SWAP and maximum absolute errors of 0.000294536, 4.67896e-06, 2.72691e-06, and 0.00802997, respectively.

Compatibility and dependency boundaries

• No new package dependency is introduced.
• All development and validation use the existing uv environment.
• Normal -tb flatbuffer_direct conversion and -cotof remain behind the TensorFlow import blocker.
• TensorFlow remains optional and is reachable only for explicitly requested SavedModel/H5/Keras/TFv1-style artifacts.
• Existing float32/float16 TFLite outputs, quantization variants, weights, split/crop, custom/pseudo op handling, reports, PyTorch-derived outputs, public defaults, return behavior, and filenames are preserved.
• The 2,000-node value is only the Tier 4/Tier 5 corpus boundary; it is not a source-file line limit.

Validation performed

Focused and architecture tests

• Changed focused suites: 253 passed.
• Architecture/contract suite: 218 passed.
• Indexed-pass regression suites after the four fixes: 154 passed.
• Bulk-runner suites, including timeout parent/grandchild cleanup and SWAP process-group termination: 40 passed.
• PyTorch extraction policy/emitter suites: 59 passed.
• Ruff checks passed for changed Python files.
• py_compile checks passed.
• git diff --check passed.
• Version consistency checks passed for onnx2tf.__version__, pyproject.toml, and README Docker examples at 2.6.4.

Tier 0-4 sequential corpus run

The authoritative sanitized-environment run processed all 381 then-active models sequentially with the exact command shape -tb flatbuffer_direct -cotof -fdopt:

• total model wall time: 9,347.523 seconds;
• median model time: 9.528 seconds;
• no completed model or descendant process used SWAP;
• TFLite observations before causal correction: 343 pass, 20 accuracy fail, 14 missing report, 2 conversion error, and 2 timeout;
• 341 of 355 managed baseline passes stayed passes;
• two known failures improved to passes (yolox_nano.onnx and rtdetrv4_s.onnx);
• the 14 baseline-pass observations were split into four branch regressions, six pre-existing/stale-baseline limitations, two user-approved DEIM results, and two timeout exclusions.

After correction, the four confirmed branch regressions passed exact sequential real-model revalidation. The next managed profile contains 379 active and 41 excluded Tier 0-4 records.

Native PyTorch assessment

The run generated and evaluated the native PyTorch package for every active model. Because no managed pre-fb-refactor5 native-PyTorch baseline exists, non-passes were not automatically labeled as regressions. Eight representative high-signal failures were rerun sequentially on detached main; all eight produced the same behavior on both implementations. No fb-refactor5-specific native PyTorch regression was confirmed by that comparison.

Known limitations and review notes

• The complete 379-model corpus was not rerun from zero after the four localized fixes; the affected real models and focused neighboring/sentinel suites were rerun instead.
• The monolithic legacy tests/test_pytorch_exporter.py run was intentionally stopped at 89% when one exported-program archive test spent several minutes recompiling. At interruption it had 942 passes and 81 failures; this is characterization, not an acceptance result. Representative fast failures reproduced identically on detached main.
• Native PyTorch still has pre-existing runtime/accuracy limitations recorded in the regression report; this PR does not claim to solve them.
• Tier 5 (2,000 or more ONNX nodes) was not part of this Tier 0-4 corpus run.
• Optional TensorFlow artifact combinations were not included in the authoritative direct-path corpus run.
• Remaining work includes further op-family lowering separation, broader quantization/split/artifact matrices, and warm-run time/RSS comparison. These are documented rather than hidden by the current checkpoint.

What's Changed

• Refactor flatbuffer_direct passes and harden Tier 0-4 regression safety by @PINTO0309 in https://github.com/PINTO0309/onnx2tf/pull/951

Full Changelog: https://github.com/PINTO0309/onnx2tf/compare/2.6.3...2.6.4