fix(tracking): mlflow tracking URI and example id hardening #2
Loading…
Reference in a new issue
No description provided.
Delete branch "v2.4.0-PR-tracking-logs"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
training_step never fed train_ids while every tracker gathers {prefix}_ids for both prefixes, so _gather_store logged an empty-store ERROR for the train id store on every run. Both id feeds now share _feed_ids, which skips a batch whose ids are all NaN, applied to the new train feed and to the existing val feed (design/tracking-stores.md rule 2): return_id=False configs emit a scalar-NaN fallback per item, so an unguarded feed would trade the empty-store ERROR spam for NaN-rejection ERROR spam (AD2). The guard is per batch (torch.isnan(batch.id).all()), not per dataset assumption. Id survey (NaN-mix trigger check; return_id is decided once per dataset config, so a loader's batches are homogeneous, and the NaN fallback sites emit scalar NaN per item when the flag is false): - graph_embed_class: train return_id=False (src/config/experiments/graph_embed_class.py:81, CitationGraphDatasetConfig 'train-dataset') -> train id tensors fully NaN per batch; val return_id=True (:91, 'test-dataset') -> fully real. Cross-loader asymmetry, never mixed within one batch. - retrieval_forecast: return_id True for train and val via base_dataset_kwargs (src/config/experiments/retrieval_forecast.py:141) -> fully real per batch on both paths. - transformer_class: return_id True for train and val via base_dataset_kwargs (src/config/experiments/transformer_class.py:84) -> fully real per batch on both paths. - Fallback sites emit per-item scalar NaN, never a real/NaN row mix: src/data/datasets/polars_dataset.py:229, src/data/formaters/graph.py:45, src/data/formaters/citation_graph.py:53, src/data/datasets/text_token_dataset.py:184. No train config mixes real and fallback ids inside one batch, so the NaN-mix trigger does not fire. NaN-rejection level review: process_values and _process_value keep their ERROR level. After the guard, a NaN reaching them is a genuine bad value, not a config fact. With export enabled, train_ids.npy joins the debug export payload (release doc friction): consumers of the exported arrays gain one int64 column per example. No metric names or values change. Tests: tests/test_id_feed.py drives a minimal strategy with a recording tracker stub; real ids land in train_ids, an all-NaN batch skips the feed on both paths while non-id feeds continue, and a mixed batch still feeds (pins the .all() guard granularity).Every logger.error(e) and logger.error(str(e)) site in src/training/tracking/ becomes logger.exception("<context string>"), so swallowed failures carry a traceback and name their site (AD5: mechanical, confined to the package). 45 sites across the six tracker modules; context strings name the plot paths (e.g. "plot rendering failed: ROC curve in _log_plots") so a plot-path revert for the plot-spam trigger can find them instantly. binary_classification_tracker.py calc_metrics PR_AUC handler: the nested try building mssg = f"{e}..." existed only to pre-format the message for logger.error(mssg); it collapses to one logger.exception call, which preserves the traceback the f-string discarded. Untouched: contextual f-string logger.error sites (shape-mismatch messages), the NaN-rejection ERROR sites in metric_tracker, and the _gather_store level logic (owned by v2.3.1-store-latch; hunks disjoint). Tests: test_tracker_exception_logging.py - caplog test on poisoned metric computation asserting the contextual message and exc_info presence, plus an AST scan asserting no logger.error(<bare name>) call remains in the package. Plot-spam trigger: pending - checked at the first real eval run, not in this commit.