Two threads I'm actively working on. Each entry below is a living
record — I update it as findings change rather than starting a new
page for every increment.
Template note — each section below (findings list, benchmark table, screenshots, PDF attachment, discussion links) is optional per entry. Use what applies, drop what doesn't.
Updated Jun 28, 2026Analytic EngineeringActive
Analytic Pipeline Optimization
Data pipelines and transformations for high-performance analytics and
ML training. The current focus is vectorized operations and query
pushdown — moving computation closer to storage instead of pulling
everything into memory first.
Key findings
Pushdown filtering before joins cut shuffle volume substantially on the test dataset.
Vectorized transforms outperform row-wise UDFs once batch size clears a threshold.
Column pruning earlier in the plan reduced peak memory more than expected.
Benchmark
Throughput, 10M row synthetic dataset, single node
Star schemas designed for efficient retrieval and embedding
generation. The question here: does a dimensional model built for
BI-style aggregation also serve as a good retrieval substrate for
fine-tuning pipelines, or do the two use cases pull the schema in
different directions?
Key findings
Conformed dimensions reduced duplicate embedding generation across overlapping training samples.
Fact tables denormalized for retrieval speed cost some flexibility for ad-hoc BI queries — a real tradeoff, not a free win.
Slowly changing dimension handling matters more for training data lineage than I expected going in.