GeoStack

GeoStack (Geometric Stacking) addresses the challenge of catastrophic forgetting in Vision-Language Models (VLMs) by allowing independently trained domain experts to be composed into a unified model. By imposing geometric and structural constraints on the adapter manifold, GeoStack ensures the foundational knowledge of the base model remains preserved.

GeoStack enables modular knowledge composition through geometric constraints.

Key Highlights

• Quasi-Abelian Property: Our framework achieves a quasi-additive composition ($W_g \approx I + \sum \Delta_i$), meaning the order in which you stack domain experts is largely irrelevant.
• Weight-Folding & Efficiency: Demonstrates a weight-folding property that achieves constant-time inference complexity $O(1)$, regardless of the number of integrated experts.
• Mitigating Forgetting: Significantly outperforms standard adapters in class-incremental learning and multi-domain adaptation.
• Geometric Stability: Introduces perturbation minimization theory to ensure the stability of the composed manifold.

Abstract

We address the challenge of knowledge composition in VLMs where accumulating expertise typically leads to catastrophic forgetting. We introduce GeoStack, a modular framework that allows independently trained domain experts to be composed into a unified model. By imposing geometric and structural constraints, GeoStack ensures the foundational knowledge of the base model is preserved while enabling constant-time inference.