How to Exclude Modules and Parameters¶

FlexTensor supports both include_patterns and exclude_patterns for fine-grained control over which modules and parameters are offloaded.

Offload everything except `lm_head`¶

config = OffloadConfig(
    include_patterns=["*"],
    exclude_patterns=["lm_head"],
)
model = flextensor.offload(model, config=config)

Exclude FP8 scale tensors¶

config = OffloadConfig(
    include_patterns=["layers.*"],
    exclude_patterns=["*.scale"],
)

Exclude specific layers¶

config = OffloadConfig(
    include_patterns=["layers.*"],
    exclude_patterns=["layers.31"],
)

Keep specific module classes on GPU¶

A class:<glob> exclude pattern excludes every module whose class matches from offloading, regardless of where it sits in the tree. Matching modules stay on GPU throughout discovery, profiling, and inference. This is the recommended way to keep hybrid-architecture sub-modules on GPU without having to enumerate their paths.

config = OffloadConfig(
    include_patterns=["layers.*"],
    exclude_patterns=["class:SharedExpertMLP", "class:MoELayer"],
)

Each class: pattern is tested against both the short class name (type(module).__name__) and the fully-qualified class name (f"{cls.__module__}.{cls.__qualname__}"). Use an FQCN glob to disambiguate when multiple packages define a class with the same short name:

# Match only torch.nn.Linear, not a user-defined Linear
config = OffloadConfig(
    include_patterns=["layers.*"],
    exclude_patterns=["class:torch.nn.*.Linear"],
)

Class patterns are module-level only — a match cascades to every parameter of the matched module. For parameter-level filtering (e.g. only *.scale), use a name pattern.

See Pattern Matching for the full semantics.

Combine include and exclude patterns¶

Include patterns select which modules to offload. Exclude patterns then remove specific modules or parameters from that set.

config = OffloadConfig(
    include_patterns=["embed", "layers.*", "head"],
    exclude_patterns=["head", "*.norm"],
)
# Result: offloads embed and layers.* but NOT head or any *.norm sub-modules

Configure via environment variables¶

FT_INCLUDE_PATTERNS="*" FT_EXCLUDE_PATTERNS="lm_head,*.norm" python serve.py

In your code:

config = flextensor.load_config()
model = flextensor.offload(model, config=config)

Configure via YAML¶

include_patterns:
  - "*"
exclude_patterns:
  - "lm_head"
  - "*.scale"

Exclude a sub-module within an offload unit¶

When a layer is an offload unit, you can exclude specific sub-modules to keep their parameters on GPU:

config = OffloadConfig(
    include_patterns=["layers.*"],
    exclude_patterns=["layers.*.norm"],
)
# Result: layers.0, layers.1, ... are offloaded but norm parameters stay on GPU

The exclude pattern layers.*.norm is not itself a patched module (it belongs to its parent's offload unit), so the exclusion happens at the parameter level during tensor discovery -- all parameters under norm are excluded from offloading.

Evaluation order¶

include_patterns selects candidate modules to offload (default: ["*"] = all).
Ancestor guard skips any candidate whose ancestor is already patched, forming offload units.
exclude_patterns removes offload units from the offloaded set (default: [] = exclude nothing). Excludes targeting sub-modules within an offload unit are handled at the parameter level.
A target matching both include and exclude is not offloaded.

GPU residency behavior¶

Excluded modules and parameters are moved to GPU permanently during initialization. They stay on GPU throughout all phases (discovery, profiling, inference) and are never offloaded. This means:

Excluded tensors consume GPU memory at all times
They do not participate in the offload scheduling strategy
Forward passes through excluded modules execute entirely on GPU without tensor transfers