Lustre Striping Guide
Lustre file striping distributes file data across multiple Object Storage Targets (OSTs) using a RAID-0 pattern to improve performance, scalability, and bandwidth. Striping is managed by the Logical Object Volume (LOV) layer and stored as extended attributes. It enables parallel I/O. This guide covers Lustre 2.17.0 (January 2026), based on the Lustre Operations Manual (updated 2025). For details, see Lustre Manual.
Core Concepts
| Concept | Description |
|---|
| Stripe Count | Number of OSTs per file. Default: 1. Max: 2000 (2.13+ with ea_inode or ZFS). |
| Stripe Size | Size of each chunk. Default: 1 MiB. Multiple of 64 KiB. Max: <4 GiB. |
| Stripe Offset | Starting OST index. Default: -1 (MDS chooses, strongly preferred). |
| RAID-0 Striping | Default round-robin without redundancy. |
| Progressive File Layout (PFL) | Dynamic extents with varying stripes and other parameters (2.10+). |
| Self-Extending Layout (SEL) | Extends PFL components dynamically (2.13+). |
| Data on MDT (DoM) | Small files on MDT (2.11+). |
| Directory Inheritance | Striping propagates to subdirs/files (with limited inheritance 2.14+).(with limited inheritance 2.14+). |
| File Level Redundancy (FLR) | Mirroring for multi-tiered storage, fault tolerance (2.11+). |
| Overstriping | Multiple stripes per OST for highly-contended files(2.13+). |
| Striped Directories | Directories spread across MDTs (2.8+). |
Commands
| Command | Purpose | Key Options |
|---|
| lfs setstripe | Set striping | -c (count), -S (size), -E (component end), -p (pool), -L / --layout / --layout / --layout / --layout / --layout (layout) |
| lfs getstripe | Display striping | -v (verbose) |
| lfs setdirstripe | Set dir striping (2.13+) | -D, -c, -i |
| lfs mkdir | Create striped dir | -c (count) |
| lfs find | Locate by attributes | --layout, --pool |
| lfs mirror | Manage FLR | create, resync, -N (mirrors) |
| lfs migrate | Re-layout files | (same as lfs setstripe) |
| filefrag | Check fragmentation | -v |
| lfs df | Space usage | -h, -i |
| lctl set_param | Tuning | mdc.dom_stripes=1 |
| getfattr | Inspect xattrs | -d, -m |
Options
| Option | Description | Example |
|---|
| -c / --stripe-count | OST count | -c 8 |
| -S / --stripe-size | Chunk size | -S 128K |
| -E / --comp-end | Component end | -E 1G |
| -C / --overstripe-count | Per OST stripes | -C -2 |
| -p / --pool | OST pool | -p flash |
| -z / --extension-size | SEL extend size | -z 64M |
| -L | Layout type | -L / --layout / --layout / --layout / --layout / --layout mdt |
| --mirror-count / -N | Mirrors | -N 2 |
| --flags | FLR flags | --flags prefer |
Types & Layouts
| Type | Description |
|---|
| RAID-0 | Default no-redundancy striping. |
| PFL | Dynamic extents (2.10+). |
| SEL | Auto-extending PFL (2.13+). |
| DoM | Small files on MDT (2.11+). |
| FLR | Mirroring (2.11+). |
| Foreign Layout | External references. |
Best Practices
- Match stripe size to I/O: 1-4 MiB sequential, 64-512 KiB random.
- Use stripe_index=-1 for balance OST space usage and workload.
- Set PFL layout at root directory for whole filesystem default
- Set PFL layout at parent directory level for new files with very different workloads.
- Set PFL layout at parent directory level for new files with very different workloads.
- Set PFL layout at parent directory level for new files with very different workloads.
- Set PFL layout at parent directory level for new files with very different workloads.
- Set PFL layout at parent directory level for new files with very different workloads.
- DoM for small files; reserve MDT space during formatting for this.
- PFL recommended for most files.
- FLR for critical data; different fault domains.
- Pools for workload isolation.
- Monitor/rebalance poorly configured files with lfs df, lfs migrate.
Performance Considerations
- Higher stripe count improves bandwidth, but also increases overhead.
- Small stripe size adds overhead for most workloads.
- Overstriping improves many threads writing to a single file.
- DoM reduces small file latency, increases load on MDTs.
- Tune RPC sizes via osd-ldiskfs.*.brw_size up to 32 MiB, mostly for HDD.
- OST free space imbalance >17% triggers free space weighted allocation.
- ZFS: dynamic inode allocation, compression, recordsize >= brw_size.
- ldiskfs: lowest overhead, but needs underlying RAID layer.
- ldiskfs: lowest overhead, but needs underlying RAID layer.
- ldiskfs: lowest overhead, but needs underlying RAID layer.
- ldiskfs: lowest overhead, but needs underlying RAID layer.
- ldiskfs: lowest overhead, but needs underlying RAID layer.
Recent Features (Up to 2.17.0)
- 2.10: PFL.
- 2.11: DoM, FLR, LSoM.
- 2.12: Enhanced DoM.
- 2.13: 2000 stripes, overstriping, SEL.
- 2.14: Pool quotas.
- 2.15: Default striping.
- 2.16: Per-target tunables.
- 2.17: Minor improvements.
Examples
# Plain striping across eight OSTs with 4MiB stripe size
lfs setstripe -c 8 -S 4M /mnt/lustre/file
# PFL to have wide striping after first 1GiB of file to have wide striping after first 1GiB of file to have wide striping after first 1GiB of file to have wide striping after first 1GiB of file to have wide striping after first 1GiB of file
lfs setstripe -E 1G -c 1 -E eof -c -1 /mnt/lustre/bigfile
# DoM to put first 1MiB of files on MDT flash, rest of files on HDD OSTs to put first 1MiB of files on MDT flash, rest of files on HDD OSTs to put first 1MiB of files on MDT flash, rest of files on HDD OSTs to put first 1MiB of files on MDT flash, rest of files on HDD OSTs to put first 1MiB of files on MDT flash, rest of files on HDD OSTs
lfs setstripe -E 1M -L / --layout / --layout / --layout / --layout / --layout mdt -E -1 -S 4M -c -1 /mnt/lustre/smallfile
# SEL to avoid filling small SSD devices to avoid filling small SSD devices to avoid filling small SSD devices to avoid filling small SSD devices to avoid filling small SSD devices
lfs setstripe -E 1G -z 64M --pool ssd -E -1 -z 256M /mnt/lustre/growfile
# FLR mirrors with tiered data copy on HDD mirrors with tiered data copy on HDD mirrors with tiered data copy on HDD mirrors with tiered data copy on HDD mirrors with tiered data copy on HDD
lfs mirror create -N 2 -S 4M -c 2 --pool ssd -N -c -1 --pool hdd /mnt/lustre/critical
# Overstriping for highly concurrent write workload for highly concurrent write workload for highly concurrent write workload for highly concurrent write workload for highly concurrent write workload
lfs setstripe -c -1 -C -2 /mnt/lustre/highthread
# Get layout information for a file layout information for a file layout information for a file layout information for a file layout information for a file
lfs getstripe -vy /mnt/lustre/file
Warnings
- Need OST-level redundancy for RAID-0 striped files.
- Avoid fixed OST starting offsets to avoid OST imbalance.
- DoM max component size limited by lod.*.dom_stripesize on MDT.
- FLR mirrors need external resync after file is written.
- Backup xattrs on MDTs when migrating storage.
- MDT failure affects all files.files.files.files.files.
- PFL file without EOF component returns ENODATA on IO beyond last component.
- Deactivate OSTs after migrate.
Monitoring & Maintenance
- lfs df -h/i for space/inodes.
- lctl get_param for tunables.
- lfs_migrate for rebalance.
- lctl lctl lctl lctl lctl lfsck_start -t layout for repairs.
- e2fsck after restore.