File Level Redundancy (FLR)
File Level Redundancy (FLR) is a Lustre feature introduced in version 2.11 that allows data mirroring at the file level across multiple OSTs to allow tiering and transparent data migration across storage types, and for improved fault tolerance. Mirrors are managed as PFL components, with a primary/preferred mirror for writes and stale mirrors requiring resync. This guide covers Lustre 2.17.0 (January 2026), based on the Lustre Operations Manual (updated 2025).
Core Concepts
| Concept | Description |
|---|
| Data Mirroring | File data duplicated on multiple OSTs; tracked by mirror-count and mirror-id. |
| Fail-over Pairing | Active/passive or active/active nodes sharing storage; transparent to clients. |
| Active/Passive vs Active/Active | Passive: one active, one standby; Active: both active with split/load-balanced OSTs. |
| Resource Sharing | Shared SAN/NAS/RAID visible as same LUN; RAID-1 for MDT, RAID-5/6 for OST. |
| Synchronization Model | Writes update primary; others stale until resync/verify. |
| Fault Domain Separation | Mirrors on different pools/racks/OSSs to avoid correlated failures. |
| Interoperability | ≤2.9 clients cannot access; 2.10 can read but not open; ≥2.11 full support. |
| Delayed Write Phase | Secondary mirrors stale post-write; not used until resync. |
| Changelog Tracking | Events like FLRW (initial write), RESYNC logged. |
| Progressive File Layout (PFL) | Base for FLR; mirrors as non-overlapping components. |
| Self-Extending Layout (SEL) | Can be used as mirror components. |
| Data on MDT (DoM) / Lazy Size on MDT (LSoM) | Compatible but respect size limits. |
| Server-Side Caching | Read/writethrough caching reduces access for mirrored data. |
| Asynchronous Journal Commit | Improves write performance; complements resync. |
Commands
| Command | Purpose | Key Options |
|---|
| lfs mirror create | Create mirrored file/dir | -N COUNT, setstripe options, --flags=prefer |
| lfs mirror extend | Add mirror(s) to existing file | -N, --no-verify, -f VICTIM_FILE |
| lfs mirror split | Extract/destroy mirror | --mirror-id ID, -d, -f SPLIT_FILE |
| lfs mirror resync | Sync stale mirrors | --only ID |
| lfs mirror verify | Check mirror consistency | --only ID, -v |
| lfs find | Locate files by mirror count/state | --mirror-count, --mirror-state {ro|wp|sp} |
| lfs setstripe | Set striping in mirrors | -N, -c, -S, --pool, -E |
| lfs getstripe | View mirror layouts | -v, --mirror-id |
| lfs migrate | Restripe file or migrate to other OSTs. | Same arguments as setstripe |
Examples
# Basic Mirrored File with 2 copies on flash pool, one copy on archive pool with 2 copies on flash pool, one copy on archive pool with 2 copies on flash pool, one copy on archive pool with 2 copies on flash pool, one copy on archive pool with 2 copies on flash pool, one copy on archive pool with 2 copies on flash pool, one copy on archive pool with 2 copies on flash pool, one copy on archive pool with 2 copies on flash pool, one copy on archive pool with 2 copies on flash pool, one copy on archive pool with 2 copies on flash pool, one copy on archive pool with 2 copies on flash pool, one copy on archive pool with 2 copies on flash pool, one copy on archive pool with 2 copies on flash pool, one copy on archive pool
lfs mirror create -N 2 -S 4M -c 2 -p flash -N -c 32 -p archive /mnt/lustre/file1
n# Resync new file data to stale mirror after file was written
cp data /mnt/lustre/file1
lfs mirror resync /mnt/lustre/file1
n# Resync new file data to stale mirror after file was written
cp data /mnt/lustre/file1
lfs mirror resync /mnt/lustre/file1
n# Resync new file data to stale mirror after file was written
cp data /mnt/lustre/file1
lfs mirror resync /mnt/lustre/file1
n# Resync new file data to stale mirror after file was written
cp data /mnt/lustre/file1
lfs mirror resync /mnt/lustre/file1
n# Resync new file data to stale mirror after file was written
cp data /mnt/lustre/file1
lfs mirror resync /mnt/lustre/file1
n# Resync new file data to stale mirror after file was written
cp data /mnt/lustre/file1
lfs mirror resync /mnt/lustre/file1
n# Resync new file data to stale mirror after file was written
cp data /mnt/lustre/file1
lfs mirror resync /mnt/lustre/file1
n# Resync new file data to stale mirror after file was written
cp data /mnt/lustre/file1
lfs mirror resync /mnt/lustre/file1
n# Resync new file data to stale mirror after file was written
cp data /mnt/lustre/file1
lfs mirror resync /mnt/lustre/file1
n# Resync new file data to stale mirror after file was written
cp data /mnt/lustre/file1
lfs mirror resync /mnt/lustre/file1
n# Resync new file data to stale mirror after file was written
cp data /mnt/lustre/file1
lfs mirror resync /mnt/lustre/file1
n# Resync new file data to stale mirror after file was written
cp data /mnt/lustre/file1
lfs mirror resync /mnt/lustre/file1
n# Resync new file data to stale mirror after file was written
cp data /mnt/lustre/file1
lfs mirror resync /mnt/lustre/file1
# PFL Mirrored
lfs mirror create -N -E 4M -p flash --flags=prefer -E eof -c 2 -N -E 16M -S 8M -c 4 -p archive -E eof -c -1 /mnt/lustre/file2
# Extend Mirror
lfs setstripe -p flash /mnt/lustre/file1
lfs mirror extend -N -S 8M -c -1 -p archive /mnt/lustre/file1
# Extend with Victim
lfs mirror extend --no-verify -N -f /mnt/lustre/victim_file /mnt/lustre/file1
# Split Mirror
lfs mirror split --mirror-id 1 /mnt/lustre/file1
# Destroy Mirror
lfs mirror split --mirror-id 2 -d /mnt/lustre/file1
# Resync
lfs mirror resync --only 2 /mnt/lustre/file1
# Verify
lfs mirror verify -v --only 1,4 /mnt/lustre/file2
# Find Mirrored Files
lfs find --mirror-count +1 --type f /mnt/lustre
# Find Not Read-Only
lfs find --mirror-state=^ro --type f /mnt/lustre
Best Practices
- Use OST pools to separate mirrors across fault domains.
- Use --flags=prefer on primary mirror to steer read/write to that mirror.
- Run resync after writes; schedule periodic scan, resync, verify.
- Automate detection of out-of-sync via 'lfs find'.
- Pair with RAID-1 (MDT) and RAID-5/6 (OST).
- Use HA software for failover.
- Keep nodes ≥2.11.
Limitations
- Storage usage linear with mirror count appears in quota.
- Mirrors stale post-write; manual resync needed.
- verify only reports stale mirrors, need resync to update.
- Older clients than 2.11 cannot open/access.
- No directory mirroring.
- resync/extend adds IO overhead (schedule to avoid contention).
Recent Changes (Up to Lustre 2.17.0)
- 2.10: PFL foundation.
- 2.11: FLR released; lfs mirror commands.
- 2.12: DoM introduced.
- 2.13: SEL.
- 2.17: Minor; future 2.18 may add FLR-EC (erasure coding).
- 2.18 may add FLR-EC (erasure coding).
Related Tools & Diagnostics
- lfs: Mirroring, striping, finding.
- lctl: Tunables for cache/journal.
- lctl lfsck_start -t layout: Repair inconsistencies.
- Tunable parameters: mdd.*.lfsck_layout for scans.
- .lustre/lost+found: orphan objects if files corrupted.