PFL SEL Resync Examples

Resync in Lustre FLR synchronizes stale mirrors after writes or failures. When using PFL (Progressive File Layouts) or SEL (Self-Extending Layouts) within FLR mirrors, resync applies to the dynamic components. Examples below show creation of FLR files with PFL/SEL layouts and resync operations for Lustre 2.17.0 (January 2026), based on the Lustre Operations Manual (updated 2025) and community resources.

Prerequisites

• FLR-enabled filesystem (Lustre ≥2.11).
• OST pools for mirror separation (e.g., primary and mirror pools).
• ea_inode enabled for large stripe counts.
• Clients ≥2.11 for full support.

Setup OST Pools for Mirrors

# Create pools
lctl pool_new testfs.primary
lctl pool_add testfs.primary OST[0-4]
lctl pool_new testfs.mirror
lctl pool_add testfs.mirror OST[5-9]

PFL with FLR Resync Example

Create a mirrored file with PFL layouts, simulate a write (making mirrors stale), then resync.

# Create FLR file with PFL layouts per mirror
lfs mirror create -N -E 4M -c 1 -p primary --flags=prefer -E eof -c 2 -N -E 16M -c 4 -p mirror -E eof -c -1 /mnt/lustre/dir

# View layout
lfs getstripe -vd /mnt/lustre/dir

# Simulate write (makes non-primary stale)
echo "Test data" >> /mnt/lustre/dir/pfl_flr_file

# Find stale mirrors and resync them
lfs find --mirror-state=^ro -0 /mnt/lustre/dir | xargs -0 lfs mirror resync


# Verify
lfs mirror verify -v /mnt/lustre/dir/pfl_flr_file

SEL with FLR Resync Example

Create a mirrored file with SEL extensions, extend via write, then resync.

# Create FLR file with SEL layouts
lfs mirror create -N -E 1G -c 1 -z 64M -p primary -E -1 -c 4 -z 256M -N -c -1 -p mirror /mnt/lustre/sel_flr_file

# View initial layout
lfs getstripe -v /mnt/lustre/sel_flr_file

# Write to trigger extension (makes mirrors stale)
dd if=/dev/zero of=/mnt/lustre/sel_flr_file bs=1G count=2

lfs mirror resync /mnt/lustre/sel_flr_file

# Verify consistency
lfs mirror verify /mnt/lustre/sel_flr_file

Batch Resync for Multiple Files

# Find all stale files in directory
lfs find --mirror-state=^ro /mnt/lustre/dir > stale_files.txt

# Batch resync
cat stale_files.txt | xargs lfs mirror resync

# Batch verify
cat stale_files.txt | xargs lfs mirror verify

Resync After Failover

After OST failover, resync affected mirrors.

# Simulate OST failure
lctl set_param obdfilter.testfs-OST0005.degraded=1

lfs find --mirror-state=^ro -0 /mnt/lustre/dir | xargs -0 lfs mirror resync

# Reactivate and verify
lctl set_param obdfilter.testfs-OST0005.degraded=0
lfs find --mirror-state=^ro -0 /mnt/lustre/dir | xargs -0 lfs mirror verify

Best Practices

• Use --flags=prefer on primary mirror for reads.
• Resync after writes or failovers to maintain redundancy.
• Separate mirrors across pools/racks for fault isolation.
• Monitor with jobstats: lctl get_param job_stats.
• Automate resync in HA scripts post-failover.
• For large files, resync selectively with --only <id>.

For troubleshooting, check logs for ENOSPC/EIO during resync. Use LFSCK if inconsistencies persist: lctl lfsck_start -t layout.