bbcp: High-Speed Large File Transfers on Linux

What Is bbcp?

bbcp (Balanced, Broadcast, and Cache Protocol) is a point-to-point network file transfer tool designed for moving large files and datasets at maximum speed across wide-area networks. Developed at SLAC (Stanford Linear Accelerator Center), bbcp can saturate network links that traditional tools like scp or rsync leave underutilised.

Where scp uses a single TCP stream and is limited by TCP window sizes and latency, bbcp opens multiple parallel streams and uses direct I/O to bypass kernel buffering, achieving throughput several times faster on high-bandwidth, high-latency links.

When to Use bbcp

• Large scientific datasets — Genomics FASTQ files, physics simulation outputs, telescope imagery.
• Database dumps — Multi-gigabyte SQL or binary backups.
• Media production — Raw video files, post-production assets.
• Data centre migrations — Moving terabytes between geographically distant sites.
• Any scenario where scp/rsync is too slow on high-bandwidth links.

Installation

# On RHEL/CentOS/Oracle Linux — build from source
sudo dnf install -y gcc gcc-c++ make openssl-devel zlib-devel

# Clone the repository
git clone https://github.com/eeertekin/bbcp.git
cd bbcp/src

# Build
make

# Install
sudo cp ../bin/*/bbcp /usr/local/bin/
bbcp --version

# On Debian/Ubuntu
sudo apt install -y build-essential libssl-dev zlib1g-dev
git clone https://github.com/eeertekin/bbcp.git
cd bbcp/src && make
sudo cp ../bin/*/bbcp /usr/local/bin/

Important: bbcp must be installed on both the source and destination hosts, and it must be in the PATH for the remote user.

Basic Usage

# Simple file transfer (like scp)
bbcp source-file user@remote:/destination/path/

# Transfer a directory recursively
bbcp -r /data/dataset/ user@remote:/backup/dataset/

# Specify number of parallel streams (default: 4)
bbcp -s 8 large-file.tar.gz user@remote:/data/

# Set TCP window size (bytes) for high-latency links
bbcp -w 8M -s 8 big-dataset.tar user@remote:/storage/

Key Options Explained

-s <streams>   Number of parallel TCP streams (default: 4)
-w <size>      TCP window size (e.g., 2M, 8M, 16M)
-r             Recursive directory transfer
-P 2           Progress reporting every 2 seconds
-V             Verbose output
-f             Force overwrite existing files
-E <prog>      Run checksum program after transfer
-S <cmd>       Source staging command
-T <cmd>       Target staging command
-p             Preserve permissions and timestamps
-z             Compress data in transit

Performance Tuning

Optimal Stream Count

# Rule of thumb:
# Streams = (BDP in bytes) / (TCP window size)
# BDP = Bandwidth × RTT

# Example: 10 Gbps link, 50ms RTT
# BDP = 10,000,000,000 × 0.050 / 8 = 62,500,000 bytes
# With 8M windows: 62.5M / 8M ≈ 8 streams

bbcp -s 8 -w 8M /data/huge-file user@remote:/data/

Kernel Tuning

# Increase TCP buffer sizes on both hosts
sudo sysctl -w net.core.rmem_max=67108864
sudo sysctl -w net.core.wmem_max=67108864
sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 33554432"
sudo sysctl -w net.ipv4.tcp_wmem="4096 65536 33554432"

# Make persistent in /etc/sysctl.conf

bbcp vs scp vs rsync

Practical Example: Transferring a Database Backup

# Create compressed backup
pg_dump -Fc mydb > /tmp/mydb_backup.dump

# Transfer with 8 streams, progress reporting
bbcp -s 8 -w 4M -P 5 -p /tmp/mydb_backup.dump admin@backup-server:/backups/

# Verify with checksum
bbcp -s 8 -E "%md5" /tmp/mydb_backup.dump admin@backup-server:/backups/

Summary

bbcp is a specialised tool for one job: moving large files fast. When scp and rsync leave bandwidth on the table, bbcp’s parallel streams and tunable TCP parameters can deliver dramatic speed improvements. Install it on both endpoints, tune your stream count and window size for the link, and watch your transfers fly.