使用rsync解决同步问题

发表评论 / 作者： /

缘由

在Windows生态中实现高效文件同步一直是个技术痛点，有各种各样的工具，图形的winscp、命令行的scp、rsync等等工具。

最近有一个项目，因保密的需要而在多个windows电脑中同步代码文件。一开始使用scp工具全量同步，随着进度的推进，同步的效率越来越跟不上，因为scp每次都是全量同步导致同步耗时较长，再加上编译和打包的时候，每次改动需要花20来分钟的准备时间，严重拉慢进度。

这里有一个痛点，其实不需要每次都全量同步文件，而只需要针对发生过变更的文件进行同步即可，而这正是rsync的优势所在。

rsync拥有以下优势

  • 增量传输：仅传输差异部分（Delta），节约75%以上带宽
  • 断点续传：网络中断后可恢复传输进度
  • 加密传输：支持SSH隧道加密
  • 权限保留：完整保留ACL权限属性
  • 跨平台兼容：Linux/macOS/Windows全支持

实测数据：同步2GB开发项目目录（约3万文件），传统复制耗时22分钟，rsync首次同步21分钟，后续变更同步仅需47秒。

部署Rsync

由于rsync是一款原生的linux应用，需要运行在linux环境中，在windows平台上已经有大佬移植了大量的Linux库并集成到msys2中。

正好在windows平台上，git在windows平台上需要运行在msys2之上，因此可以借用git for windows来部署rsync，这会简单很多，而且这个git for windows还是一套开源工具，让windows可以拥有一部分linux工具。

下载Git for windows，并安装到C:/Tools目录，这里只是为了避免目录名中间有空格，原因是-e选项需要指定完整路径的ssh，如果目录名带空格会变成两个参数。

git install

安装完成之后，在开始菜单中就可以找到git bash，以管理员身份打开一个终端窗口，这样就拥有了一个类似linux的bash终端，里面的环境也跟linux环境类似。

管理员身份的bash

安装pacman

msys2官方使用pacman为软件管理器，但在git for windows中并没有集成这个管理器，如果不安装pacman的话，那需要自己到msys2的软件仓库中下载rsync压缩包再解压到git安装目录，这个手工过程，需要连rsync的依赖也安装，分别有xxhash、zstd、rsync。而使用pacman管理安装的话，只需要一行命令：pacman -S rysnc，即可全自动安装，包括所需要的依赖。

在bash终端中执行以下命令，为msys2安装pacman包管理器，可以先上仓库看看最新的版本号，需要管理员权限

cd /
for f in pacman-6.1.0-9-x86_64.pkg.tar.zst pacman-mirrors-20250220-1-any.pkg.tar.zst msys2-keyring-1~20250214-1-any.pkg.tar.zst;   do curl https://repo.msys2.org/msys/x86_64/$f -o ~/Downloads/$f; 
tar x --zstd -vf ~/Downloads/$f usr etc var 
done
mkdir -p /var/lib/pacman
pacman-key --init
pacman-key --populate msys2
pacman -Sy
; 注册一些包，这样后续的安装才不会出错
export URL=https://ghfast.top/https://github.com/git-for-windows/git-sdk-64/raw/main
cat /etc/package-versions.txt | while read p v; do d=/var/lib/pacman/local/$p-$v; mkdir -p $d; echo $d; for f in desc files install mtree; do curl -sSL "$URL$d/$f" -o $d/$f; done; done

安装完成之后的结果

安装rsync

在管理员身份下的bash终端中执行pacman -S rsync cygrunsrv，安装rsync和cygrunsrv，其中的cygrunsrv为系统服务注册器，主要是用来向windows注册rsyncd常驻服务，让windows能通过873端口提供rsync服务

配置rsyncd服务

这是可选项，如不需要在linux配置服务端，则不需要部署这个，因为也可以使用SSH服务来完成服务端的功能，但我是两台Windows之间的同步，并没有使用openssh的服务端，所以也部署了rsyncd

rsync配置

cat > /etc/rsyncd.conf <<EOF
port = 873
uid = 0
gid = 0
use chroot =false
strict modes = false
hosts allow = *
log file = /d/rsyncd.log
secrets file = /etc/rsyncd.secrets  #密码文件
 
[syncdata]
path = /e/projects/code
read only = no
auth users = sync
EOF

密码文件，格式：username:passwd，自行加密

cat > /etc/rsyncd.secrets
sync:123456

Rsync服务注册，并配置为开机启动

# 注册
cygrunsrv.exe -I "Rsync" -p /bin/rsync.exe -a "--config=/etc/rsyncd.conf --daemon --no-detach" -f "Rsync daemon service" --desc "Starts a rsync daemon for accepting incoming rsync connections"  --disp "Rsync Daemon" --type auto
# 启动服务
net start Rsync
# 放开防火墙
netsh advfirewall firewall add rule name="rsync service 873" dir=in action=allow protocol=TCP localport=873
# 或者
netsh firewall add portopening TCP 873 "rsync service 873"

服务验证netstat -ano | findstr 873，有以下端口在监听就表示服务启动成功

同步应用

6种运行模式

  1. 拷贝本地文件；当 SRC 和 DEST 路径信息都不包含有单个冒号”:”分隔符时就启动这种工作模式。
  2. 使用一个远程 shell 程序（如 rsh、ssh）来实现将本地机器的内容拷贝到远程机器。当 DEST 路径地址包含单个冒号”:”分隔符时启动该模式。
  3. 使用一个远程 shell 程序（如 rsh、ssh）来实现将远程机器的内容拷贝到本地机器。当 SRC 地址路径包含单个冒号”:”分隔符时启动该模式。
  4. 从远程 rsync 服务器中拷贝文件到本地机。当 SRC 路径信息包含”::”分隔符时启动该模式。
  5. 从本地机器拷贝文件到远程 rsync 服务器。当 DEST 路径信息包含”::”分隔符时启动该模式。
  6. 列远程机的文件列表。这类似于 rsync 传输，不过只要在命令中省略掉本地机信息即可

个人应用

在A机安装了rsyncd，成功监听873端口，并在/etc/rsyncd.conf中配置了所需要同步的目录，如syncdata，那就可以使用以下的方式来运行同步，首次同步为全量同步，后续都是增量同步。

A=>B，并在B机运行

export RSYNC_PASSWORD="123456"
rsync -zav --del --exclude=.svn --exclude=.cvs --exclude=.idea --exclude=.DS_Store --exclude=.hg --exclude=*.hprof --exclude=*.pyc sync@192.168.5.5::syncdata ./code

B=>A，即把变更推送为服务端，在B机运行

export RSYNC_PASSWORD="123456"
rsync -zav --del --exclude=.svn --exclude=target --exclude=.cvs --exclude=.idea --exclude=.hg --exclude=*.hprof --exclude=*.pyc ./code/* sync@192.168.5.5::syncdata

两类选项都加–del，属于会在接收侧删除不存在的文件，即镜像模式

列举远程服务信息

如在B机执行：sync -v rsync://remoteip /www

wisfern@Dev10 MINGW64 /d/work
$ rsync -v rsync://192.168.5.5
syncdata

指定远程shell

可以使用-e选项来指向执行的shell，如ssh

rsync -e "ssh -p 22 -i ~/.ssh/id25519" xx其x他x选项....xxx

#注意
#这里只能使用同体系下的rsync和ssh，如都是linux，不然会报以下的错误，应该是响应码不匹配的原因
# rsync -rvtz -e ssh test wisfern@192.168.1.46::OnWorking/test
rsync: did not see server greeting
rsync error: error starting client-server protocol (code 5) at main.c(1814) [sender=3.2.3]
      0 [sig] rsync 1791! sigpacket::process: Suppressing signal 30 to win32 process (pid 31888)
#解决：排查是否同一套体系
#where rsync
#where ssh

#解决
C:\Tools\Git\usr\bin\rsync -rvtz -e "C:\Tools\Git\usr\bin\ssh.exe"

常用选项说明

-z   压缩传输，节省网络带宽
-a   归档模式，目录、文件、权限、归属、时间
-v   详细输出
-q   少输出模式
-e   指定连接远程的shell
--process  输出文件的处理进度，针对大文件需要长时间传输有用
--exclude= 忽略指定目录或文件，一部分匹配就命中
--exclude-from=   用于都是进行忽略的匹配列表文件
--partial  断点续传
-P   等于--partial --process
--password-file=  密码文件，只能是600权限，但在windows，无法设置为600权限，因此密码文件使用不了

问题：无法使用–password-file

在windows平台下，无法使用--password-file指令

$ rsync -av sync@10.3.18.143::syncdata --password-file=/etc/rsyncd.secrets
ERROR: password file must not be other-accessible
rsync error: syntax or usage error (code 1) at authenticate.c(197) [Receiver=3.2.3]

附属

MSYS

当 MinGW 出现之后，程序员们发现光有 GCC 不行啊，面对着如此 naive 的 Windows 命令行也照样不能干活，Cygwin 又太大了。所以 MinGW 的一群人再次 fork 了 Cygwin，把它精简了一番，就成了 MSYS。后来，MinGW 咕咕咕了，MSYS 也随之咕咕咕了，又一群程序员 fork 了新版本的 Cygwin，精简了一番，给它加上了 Pacman 包管理器（没错就是 Arch Linux 用的那个），最后给 MSYS 的名字后面加了个 2，成了 MSYS2。

Git for Windows

随着 Git 的快速完善，将 Git 移植到 Windows 的呼声越来越高，可是 Git 需要很多 Linux 下的命令行工具，所以不想重复造轮子的 Git 开发者们找到了 MSYS2，又将 MSYS2 做了一番精简和针对 Git 的修改，把 Git 放了进去。

手工快速安装rsync

curl -L https://repo.msys2.org/msys/x86_64/rsync-3.2.7-2-x86_64.pkg.tar.zst --output xxx
tar -I zstd -xvf xxx
cp usr/bin/rsync.exe 'c:\Tools\Git\usr\bin\'
rm -r * .*

curl -L https://repo.msys2.org/msys/x86_64/libzstd-1.5.5-1-x86_64.pkg.tar.zst --output xxx
tar -I zstd -xvf xxx
cp usr/bin/msys-zstd-1.dll 'c:\Tools\Git\usr\bin\'
rm -r * .*

curl -L https://repo.msys2.org/msys/x86_64/libxxhash-0.8.1-1-x86_64.pkg.tar.zst --output xxx
tar -I zstd -xvf xxx
cp usr/bin/msys-xxhash-0.dll 'c:\Tools\Git\usr\bin\'
rm -r * .*

curl -L https://repo.msys2.org/msys/x86_64/liblz4-1.9.4-1-x86_64.pkg.tar.zst --output xxx
tar -I zstd -xvf xxx
cp usr/bin/msys-lz4-1.dll 'c:\Tools\Git\usr\bin\'

curl -L https://repo.msys2.org/msys/x86_64/libopenssl-3.1.1-1-x86_64.pkg.tar.zst --output xxx
tar -I zstd -xvf xxx
cp usr/bin/msys-crypto-3.dll 'c:\Program Files\Git\usr\bin\'

Rsync选项说明

$ rsync -h
rsync  version 3.4.1  protocol version 32
Copyright (C) 1996-2025 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, symlinks, symtimes, hardlinks, no hardlink-specials,
    no hardlink-symlinks, IPv6, atimes, batchfiles, inplace, append, ACLs,
    xattrs, optional secluded-args, iconv, prealloc, stop-at, crtimes
Optimizations:
    no SIMD-roll, no asm-roll, openssl-crypto, no asm-MD5
Checksum list:
    xxh128 xxh3 xxh64 (xxhash) md5 md4 sha1 none
Compress list:
    zstd lz4 zlibx zlib none
Daemon auth list:
    sha512 sha256 sha1 md5 md4

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

rsync is a file transfer program capable of efficient remote update
via a fast differencing algorithm.

Usage: rsync [OPTION]... SRC [SRC]... DEST
  or   rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST
  or   rsync [OPTION]... SRC [SRC]... [USER@]HOST::DEST
  or   rsync [OPTION]... SRC [SRC]... rsync://[USER@]HOST[:PORT]/DEST
  or   rsync [OPTION]... [USER@]HOST:SRC [DEST]
  or   rsync [OPTION]... [USER@]HOST::SRC [DEST]
  or   rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST]
The ':' usages connect via remote shell, while '::' & 'rsync://' usages connect
to an rsync daemon, and require SRC or DEST to start with a module name.

Options
--verbose, -v            increase verbosity
--info=FLAGS             fine-grained informational verbosity
--debug=FLAGS            fine-grained debug verbosity
--stderr=e|a|c           change stderr output mode (default: errors)
--quiet, -q              suppress non-error messages
--no-motd                suppress daemon-mode MOTD
--checksum, -c           skip based on checksum, not mod-time & size
--archive, -a            archive mode is -rlptgoD (no -A,-X,-U,-N,-H)
--no-OPTION              turn off an implied OPTION (e.g. --no-D)
--recursive, -r          recurse into directories
--relative, -R           use relative path names
--no-implied-dirs        don't send implied dirs with --relative
--backup, -b             make backups (see --suffix & --backup-dir)
--backup-dir=DIR         make backups into hierarchy based in DIR
--suffix=SUFFIX          backup suffix (default ~ w/o --backup-dir)
--update, -u             skip files that are newer on the receiver
--inplace                update destination files in-place
--append                 append data onto shorter files
--append-verify          --append w/old data in file checksum
--dirs, -d               transfer directories without recursing
--old-dirs, --old-d      works like --dirs when talking to old rsync
--mkpath                 create destination's missing path components
--links, -l              copy symlinks as symlinks
--copy-links, -L         transform symlink into referent file/dir
--copy-unsafe-links      only "unsafe" symlinks are transformed
--safe-links             ignore symlinks that point outside the tree
--munge-links            munge symlinks to make them safe & unusable
--copy-dirlinks, -k      transform symlink to dir into referent dir
--keep-dirlinks, -K      treat symlinked dir on receiver as dir
--hard-links, -H         preserve hard links
--perms, -p              preserve permissions
--executability, -E      preserve executability
--chmod=CHMOD            affect file and/or directory permissions
--acls, -A               preserve ACLs (implies --perms)
--xattrs, -X             preserve extended attributes
--owner, -o              preserve owner (super-user only)
--group, -g              preserve group
--devices                preserve device files (super-user only)
--copy-devices           copy device contents as a regular file
--write-devices          write to devices as files (implies --inplace)
--specials               preserve special files
-D                       same as --devices --specials
--times, -t              preserve modification times
--atimes, -U             preserve access (use) times
--open-noatime           avoid changing the atime on opened files
--crtimes, -N            preserve create times (newness)
--omit-dir-times, -O     omit directories from --times
--omit-link-times, -J    omit symlinks from --times
--super                  receiver attempts super-user activities
--fake-super             store/recover privileged attrs using xattrs
--sparse, -S             turn sequences of nulls into sparse blocks
--preallocate            allocate dest files before writing them
--dry-run, -n            perform a trial run with no changes made
--whole-file, -W         copy files whole (w/o delta-xfer algorithm)
--checksum-choice=STR    choose the checksum algorithm (aka --cc)
--one-file-system, -x    don't cross filesystem boundaries
--block-size=SIZE, -B    force a fixed checksum block-size
--rsh=COMMAND, -e        specify the remote shell to use
--rsync-path=PROGRAM     specify the rsync to run on remote machine
--existing               skip creating new files on receiver
--ignore-existing        skip updating files that exist on receiver
--remove-source-files    sender removes synchronized files (non-dir)
--del                    an alias for --delete-during
--delete                 delete extraneous files from dest dirs
--delete-before          receiver deletes before xfer, not during
--delete-during          receiver deletes during the transfer
--delete-delay           find deletions during, delete after
--delete-after           receiver deletes after transfer, not during
--delete-excluded        also delete excluded files from dest dirs
--ignore-missing-args    ignore missing source args without error
--delete-missing-args    delete missing source args from destination
--ignore-errors          delete even if there are I/O errors
--force                  force deletion of dirs even if not empty
--max-delete=NUM         don't delete more than NUM files
--max-size=SIZE          don't transfer any file larger than SIZE
--min-size=SIZE          don't transfer any file smaller than SIZE
--max-alloc=SIZE         change a limit relating to memory alloc
--partial                keep partially transferred files
--partial-dir=DIR        put a partially transferred file into DIR
--delay-updates          put all updated files into place at end
--prune-empty-dirs, -m   prune empty directory chains from file-list
--numeric-ids            don't map uid/gid values by user/group name
--usermap=STRING         custom username mapping
--groupmap=STRING        custom groupname mapping
--chown=USER:GROUP       simple username/groupname mapping
--timeout=SECONDS        set I/O timeout in seconds
--contimeout=SECONDS     set daemon connection timeout in seconds
--ignore-times, -I       don't skip files that match size and time
--size-only              skip files that match in size
--modify-window=NUM, -@  set the accuracy for mod-time comparisons
--temp-dir=DIR, -T       create temporary files in directory DIR
--fuzzy, -y              find similar file for basis if no dest file
--compare-dest=DIR       also compare destination files relative to DIR
--copy-dest=DIR          ... and include copies of unchanged files
--link-dest=DIR          hardlink to files in DIR when unchanged
--compress, -z           compress file data during the transfer
--compress-choice=STR    choose the compression algorithm (aka --zc)
--compress-level=NUM     explicitly set compression level (aka --zl)
--skip-compress=LIST     skip compressing files with suffix in LIST
--cvs-exclude, -C        auto-ignore files in the same way CVS does
--filter=RULE, -f        add a file-filtering RULE
-F                       same as --filter='dir-merge /.rsync-filter'
                         repeated: --filter='- .rsync-filter'
--exclude=PATTERN        exclude files matching PATTERN
--exclude-from=FILE      read exclude patterns from FILE
--include=PATTERN        don't exclude files matching PATTERN
--include-from=FILE      read include patterns from FILE
--files-from=FILE        read list of source-file names from FILE
--from0, -0              all *-from/filter files are delimited by 0s
--old-args               disable the modern arg-protection idiom
--secluded-args, -s      use the protocol to safely send the args
--trust-sender           trust the remote sender's file list
--copy-as=USER[:GROUP]   specify user & optional group for the copy
--address=ADDRESS        bind address for outgoing socket to daemon
--port=PORT              specify double-colon alternate port number
--sockopts=OPTIONS       specify custom TCP options
--blocking-io            use blocking I/O for the remote shell
--outbuf=N|L|B           set out buffering to None, Line, or Block
--stats                  give some file-transfer stats
--8-bit-output, -8       leave high-bit chars unescaped in output
--human-readable, -h     output numbers in a human-readable format
--progress               show progress during transfer
-P                       same as --partial --progress
--itemize-changes, -i    output a change-summary for all updates
--remote-option=OPT, -M  send OPTION to the remote side only
--out-format=FORMAT      output updates using the specified FORMAT
--log-file=FILE          log what we're doing to the specified FILE
--log-file-format=FMT    log updates using the specified FMT
--password-file=FILE     read daemon-access password from FILE
--early-input=FILE       use FILE for daemon's early exec input
--list-only              list the files instead of copying them
--bwlimit=RATE           limit socket I/O bandwidth
--stop-after=MINS        Stop rsync after MINS minutes have elapsed
--stop-at=y-m-dTh:m      Stop rsync at the specified point in time
--fsync                  fsync every written file
--write-batch=FILE       write a batched update to FILE
--only-write-batch=FILE  like --write-batch but w/o updating dest
--read-batch=FILE        read a batched update from FILE
--protocol=NUM           force an older protocol version to be used
--iconv=CONVERT_SPEC     request charset conversion of filenames
--checksum-seed=NUM      set block/file checksum seed (advanced)
--ipv4, -4               prefer IPv4
--ipv6, -6               prefer IPv6
--version, -V            print the version + other info and exit
--help, -h (*)           show this help (* -h is help only on its own)

Use "rsync --daemon --help" to see the daemon-mode command-line options.
Please see the rsync(1) and rsyncd.conf(5) manpages for full documentation.
See https://rsync.samba.org/ for updates, bug reports, and answers

发表评论

您的电子邮箱地址不会被公开。 必填项已用*标注

Scroll to Top