2

I am trying to keep 3 large directories (9G, 400G, 800G) in sync between our home site and another in a land far, far away across a network link that is a bit dodgy (slow and drops occasionally). Data was copied onto disks prior to installation so the rsync only needs to send updates. The problem I'm having is the rsync hangs for hours on the client side. The smaller 9G job completed, the 400G job has been in limbo for 15 hours - no output to the log file in that time, but has not timed out.

What I've done to setup for this (after reading many forum articles about rsync/rsync server/partial since I am not really a system admin)

I setup rsync server (/etc/rsyncd.conf) on our home system, entred it into xinetd and wrote a script to run rsync on the distant server, it loops if rsync fails in an attempt to deal with the dodgy network. The rsync command in the script looks like this:

rsync -avzAXP --append root@homesys01::tools /disk1/tools

Note the "-P" option is equivalent to "--progress --partial"

I can see in the log file that rsync did fail at one point and the loop restarted rsync, data was transferred after that based on entries in the log file, but the last update to the log file was 15 hours ago, and the rsync process on the client is still running.

   CNT=0
   while [ 1 ]
   do
       rsync -avzAXP --append root@homesys01::tools /disk1/tools
       STATUS=$?
       if [ $STATUS -eq 0 ] ; then
           echo "Successful completion of tools rsync." 
           exit 0
       else
           CNT=`expr ${CNT} + 1`
           echo " Rsync of tools failure. Status returned: ${STATUS}"
           echo "   Backing off and retrying(${CNT})..."
           sleep 180
       fi
   done

So I expected these jobs to take a long time, I expected to see the occasional failure message in the log files (which I have) and to see rsync restart (which it has). Was not expecting rsync to just hang for 15 hours or more with no progress and no timeout error.

Is there a way to tell if rsync on the client is hung versus dealing with the dodgy network?

I set no timeout in the /etc/rsyncd.conf file. Should I and how do I determin a reasonable timeout setting?

I set rsync up to be available through xinetd, but don't always see the "rsync --daemon" process running. It restarts if I run rsync from the remote system. But shouldn't it be always running?

Any guidance or suggestions would be appreciated.

brontolo
  • 39
  • 5

2 Answers2

1

to tell the rsync client working status , with verbose option and keep a log file change this line

 rsync -avzAXP --append root@homesys01::tools /disk1/tools

to 

 rsync -avzAXP --append root@homesys01::tools /disk1/tools  >>/tmp/rsync.log.`date +%F`

this would produce one log file per day under /tmp directory then you can use tail -f command to trace the most recent log file , if it is rolling , it is working

see also rsync - what means the f+++++++++ on rsync logs? to understand more about the log

James Li
  • 469
  • 3
  • 7
  • Thanks for the tail -f suggestion, it does facilitate monitoring. The wrapper script redirects to a log file and there is an echo in the loop to log if it had to restart rsync. – brontolo Aug 29 '19 at 18:45
1

I thought I would post my final solution, in case it can help anyone else. I added --timeout 300 and --append-verify. The timeout eliminates the case of rsync getting hung indefinitely, the loop will restart it after the timeout. The append-verify is necessary to have it check any partial file it updated.

Note the following code is in a shell script and the output is redirected to a log file.

CNT=0
while [ 1 ]
do
    rsync -avzAXP --append-verify --timeout 300  root@homesys01::tools /disk1/tools
    STATUS=$?
    if [ $STATUS -eq 0 ] ; then
        echo "Successful completion of tools rsync." 
        exit 0
    else
        CNT=`expr ${CNT} + 1`
        echo " Rsync of tools failure. Status returned: ${STATUS}"
        echo "   Backing off and retrying(${CNT})..."
        sleep 180
    fi
done
brontolo
  • 39
  • 5