DSM-G600, DNS-3xx and NSA-220 Hack Forum

Unfortunately no one can be told what fun_plug is - you have to see it for yourself.

You are not logged in.

#1 2008-03-19 13:49:53

groovyjames
Member
Registered: 2008-03-19
Posts: 6

BackupNetClone - Speed Issues

Hello everyone!

I've just got myself a DNS-323 and have been spending the last week taking in as much information as possible from this site - what a fantastic resource!

Fun_plug 1.04 has been installed, and now I've been trying to get automated backups working with BackupNetClone.  I just want to backup disk 1 to disk 2.  After a bit of trial-and-error, I've managed to get BackupNetClone to do something - however it seems VERY slow, and I'm not entirely sure what it's doing.  The harddrives are spinning, and my data appears to be being copied to disk 2 - but as I said, it's very slow.  For the initial transfer I need to move around 300GB of data from disk 1 to disk 2 - the process started around 35 hours ago, but according to Windows Explorer, only around 40GB has been transfered. 

It would be great if someone could answer a few questions I have:

1). Is this sort of speed normal?
2). Is it because of ssh?  Is it possible to "switch off" ssh withing BackupNetClone when transfering locally inside the DNS?
3). Is it okay to have mapped drives, open explorer windows with the DNS when syncing?
4). Does Telnet have to be continually connect during the sync process?
5). How can I stop the process?

Many thanks for your help, and for the great wiki and forum,

James

Offline

 

#2 2008-03-19 17:24:52

blbrown
Member
Registered: 2007-11-02
Posts: 88
Website

Re: BackupNetClone - Speed Issues

groovyjames wrote:

Hello everyone!

I've just got myself a DNS-323 and have been spending the last week taking in as much information as possible from this site - what a fantastic resource!

Fun_plug 1.04 has been installed, and now I've been trying to get automated backups working with BackupNetClone.  I just want to backup disk 1 to disk 2.  After a bit of trial-and-error, I've managed to get BackupNetClone to do something - however it seems VERY slow, and I'm not entirely sure what it's doing.  The harddrives are spinning, and my data appears to be being copied to disk 2 - but as I said, it's very slow.  For the initial transfer I need to move around 300GB of data from disk 1 to disk 2 - the process started around 35 hours ago, but according to Windows Explorer, only around 40GB has been transfered. 

It would be great if someone could answer a few questions I have:

1). Is this sort of speed normal?
2). Is it because of ssh?  Is it possible to "switch off" ssh withing BackupNetClone when transfering locally inside the DNS?
3). Is it okay to have mapped drives, open explorer windows with the DNS when syncing?
4). Does Telnet have to be continually connect during the sync process?
5). How can I stop the process?

Many thanks for your help, and for the great wiki and forum,

James

Ok, here's a couple thoughts:

- to check on what BNC is doing at any particular moment, do a 'tail backupnetclone.log' from the BackupNetClone/logs/ directory
- my guess is that BNC is in the middle of doing a full rsync of the data, which only happens once
- unfortunately, the speed you're getting sounds about right--this slow speed, as you guessed, is due to the fact that BNC sends all data through an SSH tunnel, even when the connection is local (from one DNS-323 disk to the other); due to the secure nature of SSH it is rather processor-intensive, which keeps the DNS-323 churning on encrypting and decrypting the data throughout the simple transfer from one disk to the other; in a future version I might try to add the feature you suggest where it skips/bypasses the SSH tunnel if performing local or even within the same network
- mapping drives to either drive is just fine while rsync is running; any file changes made will be resynced during the next BNC session
- whatever shell you used to start BNC (by running start_here.sh) does need to remain open; however, if BNC was started from cron, then you don't need any session open
- you can stop BNC in a few ways:
   a) stop the telnet session that's running start_here.sh (quick and dirty)
   b) use a telnet session to kill the start_here.sh or rsync process (not really necessary)
   c) open another telnet session and issue the command 'sh interrupt_transfer.sh' from the BackupNetClone/ directory (cleanest way)
- BNC will clean up after itself no matter which way you stop it, but I still recommend option C above
- for the initial transfer of your 300GB quickly, I recommend you stop the current transfer and do a 'cp' of all of your data from the source drive to the latest target directory; respond back here if you need help with that

I'm glad you got BNC running--it's no small task!  I hope I can help any way possible, and want to answer any questions I can so you can successfully use it.

Offline

 

#3 2008-03-20 10:31:39

groovyjames
Member
Registered: 2008-03-19
Posts: 6

Re: BackupNetClone - Speed Issues

Wow, what a difference that made! 

I killed the transfer and did a 'cp' which copied the data in a few hours.  I then ran BNC which started to do a full backup again yikes(  I figured out that this was because of the lack of a 'last-rsync' file because I didn't end the process properly.  I quickly ran 'interrupt_transfer.sh' which stopped the transfer and created the 'last-rsync' file.  The file needed to be updated with the correct directory, but after that BNC ran completely through in about an hour or two yikes)

I'm now trying to get the email reports working.  I'm having a problem pinging a server - it's pretty weird because pinging smtp.google.com works, but smtpout.secureserver.net (for godaddy) doesn't (although it does from the windows command prompt ?!?).  Is there anything in the DNS which could be stopping certain pings getting through? 

Thanks for your great little program and the support.  I found the instructions actually pretty easy to follow and very complete if you have a bit of patience and common sense.  I managed it without knowing literally more than 'cd', 'mv', 'cp', 'ls', and of course 'vi'

Offline

 

#4 2008-03-20 15:51:28

blbrown
Member
Registered: 2007-11-02
Posts: 88
Website

Re: BackupNetClone - Speed Issues

Glad to hear you figured things out!  I hope that means others have done similar, since I haven't had much feedback about how successful it has been for anyone.

Let me know how the email thing goes.  I could only get mine to work with a 'dumb' SMTP server that only does basic authentication.  It's my goal to someday get mailx compiled with openSSL support so that I can use SMTP servers that require SSL authentication (which most newer ones do, such as Yahoo, gmail, and maybe even your godaddy server).

In the meantime I'm working on auto-cleanup where BNC will remove old directories as the disk gets full.  I've been sort of pushed to finally add this feature since my snapshots go back to October 2007, and the disk is finally getting to 90% full...

Offline

 

#5 2008-07-03 11:47:58

groovyjames
Member
Registered: 2008-03-19
Posts: 6

Re: BackupNetClone - Speed Issues

Success!

I thought I'd give the email thing another go last night.  I couldn't find an email service just using SMTP - the most basic ones all used SMTP-AUTH. 

At first I still couldn't get the emails to get sent - pings to the SMTP server were okay, but I kept getting disconnected.  I looked at the script to see where the problem could be. 

Now, I didn't know anything about HELO and EHLO, but a bit of googling made me thing this could be where the problem was.  I changed the script in sendmail.sh to try to send the mail even if the wrong response came back from the server (very dirty, I know, but I just changed SHALLCONTINUE in line 70 to yes.  I tried it out, and it worked! The only problem was that I was getting the email multiple times as the script kept sending it because it thought it hadn't gotten through.  Next step I took to resolve this will probably make you cringe - I deleted all the if statements in lines 116 to 148 which check the status and resend if necessary, but kept the EMAILSENDOK="yes" in line 119 to make sure the other scripts ran okay.

Now the email status reports are working just fine - they make a great product even better.  Now I can see what BNC has been up to, and almost more importantly to me, see how much space is being used.

Thanks!

Offline

 

#6 2008-07-03 17:45:09

blbrown
Member
Registered: 2007-11-02
Posts: 88
Website

Re: BackupNetClone - Speed Issues

Good!  I'm just impressed that you went in and modified things to make it work.  As long as it's stable for you, I think it's great.  I hope the code made some sense for you, since I didn't comment it very well.

For line 70 and the other if statements, you might try looking at the backupnetclone.log and then modify the "Setting up send & response list" lines (in RESPONSEMATRIX variable) of complete_email.sh.  This variable essentially lists the send and receive order that BNC expects for the SMTP server, and you might be able to change sendmail.sh back to normal by just making this simple modification in complete_email.sh.

Or since it works for you now, you could just leave it...  You might run into some issues in upgrading to newer versions of the scripts though.

Offline

 

Board footer

Powered by PunBB
© Copyright 2002–2010 PunBB