Author Topic: Creating a synced backup  (Read 7444 times)

Offline rmcellig

  • Member
  • ***
  • Posts: 207
  • Karma: 14
  • New Forum User
    • View Profile
Creating a synced backup
« on: April 07, 2014, 05:42:18 pm »
This is my current issue I have with digitizing my LP's. At the moment, I am using my Dell 3000 (peppermint4) with Audacity. The files are saved as flac files to the following external USB drive location:

/passport/digitized-lps
In this directory, I have the following directory structure, A B C D....

I manually make backups by copying the files to computers on my LAN (they are all running Linux and shared using Samba) :

/drive1/lps-backup/lps (My wife's PC)
/cdlibrary/lps (USB drive hanging off my iMac)
/sdc1/lps (USB drive hanging off my HP laptop)
/sdb1/lps (USB drive hanging off the same HP laptop)
/passport/lps (USB drive hanging off my Dell 3000)

What would be a better way of making sure that my digitized LP's are being backed up properly without physically dragging the files from one window to another, and how would I go about setting this up? Looking for a simple and practical solution.

Is rsync a viable solution using it recursively with the digitized-lps directory as the source and the samba shares as the destinations?

Not sure how to set all of this up. A friend of mine suggested setting up rsync seperately for each samba destination and then putting all of this in a script, and using the script in something like gnome-schedule to schedule the syncing.

Looking forward as always to the possible solutions I can try out. :)

Thanks!!

Offline emegra

  • Administrator
  • Veteran
  • *****
  • Posts: 1946
  • Karma: 450
  • New Forum User
    • View Profile
  • Peppermint version(s): Peppermint 10 64bit
Re: Creating a synced backup
« Reply #1 on: April 07, 2014, 10:48:01 pm »
Hi rmcellig

You could try Bittorrentsync http://www.bittorrent.com/sync and set up shares on each PC/Laptop, once you have the shares set up they will all automatically sync with each other, shares don't even have to be on the same LAN .

I'm not sure if this is any better than rsync for your purpose but it might be worth having a look




Good luck




Graeme
If you can keep your head while all around are losing theirs then you're not quite grasping the situation

Offline rmcellig

  • Member
  • ***
  • Posts: 207
  • Karma: 14
  • New Forum User
    • View Profile
Re: Creating a synced backup
« Reply #2 on: April 08, 2014, 03:44:21 am »
Thanks! I will look into Bittorrent sync and see if this is what I am looking for. Sounds interesting!

Offline PCNetSpec

  • Administrator
  • Hero
  • *****
  • Posts: 26452
  • Karma: 65531
  • "-rw-rw-rw-" .. The Number Of The Beast
    • View Profile
    • PCNetSpec
  • Peppermint version(s): Peppermint 10
Re: Creating a synced backup
« Reply #3 on: April 08, 2014, 05:26:19 am »
I'd have also used rsync/grsync and cron/gnome-scedule

But then I must admit I don't know much about Bittorrentsync
WARNING: You are logged into reality as 'root' .. logging in as 'insane' is the only safe option.

Team Peppermint
PCNetSpec

Offline rmcellig

  • Member
  • ***
  • Posts: 207
  • Karma: 14
  • New Forum User
    • View Profile
Re: Creating a synced backup
« Reply #4 on: April 08, 2014, 10:27:21 am »
I just installed Bittorrent sync on all my computers. It's very easy to set up. It's now synching the folders I want to sync. Very straight forward.

I could try rsync/grdync again with SSH and see how that goes just so I know how to do it so that I can learn one more thing regarding Linux. :)

Offline rmcellig

  • Member
  • ***
  • Posts: 207
  • Karma: 14
  • New Forum User
    • View Profile
Re: Creating a synced backup
« Reply #5 on: April 09, 2014, 06:23:06 am »
I am trying out rsync and am making headway!! I am trying to set ssh up so that it does not always require a password when I use rsync. Once I get the rsync code the way it should be, I will put it into Gnome-schedule and try it out. At the moment, this is what I am getting. You can see that I can copy the files no problem but I am being asked for a password.

I ran ssh-keygen from my Dell3000 and then ran ssh-copy-id randy@192.168.2.13. That went fine so I tested my connection by doing ssh 192.168.2.13. That went fine until I ran  rsync that I mentioned above.

Am I missing something?

The Source machine is the Dell3000
The Destination machine is the hpdv6

Code: [Select]
randy@pepp3 ~ $ rsync -avr --delete /media/randy/passport/lpslibrary/D randy@192.168.2.13:/media/randy/stuff/D
Agent admitted failure to sign using the key.
randy@192.168.2.13's password:
sending incremental file list
D/
D/3-4-mosaic111-charles-mingus.flac
D/drake-nick - Made to Love Magic.flac
D/drake-nick-bryter-layter.flac
D/drake-nick-five-leaves-left.flac
D/drake-nick-pink-moon.flac
D/drake-nick-time-of-no-reply.flac

sent 1270873935 bytes  received 130 bytes  8674908.29 bytes/sec
total size is 1270718302  speedup is 1.00
randy@pepp3 ~ $

Offline PCNetSpec

  • Administrator
  • Hero
  • *****
  • Posts: 26452
  • Karma: 65531
  • "-rw-rw-rw-" .. The Number Of The Beast
    • View Profile
    • PCNetSpec
  • Peppermint version(s): Peppermint 10
Re: Creating a synced backup
« Reply #6 on: April 09, 2014, 06:33:09 am »
You need to add the SSH key

see
Code: [Select]
man ssh-add
WARNING: You are logged into reality as 'root' .. logging in as 'insane' is the only safe option.

Team Peppermint
PCNetSpec

Offline rmcellig

  • Member
  • ***
  • Posts: 207
  • Karma: 14
  • New Forum User
    • View Profile
Re: Creating a synced backup
« Reply #7 on: April 09, 2014, 06:39:16 am »
I tried that on the dv6. I forgot to mention it in my last post. It did that no problem, but when I try to rsync from the Dell to the DV6, I get prompted for a password. Is there a simple test I can try and then post back?

Offline rmcellig

  • Member
  • ***
  • Posts: 207
  • Karma: 14
  • New Forum User
    • View Profile
Re: Creating a synced backup
« Reply #8 on: April 09, 2014, 07:17:47 am »
I am running a test using Gnome-schedule and so far so good. I think I will use rsync instead of Bittorrent Sync. There seem to be some issues with it. At least rsync is a tried true and tested way to sync folders.

If I have any other issues, I will post back. Once I am happy I will mark this thread as solved. :) Thanks again for the help!!!

Offline rmcellig

  • Member
  • ***
  • Posts: 207
  • Karma: 14
  • New Forum User
    • View Profile
Re: Creating a synced backup
« Reply #9 on: April 13, 2014, 02:21:56 pm »
I am using ssh and rsync to backup my digitized LP flac files. I have the code (listed below) running on my Dell 3000 using Gnome-schedule. I am finding that the Dell is slowing down considerably.

Is this because I am running rsync and ssh?

Should I be running this from the HP DV6 instead? I could copy all the flac files to the internal drive on the DV6. It's a 500GB drive. Would this be faster than the way I am currently doing it on the Dell 3000?

Is this the best and most practical way to backup my FLAC files? What would you recommend? Because this is such a big project with a lot of invested time, I want to make sure that I have multiple backups. I am also considering backing up to Crashplan which I already use.

My HP laptop has a few distros installed each in their own partition. Right now I am booted into Peppermint 4. I could easyily free up some space by deleting some of the partitions (I like to try out different distros :) ).

The Dell 3000 has Crunchbang 11 and Peppermint 4 installed.


/media/drive1/lps is a usb drive hanging off a computer I built a few years ago
/volumes/CDLIBRARY/lpsflacfiles is a usb drive hanging off my 2006 iMac 2GB RAM
/media/lps/flaclps is a usb drive hanging off my HP dv6 laptop Intel(R) Core(TM) i3 CPU       M 350  @ 2.27GHz 3828 MiB RAM
/media/passport/lps/ is a usb drive hanging off my Dell 3000 desktop (Intel(R) Celeron(R) CPU 2.66GHz 2GB RAM)


Code: [Select]
rsync -avr  /media/passport/lps/ randy@192.168.2.12:/media/lps/flaclps

rsync -avr /media/passport/lps/ randymac@192.168.2.10:/volumes/CDLIBRARY/lpsflacfiles

rsync -avr  /media/passport/lps/ nicole@192.168.2.14:/media/drive1/lps

If you require more information, please let me know!! Much appreciated (as always :))

Offline PCNetSpec

  • Administrator
  • Hero
  • *****
  • Posts: 26452
  • Karma: 65531
  • "-rw-rw-rw-" .. The Number Of The Beast
    • View Profile
    • PCNetSpec
  • Peppermint version(s): Peppermint 10
Re: Creating a synced backup
« Reply #10 on: April 14, 2014, 05:51:01 am »
Can you try to explain this a little clearer .. what is prompting you for a password, and when ?

Are you using the the grsync GUI ?



[EDIT]

There are other backup tools .. such as simple backup (sbackup) etc. that can schedule backups themselves.

But I hate trying to give advice on the "best" backup solution .. that's nearly impossible to do without FULLY understanding the network, how it's used, what's considered critical data, size constraints, and the people that may need to use the backups for recovery.
« Last Edit: April 14, 2014, 06:02:48 am by PCNetSpec »
WARNING: You are logged into reality as 'root' .. logging in as 'insane' is the only safe option.

Team Peppermint
PCNetSpec

Offline rmcellig

  • Member
  • ***
  • Posts: 207
  • Karma: 14
  • New Forum User
    • View Profile
Re: Creating a synced backup
« Reply #11 on: April 14, 2014, 06:00:01 am »
I am not being prompted for a password when running rsync over ssh. It seems to work but is really slowing down my computer. It's also taking a long time for the backups to occur to the other computers.

Is it something in the sync code I am using that is not set correctly?

Should I use rsync to backup one sub directory at a time (Folder A, folder B, folder C etc....), instead of backing up the master folder all at once?

I'm really trying to wrap my head around all of this and really appreciate your help!!

Offline PCNetSpec

  • Administrator
  • Hero
  • *****
  • Posts: 26452
  • Karma: 65531
  • "-rw-rw-rw-" .. The Number Of The Beast
    • View Profile
    • PCNetSpec
  • Peppermint version(s): Peppermint 10
Re: Creating a synced backup
« Reply #12 on: April 14, 2014, 06:05:32 am »
The good thing about rsync is it ONLY syncs changes, where other backup solutions might want to use more bandwidth.

BUT

As with any backup solution, the first run WILL be bandwidth intensive .. successive runs will be MUCH quicker.



In an ideal world you'd have a dedicated machine doing all the syncing .. so the only drain on the clients would be I/O (disk reads)
Obviously this isn't really possible in the home, but as I said it's only really the first run of rsync that's intensive .. schedule it for when you're in bed ;)



Hint -

Have you tried using the grsync GUI to test and/or generate your rsync commands ?

If you set up a sync in there, then do a "dry run" .. the actual command it used will be listed at the top of the Rsync output window.
You can then use that from gnome-schedule or the command line if you wish.
« Last Edit: April 14, 2014, 06:18:46 am by PCNetSpec »
WARNING: You are logged into reality as 'root' .. logging in as 'insane' is the only safe option.

Team Peppermint
PCNetSpec

Offline rmcellig

  • Member
  • ***
  • Posts: 207
  • Karma: 14
  • New Forum User
    • View Profile
Re: Creating a synced backup
« Reply #13 on: April 14, 2014, 06:18:47 am »
Great points! I will try them out.

Offline PCNetSpec

  • Administrator
  • Hero
  • *****
  • Posts: 26452
  • Karma: 65531
  • "-rw-rw-rw-" .. The Number Of The Beast
    • View Profile
    • PCNetSpec
  • Peppermint version(s): Peppermint 10
Re: Creating a synced backup
« Reply #14 on: April 14, 2014, 06:20:31 am »
Check the hint in my last reply .. I think I was adding that as you posted.
WARNING: You are logged into reality as 'root' .. logging in as 'insane' is the only safe option.

Team Peppermint
PCNetSpec