Peppermint OS Community Forum

General => GNU/Linux Discussion => Topic started by: alynur on March 19, 2019, 09:52:43 pm

Title: Cloning Hard drives refresher course
Post by: alynur on March 19, 2019, 09:52:43 pm
Hello Peppermint people. I just want to be sure I'm going to do this right. I currently have a 250gb UEFI hard drive installed in my desktop.  I am going to replace it with a 500gb Hard drive. I'm planning on installing the new hard drive where the optical drive is, or at least use it's sata and power cables for this operation. So first of all, while in Peppermint, I'll run gparted and create  a GPT partition table in the new hard drive, or is that even necessary? I should then get a  disc ID such as sdb.  According to what I've found, I can then reboot into a live session and run in the terminal either
 
Code: [Select]
dd if=/dev/sda of=/dev/sdb bs=64k
  or
 
Code: [Select]
sudo ddrescue -v /dev/sda /dev/sdb
   of course I'll have to ensure ddrescue is installed for this method.
How am I doing, is this correct? Am I missing anything? This is my first time doing something like this so I just want a little assurance. Once I'm done, I'll take the 250gb hard drive and put it into my laptop and I'll have SSD's in both my computers. Thanks guys.   :)
Title: Re: Cloning Hard drives refresher course
Post by: pin on March 20, 2019, 01:06:53 am
Hi alynur!
There might be other tools to do this, but yes, dd should work.
There're a few things, though...

-Back-up your data!
-The target  drive should be larger than the origin drive. This seems to be your case.
-Extra space on the target will be unused space. You can use gparted from a live-usb to fix that afterwards.
-Why bother with partions? dd will overwrite everything!
-Why using a block size of 64k? Any reason for this? 1M will do it!
-Be aware that it will take some time. I nearly always, use dd to create my bootable usbs, in these cases 1GB takes about 3 to 5min. Copying 250GB will take a loooong time.
-You might want to add conv=fsync  to the command to flush data to disk after the write.

NOTE: I've never used dd to write such a large amount of data, so I have no clue how this will work and, above all, how long it will take!

Wouldn't it be easier/faster to, back-up your data and config files and, do a fresh install?
Title: Re: Cloning Hard drives refresher course
Post by: murraymint on March 20, 2019, 05:20:10 am
I'm planning on installing the new hard drive where the optical drive is, or at least use it's sata and power cables for this operation.

Have you checked that the power cable fits? I remember the optical drives having a different connector. You might have to use a splitter cable for the SATA power to both HDs.

Title: Re: Cloning Hard drives refresher course
Post by: PCNetSpec on March 20, 2019, 07:08:59 am
With the caveats pin mentions, the 'dd' command looks okay to me (though it'll likely take ages) .. you just have to be 100% SURE you get the input file (if=), and output file (of=) correct .. mix em up and you're headed for disaster.

bite size (bs=) is less important, it's just a measure of how much is read from the input file before being written to the output file .. though larger numbers here can save considerable time, personally I usually use bs=1M (for a 1MB bite size)
Title: Re: Cloning Hard drives refresher course
Post by: rayzer on March 20, 2019, 01:25:35 pm
you might be able to use gnome disk utility to create a restore image which you can then restore to the new drive
Title: Re: Cloning Hard drives refresher course
Post by: murraymint on March 20, 2019, 01:30:03 pm
I should then get a  disc ID such as sdb.  According to what I've found, I can then reboot into a live session

Be very careful with this. If you reboot into a live session the "disc IDs" can change so you really need to check them from within that live session.
Title: Re: Cloning Hard drives refresher course
Post by: alynur on March 20, 2019, 04:02:56 pm
Hi alynur!
There might be other tools to do this, but yes, dd should work.
There're a few things, though...

-Why using a block size of 64k? Any reason for this? 1M will do it!

Wouldn't it be easier/faster to, back-up your data and config files and, do a fresh install?

Hi pin, the 64k came from the site I was reading, they had mentioned that without any reference it would be even slower. So bs=1M it is. I try to avoid reinstalls because I never end up with what I had and I don't remember everything I've added to my system. I'm not too worried about the time it takes, I can set it up in the evening and let it copy away all night long(now I've got Lionel Richie on my mind).
So are you saying I don't even need to create the GPT partition table because the cloning will make the hard drive as such? I guess it would, huh?

I'm planning on installing the new hard drive where the optical drive is, or at least use it's sata and power cables for this operation.

Have you checked that the power cable fits? I remember the optical drives having a different connector. You might have to use a splitter cable for the SATA power to both HDs.



Hi murraymint, I have hooked up a second hard drive in the past trying to remember what was on the hard drive, so yes I'm certain this will work.

I should then get a  disc ID such as sdb.  According to what I've found, I can then reboot into a live session

Be very careful with this. If you reboot into a live session the "disc IDs" can change so you really need to check them from within that live session.


I get it, this is probably the most important thing I have to be aware of. Thanks for the additional precautions. Oh, and while I got you, is it necessary to boot into a live session to do this?

I should get the new SSD tomorrow so Saturday it should be all done.
Title: Re: Cloning Hard drives refresher course
Post by: alynur on March 20, 2019, 04:10:55 pm
Forgot to ask
Hi alynur!
There might be other tools to do this, but yes, dd should work.
There're a few things, though...

-You might want to add conv=fsync  to the command to flush data to disk after the write.


Should the new command read
Code: [Select]
dd if=/dev/sda of=/dev/sdx bs=1M conv=fsync
  ?
Title: Re: Cloning Hard drives refresher course
Post by: alynur on March 20, 2019, 04:26:39 pm
With the caveats pin mentions, the 'dd' command looks okay to me (though it'll likely take ages) .. you just have to be 100% SURE you get the input file (if=), and output file (of=) correct .. mix em up and you're headed for disaster.

bite size (bs=) is less important, it's just a measure of how much is read from the input file before being written to the output file .. though larger numbers here can save considerable time, personally I usually use bs=1M (for a 1MB bite size)

Hi PCNetSpec, how big of a bite size can I use? Of the 250 gb hard drive, about 120gb is used and 25gb is unallocated.
Title: Re: Cloning Hard drives refresher course
Post by: murraymint on March 20, 2019, 04:57:08 pm
is it necessary to boot into a live session to do this?

dd will still work if you're booted to your usual drive but things may change during the copy so it's not advisable.

You could alternatively do the whole thing graphically instead by using GParted, in which case you would have to create the GPT partition table and then copy the partition, then resize it if you like.
Title: Re: Cloning Hard drives refresher course
Post by: PCNetSpec on March 20, 2019, 05:22:47 pm
Forgot to ask
Hi alynur!
There might be other tools to do this, but yes, dd should work.
There're a few things, though...

-You might want to add conv=fsync  to the command to flush data to disk after the write.


Should the new command read
Code: [Select]
dd if=/dev/sda of=/dev/sdx bs=1M conv=fsync
  ?

That looks good to me as long as sda is the source and sdx is the target .. but as murraymint suggests, check these in the live session, don't assume they'll be the same as they were when booted to the installed OS.

Quote
Hi PCNetSpec, how big of a bite size can I use? Of the 250 gb hard drive, about 120gb is used and 25gb is unallocated.

Theoretically as big as the amount of free RAM you have, but probably best to stick to 1M .. you'll likely see diminishing speed returns above this.
Title: Re: Cloning Hard drives refresher course
Post by: pin on March 20, 2019, 05:49:59 pm
Back-up!!
Then give it a go  :)
Expert 20+h, dd will copy 'empty space' as well  :D
Your command looks good. Be sure you don't have a power loss in the middle of it  ;)

Psst,... once I did
Code: [Select]
sudo dd if=/dev/zero of=/dev/sda bs=1M
It took 22h  ;)
sda was 250GB...

WARNING: Don't issue the command above!!!
Title: Re: Cloning Hard drives refresher course
Post by: alynur on March 20, 2019, 06:16:40 pm
Back-up!!
Then give it a go  :)
Expert 20+h, dd will copy 'empty space' as well  :D
Your command looks good. Be sure you don't have a power loss in the middle of it  ;)

Psst,... once I did
Code: [Select]
sudo dd if=/dev/zero of=/dev/sda bs=1M
It took 22h  ;)
sda was 250GB...


WARNING: Don't issue the command above!!!


Uh oh, that's something I forgot about, power outages. This place has plenty of power outages. What would be a quicker method? How much space on a memory would be needed to create an iso image of 25gb, 12gb and 120gb?
Title: Re: Cloning Hard drives refresher course
Post by: PCNetSpec on March 20, 2019, 06:34:30 pm
Quote
What would be a quicker method? How much space on a memory would be needed to create an iso image of 25gb, 12gb and 120gb?

I'm not understanding your question .. are you talking about making backups of partitions with something like Gnome Disks ?

If so that's unanswerable .. it depends on how much of those partitions are full.

AFAIK Gnome Disks creates file by file backups (so will be as big as the used space), as opposed to sector by sector backups (which will be as big as the partitions).

[EDIT]

Theoretically, a power outage during a 'dd' operation should be safe for the data on the source drive, as it's not getting modified (written to) .. nothing is ever 100% safe, a power outage can always do damage, but there should theoretically be no more risk to the data on the source drive than a power outage any other time.

The data on the target would likely be toast corrupt because the cached data wouldn't have time to be written, but then you'd have to start again anyway as it wouldn't have completed so data would be missing
Title: Re: Cloning Hard drives refresher course
Post by: murraymint on March 20, 2019, 06:46:22 pm
I think GParted would be faster but I don't normally copy such large partitions. If you're talking about trying to compress it onto a smaller drive, that would take even longer, even if you had a big enough USB drive.

If these are both actually SSDs rather than HDs then it will all be much quicker.
Title: Re: Cloning Hard drives refresher course
Post by: alynur on March 20, 2019, 07:37:37 pm
I think GParted would be faster but I don't normally copy such large partitions. If you're talking about trying to compress it onto a smaller drive, that would take even longer, even if you had a big enough USB drive.

If these are both actually SSDs rather than HDs then it will all be much quicker.


I was just reading about cloning, thinking about using clonezilla, and it was mentioned that both hard drives need to have the same sector sizes and that using dd command is more secure. The source SSD is 250gb and the target SSD is 500gb. Pin didn't mention if his discs were HDs or SSDs. I think I'll stick with the original plan but if I knew how to check the sector sizes of the new, blank SSD, I might give clonezilla a go. Decisions decisions. Worse case scenario, I can start from scratch with both SSDs.
Title: Re: Cloning Hard drives refresher course
Post by: pin on March 21, 2019, 05:11:28 am
Hi alynur!
I only have SSDs.
If a disk has no partitions on it, does it even make sense to talk about disk sectors?  :-\  :-\
I honestly don't know!!

Anyway, if you want to check your disk, you can use fdisk.
I've only done this on *BSD, and the command is

Code: [Select]
fdisk wd0

I guess that on Linux it should be

Code: [Select]
fdisk /dev/sda 
or whatever your new disk is called. Check https://linux.die.net/man/8/fdisk for the syntax if you want, but I've never had any issues with dd. Do let us know how it goes  ;)
Title: Re: Cloning Hard drives refresher course
Post by: alynur on March 21, 2019, 10:34:43 am
Hi alynur!
I only have SSDs.
If a disk has no partitions on it, does it even make sense to talk about disk sectors?  :-\  :-\
I honestly don't know!!

Anyway, if you want to check your disk, you can use fdisk.
I've only done this on *BSD, and the command is

Code: [Select]
fdisk sd0



I guess that on Linux it should be

Code: [Select]
fdisk /dev/sda 
or whatever your new disk is called. Check https://linux.die.net/man/8/fdisk for the syntax if you want, but I've never had any issues with dd. Do let us know how it goes  ;)

Hi pin, I should have known you only have SSDs. I thought the minimum rewrite speed for SSDs was 3gb a minute. I guess it depends on cpu. Anyway, the above site you mentioned had this
Code: [Select]
fdisk doesn't understand GUID Partition Table (GPT) and it is not designed for large partitions. In particular case use more advanced GNU parted(8).

I think I can just run
Code: [Select]
sudo parted -ls
and get what I'm looking for, I ran that last night after my last post here and I was finally able to decipher the info given. I'll let you know what I do and how it turned out, thanks.
Title: Re: Cloning Hard drives refresher course
Post by: PCNetSpec on March 21, 2019, 12:26:01 pm
"sectors" have nothing to do with "partitions".

What's the problem/confusion now ?
Title: Re: Cloning Hard drives refresher course
Post by: pin on March 21, 2019, 02:25:03 pm
"sectors" have nothing to do with "partitions".

What's the problem/confusion now ?
Yes, that I know  :)
I was just wondering if an unpartitioned disk contains sectors or not  :-\
Sorry if that sounds dumb  ::)
Title: Re: Cloning Hard drives refresher course
Post by: PCNetSpec on March 21, 2019, 02:43:25 pm
"sectors" have nothing to do with "partitions".

What's the problem/confusion now ?
Yes, that I know  :)
I was just wondering if an unpartitioned disk contains sectors or not  :-\
Sorry if that sounds dumb  ::)
Not really "sectors" as such, maybe I should have said dd copies "block by block" ;)

Sent from my SM-N960F using Tapatalk
Title: Re: Cloning Hard drives refresher course
Post by: pin on March 21, 2019, 03:08:00 pm
That was what I thought, but couldn't find enough evidence to convince myself that it was so  :)
You see, @alynur is worried about mismatched sector sizes between origin and target. While, excuse me the wording, I don't think dd gives a f**k about it!! As it doesn't care if you already have a partition scheme or not  ;)

I must say, I really like dd, it does what you tell it to do without asking a second time, ...for the better or worst  :D
Title: Re: Cloning Hard drives refresher course
Post by: PCNetSpec on March 21, 2019, 03:17:48 pm
Sorry my mistake, I shoulda read this topic more closely :-[

Exactly, dd will copy block by block .. so sector sizes on the target will reflect whatever was on the source (as they get reproduced too).

As you elegantly put it - dd doesn't give a f**k, it just copies EXACLTY the source (sectors/partitions/data/everything .. even bad blocks) .. dd is NOT copying "data", it's cloning blocks.

Basically it's copying a bunch of 'zero's and 'ones' .. it doesn't care what they represent, it just copies them in the same order.
Title: Re: Cloning Hard drives refresher course
Post by: alynur on March 21, 2019, 03:36:21 pm
This I understand. In a thread I was reading about using clonezilla, the poster mentioned that if the sector sizes didn't match problems would appear, he didn't say what problems. He went on to say that if not sure then use dd.
Title: Re: Cloning Hard drives refresher course
Post by: pin on March 21, 2019, 04:08:44 pm
Crystal clear now  :)
Thx alynur, now I understand where it all came from.

Good dding  :D
Title: Re: Cloning Hard drives refresher course
Post by: alynur on March 21, 2019, 07:06:03 pm
Ha! just went through a 4 hour power outage!
Title: Re: Cloning Hard drives refresher course
Post by: PCNetSpec on March 21, 2019, 07:15:00 pm
Ouch :(
Title: Re: Cloning Hard drives refresher course
Post by: alynur on March 23, 2019, 12:38:52 pm
Well that was fun. I thought I would try the quickest method first and work towards the slowest method as each method failed. First after identifying that both hard drives have sector sizes of 512g, I used clonezilla which was installed in my usb P9 instillation. That was lightning fast! Only took 6 minutes to complete. But, clonezilla screwed up, it didn't clone either of my o/s partitions, I have P 9 and Lubuntu installed. It did copy the efi partition and the DATA partition. Oh, I should have mentioned that after completing with clonezilla, I replaced the hard drives, set the old one apart, and when I tried to boot up, I got a blank screen and then a grub screen. so I thought i would try reinstalling P 9's bootloader into the device and that's where I got a message that it couldn't mount sda2 and then I started gparted and noticed that the two o/s partitions had unrecognizable file types. Well hell, so I tried the disc method and copied both o/s partitions and the efi partition restored them into the new hard drive. That worked but boot times were horrific. Now that I think of it, that could have been because I didn't copy and restore the swap partition, didn't think of that last night, damn it! Anyway, I tried a Timeshift restore of P9 but that didn't help the boot time any. So I figured I would do a fresh install of P9 and that fixed the boot time but after doing Timeshift and Aptik restores, it wouldn't boot, don't remember what the message was, it was late or early depending how you look at it, about 3 AM. So I thought "oh hell, I might as well" so I opened the computer again and hooked up the old hard drive up again with it in the primary hard drive position, cable position, and booted up my usb and ran the dd code,. Since it doesn't show what it's doing, I figured I would check it when I get up, this was about 3:15AM. I had a really hard time going to sleep, probably due to the two gallons of coffee I drank the prior afternoon, and at 3:45, I checked on it and it was done. Wow, less than half an hour. I turned it off and went back to bed. at 4AM, I got up and opened the computer, took the old hard drive out, put the primary cable back on the new hard drive, repositioned the computer and fired it up. BAM, it was up and running in seconds! Rebooted into Lubuntu and it too was up and running in seconds. Everything works, symlinks,everything. I'm done, course I have to make new partitions so I can start playing with different distributions again. It's all your fault pin, you scared me with that 20 hours stuff. Don't know why clonezilla screwed up. Don't remember how I installed it. Anyway, one more notch of experience on my Peppermint belt. Thanks guys.  :)
Title: Re: Cloning Hard drives refresher course
Post by: pin on March 23, 2019, 12:45:18 pm
Sorry about that  ;)
Although, I must say I'm surprised it went so fast  :o
Title: Re: Cloning Hard drives refresher course
Post by: alynur on March 23, 2019, 05:44:06 pm
Sorry about that  ;)
Although, I must say I'm surprised it went so fast  :o

Ha ha, it was in jest. I too was surprised, I was expecting it to still be running in the morning when I got up. Now I know. I would like to know what happened to clonezilla though. I just swapped the "old SSD" into my laptop and now I have to find that thread talking about changing computer name. Thanks pin, dd command rocks!  :)
Title: Re: Cloning Hard drives refresher course
Post by: PCNetSpec on March 23, 2019, 05:54:10 pm
The bs=1M would have helped (a lot) and obviously copying between SSD's is going to be much faster than between HDD's (specially if they're both on SATA).

'dd' is a terrific utility .. right up until the first time you make a syntax mistake .. from that point on you'll cack your pants every time you use it.