3: Connect NVME to raspi, boot from SD-card and copy the system from
SSD to NVME. But how?
On Wed, 31 Jul 2024 09:51:00 +0200, Jesper wrote:
3: Connect NVME to raspi, boot from SD-card and copy the system from
SSD to NVME. But how?
Use rsync to do a file-level copy. Then the difference in volume sizes won’t matter.
I haven’t done this on ARM, only on x86 machines (several times), but the procedure should be similar: after copying the installation across, you
will need to fix up /etc/fstab for the changed filesystem IDs, and also reinstall the bootloader. Then you should be good to go.
As long as your new NVME is larger than the SSD, you can just do low
level copy with the dd command, then resize the rootfs partition on the
new drive to use any extra space with the gparted program (if not
installed use: apt install gparted).
Then the system to copy is on the 2 last lines. Correct?
And following drucks first suggestion I should run these 2 commands:
1: dd if=/dev/sda2 of=/dev/nvme0n1 bs=1M status=progress
and
2: dd if=/dev/sda1 of=/dev/nvme0n1 bs=1M status=progress
replacing the name of the NVME to what I see when it is installed on the raspi.
Best regards, and thank you for the help.
On 01.08.2024 13:55, The Natural Philosopher wrote:
On 01/08/2024 12:31, Jesper wrote:
Then the system to copy is on the 2 last lines. Correct?No.
In order to preserve the partition information *you must dd the raw
disk*
dd id=/dev/sda od = /dev/nvme0n1 (or whatever)
That will create a two partition disk with the UUIDS of the partitions
the same as is mentioned in the boot data: If they don't match it wont
boot.
Are you saying that this command (the one right above) will clone the
whole disc, and the job is done? Probably not. When I run "dd --help" on
the system i want to clone, there is no information about the switch
"id" you use in the command.
On 01.08.2024 13:55, The Natural Philosopher wrote:
On 01/08/2024 12:31, Jesper wrote:
Then the system to copy is on the 2 last lines. Correct?No.
In order to preserve the partition information *you must dd the raw disk*
dd id=/dev/sda od = /dev/nvme0n1 (or whatever)
That will create a two partition disk with the UUIDS of the partitions
the same as is mentioned in the boot data: If they don't match it wont
boot.
Are you saying that this command (the one right above) will clone the
whole disc, and the job is done? Probably not. When I run "dd --help" on
the system i want to clone, there is no information about the switch
"id" you use in the command.
Running that command on my system gives:
Viz:
df -h | grep ^/dev/
raspberrypi@raspberrypi:~ $ df -h | grep ^/dev/
/dev/sda2 234G 19G 203G 9% / /dev/sda1 511M 76M 436M 15% /boot/firmware For /dev/sda1 it says "firmware", so it probably should/can not be
copied, and is permanent on the raspi5-system
/dev/root 15G 1.5G 13G 11% /
/dev/mmcblk0p1 255M 51M 205M 20% /boot
These are the TWO partitions on a bootable PI device
In /boot which is DOS style formatted will be instructions on how to
boot the main system
In the main bootable system there will be the fstab file which needs
to tally with the partition ids.
more /etc/fstab:
proc /proc proc defaults 0 0
PARTUUID=b8c9fbb7-01 /boot vfat defaults 0 2
PARTUUID=b8c9fbb7-02 / ext4 defaults,noatime 0 1
In the BOOT partition is this file
more *.txt
::::::::::::::
cmdline.txt
::::::::::::::
console=serial0,115200 console=tty1 root=PARTUUID=b8c9fbb7-02
rootfstype=ext4 fs
ck.repair=yes rootwait modules-load=dwc2,g_ether
Unless the bootloader finds that partition ID, it will *not load Linux*
AIUI the boot sequence is this:
Look for a DOS style VFAT partition on SD card, then USB, then NVME.
Look for a file named 'cmdline.txt' parse the root partition ID and
attempt to load a linux image from the boot partition and have the
kernel image mount the aforementioned PARTUUID as root partition.
If the PARTUUIDs don't match, the boot sequence hangs
So it is important to have the same PARTUUID in /boot/cmdline.txt, and
in /etc/fstab, and in the partition label on the boot partition
The easy way to do this is not to clone the partitions, but the RAW DISK
And following drucks first suggestion I should run these 2 commands:
1: dd if=/dev/sda2 of=/dev/nvme0n1 bs=1M status=progress
and
2: dd if=/dev/sda1 of=/dev/nvme0n1 bs=1M status=progress
replacing the name of the NVME to what I see when it is installed on
the raspi.
Best regards, and thank you for the help.
That wont work, but you wont destroy anything by trying
So I will not try :-)
I think I should not take any more of your time for this project. Using
a PC and a windows program (Macrium Reflect) to clone the disc will take maybe 30 minutes. Learning the ins and outs of how it should be done in raspi-os can take 14 days for a windows-addict:-)
Best regards, and thank you for the help so far.
On Thu, 1 Aug 2024 17:47:32 +0200
Jesper <Vitsky.kasperski@gmail.com> wrote:
On 01.08.2024 13:55, The Natural Philosopher wrote:
On 01/08/2024 12:31, Jesper wrote:>boot.
Then the system to copy is on the 2 last lines. Correct?No.
In order to preserve the partition information *you must dd the raw
disk*
dd id=/dev/sda od = /dev/nvme0n1 (or whatever)
That will create a two partition disk with the UUIDS of the partitions
the same as is mentioned in the boot data: If they don't match it wont
Are you saying that this command (the one right above) will clone the
whole disc, and the job is done? Probably not. When I run "dd --help" on
the system i want to clone, there is no information about the switch
"id" you use in the command.
That command is wrong (or for an odd version of dd - but I think
it's a finger slip) make that:
dd if=/dev/sda of=/dev/nvme0n1 bs=1m
The add bs=1m will probably work wonders for performance.
But I can try running both commands:
dd if=/dev/sda2 of=/dev/nvme0n1 bs=1m
dd if=/dev/sda1 of=/dev/nvme0n1 bs=1m
On 01.08.2024 18:10, Ahem A Rivet's Shot wrote:
dd if=/dev/sda of=/dev/nvme0n1 bs=1m
The add bs=1m will probably work wonders for performance.
On 01.08.2024 17:47, Jesper wrote:
raspberrypi@raspberrypi:~ $ df -h | grep ^/dev/ /dev/sda2 234G 19G 203G 9% / /dev/sda1 511M 76M 436M 15% /boot/firmware For /dev/sda1 it says "firmware", so it probably should/can not be
copied, and is permanent on the raspi5-system
The Natural philosopher says to clone both sda1 and sda2. I still wonder
if sda1 should be cloned. It is listed as "firmware".
like it is on a flashmemory directly on the raspi5, and you modify it
with raspi-config->Advanced options->Boot order.
But I can try running both commands:
dd if=/dev/sda2 of=/dev/nvme0n1 bs=1m
dd if=/dev/sda1 of=/dev/nvme0n1 bs=1m
On Thu, 1 Aug 2024 19:50:20 +0200
Jesper <Vitsky.kasperski@gmail.com> wrote:
But I can try running both commands:
dd if=/dev/sda2 of=/dev/nvme0n1 bs=1m
This will copy sda2 to nvme0n1
dd if=/dev/sda1 of=/dev/nvme0n1 bs=1m
This will overwrite that copy with sda1. You would probably be
better off cloning sda as a whole - that will get all partitions, boot sectors etc.
On 01.08.2024 20:29, Ahem A Rivet's Shot wrote:
On Thu, 1 Aug 2024 19:50:20 +0200
Jesper <Vitsky.kasperski@gmail.com> wrote:
But I still do not know what a command that clones both sda1 and sda2 to
NVME should look like. Please?
The Natural philosopher says to clone both sda1 and sda2. I still wonder
if sda1 should be cloned. It is listed as "firmware". That sounds to me
like it is on a flashmemory directly on the raspi5, and you modify it
with raspi-config->Advanced options->Boot order.
But I can try running both commands:
dd if=/dev/sda2 of=/dev/nvme0n1 bs=1m
dd if=/dev/sda1 of=/dev/nvme0n1 bs=1m
Comments?
On 01/08/2024 09:20, The Natural Philosopher wrote:
On 31/07/2024 21:33, druck wrote:
As long as your new NVME is larger than the SSD, you can just do low
level copy with the dd command, then resize the rootfs partition on
the new drive to use any extra space with the gparted program (if not
installed use: apt install gparted).
That will at least get over the problem of having a different partuuid
etc
Yes the dd method will avoid any changes to partuuids, where as if you
use the rsync method you will have to change the values in /etc/fstab
and also potentially /boot/cmdline.txt (/boot/firmware/cmdline.txt on Bookworm).
Is it possible to then create a DIFFERENT partition on the empty part
of the new disk? And mount it as - say - /home?
Yes, you can do this with gparted, move the existing /home to the new partition and edit /etc/fstab to contain a line similar to:-
PARTUUID=abcd1234-03 /home ext4 nofail,noatime,errors=remount-ro 0 0
Then do a:-
mount -a
---druck
So with the disks attached try
mount | grep '^/dev'
To identify the device name.
On 01.08.2024 20:29, Ahem A Rivet's Shot wrote:
On Thu, 1 Aug 2024 19:50:20 +0200Yes, that's also what The Natural Philosopher says in his reply. And I
Jesper <Vitsky.kasperski@gmail.com> wrote:
But I can try running both commands:
dd if=/dev/sda2 of=/dev/nvme0n1 bs=1m
This will copy sda2 to nvme0n1
dd if=/dev/sda1 of=/dev/nvme0n1 bs=1m
This will overwrite that copy with sda1. You would probably be
better off cloning sda as a whole - that will get all partitions, boot
sectors etc.
know that you both of course are right.
But I still do not know what a command that clones both sda1 and sda2 to
NVME should look like. Please?
When I run ls /media i get this:
raspberrypi@raspberrypi:~ $ ls /media
raspberrypi
Could this work to clone the whole disc:
dd if=/dev/raspberrypi of=/dev/nvme0n1 bs=1m
Best regards
On 2024-08-01 21:02, Jesper wrote:
On 01.08.2024 20:29, Ahem A Rivet's Shot wrote:
On Thu, 1 Aug 2024 19:50:20 +0200
Jesper <Vitsky.kasperski@gmail.com> wrote:
But I still do not know what a command that clones both sda1 and sda2
to NVME should look like. Please?
Really? It is given to you more than once
dd if=/dev/sda of=/dev/nvme0n1 bs=1m
/dev/sda is the WHOLE disk called /dev/sda
/dev/sdaX where X is a number is that PARTITION on the disk /dev/sda
On 2024-08-01 21:38, Björn Lundin wrote:
On 2024-08-01 21:02, Jesper wrote:
On 01.08.2024 20:29, Ahem A Rivet's Shot wrote:
On Thu, 1 Aug 2024 19:50:20 +0200
Jesper <Vitsky.kasperski@gmail.com> wrote:
But I still do not know what a command that clones both sda1 and sda2
to NVME should look like. Please?
Really? It is given to you more than once
dd if=/dev/sda of=/dev/nvme0n1 bs=1m
/dev/sda is the WHOLE disk called /dev/sda
/dev/sdaX where X is a number is that PARTITION on the disk /dev/sda
I have not seen anybody mentioning sync
Once the dd command is done, type 'sync'
In some settings the writing is not quite done yet, the sync forces the
write of the caches to the device. I see this on my pc running ubuntu
with 32 Gb RAM.
On 02/08/2024 15:03, Björn Lundin wrote:
On 2024-08-02 14:51, The Natural Philosopher wrote:Even sync may not be enough.
Never mind sync.
It's important to wait anyway on an SSD/nvm until it has finished its
internal business.
For any copy process.
dd'ing 500 Gb to an ssd disk, I've seen sync taking 30 s or more
and sync is of course started AFTER dd is done
So - how do you know it is done its internal business?
Not all drives have blinking LEDs
sync makes it easy to know
SSDS/NVM have their own internal caching.
SSDS/NVM have their own internal caching.
On 02/08/2024 15:03, Björn Lundin wrote:
dd'ing 500 Gb to an ssd disk, I've seen sync taking 30 s or more
and sync is of course started AFTER dd is done
So - how do you know it is done its internal business?
Not all drives have blinking LEDs
sync makes it easy to know
Even sync may not be enough.
SSDS/NVM have their own internal caching.
On 2024-08-02 16:12, The Natural Philosopher wrote:
On 02/08/2024 15:03, Björn Lundin wrote:
On 2024-08-02 14:51, The Natural Philosopher wrote:Even sync may not be enough.
Never mind sync.
It's important to wait anyway on an SSD/nvm until it has finished
its internal business.
For any copy process.
dd'ing 500 Gb to an ssd disk, I've seen sync taking 30 s or more
and sync is of course started AFTER dd is done
So - how do you know it is done its internal business?
Not all drives have blinking LEDs
sync makes it easy to know
SSDS/NVM have their own internal caching.
Then the question remains - how do you know when its done?
from man sync you get
"sync - Synchronize cached writes to persistent storage"
In other words, doing sync at least clear the cache,
which must be better than nothing? or?
Or are you saying that ripping the power to a newly sync()ed disk may
mess it up?
as in
dd ....
sync
rip out usb cord for a 2.5" ssd sata in a usb case or something similar ?
I am on the edge of my comfort patch here.
If I were building an SSD I would have a diode and a large capacitor
inside it to make sure all its caches were dumped to NVRAM before the
voltage collapsed completely.
But on a big unit this could take a bit of time.
What happens between a SATA/USB plug and the actual NVRAM is a bit of a mystery.
We know its nothing like a 1:1 correlation between 'sector' and physical
RAM location.
We knows that physical RAM locations are regularly shuffled for 'wear levelling'
When is all this done?
What happens if, during it, there is power failure?
I honestly do not know, hence the warning to leave the SSD for a few
seconds before yanking any power cords
It can do no harm
On Fri, 2 Aug 2024 15:12:57 +0100, The Natural Philosopher wrote:
SSDS/NVM have their own internal caching.
True for all drives, unfortunately.
It’s bloody stupid, because the drive caching is on the wrong side of the drive interface. Better to leave it to the OS, which can use main RAM for
its filesystem caching, on the fast side of that drive interface.
When a drive says to the OS driver “write is done”, it should mean “write
has gone to actual persistent storage”, not “write is in my cache”.
On 2024-08-03 12:07, The Natural Philosopher wrote:
I am on the edge of my comfort patch here.
If I were building an SSD I would have a diode and a large capacitor
inside it to make sure all its caches were dumped to NVRAM before the
voltage collapsed completely.
But on a big unit this could take a bit of time.
What happens between a SATA/USB plug and the actual NVRAM is a bit of
a mystery.
We know its nothing like a 1:1 correlation between 'sector' and
physical RAM location.
We knows that physical RAM locations are regularly shuffled for 'wear
levelling'
When is all this done?
What happens if, during it, there is power failure?
I honestly do not know, hence the warning to leave the SSD for a few
seconds before yanking any power cords
It can do no harm
And that is my point. sync will do no harm either,
but it might save you, especially when you are dd()ing an image smaller
that the RAM of the computer.
Like I have 32 Gb RAM, and dd an image of 4 Gb (like a headless) onto a SD-card to run an old pi. The interface is slow, yet dd reports done
within a minute. sync takes a long time to flush it over.
Just waiting a few secs will end in disappointment
I simply do not know how a command to something purporting to resemble a
hard drive, that isnt can guarantee all data in it is flushed.
On 2024-08-03 03:27, Lawrence D'Oliveiro wrote:
On Fri, 2 Aug 2024 15:12:57 +0100, The Natural Philosopher wrote:
SSDS/NVM have their own internal caching.True for all drives, unfortunately.
It’s bloody stupid, because the drive caching is on the wrong side
of the
drive interface. Better to leave it to the OS, which can use main RAM for
its filesystem caching, on the fast side of that drive interface.
When a drive says to the OS driver “write is done”, it should mean
“write
has gone to actual persistent storage”, not “write is in my cache”.
I think they cause grief in the postgres mail lists some 15-20 years ago. They were called 'lying IDE-disks' and were not popular in that crowd.
looking at <https://linux.die.net/man/8/sync>
"""
The kernel keeps data in memory to avoid doing (relatively slow) disk
reads and writes. This improves performance, but if the computer
crashes, data may be lost or the file system corrupted as a result. sync ensures that everything in memory is written to disk.
"""
So, what I think is that dd writes to a slow device, and it is cached by
the OS.
sync forces the OS to actually write to slow device.
This may not be true with new and shiny fast NVM storage, but the
principle holds.
The alternative is absurdly slow write IO.
I've used the "conv=fsync" option to dd so that sync is done
automatically. Since the man page is rather vague about it, I did
a web search to double check that I wasn't imagining things and
On 04/08/2024 00:30, Computer Nerd Kev wrote:
I've used the "conv=fsync" option to dd so that sync is done
automatically. Since the man page is rather vague about it, I did
a web search to double check that I wasn't imagining things and
Apropos of not too much, I finally got around to copying my old 120GB Pi
boot disk to an image file, and splatting that back onto a 2TB SSD that
now has according to a GUI based disk editor shitloads of 'unused' space.
(The reason I didnt do it directly was on account of a lack of
functional and accessible USB slots in the computer I used.)
All looks ok, with a bootfs and a rootfs partition on it, but I haven't actually booted it yet.
I didn't sync it, I went to the supermarket instead, and left it
running, and then left it running on the second transfer while I had the evening meal.
It appears to be OK
More later... I overslept badly and it may be Monday before more gets done.
On 01.08.2024 21:38, Björn Lundin wrote:
On 2024-08-01 21:02, Jesper wrote:
On 01.08.2024 20:29, Ahem A Rivet's Shot wrote:
On Thu, 1 Aug 2024 19:50:20 +0200
Jesper <Vitsky.kasperski@gmail.com> wrote:
But I still do not know what a command that clones both sda1 and sda2
to NVME should look like. Please?
Really? It is given to you more than once
dd if=/dev/sda of=/dev/nvme0n1 bs=1m
Now I have the nvme installed and showing in a lsblk command.
Booted from a SD-card and did a few tries to copy the system from SSD to nvme.
First there was a complaint about the switch "1m". Changed it to "1b"
and got a complaint about missing permission to open SDA (the SSD I want
to copy from). Threw a sudo at it, and it ran for maybe half an hour,
until it stopped with error "writing nvme0n1, No space left on device".
The SSD and the nvme have the same size, and that seems to be a problem.
Bright ideas are welcome :-)
Below I have copied in what happend in the command line: raspberrypi@raspberrypi:~ $ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 238.5G 0 disk
├─sda1 8:1 0 512M 0 part /media/raspberrypi/bootfs
└─sda2 8:2 0 238G 0 part /media/raspberrypi/rootfs
mmcblk0 179:0 0 58G 0 disk
├─mmcblk0p1 179:1 0 512M 0 part /boot/firmware └─mmcblk0p2 179:2 0 57.5G 0 part /
nvme0n1 259:0 0 238.5G 0 disk
raspberrypi@raspberrypi:~ $ dd if=/dev/sda of=/nvme0n1 bs=1m
dd: invalid number: ‘1m’
raspberrypi@raspberrypi:~ $ dd if=/dev/sda of=/nvme0n1 bs=1b
dd: failed to open '/dev/sda': Permission denied
raspberrypi@raspberrypi:~ $ dd if=/dev/sda of=/nvme0n1
dd: failed to open '/dev/sda': Permission denied
raspberrypi@raspberrypi:~ $ sudo dd if=/dev/sda of=/nvme0n1 bs=1b
dd: error writing '/nvme0n1': No space left on device
107929249+0 records in
107929248+0 records out
55259774976 bytes (55 GB, 51 GiB) copied, 1476.65 s, 37.4 MB/s raspberrypi@raspberrypi:~
Best regards
On 16.08.2024 12:40, The Natural Philosopher wrote:
On 16/08/2024 10:04, Jesper wrote:Thank you for the reply.
On 01.08.2024 21:38, Björn Lundin wrote:
On 2024-08-01 21:02, Jesper wrote:
On 01.08.2024 20:29, Ahem A Rivet's Shot wrote:
On Thu, 1 Aug 2024 19:50:20 +0200
Jesper <Vitsky.kasperski@gmail.com> wrote:
But I still do not know what a command that clones both sda1 and
sda2 to NVME should look like. Please?
Really? It is given to you more than once
dd if=/dev/sda of=/dev/nvme0n1 bs=1m
Now I have the nvme installed and showing in a lsblk command.
Booted from a SD-card and did a few tries to copy the system from SSD
to nvme.
First there was a complaint about the switch "1m". Changed it to "1b"
and got a complaint about missing permission to open SDA (the SSD I
want to copy from). Threw a sudo at it, and it ran for maybe half an
hour, until it stopped with error "writing nvme0n1, No space left on
device".
The SSD and the nvme have the same size, and that seems to be a problem. >>> Bright ideas are welcome :-)
Below I have copied in what happend in the command line:
raspberrypi@raspberrypi:~ $ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 238.5G 0 disk
├─sda1 8:1 0 512M 0 part /media/raspberrypi/bootfs
└─sda2 8:2 0 238G 0 part /media/raspberrypi/rootfs
mmcblk0 179:0 0 58G 0 disk
├─mmcblk0p1 179:1 0 512M 0 part /boot/firmware
└─mmcblk0p2 179:2 0 57.5G 0 part /
nvme0n1 259:0 0 238.5G 0 disk
raspberrypi@raspberrypi:~ $ dd if=/dev/sda of=/nvme0n1 bs=1m
dd: invalid number: ‘1m’
raspberrypi@raspberrypi:~ $ dd if=/dev/sda of=/nvme0n1 bs=1b
dd: failed to open '/dev/sda': Permission denied
raspberrypi@raspberrypi:~ $ dd if=/dev/sda of=/nvme0n1
dd: failed to open '/dev/sda': Permission denied
raspberrypi@raspberrypi:~ $ sudo dd if=/dev/sda of=/nvme0n1 bs=1b
dd: error writing '/nvme0n1': No space left on device
107929249+0 records in
107929248+0 records out
55259774976 bytes (55 GB, 51 GiB) copied, 1476.65 s, 37.4 MB/s
raspberrypi@raspberrypi:~
Best regards
WTF is '/nvme0n1' ?"nvme0n1" is the "fucking" NVME I have installed in the raspi5.
I see no trace of the new file you are referring to.
It looks like you have created a file in the root directory of the
device you are copying from....
Best regards
On 16/08/2024 10:04, Jesper wrote:
The SSD and the nvme have the same size ...
But are they?
On Fri, 16 Aug 2024 21:34:42 +0100, druck wrote:
On 16/08/2024 10:04, Jesper wrote:
The SSD and the nvme have the same size ...
But are they?
This is why rsync is so much easier.
| Sysop: | Coz |
|---|---|
| Location: | Anoka, MN |
| Users: | 2 |
| Nodes: | 4 (0 / 4) |
| Uptime: | 139:57:28 |
| Calls: | 166 |
| Files: | 5,389 |
| Messages: | 223,236 |