• Re: A degrading micro-SD card ?

    From Computer Nerd Kev@3:770/3 to R.Wieser on Thu Sep 30 08:09:48 2021
    R.Wieser <address@not.available> wrote:
    Hello all,

    Question #1 :

    At some point I noticed that my RPi 3, used as a YouTube viewer, sometimes seemed to freeze and also responding very slowly (20-30 sec) when I, using the "Windows" key, call the system menu to shut down. After than it sometimes responded almost as slowly to present me with the shutdown confirmation dialog.

    I'm /assuming/ that it has something to do with the micro-SD cards performance degrading, perhaps because of having to swap blocks around to egalize wear-and-tear. Could someone confirm this or, if this is not the case, tell me what is going on ?

    This describes how to check the condition using mmc-utils, if the
    Micro SD card supports it (and you can understand what it says): https://developer.toradex.com/knowledge-base/emmc-linux#Configuration_and_Monitoring

    I ought to try it myself some time...

    Question #2:
    I attempted to make a DD copy of the micro-SD card to another (to check if
    it was the card or perhaps the installed software), but that failed because the newer, also 16 GB card, was 20 MB smaller than the other origional card (rather unexpected I might say). :-\

    Is there a way to tell a micro-SD card that its a bit smaller than it actually is, so that the installed image (NOOBS 3.2.1) will be smaller and a DD of it will always fit on a new card ?

    I don't think that's the right approach. You'd probably be better
    off by making the partitions stop some safe distance from the end,
    then using the "count" option to dd (which is multiplied by the
    "bs" setting) to limit the length of data read or written.

    Or if you want to be able to downgrade SD card size more freely,
    you could try using fsarchiver instead of dd. It can restore to
    differently sized partitions, but you have to archive individual
    partitions instead of the whole disk. That means you have to create
    the partitions again before restoring the archive to a new SD card.

    --
    __ __
    #_ < |\| |< _#
    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From druck@3:770/3 to Computer Nerd Kev on Thu Sep 30 11:51:38 2021
    On 30/09/2021 09:09, Computer Nerd Kev wrote:
    R.Wieser <address@not.available> wrote:
    Is there a way to tell a micro-SD card that its a bit smaller than it
    actually is, so that the installed image (NOOBS 3.2.1) will be smaller and a >> DD of it will always fit on a new card ?

    I don't think that's the right approach. You'd probably be better
    off by making the partitions stop some safe distance from the end,
    then using the "count" option to dd (which is multiplied by the
    "bs" setting) to limit the length of data read or written.

    dd is pretty safe as it will stop when it gets to the end of the source
    or destination.

    The key bit is to make the last partition a bit smaller so your image
    will fit on any SD card claiming to be the same size, as they can vary
    by up to 100s of MB.

    That's easy with a normal single OS installation as it will have just
    the boot (FAT) and root (ext4) partitions. Put the SD card in another
    Linux machine and use gparted resize the last partition so there is
    enough space after it to allow it fir on another card.

    NOOBs is a bit more complicated as it has multiple OS's and puts their
    boot and root partitions inside extended partitions. It is possible to
    resize the last root partition and then shrink the extended partition,
    but it's more involved.

    As ever the recommendation is try out different OS's on a NOOBs card,
    but once you've decided which one(s) you want to use, put them on a
    dedicated SD card with a simple partition layout, and avoid a lot of grief.

    ---druck

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From R.Wieser@3:770/3 to All on Thu Sep 30 12:36:24 2021
    Kev,

    This describes how to check the condition using mmc-utils, if
    the Micro SD card supports it (and you can understand what
    it says): https://developer.toradex.com/knowledge-base/emmc-linux#Configuration_and_Monitoring

    Thank you. I was hoping for something like that.

    I don't think that's the right approach. You'd probably be
    better off by making the partitions stop some safe distance
    from the end,

    As mentioned, I'm using a NOOBS image. AFAIK I've got zero control over
    which, and with which sizes it creates the partitions (it will just fill out all available space).

    Besides, DD doesn't care about partitions, so I don't think that that
    solution will solve much of anything - unless I could somehow tell DD to
    stop where the last partition ends ...

    Or if you want to be able to downgrade SD card size more
    freely, you could try using fsarchiver instead of dd.
    ....
    That means you have to create the partitions again before
    restoring the archive to a new SD card.

    Not what I expect to be doing on a simple restoring/duplication ...

    It can restore to differently sized partitions, but you have
    to archive individual partitions instead of the whole disk.

    I take it that it only backups the actual used space in the partition. If
    so, the actual backup would become smaller, which is a pre. On the other
    hand the ammount and complexity of work to restore it would increase, which
    is a con - especially having to recreate those partitions by hand (from
    what information?).

    Facing that I would probably go for the simpler method, and just grab myself
    a handfull of identical SD cards. :-)

    Thanks for the info.

    Regards,
    Rudy Wieser

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From R.Wieser@3:770/3 to All on Thu Sep 30 13:54:46 2021
    Druck,

    dd is pretty safe as it will stop when it gets to the end of the source or destination.

    :-) It would be rather safe if it would continue too. Its just that on reading you get large image files, and on writing you waste a lot of time
    with out anything to show for.

    Also, it depends on your definition of "safe" : if it stops with writing because the target (micro-SD card) is smaller than the source (image file)
    than you stand a good chance of an unusable, or perhaps even crashing copy.
    :-|

    Though in my case my DD program just told me that the target micro-SD card
    was to small to contain the source image file, and stopped. Rightly so.

    Put the SD card in another Linux machine and use gparted resize the last partition so there is enough space after it to allow it fir on another
    card.

    Yikes! And than just run a DD in a "just do it, ignore that its too big"
    mode ? That feels to me as if I'm just asking for trouble (see above) ...

    On the other hand, I would not mind telling DD to include all partitions, ignoring any (slack) space after the last one though.

    It is possible to resize the last root partition and then shrink the
    extended partition, but it's more involved.

    Hence my idea to make the NOOBS installation think the micro-SD card is actually smaller than it is ...

    As ever the recommendation is try out different OS's on a NOOBs card, but once you've decided which one(s) you want to use, put them on a dedicated
    SD card with a simple partition layout, and avoid a lot of grief.

    Do you have a bit more info about that ? I would not mind having the NOOBS pre-boot "press shift to reinstall" stuff removed.

    Regards,
    Rudy Wieser
    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From Nikolaj Lazic@3:770/3 to All on Thu Sep 30 13:02:00 2021
    Dana Thu, 30 Sep 2021 12:36:24 +0200, R.Wieser <address@not.available> napis'o:
    Kev,

    This describes how to check the condition using mmc-utils, if
    the Micro SD card supports it (and you can understand what
    it says):
    https://developer.toradex.com/knowledge-base/emmc-linux#Configuration_and_Monitoring

    Thank you. I was hoping for something like that.

    I don't think that's the right approach. You'd probably be
    better off by making the partitions stop some safe distance
    from the end,

    As mentioned, I'm using a NOOBS image. AFAIK I've got zero control over which, and with which sizes it creates the partitions (it will just fill out all available space).

    Besides, DD doesn't care about partitions, so I don't think that that solution will solve much of anything - unless I could somehow tell DD to
    stop where the last partition ends ...

    It knows what the size of partition is. It is written in the
    partition table.

    You could make partition smaller.
    Look at resize2fs

    After that you could also make partition smaller in the partition table.
    That way you will be able to copy from one card to another.
    In case you want to "reserve" the end of the card... you can make dummy partition at the end. With some kind of "weird" type your pi would not
    try to mount.
    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From Dennis Lee Bieber@3:770/3 to All on Thu Sep 30 10:26:22 2021
    On Thu, 30 Sep 2021 13:54:46 +0200, "R.Wieser" <address@not.available> declaimed the following:


    Hence my idea to make the NOOBS installation think the micro-SD card is >actually smaller than it is ...


    I don't think the Raspberry-Pi foundation even distributes NOOBS format any more. The closest is the "Raspberry-Pi Imager" -- which runs on Windows/Macs (and strangely, listed for Ubuntu even though R-Pi OS is based
    on Debian).

    As I understand it, this program does a network download of the selected OS and writes just it to an SD card. One does not have the NOOBS overhead of having a couple of different OS on the card (along with the
    network install OS). {the imager installer is just 20MB for Windows
    machines -- so it obviously can not contain a 3GB OS image itself}

    [Trimmed]
    C:\Users\Wulfraed>dir e:\MicroSupport\RPI3_4
    Directory of e:\MicroSupport\RPI3_4

    03/28/2021 06:28 PM 3,007,638,235
    2021-03-04-raspios-buster-armhf-full.zip

    06/23/2021 06:27 PM 3,005,999,855
    2021-05-07-raspios-buster-armhf-full.zip

    09/18/2021 12:29 PM 19,772,672 imager_1.6.2.exe

    10/13/2020 03:09 PM 2,441,652,758 NOOBS_v3_5_0.zip

    Do you have a bit more info about that ? I would not mind having the NOOBS >pre-boot "press shift to reinstall" stuff removed.

    Not using an out-of-date NOOBS image might be an option... https://www.raspberrypi.org/software/operating-systems/
    The last NOOBS image was 3.5.0, and I show a download date a year ago. In
    the meantime, there have been at least TWO RaspiOS images made available.
    (See above listing)

    Still won't do anything for the partition size filling the SD card -- when different brands of cards may reserve different amounts of blocks for bad-block remapping.




    --
    Wulfraed Dennis Lee Bieber AF6VN
    wlfraed@ix.netcom.com http://wlfraed.microdiversity.freeddns.org/

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From Martin Gregorie@3:770/3 to Dennis Lee Bieber on Thu Sep 30 16:08:40 2021
    On Thu, 30 Sep 2021 10:26:23 -0400, Dennis Lee Bieber wrote:

    On Thu, 30 Sep 2021 13:54:46 +0200, "R.Wieser" <address@not.available> declaimed the following:


    Hence my idea to make the NOOBS installation think the micro-SD card is >>actually smaller than it is ...


    I don't think the Raspberry-Pi foundation even distributes NOOBS
    format
    any more. The closest is the "Raspberry-Pi Imager" -- which runs on Windows/Macs (and strangely, listed for Ubuntu even though R-Pi OS is
    based on Debian).

    As I understand it, this program does a network download of the selected OS and writes just it to an SD card. One does not have the
    NOOBS overhead of having a couple of different OS on the card (along
    with the network install OS). {the imager installer is just 20MB for
    Windows machines -- so it obviously can not contain a 3GB OS image
    itself}

    [Trimmed]
    C:\Users\Wulfraed>dir e:\MicroSupport\RPI3_4
    Directory of e:\MicroSupport\RPI3_4

    03/28/2021 06:28 PM 3,007,638,235 2021-03-04-raspios-buster-armhf-full.zip

    06/23/2021 06:27 PM 3,005,999,855 2021-05-07-raspios-buster-armhf-full.zip

    09/18/2021 12:29 PM 19,772,672 imager_1.6.2.exe

    10/13/2020 03:09 PM 2,441,652,758 NOOBS_v3_5_0.zip

    Do you have a bit more info about that ? I would not mind having the
    NOOBS pre-boot "press shift to reinstall" stuff removed.

    Not using an out-of-date NOOBS image might be an option... https://www.raspberrypi.org/software/operating-systems/
    The last NOOBS image was 3.5.0, and I show a download date a year ago.
    In the meantime, there have been at least TWO RaspiOS images made
    available. (See above listing)

    Still won't do anything for the partition size filling the SD
    card --
    when different brands of cards may reserve different amounts of blocks
    for bad-block remapping.

    You'll need a USB connected SD card reader. Put a new SD card in it and
    connect it to the RPi.

    Use the 'parted' command-line to format the new SD card with two
    partitions:
    - partition no 1: a 1 GB fat32 partition with its boot flag set
    - partition no 2: an ext4 partition occupying the rest of the card

    Use the 'dd' utility to copy /dev/mmcblk0p1 into the first partition and /dev/root into the second partition.

    Now the RPi should boot from the new card.

    Yes, I know that the RPi OS installation is currently only 50 MB, but
    disk space is cheap these days and the process of upgrading the OS will
    require the boot partition to be at least twice the size of the installed operation system, because at one point in the upgrade process both the
    old and new versions are on the partition at the same time.

    If you're using a tiny SD card, you can make the boot partition somewhat smaller, but don't make it less than 200GB or an upgrade may fail. Don't
    ask how I know this...

    Lastly, if you're not already using 'apt' to do regular system upgrades
    over the 'net, start using it: its faster than messing round with Noobs
    and easy once you've done it the first time. Just run:

    sudo apt-get update
    sudo apt-get upgrade
    sudo apt-get dist-upgrade
    sudo apt-get autoremove
    sudo apt-get autoclean
    sudo reboot

    And, of course you can always save time and typing by putting that
    command sequence in a shell script and running that.

    I always make a system backup before doing an upgrade but ymmv.


    --
    --
    Martin | martin at
    Gregorie | gregorie dot org
    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From druck@3:770/3 to R.Wieser on Fri Oct 1 07:48:08 2021
    On 30/09/2021 12:54, R.Wieser wrote:
    Druck,

    dd is pretty safe as it will stop when it gets to the end of the source or >> destination.

    :-) It would be rather safe if it would continue too. Its just that on reading you get large image files, and on writing you waste a lot of time with out anything to show for.

    Also, it depends on your definition of "safe" : if it stops with writing because the target (micro-SD card) is smaller than the source (image file) than you stand a good chance of an unusable, or perhaps even crashing copy. :-|

    Not if you have resized the partitions first with gparted as I advised.

    Though in my case my DD program just told me that the target micro-SD card was to small to contain the source image file, and stopped. Rightly so.

    Put the SD card in another Linux machine and use gparted resize the last
    partition so there is enough space after it to allow it fir on another
    card.

    Yikes! And than just run a DD in a "just do it, ignore that its too big" mode ? That feels to me as if I'm just asking for trouble (see above) ...

    Right this is turning in to another one of these troll threads where
    you ask for advice, and then deliberately ignore / misinterpret it to
    cause a pointless argument. Pack it in.

    ---druck

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From druck@3:770/3 to Martin Gregorie on Fri Oct 1 07:54:06 2021
    On 30/09/2021 17:08, Martin Gregorie wrote:
    Use the 'parted' command-line to format the new SD card with two
    partitions:
    - partition no 1: a 1 GB fat32 partition with its boot flag set
    - partition no 2: an ext4 partition occupying the rest of the card

    Use the 'dd' utility to copy /dev/mmcblk0p1 into the first partition and /dev/root into the second partition.

    Now the RPi should boot from the new card.

    This wont work unless the two partitions are exactly the same size on
    both cards, which they can't be if the new card is smaller.

    If you are manually creating new partitions as above, don't use dd, but
    copy all the files with:-

    rsync -axP /mnt/source/boot/ /mnt/dest/boot
    rsync -axVAXP /mnt/source/root /mnt/dest/root

    ---druck

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From druck@3:770/3 to Nikolaj Lazic on Fri Oct 1 07:44:02 2021
    On 30/09/2021 14:02, Nikolaj Lazic wrote:
    You could make partition smaller.
    Look at resize2fs

    After that you could also make partition smaller in the partition table.

    This is what gparted does graphically, in a much more friendly way and
    less disaster prone way than using resize2fs and parted manually.

    That way you will be able to copy from one card to another.

    Once the partitions are correct, you can just copy with dd, and it will
    stop automatically when it has filled the new card.

    In case you want to "reserve" the end of the card... you can make dummy partition at the end. With some kind of "weird" type your pi would not
    try to mount.

    No don't do this, the system will object is there isn't room for this
    dummy partition on the new card. Just leave the space empty, then the
    system doesn't care if it is there or isn't there.

    ---druck

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From druck@3:770/3 to druck on Fri Oct 1 08:59:04 2021
    On 01/10/2021 07:54, druck wrote:
    If you are manually creating new partitions as above, don't use dd, but
    copy all the files with:-

    rsync -axP /mnt/source/boot/ /mnt/dest/boot
    rsync -axVAXP /mnt/source/root /mnt/dest/root

    This should be /mnt/source/root/ note the trailing slash or it will copy
    to /mnt/dest/root/root

    ---druck
    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From R.Wieser@3:770/3 to All on Fri Oct 1 14:10:38 2021
    Dennis,

    I don't think the Raspberry-Pi foundation even distributes NOOBS
    format any more.

    That I'm using a NOOBS installation is irrelevant. My question was aimed
    at making *an* installation think the micro-SD card would be smaller than it actually is. IOW, changing something about the micro-SD card /before/
    throwing an OS at it.

    As I understand it, this program does a network download of
    the selected OS and writes just it to an SD card.

    I only install from stuff I have already downloaded. That way I can re-install and have exactly the same as I did before. IOW, I rather
    dislike (understatement) the "just download it again and deal with whatever changes they made" approach.

    Not using an out-of-date NOOBS image might be an option...

    :-) Don't try to tell me that removing the "repair" part is something they only thought of after NOOBS 3.5.0.

    Still won't do anything for the partition size filling the SD card --

    Indeed. And as I've not seen you mention anything about that in regard to whatever the latest available RPi OS version is (if-and-when it still
    supports my aging RPi ofcourse) ...

    Regards,
    Rudy Wieser
    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From R.Wieser@3:770/3 to All on Fri Oct 1 14:51:58 2021
    Martin,

    Use the 'parted' command-line to format the new SD card with two
    partitions:
    - partition no 1: a 1 GB fat32 partition with its boot flag set
    - partition no 2: an ext4 partition occupying the rest of the card

    Use the 'dd' utility to copy /dev/mmcblk0p1 into the first partition and /dev/root into the second partition.

    Thanks, but I'm not sure I follow. Should I understand from this that
    RPis 'dd' utility will automatically resize the source partition onto the target one ? Or is something else going on ?

    Regards,
    Rudy Wieser
    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From R.Wieser@3:770/3 to All on Fri Oct 1 15:21:50 2021
    druck

    Not if you have resized the partitions first with gparted as I advised.

    You misunderstood what I was trying to say : You're suggesting to use a
    hack, ignoring the "target is too small" warning I take your 'dd' gives
    (mine on Windows won't even start the copy) and have not mentioning anything
    in regard to checking if it actually went alright. Not cool dude, not cool.

    Right this is turning in to another one of these troll threads where you
    ask for advice, and then deliberately ignore / misinterpret it to cause a pointless argument.

    I understood your advice well enough. In fact well enough to point out to you that the result of your hack could not be trusted. And now *I'm* the
    bad guy ?

    Pack it in.

    To use another, similar phrase : "go take a hike".

    Regards,
    Rudy Wieser
    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From The Natural Philosopher@3:770/3 to R.Wieser on Fri Oct 1 14:40:56 2021
    On 01/10/2021 13:51, R.Wieser wrote:
    Martin,

    Use the 'parted' command-line to format the new SD card with two
    partitions:
    - partition no 1: a 1 GB fat32 partition with its boot flag set
    - partition no 2: an ext4 partition occupying the rest of the card

    Use the 'dd' utility to copy /dev/mmcblk0p1 into the first partition and
    /dev/root into the second partition.

    Thanks, but I'm not sure I follow. Should I understand from this that
    RPis 'dd' utility will automatically resize the source partition onto the target one ? Or is something else going on ?

    See drucks post. The problem is that it *wont*.

    create partitions manually and then transfer files as per drucks rsync cribs

    I see no merit whatsoever in using DD on partitions - once you are at
    the file/directory level, use rsync to copy files

    The merit of DD is that it copies the whole disk image. So can recreate partitions.


    --
    “The urge to save humanity is almost always only a false face for the
    urge to rule it.”
    – H. L. Mencken
    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From Ahem A Rivet's Shot@3:770/3 to The Natural Philosopher on Fri Oct 1 17:29:26 2021
    On Fri, 1 Oct 2021 14:40:56 +0100
    The Natural Philosopher <tnp@invalid.invalid> wrote:

    I see no merit whatsoever in using DD on partitions - once you are at
    the file/directory level, use rsync to copy files

    There's are reasons to use it, dd will copy any kind of partition correctly even ones you can't mount while rsync can be troublesome in the
    face of devices, sparse files, named pipes, mounts and the like. It may
    also be faster if the filesystem is fairly full with a great many small
    files.

    There are reasons not to use it too, primarily that it will copy all the unused space and can only copy to a target at least as large as the partition.

    --
    Steve O'Hara-Smith
    Odds and Ends at http://www.sohara.org/
    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)