1. Forum
  2. FidoNet
  3. UTF-8

  • "Soft CRs" and UTF-8

    From Carlos Navarro@2:341/234.1 to All on Wed Mar 8 18:35:42 2023
    FTN mail processors should not strip the so called soft CRs, or else any Unicode character that contains a 0x8D byte in its UTF-8 encoding will be broken.

    This was already mentioned in Michiel's article "A plea for UTF-8 in Fidonet part 2" (FidoNews Volume 29, Number 01 - 2 Jan 2012).

    Carlos

    --- GoldED+/W32-MSVC 1.1.5-b20180707
    * Origin: cyberiada (2:341/234.1)
  • From Oli@2:280/464.47 to Carlos Navarro on Sat Mar 18 08:48:34 2023
    Carlos wrote (2023-03-08):

    FTN mail processors should not strip the so called soft CRs, or else any Unicode character that contains a 0x8D byte in its UTF-8 encoding will be broken.

    FTN mail processors should not modify the message at all, besides some necessary kludges for routing. Is there any software that strips Soft CRs or breaks UTF-8 encoding?

    ---
    * Origin: This site requires JavaScript (2:280/464.47)
  • From Carlos Navarro@2:341/234.1 to Oli on Sat Mar 18 13:51:56 2023
    18 Mar 2023 08:48, you wrote to me:

    FTN mail processors should not strip the so called soft CRs, or
    else any Unicode character that contains a 0x8D byte in its UTF-8
    encoding will be broken.

    FTN mail processors should not modify the message at all, besides some necessary kludges for routing. Is there any software that strips Soft
    CRs or breaks UTF-8 encoding?

    After some recent UTF-8 test in the JAMNNTPD echo, Tommi had to disable soft CR stripping in GEcho.

    FMail has the option to remove those, but it's disabled by default. I think that it only works for importing to the messagebase.

    Synchronet's SBBS by default removes soft CRs, but it seems it does not apply to UTF-8 messsages. I don't know if it's only when importing to its bases.

    Don't know about others...

    Carlos

    --- GoldED+/W32-MSVC 1.1.5-b20180707
    * Origin: cyberiada (2:341/234.1)
  • From Wilfred van Velzen@2:280/464 to Carlos Navarro on Sat Mar 18 17:15:04 2023
    Hi Carlos,

    On 2023-03-18 13:51:56, you wrote to Oli:

    FMail has the option to remove those, but it's disabled by default. I think that it only works for importing to the messagebase.

    Actually this is applied to messages from the 'scan' function, so when they are exported from the messagebases. (When these options are turned on)

    The documentation says this about the options:

    Remove soft cr
    Remove lf

    Line feeds and soft carriage returns are unnecessary in
    echo mail messages. In order to save some disk space, you
    may want to remove them. It will, however, slow the
    program down a bit.


    So this is from a time when disk space, processor cycles and band with were much more limited then now... ;-)


    Bye, Wilfred.

    --- FMail-lnx64 2.2.0.0
    * Origin: FMail development HQ (2:280/464)