Hello Eugene,
On 30 Aug 26 03:26, you wrote to me:
The pieces are all in place, so writing the detection is an afternoon. What I do not want to get wrong is the policy, because once it ships people will have configurations leaning on it.
OK..
It only becomes a question when the text has bytes above 0x7F - pure
ASCII is valid UTF-8 and valid in every eight-bit set alike. The rule
I would start from is: treat it as UTF-8 only if there is at least one well-formed multi-byte sequence and no ill-formed one. And judge the
whole message rather than line by line, both because FTS-5003 says the identifier covers all parts of a message, and because answering a man
in the charset he wrote in needs the message to have one.
So far so good...
Here is the case I keep coming back to. A man writes in CP866 and
quotes a paragraph of UTF-8 his editor did not recode. The check fails
on his own text, we fall back to the assumed charset, his words read
and the quote does not - which is what happens today, so nothing is
lost.
Plus that the reader can always change the coding with control-J so that he/she may still be able to read the quotes.
Now turn it around: he writes in UTF-8 and quotes CP866. The
check fails on the few bad bytes in the quote, we read the lot as the assumed eight-bit set, and now his own words are the rubbish while the quote is no better. That is where detection would earn its keep, and
the strict rule gives nothing. One bad line poisons the message.
But the reader can still change the coding with control-J...
One way out is to look only at the lines the author typed and skip the quoted ones - the charset of a message is the charset its author wrote
in, and a quote is already damaged by whoever sent it. That is not
much work,
Yes, I think that woud be a strategy that makes sense.
and only mildly circular, in that we would be deciding what is a quote before deciding how to read the bytes.
Golded already has a mechanisme to detect what is a quote, the qoutes are in a different colour.
So the question for you: have you run into that second case? It bites hardest where the traffic is not ASCII to begin with, so I can go
looking in the Russian echoes - what I do not know is how it looks
from your side.
I haven't seen it yet. I don't read Russian echos, until recently I couldn't even correctly display it and I can't read Russian anyway. It is only in the German and Dutch areas that I read non-ASCII and that is just accents and umlauts, coded in CP850 or Latin-1. I don't recall ever having seen main text and quotes not being in the same encoding, let alone one of them being UTF-8. So I am afraid I am of not much help here. Maybe now that I have an UTF-8 capable reader I may come across it in the future...
On the future you are aiming at, the precedents are on your side, with
one caveat about how such a change arrives. Mail has the same
arrangement we do: the charset rides as a parameter on Content-Type,
and when it is missing the reader does not guess - plain text is
US-ASCII, as it has been for thirty years. That is our rule almost
word for word. And it has held even where it is arguably redundant:
HTML mandates UTF-8, and a document is still required to say so, on
the Content-Type or in a meta tag. The one place the declaration was dropped is JSON, and for your exact reason: once UTF-8 was made the
only legal encoding there was nothing left to declare.
That was the future I had in mind and was aiming for...
But each of those was settled on a date, by a body with the standing
to settle it. Fidonet will not do that.
Indeed, we having no authoritative body to make such decrees. The FTSC just documents "common practise". Whatever that is, but in any case, after the fact.
It will drift, and there will be no day on which a reader may simply
start assuming - which is the situation a detector is for.
Yep!
Which is also why I would rather put it in now than when it is wanted. This code can go years with nobody in it, and the day UTF-8 without
CHRS becomes ordinary is not necessarily a day when anyone is here to
add the handling. Better it sits ready and switched off, and the
default moves later, when the network has.
These were exactly my thougts when I requested MakeNl to be updated to handle non ASCII some two decades ago....
So go ahead and implement it.
UTFautodetect Yes|no|override (Default: no)
Settable globally or per group.
And this is about reading only. What GoldED writes in its own mail is
the sysop's business and already his to configure. A keyword, off by default, settable per group.
Go ahaed!
Cheers, Michiel
--- GoldED+/W32-MINGW 1.1.5-b20260829
* Origin: Klein Schnøørd (2:280/5555.1)