Hi g00r00,

I've started seeing some duplicates in my Fidonet bases. I have three Fidonet hubs, all roughly with the same bases available (with the odd exception of some local/regional echos only available on one or the other).

I have these two lines set in my mailin.ini:

;dupe_msg_index = -1
dupe_db_size = 250000

Here is an example of a dupe that made it into my FIDOTEST base:

Via 1:229/426: https://paste.ee/p/u0Us9
Via 3:770/1: https://paste.ee/p/GIS7F

All of the kludges are the same with the exception of the seen-by and path lines. There does seem to be a difference in the message body where the quoted seen-by's have been stripped out in the message I received via 1:229/426.


Jay

... I call things as I see them; If I didn't see them, I make them up!

--- Mystic BBS v1.12 A47 2021/09/29 (Raspberry Pi/32)
* Origin: Northern Realms (1:229/664)

On 10 Oct 2021, Jay Harris said the following...

I've started seeing some duplicates in my Fidonet bases.

Here are two more than came through today. I've turned off my feeds to
2:221/6 & 3:770/1 for now and am just hooked up to 1:229/426:

https://paste.ee/p/hBhCR

https://paste.ee/p/nN9aV

Again, the kludges look the same (except for the path and seen-by lines) but the message body has again been modified. One message has a SLMR line where the other one doesn't.


Jay

... It takes a long time to understand nothing.
--- Mystic BBS v1.12 A47 2021/09/29 (Raspberry Pi/32)
* Origin: Northern Realms (1:229/664)

Hello Jay,

Here are two more than came through today. I've turned off my feeds
to 2:221/6 & 3:770/1 for now and am just hooked up to 1:229/426:

https://paste.ee/p/hBhCR

https://paste.ee/p/nN9aV

Again, the kludges look the same (except for the path and seen-by
lines) but the message body has again been modified. One message has
a SLMR line where the other one doesn't.

I don't think that is a mystic issue, something is happening to messages arriving at your node. The message body should never be modified or altered in any way or dupes could happen.

I only have one copy of those messages here that include the SLMR line.

Ttyl :-),
Al

... When you learn the answers, they change the questions.
--- GoldED+/LNX 1.1.5-b20180707
* Origin: The Rusty MailBox - Penticton, BC Canada (1:153/757.4)

On 11 Oct 2021 at 08:20p, Jay Harris pondered and said...

I've started seeing some duplicates in my Fidonet bases.

Here are two more than came through today. I've turned off my feeds to 2:221/6 & 3:770/1 for now and am just hooked up to 1:229/426:

Hi Jay

I run the 3:770/1 HUB as you know.

Not sure if this is a Mystic thing. But to start with the two links below

https://paste.ee/p/hBhCR
https://paste.ee/p/nN9aV

neither contain 3:770/1 in the PATH line so in those examples the system is not depicted. If you did get a packet direct from 3:770/1 it would show as the last system in your PATH before your system imported it.

What I can see is in the two examples above the common systems in both PATH lines are 261/38 and 229/426 so I would start with those systems to check if any modifications occurred there.

Hope that helps.

Best, Paul

--- Mystic BBS v1.12 A47 2021/09/29 (Linux/64)
* Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (3:770/100)

On 11 Oct 2021, Al said the following...

I don't think that is a mystic issue, something is happening to messages arriving at your node.

The main reason I'm reporting it here is because Mystic doesn't seem to be catching these dupes. The MSGID & date/time are the same which to my mind means the 2nd one should be considered a dupe, even if the body of the message isn't exactly (but mostly) the same.


Jay

... Assassination is the extreme form of censorship.

--- Mystic BBS v1.12 A47 2021/09/29 (Raspberry Pi/32)
* Origin: Northern Realms (1:229/664)

On 12 Oct 2021, Paul Hayton said the following...

Here are two more than came through today. I've turned off my feeds 2:221/6 & 3:770/1 for now and am just hooked up to 1:229/426

neither contain 3:770/1 in the PATH line so in those examples the system is not depicted. If you did get a packet direct from 3:770/1 it would
show as the last system in your PATH before your system imported it.

Yup, as mentioned above I've unsubscribed from all echos on 2:221/6 & 3:770/1 while I try and figure out where this is happening. I've been in touch with Nick who says he doesn't believe the issue is at his system.

What I can see is in the two examples above the common systems in both PATH lines are 261/38 and 229/426 so I would start with those systems to check if any modifications occurred there.

Yup, I've also noticed 301/1 is another common node in the path of the messages where the body has been modified.


Jay

... WARNING! Removal of this tagline prohibited by law!
--- Mystic BBS v1.12 A47 2021/09/29 (Raspberry Pi/32)
* Origin: Northern Realms (1:229/664)

* Originally in mystic
* Crossposted in fn_sysop

Hi Jay.

12 Oct 21 09:46, you wrote to Paul Hayton:

Yup, I've also noticed 301/1 is another common node in the path of the messages where the body has been modified.

AHA!

I have noted before that 2:301/1 will remove multiple empty lines from the end of the message. It will cause messages without MSGID are not detected as dupes.

Matthias? Are you reading? Is your software modifying in-transit echomail?

'Tommi
--- GoldED+/LNX 1.2.5-b20210401
* Origin: rbb.fidonet.fi (2:221/360)

* Originally in fn_sysop
* Crossposted in mystic

Hi Michiel.

12 Oct 21 17:11:58, you wrote to me:

Ah, That's the culprit! I too noticed these modifications of in transit mail. :(

It will cause messages without MSGID are not detected as dupes.

Among other things...

Matthias? Are you reading? Is your software modifying in-transit
echomail?

Looks like it.

Z2DAILY echo is a good place to notice this. There is a certain robot posting messages without MSGID, and every message is seen twice in my msgbase. Two empty lines are missing in a message via 2:301/1.

=== Cut ===
= z2daily (2:221/360) =========================================================
Msg : 117 of 118
From : ZC2 2:292/854 12 Oct 21 00:10:00
To : All
Subj : Zone2 daily input-files and generation report ===============================================================================
Processing Region 20 -- file 'h:\dailylst\files\REGION20.281'
Processing Region 24 -- file 'h:\dailylst\files\REGION24.288'
Processing Region 25 -- file 'h:\dailylst\files\REGION25.281'
Processing Region 28 -- file 'h:\dailylst\files\REGION28.253'
Processing Region 29 -- file 'h:\dailylst\files\REGION29.288'
Processing Region 30 -- file 'h:\dailylst\files\REGION30.284'
Processing Region 31 -- file 'h:\dailylst\files\REGION31.281'
Processing Region 33 -- file 'h:\dailylst\files\REGION33.280'
Processing Region 34 -- file 'h:\dailylst\files\REGION34.253'
Processing Region 40 -- file 'h:\dailylst\files\REGION40.113'
Processing Region 41 -- file 'h:\dailylst\files\REGION41.270'
Processing Region 42 -- file 'h:\dailylst\files\REGION42.281'
Processing Region 45 -- file 'h:\dailylst\files\REGION45.281'
Processing Region 46 -- file 'h:\dailylst\files\REGION46.281'
Processing Region 48 -- file 'h:\dailylst\files\REGION48.180'
Processing Region 50 -- file 'h:\dailylst\files\REGION50.288'
Processing Region 56 -- file 'h:\dailylst\files\REGION56.281'
Processing Zone 1 -- file 'h:\dailylst\files\ZONE1.284'
Processing Zone 3 -- file 'h:\dailylst\files\ZONE3.285'
Processing Zone 4 -- file 'h:\dailylst\files\ZONE4.281'
-+- Robo 0.37 by Kash
-+- D'Bridge 4
+ Origin: (2:292/854)
SEEN+BY: 30/0 221/1 6 280/464 282/464 292/854 8125 301/0 1 101 113 123 335/364 SEEN+BY: 371/0 920/1 4500/1 5058/104
@PATH: 292/854 301/1 221/6

= z2daily (2:221/360) =========================================================
Msg : 118 of 118
From : ZC2 2:292/854 12 Oct 21 00:10:00
To : All
Subj : Zone2 daily input-files and generation report ===============================================================================
Processing Region 20 -- file 'h:\dailylst\files\REGION20.281'
Processing Region 24 -- file 'h:\dailylst\files\REGION24.288'
Processing Region 25 -- file 'h:\dailylst\files\REGION25.281'
Processing Region 28 -- file 'h:\dailylst\files\REGION28.253'
Processing Region 29 -- file 'h:\dailylst\files\REGION29.288'
Processing Region 30 -- file 'h:\dailylst\files\REGION30.284'
Processing Region 31 -- file 'h:\dailylst\files\REGION31.281'
Processing Region 33 -- file 'h:\dailylst\files\REGION33.280'
Processing Region 34 -- file 'h:\dailylst\files\REGION34.253'
Processing Region 40 -- file 'h:\dailylst\files\REGION40.113'
Processing Region 41 -- file 'h:\dailylst\files\REGION41.270'
Processing Region 42 -- file 'h:\dailylst\files\REGION42.281'
Processing Region 45 -- file 'h:\dailylst\files\REGION45.281'
Processing Region 46 -- file 'h:\dailylst\files\REGION46.281'
Processing Region 48 -- file 'h:\dailylst\files\REGION48.180'
Processing Region 50 -- file 'h:\dailylst\files\REGION50.288'
Processing Region 56 -- file 'h:\dailylst\files\REGION56.281'
Processing Zone 1 -- file 'h:\dailylst\files\ZONE1.284'
Processing Zone 3 -- file 'h:\dailylst\files\ZONE3.285'
Processing Zone 4 -- file 'h:\dailylst\files\ZONE4.281'


-+- Robo 0.37 by Kash
-+- D'Bridge 4
+ Origin: (2:292/854)
SEEN+BY: 203/0 221/1 6 360 280/464 5003 292/854 8125 301/1 335/364 423/81 SEEN+BY: 4500/1
@PATH: 292/854 221/1 6

=== Cut ===

'Tommi

---
* Origin: rbb.fidonet.fi (2:221/360)

Here are two more than came through today. I've turned off my feeds to 2:221/6 & 3:770/1 for now and am just hooked up to 1:229/426:

https://paste.ee/p/hBhCR

https://paste.ee/p/nN9aV

Again, the kludges look the same (except for the path and seen-by lines) but the message body has again been modified. One message has a SLMR
line where the other one doesn't.

The short answer to your messages here is: These are not considered duplicate messages as the message content is different. But I'll explain a little more detail for you if you're interested:

Mystic doesn't only rely on MSGID because some software or messages/networks may not use them at all. Instead, Mystic takes all things into consideration in totality which includes: MSGID, actual message content, PATH addresses (to prevent importing circular distributed messages) and SEEN-BY (to prevent distributing circular exported packets).

As you noted these messages aren't really duplicates because the message content is different and Mystic knows this (despite what the MSGID alone says). Because it knows they aren't the same, it doesn't treat it as a duplicate.

I can understand why some may think a duplicate MSGID alone would/should be enough, but legacy software (especially in the art scene where Mystic was popular) didn't use them at all and other prominent software (like EleBBS I remember testing, among others) were prone to spitting out two completely different messages with the same MSGID in the same echo.

As a result of all of that Mystic looks at everything and will not block a message unless its very confident that it truly is a duplicate message. In your case those messages are not duplicates in message content.

... Fer sell cheep: BBS spel chekker. Wurks grate.

--- Mystic BBS v1.12 A47 2021/09/23 (Windows/64)
* Origin: Sector 7 * Mystic WHQ (1:129/215)

On 12 Oct 2021, g00r00 said the following...

The short answer to your messages here is: These are not considered duplicate messages as the message content is different.

Mystic doesn't only rely on MSGID because some software or messages/networks may not use them at all. Instead, Mystic takes all things into consideration in totality

Cool, thanks for the clarification g00r00. Hopefully we've tracked down the node that is modifying the messages. I believe I also saw Nick released an update to D'Bridge that may also help. (Thanks Nick!)


Jay

... Nothing matters very much, and very few things matter at all.

--- Mystic BBS v1.12 A47 2021/09/29 (Raspberry Pi/32)
* Origin: Northern Realms (1:229/664)

Re: Re: Dupes?
By: Jay Harris to g00r00 on Mon Oct 11 2021 08:20 pm

Here are two more than came through today. I've turned off my feeds to 2:221/6 & 3:770/1 for now and am just hooked up to 1:229/426: https://paste.ee/p/hBhCR
https://paste.ee/p/nN9aV
Again, the kludges look the same (except for the path and seen-by lines) but the message body has again been modified. One message has a SLMR line where the other one doesn't.

Howdy,

For me it looks like three things:

* Your BBS didnt consider it a dupe

* 229/426 didnt consider it a dupe - since it sent it to you twice.

* Somebody after 261/38 modified the message - possibly 301/1, 320/219 or 240/5832, since it looks like 261/38 sent it 2 ways, that end up at 229/426. So either 301/1 removed the bottom 2 lines, or one of 320/219, 240/5832 added them.

This could go 2 ways, depending if the authors of the software involved think they are doing the "right thing" or not.

1) A consistent method to identify a duplicate - which there is an FTS document that can help here,
2) "Thou should not modify message content" - dont recall if that is an FTS document somewhere

Fortunately for you, there is some software that handles this situation well, so even if something upsream is not doing what is expected you can mitigate it yourself.

Getting popcorn...


...ëîåï
--- SBBSecho 3.14-Linux
* Origin: I'm playing with ANSI+videotex - wanna play too? (3:633/509)

Cool, thanks for the clarification g00r00. Hopefully we've tracked down the node that is modifying the messages. I believe I also saw Nick released an update to D'Bridge that may also help. (Thanks Nick!)

Great to hear!

... Live every day as though it were your last. One day, you'll be right
--- Mystic BBS v1.12 A47 2021/09/23 (Windows/64)
* Origin: Sector 7 * Mystic WHQ (1:129/215)

On 13 Oct 21 17:23:13, G00R00 said the following to Jay Harris:

Cool, thanks for the clarification g00r00. Hopefully we've tracked dow the node that is modifying the messages. I believe I also saw Nick released an update to D'Bridge that may also help. (Thanks Nick!)

Great to hear!

I have "More Cowbell" in my code than you do...

All jokes aside - D'Bridge has a hook to play a batch-file when personal mail is received and comes with a sample MP3 player and the voice of HAL from 2001.

Does Mystic have the same hook to call a program on personal mail tossed?

Also... The JAM storage - I assume Mystic keeps "last read" pointers for each user in a seperate file outside of the spec? Thats what I'm trying to get my head around with. The spec seems to allow this but its... iffy.

Nick
--- Renegade vY2Ka2
* Origin: Joey, do you like movies about gladiators? (1:229/426)

All jokes aside - D'Bridge has a hook to play a batch-file when personal mail is received and comes with a sample MP3 player and the voice of HAL from 2001.

Does Mystic have the same hook to call a program on personal mail tossed?

There are some hooks in place that allows scripting while tossing, so you could do exactly this (and a lot more). I did this while chatting with NetSurge a couple years ago, but it remains unused. Unless I had a power use-case like he had at the time I don't think I would work on it much more though, since its unlikely to ever be used outside of his wild (by cool) ideas! :)

Also... The JAM storage - I assume Mystic keeps "last read" pointers for each user in a seperate file outside of the spec? Thats what I'm trying to get my head around with. The spec seems to allow this but its... iffy.

No, Mystic stores in the JAM .jlr files.

... Next time you wave at me, use more than one finger!
--- Mystic BBS v1.12 A47 2021/09/23 (Windows/64)
* Origin: Sector 7 * Mystic WHQ (1:129/215)

Greetings Jay!

Monday October 11 2021 20:20, you wrote to g00r00 about an urgent matter!:

On 10 Oct 2021, Jay Harris said the following...

I've started seeing some duplicates in my Fidonet bases.

Here are two more than came through today. I've turned off my feeds
to
2:221/6 & 3:770/1 for now and am just hooked up to 1:229/426:

I did the same.. I do run mystic but mystic does not handle the mail so I don't think its a mystic issue. I also get mail from 426, I turned off everything else for a test/ and I still get dupes mostly the "heartbeat" from nick, in any event because I do not use mystic to handle mail and have the same problem as you it must be nick's system somehow?


----
Regards,


Rick Smith (Nitro)

... TAGLINE A BBS addict is hooked when: you consider BBSing better than chocolat
--- GoldED+/LNX 1.1.5-b20180707
* Origin: ----> Abacus BBS! --->>>>bbs.abon.us:2323 (1:105/10)

On 18 Oct 2021, Rick Smith said the following...

off everything else for a test/ and I still get dupes mostly the "heartbeat" from nick, in any event because I do not use mystic to
handle mail and have the same problem as you it must be nick's system somehow?

I haven't really seen any dupes since Nick updated D'Bridge on this side, I don't subscribe to any of the heartbeat echos though.


Jay

... He who dies with the most toys, wins!

--- Mystic BBS v1.12 A47 2021/09/29 (Raspberry Pi/32)
* Origin: Northern Realms (1:229/664)

Greetings Jay!

Monday October 18 2021 11:28, you wrote to me about an urgent matter!:

* Forwarded from area 'MYSTIC'
On 18 Oct 2021, Rick Smith said the following...

off everything else for a test/ and I still get dupes mostly the
"heartbeat" from nick, in any event because I do not use mystic
to handle mail and have the same problem as you it must be nick's
system somehow?

I haven't really seen any dupes since Nick updated D'Bridge on this
side, I don't subscribe to any of the heartbeat echos though.

Yeah I dont on my BBS but on /81 I do because its a hub I carry it all... Although when I just logged in there hasnt been any dupes for the better part of today so maybe it is fixed? There are better than 5k dupes in the there up to today.


----
Regards,


Rick Smith (Nitro)

... YKYHW: you consider BBSing better than chocolate
--- GoldED+/LNX 1.1.5-b20180707
* Origin: ----> Abacus BBS! --->>>>bbs.abon.us:2323 (1:105/10)

Hello Jay!

Monday October 18 2021 11:28, you wrote to me:

* Forwarded from area 'MYSTIC'
On 18 Oct 2021, Rick Smith said the following...

off everything else for a test/ and I still get dupes mostly the
"heartbeat" from nick, in any event because I do not use mystic
to handle mail and have the same problem as you it must be nick's
system somehow?

I haven't really seen any dupes since Nick updated D'Bridge on this
side, I don't subscribe to any of the heartbeat echos though.

I have the dupes down to just the 229426HBEAT ones now, its nice even getting those a couple a day its better than the 5136 dupes I had until I reset it...lol

Regards,

Rick
--- GoldED+/LNX 1.1.5-b20180707
* Origin: Awesome Net- Oregon FTN Hub - www.awesomenet.us (1:105/81)