Hi All,

I was just wondering how well the dupe detection works with HPT. I originally had a fidoweb setup here, with multiple hubs feeding message areas, but found out that Mystic didn't do a very good job with dupes with different paths.

I'm a little leery to set any echos up with multiple feeds until I know HPT
is up to the task... :)


---

|03B|09lack |03P|09anther|03(|09RCS|03)|07

--- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
* Origin: Castle Rock BBS - bbs.castlerockbbs.com (1:317/3)

Hello Dan,

I was just wondering how well the dupe detection works with HPT. I originally had a fidoweb setup here, with multiple hubs feeding
message areas, but found out that Mystic didn't do a very good job
with dupes with different paths.

I don't link with multiple feeds for the same area but some of my links do so I
get dupes here. HPT handles that without any problems that I have seen.

I'm a little leery to set any echos up with multiple feeds until I
know HPT is up to the task... :)

I don't think you'll have any problems... :)

Ttyl :-),
Al

... A program is used to turn data into error messages.
--- GoldED+/LNX 1.1.5-b20180707
* Origin: The Rusty MailBox - Penticton, BC Canada (1:153/757)

On 2019 Jul 14 16:51:04, you wrote to All:

I was just wondering how well the dupe detection works with HPT.

it seems to work pretty well here on this point system... granted, it is fed from a synchronet system that is configured to pass on all mail even if it is a dupe... i have my HPT set to retain hashes for just over 3 years... i've not seen any dupes get by and some have been caught that really were dupes or at least using recycled serial numbers...

i say go for it :)

)\/(ark

And to this end they built themselves a stupendous super-computer which was
so amazingly intelligent that even before its data banks had been connected
up it had started from "I think therefore I am" and got as far as deducing
the existence of rice pudding and income tax before anyone managed to turn
it off.
... I just invested half my life savings in yeast!
---
* Origin: (1:3634/12.73)

On 14 Jul 2019, Alan Ianson said the following...

I was just wondering how well the dupe detection works with HPT. I originally had a fidoweb setup here, with multiple hubs feeding

I don't link with multiple feeds for the same area but some of my links
do so I get dupes here. HPT handles that without any problems that I
have seen.

That's what I wanted to hear. :) I have only seen a few messages get caught
as dupes so far with HPT, so I wasn't sure if there just wasn't many coming through, or if HPT wasn't catching them...


---

|03B|09lack |03P|09anther|03(|09RCS|03)|07

--- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
* Origin: Castle Rock BBS - bbs.castlerockbbs.com (1:317/3)

On 14 Jul 2019, mark lewis said the following...

I was just wondering how well the dupe detection works with HPT.

i say go for it :)

Alright. I'll turn on a few echos and see what happens. If it doesn't work, I'll just say that mark told me to... ;)


---

|03B|09lack |03P|09anther|03(|09RCS|03)|07

--- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
* Origin: Castle Rock BBS - bbs.castlerockbbs.com (1:317/3)

I'm a little leery to set any echos up with multiple feeds until I know HPT is up to the task... :)

If you mean multiple feeds feeding the same echo, then it isn't dupe
detection that will keep things from being dupes but seen-bys.

For example, I also cross hub with other major fidonet hubs. If I send a message in this echo it will send it to all the other hubs and since I am sending it to them it will put their node numbers in the seen-by. When they receive it they can see that the other hubs already have it and they won't
send it to them.

Clear as mud?

|15frank |08// |15netsurge
|07disksh0p|08!|07bbs |08% |07bbs.diskshop.ca |08% |07mystic goodness |11SciNet |03ftn hq |08% |07https://diskshop.ca/scinet

--- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
* Origin: % disksh0p!bbs % bbs.diskshop.ca % SciNet ftn hq % (1:229/101)

On 14 Jul 2019, Frank Linhares said the following...

If you mean multiple feeds feeding the same echo, then it isn't dupe detection that will keep things from being dupes but seen-bys.

That's true. But, it would still cause dupes... :)

For example, I also cross hub with other major fidonet hubs. If I send a message in this echo it will send it to all the other hubs and since I am sending it to them it will put their node numbers in the seen-by. When they receive it they can see that the other hubs already have it and
they won't send it to them.

That's how we have fsxNet hubs set up, and it seems to work fine. I just
didn't want to cause a bunch dupes in any fidonet echos... Right now, I do
have this echo set up with multiple feeds. I just want to make sure... :)


---

|03B|09lack |03P|09anther|03(|09RCS|03)|07

--- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
* Origin: Castle Rock BBS - bbs.castlerockbbs.com (1:317/3)

That's true. But, it would still cause dupes... :)

How would it cause dupes? If one of my peer hubs sends me a message that has another peer hubs node number in the seen-by then my system won't export it
to theres, plain and simple.

|15frank |08// |15netsurge
|07disksh0p|08!|07bbs |08% |07bbs.diskshop.ca |08% |07mystic goodness |11SciNet |03ftn hq |08% |07https://diskshop.ca/scinet

--- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
* Origin: % disksh0p!bbs % bbs.diskshop.ca % SciNet ftn hq % (1:229/101)

On 15 Jul 2019, Frank Linhares said the following...

That's true. But, it would still cause dupes... :)

How would it cause dupes? If one of my peer hubs sends me a message that has another peer hubs node number in the seen-by then my system won't export it to theres, plain and simple.

If the seen-bys are read and processed correctly, there wouldn't be any
dupes. But, not all systems do that, as I found out previously with Mystic. I was receiving messages from one hub, they would get sent out to my other hub, and caused a mess in those echos.


---

|03B|09lack |03P|09anther|03(|09RCS|03)|07

--- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
* Origin: Castle Rock BBS - bbs.castlerockbbs.com (1:317/3)

If the seen-bys are read and processed correctly, there wouldn't be any dupes. But, not all systems do that, as I found out previously with Mystic. I was receiving messages from one hub, they would get sent out
to my other hub, and caused a mess in those echos.

I wasn't referencing Mystic. I was talking about Husky.

|15frank |08// |15netsurge
|07disksh0p|08!|07bbs |08% |07bbs.diskshop.ca |08% |07mystic goodness |11SciNet |03ftn hq |08% |07https://diskshop.ca/scinet

--- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
* Origin: % disksh0p!bbs % bbs.diskshop.ca % SciNet ftn hq % (1:229/101)

Hi Dan!

14 Jul 2019 16:51, from Dan Richter -> All:

I'm a little leery to set any echos up with multiple feeds until I
know HPT is up to the task... :)

Many HPT users, including me, do that.
I have not heared (or seen) any major issues.

You should be good to go.
I have these configs:
# The type of dupebase used. See docs. # dupeBaseType <TextDupes | HashDupes | HashDupesWMsgId | CommonDupeBase> DupeBaseType HashDupesWMsgId
# Maximum days for storing you hashes in CommonDupeBase. # For other dupebase types see -DupeHistory option in EchoArea statement. AreasMaxDupeAge 1100

CU, Ricsi

... Character is what you are. Reputation is what people think you are.
--- GoldED+/LNX
* Origin: I always use the goodest English. (2:310/31)

On 15 Jul 2019, Richard Menedetter said the following...

Many HPT users, including me, do that.
I have not heared (or seen) any major issues.

Well, I have this echo set to receive from two hubs as a test.

# For other dupebase ty pes see -DupeHistory option in EchoArea
statement. AreasMaxDupeAge 1100

1100?! Wow! I thought I was doing good when I had it set to 31... :) I did change it to 1100, just to be safe.

# The type of dupebase used. See docs. # dupeBaseType <TextDupes | HashDupes | H ashDupesWMsgId | CommonDupeBase> DupeBaseType HashDupesWMsgId # Maximum days for storing you hashes in CommonDupeBase.

I didn't have the dupeBaseType set, but I did set it now to HashDupesWMsgId.

Thanks


---

|03B|09lack |03P|09anther|03(|09RCS|03)|07

--- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
* Origin: Castle Rock BBS - bbs.castlerockbbs.com (1:317/3)

Hi Dan!

15 Jul 2019 01:45, from Dan Richter -> Richard Menedetter:

# For other dupebase ty pes see -DupeHistory option in EchoArea
statement. AreasMaxDupeAge 1100

1100?! Wow! I thought I was doing good when I had it set to 31... :)
I did change it to 1100, just to be safe.

3 years
Why not ;)

# The type of dupebase used. See docs. # dupeBaseType <TextDupes
| HashDupes | H ashDupesWMsgId | CommonDupeBase> DupeBaseType
HashDupesWMsgId # Maximum days for storing you hashes in
CommonDupeBase.

I didn't have the dupeBaseType set, but I did set it now to HashDupesWMsgId.

I have forgotten what the config default is.
When I had a look I concluded for myself that that is the best type.
I hope I was right ;)

CU, Ricsi

... If you do a favor, forget it. If you receive a favor, remember it.
--- GoldED+/LNX
* Origin: Windows NT? New Technology? I don't think so... (2:310/31)

On 2019 Jul 14 19:31:46, you wrote to me:

I was just wondering how well the dupe detection works with HPT.

i say go for it :)

Alright. I'll turn on a few echos and see what happens. If it doesn't

work,

I'll just say that mark told me to... ;)

hahahahaha... :thumbsup:

)\/(ark

And to this end they built themselves a stupendous super-computer which was
so amazingly intelligent that even before its data banks had been connected
up it had started from "I think therefore I am" and got as far as deducing
the existence of rice pudding and income tax before anyone managed to turn
it off.
... Money Talks! Mine always says goodbye!
---
* Origin: (1:3634/12.73)

On 2019 Jul 14 22:57:20, you wrote to Frank Linhares:

That's true. But, it would still cause dupes... :)

How would it cause dupes? If one of my peer hubs sends me a message
that has another peer hubs node number in the seen-by then my system
won't export it to theres, plain and simple.

If the seen-bys are read and processed correctly, there wouldn't be
any dupes.

ummm... yes, there could be... but that depends on what is defined as a dupe... is it duplicated MSGID? is it duplicate message body even with a different MSGID? is it a duplicate message body that has been reformatted in transit (which should never be done)??

But, not all systems do that, as I found out previously with Mystic. I
was receiving messages from one hub, they would get sent out to my
other hub, and caused a mess in those echos.

mystic is known to have problems in a couple of areas... some versions are known to reformat messages that should not be reformatted... that reformatting will absolutely throw off tossers that hash the message bodies in addition to tracking MSGID control lines and headers... we've seen messages passing through mystic systems coming out the other side with EOLs added where there were none before, characters stripped that should not have been and similar problems... this stuff was all working perfectly in v1.11 when i was using mystic and providing feedback but something has changed that should not have and well... it just ain't right...

)\/(ark

And to this end they built themselves a stupendous super-computer which was
so amazingly intelligent that even before its data banks had been connected
up it had started from "I think therefore I am" and got as far as deducing
the existence of rice pudding and income tax before anyone managed to turn
it off.
... No woman ever shot a man while he was doing the dishes.
---
* Origin: (1:3634/12.73)

Hi Dan.

15 Jul 19 09:10, Richard Menedetter wrote to you:

Hi Dan!

14 Jul 2019 16:51, from Dan Richter -> All:

I'm a little leery to set any echos up with multiple feeds until I
know HPT is up to the task... :)

Many HPT users, including me, do that.
I have not heared (or seen) any major issues.

I second this. I've run hpt for a long time as a fidoweb hub. The only bug found was that messages without MSGID were not detected as dupes because they arrived via different paths. It was fixed. Works nice now.

Do you want another feed? ;-)

'Tommi

--- GoldED+/LNX 1.1.5-b20180707
* Origin: nntps://news.fidonet.fi (2:221/6)

On 15 Jul 2019, Richard Menedetter said the following...

1100?! Wow! I thought I was doing good when I had it set to 31... :) I did change it to 1100, just to be safe.

3 years
Why not ;)

Good point. Why not. ;) I think the default is 10...

I have forgotten what the config default is.
When I had a look I concluded for myself that that is the best type.
I hope I was right ;)

Between you and me, we'll find out... :)


---

|03B|09lack |03P|09anther|03(|09RCS|03)|07

--- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
* Origin: Castle Rock BBS - bbs.castlerockbbs.com (1:317/3)

On 2019 Jul 15 13:14:42, you wrote to Richard Menedetter:

I did change it to 1100, just to be safe.

3 years
Why not ;)

Good point. Why not. ;) I think the default is 10...

do i see some binary humor there? -=B-)

)\/(ark

And to this end they built themselves a stupendous super-computer which was
so amazingly intelligent that even before its data banks had been connected
up it had started from "I think therefore I am" and got as far as deducing
the existence of rice pudding and income tax before anyone managed to turn
it off.
... A wise man who knows proverbs, reconciles difficulties -Yoruba
---
* Origin: (1:3634/12.73)

On 15 Jul 2019, mark lewis said the following...

If the seen-bys are read and processed correctly, there wouldn't be any dupes.

ummm... yes, there could be... but that depends on what is defined as a dupe... is it duplicated MSGID? is it duplicate message body even with a different MSGID ? is it a duplicate message body that has been
reformatted in transit (which sho uld never be done)??

In this case, it would be the same message and MSGID, but with different PATH and SEEN-BY kludge lines.

mystic is known to have problems in a couple of areas... some versions
are known to reformat messages that should not be reformatted... that reformatting will a bsolutely throw off tossers that hash the message bodies in addition to tracking MSGID control lines and headers... we've

Now, I don't want to bash Mystic, as I run it here, and love it. There are
some quirks when trying to run it as a hub, that I've seen. That's part of
the reason I switched over to Husky projects. :)


---

|03B|09lack |03P|09anther|03(|09RCS|03)|07

--- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
* Origin: Castle Rock BBS - bbs.castlerockbbs.com (1:317/3)

On 15 Jul 2019, Tommi Koivula said the following...

I second this. I've run hpt for a long time as a fidoweb hub. The only
bug found was that messages without MSGID were not detected as dupes because they arrived via different paths. It was fixed. Works nice now.

So, if you have multiple feeds for an echo, let's say FIDOSOFT.HUSKY. Do you have new messages sent to both feeds for that echo?

I have two feeds set for this echo now, but I did set one of them to WRITE-ONLY. I guess I just don't want to cause problems for other systems.

Do you want another feed? ;-)

I've got that covered already. Right now, I've got access to 5 fido hubs. :) Thank you though.


---

|03B|09lack |03P|09anther|03(|09RCS|03)|07

--- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
* Origin: Castle Rock BBS - bbs.castlerockbbs.com (1:317/3)

Re: Re: HPT and dupe detection
By: Dan Richter to Tommi Koivula on Mon Jul 15 2019 13:25:44

So, if you have multiple feeds for an echo, let's say FIDOSOFT.HUSKY. Do

you have new messages sent to both feeds for that echo?

yes, absolutely... otherwise the multiple connections to ensure that mail traverses all systems won't work...

some of my echos have five or more links that are also interlinked either between themselves or even with other systems between them... it all flows fine... the main thing is the throwing away of all the dupes... it is quite a mindset change from traditional FTN echomail distribution...

i say that as one of the Z1 backbone star hubs, too ;)

)\/(ark
--- SBBSecho 3.07-Linux
* Origin: SouthEast Star Mail HUB - SESTAR (1:3634/12)

So, if you have multiple feeds for an echo, let's say FIDOSOFT.HUSKY. Do you have new messages sent to both feeds for that echo?

I have two feeds set for this echo now, but I did set one of them to WRITE-ONLY. I guess I just don't want to cause problems for other
systems.

Yes. If the message comes from your system then both of those feeds would be added to the seen-by. This will prevent them from sending it to each other.

I have two feeds set for this echo now, but I did set one of them to WRITE-ONLY. I guess I just don't want to cause problems for other
systems.

If you want to put your mind at ease, feel free to connect to FIDOSOFT.HUSKY
on my system and test it.

|15frank |08// |15netsurge
|07disksh0p|08!|07bbs |08% |07bbs.diskshop.ca |08% |07mystic goodness |11SciNet |03ftn hq |08% |07https://diskshop.ca/scinet

--- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
* Origin: % disksh0p!bbs % bbs.diskshop.ca % SciNet ftn hq % (1:229/101)

On 15.7.2019 19.25, Dan Richter wrote:

I second this. I've run hpt for a long time as a fidoweb hub. The only bug found was that messages without MSGID were not detected as dupes because they arrived via different paths. It was fixed. Works nice now.

So, if you have multiple feeds for an echo, let's say FIDOSOFT.HUSKY. Do
you have new messages sent to both feeds for that echo?

Oh yes. :)

I have two feeds set for this echo now, but I did set one of them to WRITE-ONLY. I guess I just don't want to cause problems for other systems.

Sending to both does not normally cause problems. Properly behaving
tossers check seenbys and paths to _prevent_ dupes. In fidoweb style
echomail transportation there are always dupes when one message goes
different ways from a node to another.

Do you want another feed? ;-)

I've got that covered already. Right now, I've got access to 5 fido
hubs. :) Thank you though.

You could have 6. :D

'Tommi

--- Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.
* Origin: smapinntpd/linux @ nntps://news.fidonet.fi (2:221/6.0)

On 16 Jul 2019, Tommi Koivula said the following...

Sending to both does not normally cause problems. Properly behaving tossers check seenbys and paths to _prevent_ dupes. In fidoweb style echomail transportation there are always dupes when one message goes different ways from a node to another.

Ok, I've got the messages being sent to both hubs. It's always interesting to see which one delivers the new messages first... :)

I've got that covered already. Right now, I've got access to 5 fido hubs. :) Thank you though.

You could have 6. :D

Do you have any English echos that aren't available from the other hubs? ;)
I'm connected to 1:229/426, 1:261/38, 1:15/0, 2:250/1, and 3:770/1.


---

|03B|09lack |03P|09anther|03(|09RCS|03)|07

--- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
* Origin: Castle Rock BBS - bbs.castlerockbbs.com (1:317/3)

On 16.07.2019 5:26, Dan Richter -> Tommi Koivula :

I've got that covered already. Right now, I've got access to 5 fido hubs. :) Thank you though.

You could have 6. :D

Do you have any English echos that aren't available from the other hubs? ;) I'm connected to 1:229/426, 1:261/38, 1:15/0, 2:250/1, and 3:770/1.

I have no link with these, so I can't tell... :) But "full" fidoweb feed
plus some selected usenet groups.

'Tommi

--- Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.
* Origin: smapinntpd/linux @ nntps://news.fidonet.fi (2:221/6.0)