Hello g00r00!

I've noticed that since the A48 prealpha I'm running (2022/02/02), mutil MassUpload appears to add duplicates to the file areas every now and then.

It doesn't happen all the time, and the files and areas appear to differ from time to time when this happens, which makes it hard to spot any pattern.

Example:


--------------------- MUTIL v1.12 A48 2022/02/02 Mon, Feb 07 2022 (loglevel 3) + 2022-02-07 00:00:02 Startup using maint.ini
- 2022-02-07 00:00:02 EXEC MassUpload
- 2022-02-07 00:00:02 EXEC GenerateTopLists
- 2022-02-07 00:00:02 EXEC PackUserBase
- 2022-02-07 00:00:02 EXEC PurgeMessageBases
- 2022-02-07 00:00:02 EXEC PackMessageBases
- 2022-02-07 00:00:02 EXEC LinkMessages
- 2022-02-07 00:00:02 EXEC MergeNodeLists
- 2022-02-07 00:00:02 EXEC PackFileBases
- 2022-02-07 00:00:02 EXEC FileSort
- 2022-02-07 00:00:02 EXEC GenerateAllFiles
- 2022-02-07 00:00:02 EXEC Export_FILEBONE.NA
- 2022-02-07 00:00:02 EXEC Export_AREAS.BBS
- 2022-02-07 00:00:02 EXEC Export_Golded
+ 2022-02-07 00:00:02 Process: Mass Upload Files
+ 2022-02-07 00:00:02 Uploads
+ 2022-02-07 00:00:02 Allocating 128
+ 2022-02-07 00:00:02 Free 128
+ 2022-02-07 00:00:02 FSX_ARTS: ANSI Art - Groups, Individuals, Etc.
+ 2022-02-07 00:00:02 Allocating 2,472
+ 2022-02-07 00:00:03 Free 2,472
+ 2022-02-07 00:00:03 FSX_DOOR: BBS Doors, Games, Etc.
+ 2022-02-07 00:00:03 Allocating 888
+ 2022-02-07 00:00:04 Free 888
+ 2022-02-07 00:00:04 FSX_IMGE: Image Files (Various)
+ 2022-02-07 00:00:04 Allocating 8,856
+ 2022-02-07 00:00:04 Add: asicbrwp.zip To: FSX_IMGE: Image Files (Various)
+ 2022-02-07 00:00:04 Add: astronwp.zip To: FSX_IMGE: Image Files (Various)
+ 2022-02-07 00:00:05 Add: ap220202.zip To: FSX_IMGE: Image Files (Various)
+ 2022-02-07 00:00:05 Add: ap220203.zip To: FSX_IMGE: Image Files (Various)
+ 2022-02-07 00:00:05 Add: ap220204.zip To: FSX_IMGE: Image Files (Various)
+ 2022-02-07 00:00:05 Free 8,856
+ 2022-02-07 00:00:05 FSX_INFO: Weekly Infopacks (fsxNet, Etc.)


(Cut off -- all OK in what follows; Allocating and Free with matching values all the time there as well.)

The file area in question now has e.g. a duplicate ap220203.zip file:


ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿ ³ ± File Listing (00 tagged) FSX_IMGE: Image Files (Various) ³ ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ
01 ap220203.zip 430KB 02/07/22
NASA Astronomy Picture of the
Day (plus published report)
À Uploader: Mystic BBS
02 ap220203.zip 430KB 02/03/22
NASA Astronomy Picture of the
Day (plus published report)
À Uploader: Mystic TIC


The maint.ini contains (among other things):


[MassUpload]

; this function searches all configured file directories for new
; files and will upload them into the BBS. It will attempt to
; import FILE_ID.DIZ using the configured archivers if the option
; is enabled.

; Name to save has the uploader

uploader_name = Mystic BBS

; Import FILE_ID.DIZ? 1=yes

import_fileid = 1

; No description string used when no FILE_ID.DIZ is imported.

no_description = No Description

; Ignore list one file mask per line (allows * and ? wildcards)

ignore = .htaccess
ignore = .htpasswd

ignore = *.readme
ignore = *.README

ignore = file_id.*
ignore = FILE_ID.*
ignore = files.bbs
ignore = FILES.BBS
ignore = lost+found
ignore = readme
ignore = README
ignore = readme.txt
ignore = README.txt
ignore = README.TXT

[PackFileBases]

; If true, MUTIL will check to make sure files exist, and will update their
; file size in the listing, optionally removing missing files (remove_missing)

check_files = true

; If true, MUTIL will remove any files which do not exist from the file
; list database while checking integrity. Otherwise they will be flagged as
; OFFLINE in the file listing. 0 byte files are considered "missing"

remove_missing = false


Can you think of any way to debug this, and/or anything that could cause
this?

Just a guess but could this change possibly have anything to do with it:

! MUTIL FilePack was not properly setting the OFFLINE flag when check_files
was enabled, remove_missing was disabled, and a file was physically removed

E.g. that the file area data files are being "corrupted" in some way when packing, causing the additions to be done as something doesn't quite match up?

I've collected the data files before and after (from my NAS trashcan) if you would like to take a peek at some of the files to see if they could possibly be "corrupted" in some way:

https://scbbs.nsupdate.info/files/FSX_IMGE.tar.gz

(old/ = before MassUpload; new/ = after MassUpload)

Many thanks in advance, and please let me know if there's anything I can do!

Best regards
Bj”rn
--- Mystic BBS v1.12 A48 2022/02/02 (Linux/64)
* Origin: Star Collision BBS, Uppsala, Sweden (2:201/137)

On 07 Feb 2022 at 02:06p, Bj”rn Wiberg pondered and said...

Hi there

--------------------- MUTIL v1.12 A48 2022/02/02 Mon, Feb 07 2022 (loglevel 3) + 2022-02-07 00:00:02 Startup using maint.ini
- 2022-02-07 00:00:02 EXEC MassUpload
- 2022-02-07 00:00:02 EXEC GenerateTopLists
- 2022-02-07 00:00:02 EXEC PackUserBase
- 2022-02-07 00:00:02 EXEC PurgeMessageBases
- 2022-02-07 00:00:02 EXEC PackMessageBases
- 2022-02-07 00:00:02 EXEC LinkMessages
- 2022-02-07 00:00:02 EXEC MergeNodeLists
- 2022-02-07 00:00:02 EXEC PackFileBases
- 2022-02-07 00:00:02 EXEC FileSort
- 2022-02-07 00:00:02 EXEC GenerateAllFiles
- 2022-02-07 00:00:02 EXEC Export_FILEBONE.NA
- 2022-02-07 00:00:02 EXEC Export_AREAS.BBS
- 2022-02-07 00:00:02 EXEC Export_Golded

Just an observation but that's a lot of stuff to run in a maint. I wonder if it would be better to run the mass upload process as something separate from the rest. I personally don't really see it as a nightly maint thing.
--- Mystic BBS v1.12 A47 2021/11/06 (Linux/64)
* Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (3:770/100)

Hello Paul!

Thank you for your reply!

On 08 Feb 2022, Paul Hayton said the following...

Just an observation but that's a lot of stuff to run in a maint. I

Yes =), some of it could perhaps be run less often (e.g. weekly or monthly).

Best regards
Bj”rn
--- Mystic BBS v1.12 A48 2022/02/02 (Linux/64)
* Origin: Star Collision BBS, Uppsala, Sweden (2:201/137)

On 08 Feb 2022 at 09:17a, Bj”rn Wiberg pondered and said...

Yes =), some of it could perhaps be run less often (e.g. weekly or monthly).

If it helps I need to revisit my ini files too :)

--- Mystic BBS v1.12 A47 2021/12/24 (Linux/64)
* Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (3:770/100)

Just an observation but that's a lot of stuff to run in a maint. I
wonder if it would be better to run the mass upload process as something separate from the rest. I personally don't really see it as a nightly maint thing.

I would also add to this that purging/packing the msg/file bases isn't something that I would be expected to do nightly either.

It shouldn't hurt things to run every night but it does recreate all of your msg/file base data files and is probably the "heaviest" process that MUTIL does.

I personally run it manually every month or so but if I had a public BBS I'd probably schedule it to run once per week.
--- Mystic BBS v1.12 A48 2022/02/03 (Windows/64)
* Origin: Sector 7 * Mystic WHQ (1:129/215)

On 08 Feb 2022 at 12:19p, g00r00 pondered and said...

I would also add to this that purging/packing the msg/file bases isn't something that I would be expected to do nightly either.

It shouldn't hurt things to run every night but it does recreate all of your msg/file base data files and is probably the "heaviest" process
that MUTIL does.

I personally run it manually every month or so but if I had a public BBS I'd probably schedule it to run once per week.

Yep. good thoughts thanks... I recall it was packing that was the more intensive, is that right?

Could you happily purge more often than pack?

The other question I had was regarding linking, is it better to link each time packets are processed or do you think it's more of a nightly thing?

Thanks :)
--- Mystic BBS v1.12 A47 2021/12/24 (Linux/64)
* Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (3:770/100)

On 09 Feb 2022, Paul Hayton said the following...

The other question I had was regarding linking, is it better to link
each time packets are processed or do you think it's more of a nightly thing?

I personally link each time I toss in my mailin.ini.

[General]
ImportEchoMail = true
FileToss = true
LinkMessages = true


Jay

... His eyes are so bad, he has to wear contact lenses to see his glasses.
--- Mystic BBS v1.12 A47 2021/12/24 (Raspberry Pi/32)
* Origin: Northern Realms (1:229/664)

On 08 Feb 2022 at 09:36p, Jay Harris pondered and said...

I personally link each time I toss in my mailin.ini.

All this chatter of linking... feels like we should be holding hands and singing kum-bye-ya heheh
--- Mystic BBS v1.12 A47 2021/12/24 (Linux/64)
* Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (3:770/100)

Yep. good thoughts thanks... I recall it was packing that was the more intensive, is that right?

Could you happily purge more often than pack?

You could but it makes most sense to me anyway to run them both together maybe once a week or so.

Purging will enforce the message limits on your bases (max messages and max message age) by marking them deleted. It will also go through and remove any private messages that are in the data files for users that no longer exist.

Its not as intensive as rebuilding the message bases, but its still a good amount of work being done since its scanning through a lot of data.

Packing will then physically rebuild your message bases without all of the data that purge marked to be deleted. It keeps the bases slim and the message numbers in order, etc. It can sometimes fix errors as well.

... They say there's always one weirdo on the bus, but I couldn't find them! --- Mystic BBS v1.12 A48 2022/02/03 (Windows/64)
* Origin: Sector 7 * Mystic WHQ (1:129/215)