• Re: Mystic 2.0

    From Ryan Fantus@1:218/820 to g00r00 on Mon Sep 28 15:24:26 2020
    There is almost always a "Mystic 2 build" that I am working on that has never been released. They are sort of a "proof of concept" and one was even written in Java. The current unreleased iteration focuses on an entirely new terminal engine, a entirely new MPL, and uses SQL
    databases. Maybe that will become the real Mystic 2? Or maybe some of those things will get moved into 1.x. Who knows!

    I'm curious your thoughts around SQL databases. On the one hand, it standardizes your database functions around what's been the norm in industry for decades, which is cool. But on the other hand, you're adding a dependency and you'll ultimately open yourself up to "can we use postgres" "can we use mongo" etc.

    I'm curious what you see as the benefit?

    Main reason I ask is because I've considered doing the same with daydream. A proof of concept called vmatik was already done :)

    --- Mystic BBS v1.12 A46 2020/08/06 (Linux/64)
    * Origin: monterey bbs (1:218/820)
  • From Chad Adams@1:19/37 to All on Mon Sep 28 17:53:46 2020
    Good work. When I was writing Cyberbbs I used SQLite. Worked well.

    I’d really like to see expanded httpd with php and message and file area support built in. And a full fledge email server to our local message base. Please!!!!!!

    On 06:36 28/09 , g00r00 wrote:
    Will Mystic 2.0 come to light one day?

    Probably at some point but this is a complex answer.

    Historically I've used those 2.x versions for the purpose of experimenting with
    major changes that end up in the 1.x version. For example the current server >code in MIS started off as a Mystic 2 demo that people could test. It had IPV6
    and SSL/SSH which weren't in 1.x at the time. The current -CFG menus started >off in a Mystic 2.x release, and so on.

    Both times I had a publically released Mystic 2 branch available, the community
    voted to continue with 1 and discontinue working on Mystic 2. They wanted me >to move some key features from 2 demo into 1.

    There is almost always a "Mystic 2 build" that I am working on that has never >been released. They are sort of a "proof of concept" and one was even written >in Java. The current unreleased iteration focuses on an entirely new terminal >engine, a entirely new MPL, and uses SQL databases. Maybe that will become the
    real Mystic 2? Or maybe some of those things will get moved into 1.x. Who >knows!

    --- Mystic BBS v1.12 A47 2020/09/27 (Windows/64)
    * Origin: Sector 7 | Mystic WHQ (1:129/215)


    --
    yrNews Usenet Reader for iOS
    http://appstore.com/yrNewsUsenetReader

    --- Mystic BBS/NNTP v1.12 A47 2020/09/12 (Linux/64)
    * Origin: The ByteXchange BBS | bbs.thebytexchange.com (1:19/37)
  • From g00r00@1:129/215 to Ryan Fantus on Mon Sep 28 21:35:10 2020
    I'm curious your thoughts around SQL databases. On the one hand, it standardizes your database functions around what's been the norm in industry for decades, which is cool. But on the other hand, you're
    adding a dependency and you'll ultimately open yourself up to "can we
    use postgres" "can we use mongo" etc.

    I'm curious what you see as the benefit?

    I don't think there are too many benefits considering the effort it would take to switch over to it. The TLDR version is that if I were starting from scratch today I think I'd use SQLite3, but I am not sure if the effort required to switch over to it justifies the gains at this point.

    If Mystic's data files were properly indexed I don't think there would be much or any performance gain, and it would add the first required dependancy for Mystic to function like you said.

    It would also remove JAM message bases and support for all of the third party tossers. JAM itself has its own set of limitations though, and Mystic's tosser is probably good enough to support being the only tosser these days. This may not be as big of a con as it was in the recent past.

    SQLITE has some issues with concurrancy and scaling since its not really designed for that, but its not like BBSes are huge. I've ran into some of this stuff in my real life job about 8(?) years ago when I was moving a massive DB from SQLITE3 to IBM DB2. But that was also 8 years ago and some of it could have been user error.

    I've read of issues vaccuming (packing) large databases and also issues with inserting large amounts of small inserts on large indexed databases, so I would have to "proof of concept" those to see what that really looks like first.

    (I'd be curious to see how mass uploading 10,000 new files compares to what MUTIL does today since that would need to scan for duplicates and insert 10,000 records. I'd probably start there)

    As far as pros, it would simplify the code once everything is working since all of the file locking and disk writing would be left up to SQLITE.

    It would make adding new columns into a database schema much easier since SQL does all the legwork for you and when I do it with direct data, I have to rebuild the file manually.

    It would provide better access to the data outside of Mystic for writing utilities. This is both a pro and a con because its easier for people to experiment and mess up their data, but having SQL would be better for people to whip up utility functions in whatever script language they like.

    It'd also make it easier for on-the-fly sorting of data for things like message listings and file listings, since Mystic today currently relies on sorting that data on disk. All of the internal paginating could probably be moved to SQL although that I've read is another thing that SQLite isn't amazing at doing.

    Memory usage would probably increase significantly. Mystic is very lightweight when it comes to memory used when reading data, and SQLITE would require much more. I don't know the amount it would use offhand but I do know that if you deprive it of memory it will sometimes slow it to a crawl. On the flip, Mystic today doesn't use more than maybe 8-20 kilobytes when reading data files. If there were scaled across a decent number of nodes the difference might be pretty massive.

    SQLite support would probably be best left for a Mystic 2 spin off given the amount of work it'd probably take to rewrite all the message base and FidoNet/QWK stuff. It seems like it'd be easier to just build something from scratch and that would also allow me to do things like change the directory structure, MCI code system, etc, without having to build an upgrade path for the data files.

    --- Mystic BBS v1.12 A47 2020/09/27 (Windows/64)
    * Origin: Sector 7 | Mystic WHQ (1:129/215)
  • From Ryan Fantus@1:218/820 to g00r00 on Mon Sep 28 21:17:52 2020
    I don't think there are too many benefits considering the effort it
    would take to switch over to it. The TLDR version is that if I were starting from scratch today I think I'd use SQLite3, but I am not sure
    if the effort required to switch over to it justifies the gains at this point.

    This is basically where I'm at with Daydream at the moment. On the one hand, the data files could really use an overhaul, but on the other hand...why? :)

    As far as pros, it would simplify the code once everything is working since all of the file locking and disk writing would be left up to
    SQLITE.

    Right - and making ontological changes or adding rows here and there would be pretty straightforward, without some gross migration binary each release.

    That shit scares me. Hehe.

    It would make adding new columns into a database schema much easier
    since SQL does all the legwork for you and when I do it with direct
    data, I have to rebuild the file manually.

    Yeah, what you said. :P

    It would provide better access to the data outside of Mystic for writing utilities. This is both a pro and a con because its easier for people to experiment and mess up their data, but having SQL would be better for people to whip up utility functions in whatever script language they
    like.

    As Daydream is built in c and opensource, some people have forked it to do custom things. One such thing was to use as a frontend for glftpd (the racer/trader communities were into this). The implementation was pretty
    screwy and destroyed a lot of existing bbses. User accounts were all hosed because people didn't pay attention to data structures. Such a shame.

    It'd also make it easier for on-the-fly sorting of data for things like message listings and file listings, since Mystic today currently relies
    on sorting that data on disk. All of the internal paginating could probably be moved to SQL although that I've read is another thing that SQLite isn't amazing at doing.

    I have multiple nets with upwards of 5k messages per base per net. I haven't hit a perf limit yet as far as I can tell.

    Memory usage would probably increase significantly. Mystic is very lightweight when it comes to memory used when reading data, and SQLITE would require much more. I don't know the amount it would use offhand
    but I do know that if you deprive it of memory it will sometimes slow it to a crawl. On the flip, Mystic today doesn't use more than maybe 8-20 kilobytes when reading data files. If there were scaled across a decent number of nodes the difference might be pretty massive.

    Memory is damn near free these days :) My VPS has 8GB RAM and idles around 400MB. I love efficiency and optimization but also love utility, hehe.

    ...and justifying an expense here and there..

    SQLite support would probably be best left for a Mystic 2 spin off given the amount of work it'd probably take to rewrite all the message base and FidoNet/QWK stuff. It seems like it'd be easier to just build something from scratch and that would also allow me to do things like change the directory structure, MCI code system, etc, without having to build an upgrade path for the data files.

    Did The Millionaire fire up your Mystic 2 spirit?!

    --- Mystic BBS v1.12 A46 2020/08/06 (Linux/64)
    * Origin: monterey bbs (1:218/820)
  • From g00r00@1:129/215 to Ryan Fantus on Tue Sep 29 11:02:50 2020
    Right - and making ontological changes or adding rows here and there
    would be pretty straightforward, without some gross migration binary
    each release.

    That shit scares me. Hehe.

    Yeah it is annoying to do and of course risky to some extent.

    Memory is damn near free these days :) My VPS has 8GB RAM and idles
    around 400MB. I love efficiency and optimization but also love utility, hehe.

    You're not wrong but it depends on what your target is! Lets say someone has 25 active nodes each with the need to have a DB connection. It might look something like:

    Mystic: 20kb * 25 = 500kb used for 25 active nodes
    SQLITE: 20MB * 25 = 500MB of memory
    MYSQL: 150MB

    I just made those numbers up as estimates but consider that Mystic is developed to target a Pi 1 or $5 Pi Zero that has 500MB of RAM as its minimal hardware.

    I realize I am creating scenarios that are unlikely but I still like to take scaling into consideration. Maybe its left over trauma from writing Mystic
    on a 386/16 and trying to make it run in 400kb of memory back in the day lol

    There actually has been one Mystic BBS in recent times that I've seen 20+ active nodes on which is hard to believe but still possible somehow.

    I tossed MYSQL in there because at some point MYSQL running locally becomes more efficient due to the poor scaling and concurrancy of SQLITE. Not trying to knock SQLITE its awesome for what its designed for (its just not designed to scale in a server).

    Did The Millionaire fire up your Mystic 2 spirit?!

    Nah I always have some sort of "dreamer" version of Mystic 2 in the works and occasionally I release them if they start to become usable/I want people to provide feedback. I have all types of concepts for ideas laying around on disk here (things like Threddit for example if you ever saw that)

    I have three iterations of Mystic 2.x on my disk now. The first was last released in 2007 or so, the second was last released in 2016 or so, and the third was never released.

    Unfortunately I lost the Java version which was developed between the first and second released iterations. I am sad about that it was fun to work on!

    --- Mystic BBS v1.12 A47 2020/09/27 (Windows/64)
    * Origin: Sector 7 | Mystic WHQ (1:129/215)