Read Post
Sat, 14 Oct 2023 06:16:02 -0700
Andy from
private IP
/all
Technical explanation for the simplicity of this board
I was designing this site in early September, and I realized that all it would
take to match the original JDU is a very simple filesystem-based BBS without
even using a SQL database. This drastically simplifies everything, but adds
other challenges. One advantage is that the board is fully portable and I can
have a complete backup system very easily that continuously syncs with the
primary. This eliminates the problem of SQL replication, which is otherwise
currently beyond the scope of my web application projects. The general approach
is using plain text files to contain the posts and putting them in an actual
file hierarchy with command line tools to extract the relevant data, which is
tagged. I believe such large community sites as Craigslist use this same method
to handle their global high-traffic systems, because otherwise SQL replication
and performance are the two primary challenges.
In terms of disadvantages, it's a lot harder to have an effective search box
without a proper search index as would be found in any SQL database. However, I
think I've come up with a blend of performance, convenience, and accuracy by
using standard command line tools in conjunction with plain text files. The
Linux tools "grep" and "sed" running in a simple PHP script are all I really
need. Grep will search text files with extreme speed, and sed will seek to
particular lines in the file and perform operations on them. Sed is how I
extract subject lines and metadata for posts, and grep is used to find the reply
tag to show the number of replies. The cool part about the search box is that I
will be able to add regular expression search in the near future easily, so
people who know how to use PCRE can find exactly what they are looking for. I'm
not sure Google is going to index all the posts, but I'll figure that out.
I think Admin on the original JD Underground took an approach similar to this
one, although he had a traditional login system and it was always unclear what
type of technology he was using on the backend. Either way, this was a fun
challenge to start and it will be a continuously improved project. Feedback and
feature requests are welcome.
_reply
Fri, 29 Dec 2023 19:42:15 -0800
zerosugar
from private IP
/all
i like it bro. don't sweat it too much.
_reply
Wed, 03 Jan 2024 15:09:15 -0800
cowgod from
private IP
/all
It looks good but you may wish to add more Graphics to attract Preps, or study
www.autoadmit.com if you want more Losers and Nerds to post here.
_reply
Wed, 03 Jan 2024 21:06:31 -0800
Andy from
private IP
/all
One update: I was researching the Boyer-Moore algorithm for something else, and
it turns out that grep is blindingly fast due to its use of Boyer-Moore for
string searching. For technical reasons I don't need to get into here, my
approach to this simple BBS crushes all other general purpose bulletin board
software, such as phpBB. All of those types of forum systems rely on dynamic
pages generated from a SQL database, and once you have a critical mass of users,
the performance slows down dramatically. When my techniques here are combined
with another system I developed 20 years ago when I was doing my law school
website, gigantic amounts of traffic are possible on a single server. I call my
system Mosaic, because conceptually it uses shards and slices of files to
assemble into a final HTML page without using any database. This is a great
project and one that I enjoy doing. Discuss.
_reply
Sat, 06 Jan 2024 12:32:54 -0800
Gigapi from
private IP
/all
Never made a web app that didn’t use a relational database on the backend
myself.
What is this architecture using plain text known as?
Regarding db duplication etc , true if you’re serving a huge number of users
in different localities and latency is important but for a site like this you
would never need that imho
_reply
Sat, 06 Jan 2024 12:39:48 -0800
Andy from
private IP
/all
Welcome Gigapi! I would call this a filesystem/NoSQL backend. I don't think
it's going to get to the point of needing anything more advanced, and since it
works, I use it...lol. Clicking on each individual post is basically completed
within the seek time of the RAID 5 that I use for storage (9 ms). The only more
significant pages are the index (60 ms so far) and the user profile pages (100+
ms). These pages are expected to be a bigger load on the disks, and I am
watching those closely to make sure that my backend with Linux command line
utilities remains optimal. I'm happy with this so far.
Replies require login.
