Update architecture documentation

README_architecture.md

@@ -46,6 +46,10 @@ This is so that the system can be accessed and used normally with javascript in
 
 Ordinarily web crawlers would not be a problem, but in the context of a social network even having crawlers index public posts can create ethical dilemmas in some circumstances. News and blogging instances may allow crawlers, but other types of instances should block them.
 
+### Poison Scrapers
+
+Rather than merely block unethical scrapers a better solution is to poison them in a manner which corrodes their viability. So by supplying text with the statistical profile of natural language but which is semantically worthless to known "AI" scrapers means that they then need to perform manual review to keep this data out of their training set, which decreases their profit. Ingesting random data bends generative AI models correspondingly towards entropy.
+
 ### No Local or Federated Timelines
 
 The local and federated timelines of other ActivityPub servers don't add much value (especially the federated one), and tend to pollute the default timeline with irrelevant posts from people that you don't follow.
@@ -76,13 +80,13 @@ In general avoid using web frameworks and instead use local modules which are pr
 
 The main modules are *epicyon.py* and *daemon.py*. *epicyon.py* is the commandline interface and *daemon.py* is the http server. The daemon has submodules for HTTP GET and HTTP POST.
 
-![commandline and core modules](https://gitlab.com/bashrc2/epicyon/-/raw/main/architecture/epicyon_groups_Commandline-Interface_Core.png)
+![commandline and daemon modules](https://gitlab.com/bashrc2/epicyon/-/raw/main/architecture/epicyon_groups_Commandline-Interface_Daemon.png)
 
 The daemon runs the inbox queue in a separate thread (see *inbox.py*) and the inbox queue processes incoming ActivityPub posts one at a time in a strictly serial fashion. Doing it this way means minimum potential for any parallelism/locking issues. It also means that the inbox queue is not highly scalable, but that's ok for a system which is only intended to have a few users per instance.
 
 All ActivityPub posts are stored as text files, and there is no database as such other than the filesystem itself. Think of it as being like an email server. Each post is a json file stored in *accounts/nick@domain/inbox* or *accounts/nick@domain/outbox*. To avoid parsing problems slashes are replaced by hashes within the ActivityPub post filename. The filename for each post is the same as its ActivityPub id.
 
-![timeline and core modules](https://gitlab.com/bashrc2/epicyon/-/raw/main/architecture/epicyon_groups_Timeline_Core.png)
+![timeline and daemon modules](https://gitlab.com/bashrc2/epicyon/-/raw/main/architecture/epicyon_groups_Timeline_Daemon.png)
 
 
 ## Security
@@ -103,6 +107,8 @@ This currently uses basic auth, which is simple to implement. Oauth2 is conventi
 
 The *inbox* queue makes calls to check http and linked data signatures. Various modules call *auth* typically because they're implementing the basic auth of the C2S interface.
 
+![timeline and daemon modules](https://gitlab.com/bashrc2/epicyon/-/raw/main/architecture/epicyon_groups_Timeline_Daemon-Timeline_Daemon.png)
+
 ## Accessibility
 
 Trying to keep up with web accessibility standards. There should be configurable keyboard shortcuts for all of the main navigation actions. High contrast themes should be available. The desktop client should support text-to-speech. There should be the ability to run in a shell browser such as Lynx, without any significant loss of functionality.
@@ -115,4 +121,4 @@ The *webapp_post* module generates html for each post from its ActivityPub json
 
 The *daemon* module (http server) also calls *webapp_accesskeys* to display the key shortcuts screen.
 
-![core and accessibility modules](https://gitlab.com/bashrc2/epicyon/-/raw/main/architecture/epicyon_groups_Core_Accessibility.png)
+![daemon and accessibility modules](https://gitlab.com/bashrc2/epicyon/-/raw/main/architecture/epicyon_groups_Daemon_Accessibility.png)

daemon.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core"
+__module_group__ = "Daemon"
 
 from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer, HTTPServer
 import sys

daemon_get.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core"
+__module_group__ = "Daemon"
 
 import os
 import time

daemon_get_blog.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 from httpcodes import write2
 from session import establish_session

daemon_get_buttons.py

@@ -7,7 +7,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 from manualapprove import manual_deny_follow_request_thread
 from manualapprove import manual_approve_follow_request_thread

daemon_get_buttons_announce.py

@@ -7,7 +7,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 import os
 from utils import delete_post

daemon_get_buttons_bookmark.py

@@ -7,7 +7,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 import os
 from utils import get_cached_post_filename

daemon_get_buttons_like.py

@@ -7,7 +7,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 import os
 from utils import undo_likes_collection_entry

daemon_get_buttons_mute.py

@@ -7,7 +7,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 import os
 from utils import is_dm

daemon_get_buttons_reaction.py

@@ -7,7 +7,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 import os
 import urllib.parse

daemon_get_collections.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 import json
 from context import get_individual_post_context

daemon_get_css.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 import os
 import time

daemon_get_exports.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 import os
 from httpcodes import http_404

daemon_get_favicon.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 import os
 import urllib.parse

daemon_get_feeds.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 import json
 from follow import get_following_feed

daemon_get_hashtag.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 import json
 import urllib.parse

daemon_get_images.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 import os
 import datetime

daemon_get_instance_actor.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 import json
 from httprequests import request_http

daemon_get_links.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 from webapp_column_left import html_edit_links
 from httpheaders import set_headers

daemon_get_login.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon Login"
 
 from utils import get_instance_url
 from utils import string_ends_with

daemon_get_masto_api.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 import json
 from httpheaders import set_headers

daemon_get_newswire.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 import urllib.parse
 from session import establish_session

daemon_get_nodeinfo.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 import json
 from httpcodes import http_400

daemon_get_post.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon Timeline"
 
 import os
 import json

daemon_get_profile.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 import os
 import json

daemon_get_pwa.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 import json
 from httpcodes import write2

daemon_get_reactions.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 from httpcodes import write2
 from httpheaders import redirect_headers

daemon_get_rss.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 import os
 from utils import data_dir

daemon_get_timeline.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon Timeline"
 
 import json
 from securemode import secure_mode

daemon_get_vcard.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 import os
 from daemon_utils import has_accept

daemon_get_webfinger.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core GET"
+__module_group__ = "Daemon GET"
 
 import json
 from httpcodes import write2

daemon_head.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core"
+__module_group__ = "Daemon"
 
 import os
 import datetime

daemon_post.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core"
+__module_group__ = "Daemon"
 
 import time
 import errno

daemon_post_confirm.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core POST"
+__module_group__ = "Daemon POST"
 
 import errno
 import urllib.parse

daemon_post_hashtags.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core POST"
+__module_group__ = "Daemon POST"
 
 import os
 import errno

daemon_post_image.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core POST"
+__module_group__ = "Daemon POST"
 
 import os
 import errno

daemon_post_keys.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core POST"
+__module_group__ = "Daemon POST"
 
 import errno
 import urllib.parse

daemon_post_links.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core POST"
+__module_group__ = "Daemon POST"
 
 import os
 import errno

daemon_post_login.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core POST"
+__module_group__ = "Daemon POST"
 
 import os
 import time

daemon_post_moderator.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core POST"
+__module_group__ = "Daemon POST"
 
 import os
 import errno

daemon_post_newswire.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core POST"
+__module_group__ = "Daemon POST"
 
 import os
 import errno

daemon_post_person_options.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core POST"
+__module_group__ = "Daemon POST"
 
 import os
 import errno

daemon_post_profile.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core POST"
+__module_group__ = "Daemon POST"
 
 import os
 import errno

daemon_post_question.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core POST"
+__module_group__ = "Daemon POST"
 
 import os
 import errno

daemon_post_receive.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core POST"
+__module_group__ = "Daemon POST"
 
 import os
 import time

daemon_post_remove.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core POST"
+__module_group__ = "Daemon POST"
 
 import os
 import errno

daemon_post_search.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core POST"
+__module_group__ = "Daemon POST"
 
 import errno
 import urllib.parse

daemon_post_theme.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core POST"
+__module_group__ = "Daemon POST"
 
 import errno
 import urllib.parse

daemon_utils.py

@@ -5,7 +5,7 @@ __version__ = "1.6.0"
 __maintainer__ = "Bob Mottram"
 __email__ = "bob@libreserver.org"
 __status__ = "Production"
-__module_group__ = "Core"
+__module_group__ = "Daemon"
 
 import os
 import time

tests.py

@@ -6045,30 +6045,34 @@ def _test_functions():
 
     _diagram_groups(['Commandline Interface', 'ActivityPub'], ['utils'],
                     modules, mod_groups, max_module_calls)
-    _diagram_groups(['Commandline Interface', 'Core'], ['utils'],
-                    modules, mod_groups, max_module_calls)
-    _diagram_groups(['Timeline', 'Core'], ['utils'],
-                    modules, mod_groups, max_module_calls)
-    _diagram_groups(['Web Interface', 'Core'], ['utils'],
-                    modules, mod_groups, max_module_calls)
-    _diagram_groups(['Web Interface Columns', 'Core'], ['utils'],
-                    modules, mod_groups, max_module_calls)
-    _diagram_groups(['Core'], [],
-                    modules, mod_groups, max_module_calls)
-    _diagram_groups(['ActivityPub'], [],
-                    modules, mod_groups, max_module_calls)
-    _diagram_groups(['ActivityPub', 'Core'], ['utils'],
-                    modules, mod_groups, max_module_calls)
-    _diagram_groups(['ActivityPub', 'Security'], ['utils'],
-                    modules, mod_groups, max_module_calls)
-    _diagram_groups(['Core', 'Security'], ['utils'],
+    _diagram_groups(['Commandline Interface', 'Daemon'], ['utils'],
                     modules, mod_groups, max_module_calls)
     _diagram_groups(['Timeline', 'Security'], ['utils'],
                     modules, mod_groups, max_module_calls)
     _diagram_groups(['Web Interface', 'Accessibility'],
                     ['utils', 'webapp_utils'],
                     modules, mod_groups, max_module_calls)
-    _diagram_groups(['Core', 'Accessibility'], ['utils'],
+    _diagram_groups(['Daemon', 'Accessibility'], ['utils'],
+                    modules, mod_groups, max_module_calls)
+    _diagram_groups(['Timeline', 'Daemon Timeline', 'Daemon'],
+                    ['utils'], modules, mod_groups, max_module_calls)
+    _diagram_groups(['Web Interface', 'Core', 'Daemon'], ['utils'],
+                    modules, mod_groups, max_module_calls)
+    _diagram_groups(['Web Interface Columns', 'Core', 'Daemon'], ['utils'],
+                    modules, mod_groups, max_module_calls)
+    _diagram_groups(['Daemon'], ['utils'],
+                    modules, mod_groups, max_module_calls)
+    _diagram_groups(['Core', 'Daemon', 'Daemon Login', 'Security'],
+                    ['daemon_utils', 'utils', 'daemon_post',
+                     'daemon_head', 'flags'],
+                    modules, mod_groups, max_module_calls)
+    _diagram_groups(['ActivityPub'], ['utils'],
+                    modules, mod_groups, max_module_calls)
+    _diagram_groups(['ActivityPub', 'Daemon'], ['utils'],
+                    modules, mod_groups, max_module_calls)
+    _diagram_groups(['ActivityPub', 'Security'], ['utils'],
+                    modules, mod_groups, max_module_calls)
+    _diagram_groups(['Core', 'Security'], ['utils'],
                     modules, mod_groups, max_module_calls)