Merge branch 'main' of gitlab.com:bashrc2/epicyon

README_architecture.md

@@ -16,6 +16,10 @@ Although it can be single user, this is not strictly a single user system.
 
 The design of this system is opinionated, and to a large extent informed by years of past experience in the fediverse. There is no claim to neutrality of any sort. Automatic removal of hellthreads and other common griefing tactics is an example of this.
 
+### Privacy Sensitive Defaults
+
+Follow approval should be required by default. This gives the user a chance to see who wants to follow them and make a decision. Also by default direct messages should not be permitted except with accounts that you are following. This helps to reduce spam and harrassment from random accounts in the wider fediverse. The aim is for the user to have a good experience by default, even if they have not yet built up any sort of block list.
+
 ### Resisting Centralization
 
 Centralization is characterized by the typical fixation upon "scale" within the software industry. Systems which scale, in the way which is commonly understood, mean that a few individuals can control the social lives of many, and extract value from them in often cynical and manipulative ways.
@@ -24,7 +28,7 @@ In general, methods have been preferred which do not vertically scale. This incl
 
 Being hostile towards the common notion of scaling means that this system will be of no interest to "big tech" and can't easily be used within extractive economic models without needing a substantial rewrite. This avoids the typical cooption strategies in which large companies eventually take over what was originally software developed by grassroots activists to address real community needs.
 
-This system should however be able to scale rhizomatically with the deployment of many small instances federated together. Instead of scaling up, scale out. In a network of many small instances nobody has overall control and corporate capture is much more unlikely. Small instances also minimize the bureaucratic requirements for governance processes, which at medium to large scale eventually becomes tyrannical.
+This system should however be able to scale rhizomatically with the deployment of many small instances federated together. Instead of scaling up, scale out. In a network of many small instances nobody has overall control and corporate capture is far less feasible. Small instances also minimize the bureaucratic requirements for governance processes, which at medium to large scale eventually becomes tyrannical.
 
 ### Roles
 
@@ -32,11 +36,11 @@ The roles within an instance are comparable to the crew roles onboard a ship, wi
 
 ### No Javascript
 
-This is so that the system can be accessed and used normally with javascript in the web browser turned off. If you want to have good security then this is useful, since lack of javascript greatly reduces the attack surface and constrains adversaries to a limited number of vectors.
+This is so that the system can be accessed and used normally with javascript in the web browser turned off. If you want to have good security then this is useful, since lack of javascript greatly reduces the attack surface and constrains adversaries to a limited number of vectors. Not using javascript also makes this system usable in shell based browsers such as Lynx, or other less common browsers, which helps to avoid being locked in to a browser duopoly.
 
 ### Block Crawlers
 
-Ordinarily web crawlers would not be a problem, but in the context of a social network even having crawlers index public posts can create ethical dilemmas in some circumstances. News instances may allow crawlers, but other types of instances should block them.
+Ordinarily web crawlers would not be a problem, but in the context of a social network even having crawlers index public posts can create ethical dilemmas in some circumstances. News and blogging instances may allow crawlers, but other types of instances should block them.
 
 ### No Local or Federated Timelines
 
@@ -64,7 +68,6 @@ Where Json linked data signatures are supported there should not be arbitrary sc
 
 In general avoid using web frameworks and instead use local modules which are prefixed with *webapp_*. Web frameworks are built for conventional software engineering by large companies who are designing for scale. They typically have database dependencies and contain a lot of hardcoded Google stuff or other things which will leak metadata or be incompatible with onion routing. Keeping up with web frameworks is a constant firefight. They also create a massive attack surface requiring constant vigilance.
 
-
 ## High Level Architecture
 
 The main modules are *epicyon.py* and *daemon.py*. *epicyon.py* is the commandline interface and *daemon.py* is the http server.

README_customizations.md

@@ -26,6 +26,20 @@ When a moderator report is created the message at the top of the screen can be c
 
 Extra emoji can be added to the *emoji* directory and you should then update the **emoji/emoji.json** file, which maps the name to the filename (without the .png extension).
 
+Another way to import emoji is to create a text file where each line is the url of the emoji png file and the emoji name, separated by a comma.
+
+```bash
+https://somesite/emoji1.png, :emojiname1:
+https://somesite/emoji2.png, :emojiname2:
+https://somesite/emoji3.png, :emojiname3:
+```
+
+Then this can be imported with:
+
+```bash
+python3 epicyon.py --import-emoji [textfile]
+```
+
 ## Themes
 
-If you want to create a new theme then the functions for that are within *theme.py*. These functions take the CSS templates and modify them. You will need to edit *themesDropdown* within *webinterface.py* and add the appropriate translations for the theme name. Themes are selectable from the profile screen of the administrator.
+If you want to create a new theme then copy the *default* directory within the *theme* directory, rename it to your new theme name, then you can edit the colors and fonts within *theme.json*, and change the icons and banners. Themes are selectable from the graphic design section of the profile screen of the administrator, or of any accounts having the *artist* role.

blocking.py

@@ -1016,19 +1016,29 @@ def load_cw_lists(base_dir: str, verbose: bool) -> {}:
 
 
 def add_cw_from_lists(post_json_object: {}, cw_lists: {}, translate: {},
-                      lists_enabled: str) -> None:
+                      lists_enabled: str, system_language: str) -> None:
     """Adds content warnings by matching the post content
     against domains or keywords
     """
     if not lists_enabled:
         return
     if not post_json_object['object'].get('content'):
-        return
+        if not post_json_object['object'].get('contentMap'):
+            return
     cw_text = ''
     if post_json_object['object'].get('summary'):
         cw_text = post_json_object['object']['summary']
 
-    content = post_json_object['object']['content']
+    content = None
+    if post_json_object['object'].get('contentMap'):
+        if post_json_object['object']['contentMap'].get(system_language):
+            content = \
+                post_json_object['object']['contentMap'][system_language]
+    if not content:
+        if post_json_object['object'].get('content'):
+            content = post_json_object['object']['content']
+    if not content:
+        return
     for name, item in cw_lists.items():
         if name not in lists_enabled:
             continue

content.py

@@ -1463,7 +1463,8 @@ def content_diff(content: str, prev_content: str) -> str:
 
 
 def create_edits_html(edits_json: {}, post_json_object: {},
-                      translate: {}, timezone: str) -> str:
+                      translate: {}, timezone: str,
+                      system_language: str) -> str:
     """ Creates html showing historical edits made to a post
     """
     if not edits_json:
@@ -1471,20 +1472,42 @@ def create_edits_html(edits_json: {}, post_json_object: {},
     if not has_object_dict(post_json_object):
         return ''
     if not post_json_object['object'].get('content'):
-        return ''
+        if not post_json_object['object'].get('contentMap'):
+            return ''
     edit_dates_list = []
     for modified, item in edits_json.items():
         edit_dates_list.append(modified)
     edit_dates_list.sort(reverse=True)
     edits_str = ''
-    content = remove_html(post_json_object['object']['content'])
+    content = None
+    if post_json_object['object'].get('contentMap'):
+        if post_json_object['object']['contentMap'].get(system_language):
+            content = \
+                post_json_object['object']['contentMap'][system_language]
+    if not content:
+        if post_json_object['object'].get('content'):
+            content = post_json_object['object']['content']
+    if not content:
+        return ''
+    content = remove_html(content)
     for modified in edit_dates_list:
         prev_json = edits_json[modified]
         if not has_object_dict(prev_json):
             continue
+        prev_content = None
         if not prev_json['object'].get('content'):
+            if not prev_json['object'].get('contentMap'):
+                continue
+        if prev_json['object'].get('contentMap'):
+            if prev_json['object']['contentMap'].get(system_language):
+                prev_content = \
+                    prev_json['object']['contentMap'][system_language]
+        if not prev_content:
+            if prev_json['object'].get('content'):
+                prev_content = prev_json['object']['content']
+        if not prev_content:
             continue
-        prev_content = remove_html(prev_json['object']['content'])
+        prev_content = remove_html(prev_content)
         if content == prev_content:
             continue
         diff = content_diff(content, prev_content)

daemon.py

@@ -16911,7 +16911,7 @@ class PubServer(BaseHTTPRequestHandler):
                                     self.server.domain_full,
                                     self.server.text_mode_banner,
                                     access_keys,
-                                    False)
+                                    False, self.server.system_language)
                 if msg:
                     msg = msg.encode('utf-8')
                     msglen = len(msg)
@@ -16952,12 +16952,17 @@ class PubServer(BaseHTTPRequestHandler):
                                     self.server.domain_full,
                                     self.server.text_mode_banner,
                                     access_keys,
-                                    True).encode('utf-8')
-                msglen = len(msg)
-                self._set_headers('text/calendar',
-                                  msglen, cookie, calling_domain,
-                                  False)
-                self._write(msg)
+                                    True,
+                                    self.server.system_language)
+                if msg:
+                    msg = msg.encode('utf-8')
+                    msglen = len(msg)
+                    self._set_headers('text/calendar',
+                                      msglen, cookie, calling_domain,
+                                      False)
+                    self._write(msg)
+                else:
+                    self._404()
                 fitness_performance(getreq_start_time, self.server.fitness,
                                     '_GET', 'icalendar shown',
                                     self.server.debug)
@@ -18362,7 +18367,8 @@ class PubServer(BaseHTTPRequestHandler):
                                     self.server.http_prefix,
                                     curr_etag,
                                     self.server.recent_dav_etags,
-                                    self.server.domain_full)
+                                    self.server.domain_full,
+                                    self.server.system_language)
         elif endpoint_type == 'delete':
             response_str = \
                 dav_delete_response(self.server.base_dir,

happening.py

@@ -228,7 +228,7 @@ def _event_text_match(content: str, text_match: str) -> bool:
 def get_todays_events(base_dir: str, nickname: str, domain: str,
                       curr_year: int, curr_month_number: int,
                       curr_day_of_month: int,
-                      text_match: str) -> {}:
+                      text_match: str, system_language: str) -> {}:
     """Retrieves calendar events for today
     Returns a dictionary of lists containing Event and Place activities
     """
@@ -268,8 +268,16 @@ def get_todays_events(base_dir: str, nickname: str, domain: str,
                 continue
 
             if post_json_object.get('object'):
-                if post_json_object['object'].get('content'):
-                    content = post_json_object['object']['content']
+                content = None
+                if post_json_object['object'].get('contentMap'):
+                    sys_lang = system_language
+                    if post_json_object['object']['contentMap'].get(sys_lang):
+                        content = \
+                            post_json_object['object']['contentMap'][sys_lang]
+                if not content:
+                    if post_json_object['object'].get('content'):
+                        content = post_json_object['object']['content']
+                if content:
                     if not _event_text_match(content, text_match):
                         continue
 
@@ -465,14 +473,14 @@ def get_todays_events_icalendar(base_dir: str, nickname: str, domain: str,
                                 year: int, month_number: int,
                                 day_number: int, person_cache: {},
                                 http_prefix: str,
-                                text_match: str) -> str:
+                                text_match: str, system_language: str) -> str:
     """Returns today's events in icalendar format
     """
     day_events = None
     events = \
         get_todays_events(base_dir, nickname, domain,
                           year, month_number, day_number,
-                          text_match)
+                          text_match, system_language)
     if events:
         if events.get(str(day_number)):
             day_events = events[str(day_number)]
@@ -1010,7 +1018,7 @@ def dav_report_response(base_dir: str, nickname: str, domain: str,
                         depth: int, xml_str: str,
                         person_cache: {}, http_prefix: str,
                         curr_etag: str, recent_dav_etags: {},
-                        domain_full: str) -> str:
+                        domain_full: str, system_language: str) -> str:
     """Returns the response to caldav REPORT
     """
     if '<c:calendar-query' not in xml_str or \
@@ -1083,7 +1091,8 @@ def dav_report_response(base_dir: str, nickname: str, domain: str,
                                                 search_date.year,
                                                 search_date.month,
                                                 search_date.day, person_cache,
-                                                http_prefix, text_match)
+                                                http_prefix, text_match,
+                                                system_language)
                 events_href = \
                     http_prefix + '://' + domain_full + '/users/' + \
                     nickname + '/calendar?year=' + \
@@ -1201,7 +1210,8 @@ def dav_report_response(base_dir: str, nickname: str, domain: str,
             get_todays_events_icalendar(base_dir, nickname, domain,
                                         search_date.year, search_date.month,
                                         search_date.day, person_cache,
-                                        http_prefix, text_match)
+                                        http_prefix, text_match,
+                                        system_language)
         events_href = \
             http_prefix + '://' + domain_full + '/users/' + \
             nickname + '/calendar?year=' + \

inbox.py

@@ -3937,7 +3937,8 @@ def _inbox_after_initial(server,
             # NOTE: this must be done before update_conversation is called
             edited_filename, edited_json = \
                 edited_post_filename(base_dir, handle_name, domain,
-                                     post_json_object, debug, 300)
+                                     post_json_object, debug, 300,
+                                     system_language)
 
             # If this was an edit then update the edits json file and
             # delete the previous version of the post

posts.py

@@ -4944,6 +4944,11 @@ def download_announce(session, base_dir: str, http_prefix: str,
             return None
         # Check the content of the announce
         content_str = announced_json['content']
+        using_content_map = False
+        if announced_json.get('contentMap'):
+            if announced_json['contentMap'].get(system_language):
+                content_str = announced_json['contentMap'][system_language]
+                using_content_map = True
         if dangerous_markup(content_str, allow_local_network_access):
             print('WARN: announced post contains dangerous markup ' +
                   str(announced_json))
@@ -4980,6 +4985,8 @@ def download_announce(session, base_dir: str, http_prefix: str,
         content_str = remove_text_formatting(content_str, bold_reading)
 
         # set the content after santitization
+        if using_content_map:
+            announced_json['contentMap'][system_language] = content_str
         announced_json['content'] = content_str
 
         # wrap in create to be consistent with other posts
@@ -5476,7 +5483,8 @@ def seconds_between_published(published1: str, published2: str) -> int:
 
 def edited_post_filename(base_dir: str, nickname: str, domain: str,
                          post_json_object: {}, debug: bool,
-                         max_time_diff_seconds: int) -> (str, {}):
+                         max_time_diff_seconds: int,
+                         system_language: str) -> (str, {}):
     """Returns the filename of the edited post
     """
     if not has_object_dict(post_json_object):
@@ -5545,8 +5553,17 @@ def edited_post_filename(base_dir: str, nickname: str, domain: str,
         return '', None
     if debug:
         print(post_id + ' might be an edit of ' + lastpost_id)
-    if words_similarity(lastpost_json['object']['content'],
-                        post_json_object['object']['content'], 10) < 70:
+    lastpost_content = lastpost_json['object']['content']
+    if lastpost_json['object'].get('contentMap'):
+        if lastpost_json['object']['contentMap'].get(system_language):
+            lastpost_content = \
+                lastpost_json['object']['contentMap'][system_language]
+    content = post_json_object['object']['content']
+    if post_json_object['object'].get('contentMap'):
+        if post_json_object['object']['contentMap'].get(system_language):
+            content = \
+                post_json_object['object']['contentMap'][system_language]
+    if words_similarity(lastpost_content, content, 10) < 70:
         return '', None
     print(post_id + ' is an edit of ' + lastpost_id)
     return lastpost_filename, lastpost_json

tests.py

@@ -6504,6 +6504,7 @@ def _test_word_similarity() -> None:
 def _test_add_cw_lists(base_dir: str) -> None:
     print('test_add_CW_from_lists')
     translate = {}
+    system_language = "en"
     cw_lists = load_cw_lists(base_dir, True)
     assert cw_lists
 
@@ -6514,7 +6515,8 @@ def _test_add_cw_lists(base_dir: str) -> None:
             "content": ""
         }
     }
-    add_cw_from_lists(post_json_object, cw_lists, translate, 'Murdoch press')
+    add_cw_from_lists(post_json_object, cw_lists, translate, 'Murdoch press',
+                      system_language)
     assert post_json_object['object']['sensitive'] is False
     assert post_json_object['object']['summary'] is None
 
@@ -6522,10 +6524,13 @@ def _test_add_cw_lists(base_dir: str) -> None:
         "object": {
             "sensitive": False,
             "summary": None,
-            "content": "Blah blah news.co.uk blah blah"
+            "contentMap": {
+                "en": "Blah blah news.co.uk blah blah"
+            }
         }
     }
-    add_cw_from_lists(post_json_object, cw_lists, translate, 'Murdoch press')
+    add_cw_from_lists(post_json_object, cw_lists, translate, 'Murdoch press',
+                      system_language)
     assert post_json_object['object']['sensitive'] is True
     assert post_json_object['object']['summary'] == "Murdoch Press"
 
@@ -6536,7 +6541,8 @@ def _test_add_cw_lists(base_dir: str) -> None:
             "content": "Blah blah news.co.uk blah blah"
         }
     }
-    add_cw_from_lists(post_json_object, cw_lists, translate, 'Murdoch press')
+    add_cw_from_lists(post_json_object, cw_lists, translate, 'Murdoch press',
+                      system_language)
     assert post_json_object['object']['sensitive'] is True
     assert post_json_object['object']['summary'] == \
         "Murdoch Press / Existing CW"
@@ -6945,6 +6951,7 @@ def _test_diff_content() -> None:
         '<label class="diff_remove">- This is another line</label></p>'
     assert result == expected
 
+    system_language = "en"
     translate = {
         "SHOW EDITS": "SHOW EDITS"
     }
@@ -6973,13 +6980,16 @@ def _test_diff_content() -> None:
         },
         "2020-12-14T00:07:34Z": {
             "object": {
-                "content": content2,
+                "contentMap": {
+                    "en": content2
+                },
                 "published": "2020-12-14T00:07:34Z"
             }
         }
     }
     html_str = \
-        create_edits_html(edits_json, post_json_object, translate, timezone)
+        create_edits_html(edits_json, post_json_object, translate,
+                          timezone, system_language)
     assert html_str
     expected = \
         '<details><summary class="cw">SHOW EDITS</summary>' + \

video.py

@@ -87,7 +87,11 @@ def convert_video_to_note(base_dir: str, nickname: str, domain: str,
                                post_json_object['license']['name']):
                     return None
                 content += '<p>' + post_json_object['license']['name'] + '</p>'
-    content += post_json_object['content']
+    post_content = post_json_object['content']
+    if post_json_object.get('contentMap'):
+        if post_json_object['contentMap'].get(system_language):
+            post_content = post_json_object['contentMap'][system_language]
+    content += post_content
 
     conversation_id = remove_id_ending(post_json_object['id'])
 

webapp_calendar.py

@@ -265,7 +265,7 @@ def html_calendar(person_cache: {}, css_cache: {}, translate: {},
                   base_dir: str, path: str,
                   http_prefix: str, domain_full: str,
                   text_mode_banner: str, access_keys: {},
-                  icalendar: bool) -> str:
+                  icalendar: bool, system_language: str) -> str:
     """Show the calendar for a person
     """
     domain = remove_domain_port(domain_full)
@@ -327,12 +327,13 @@ def html_calendar(person_cache: {}, css_cache: {}, translate: {},
                                                day_number,
                                                person_cache,
                                                http_prefix,
-                                               text_match)
+                                               text_match,
+                                               system_language)
         day_events = None
         events = \
             get_todays_events(base_dir, nickname, domain,
                               year, month_number, day_number,
-                              text_match)
+                              text_match, system_language)
         if events:
             if events.get(str(day_number)):
                 day_events = events[str(day_number)]

webapp_post.py

@@ -92,7 +92,8 @@ from blocking import add_cw_from_lists
 from reaction import html_emoji_reactions
 
 
-def _html_post_metadata_open_graph(domain: str, post_json_object: {}) -> str:
+def _html_post_metadata_open_graph(domain: str, post_json_object: {},
+                                   system_language: str) -> str:
     """Returns html OpenGraph metadata for a post
     """
     metadata = \
@@ -122,7 +123,11 @@ def _html_post_metadata_open_graph(domain: str, post_json_object: {}) -> str:
             "\" property=\"og:published_time\" />\n"
     if not obj_json.get('attachment') or obj_json.get('sensitive'):
         if obj_json.get('content') and not obj_json.get('sensitive'):
-            description = remove_html(obj_json['content'])
+            obj_content = obj_json['content']
+            if obj_json.get('contentMap'):
+                if obj_json['contentMap'].get(system_language):
+                    obj_content = obj_json['contentMap'][system_language]
+            description = remove_html(obj_content)
             metadata += \
                 "    <meta content=\"" + description + \
                 "\" name=\"description\">\n"
@@ -150,7 +155,11 @@ def _html_post_metadata_open_graph(domain: str, post_json_object: {}) -> str:
             description = 'Attached: 1 audio'
         if description:
             if obj_json.get('content') and not obj_json.get('sensitive'):
-                description += '\n\n' + remove_html(obj_json['content'])
+                obj_content = obj_json['content']
+                if obj_json.get('contentMap'):
+                    if obj_json['contentMap'].get(system_language):
+                        obj_content = obj_json['contentMap'][system_language]
+                description += '\n\n' + remove_html(obj_content)
             metadata += \
                 "    <meta content=\"" + description + \
                 "\" name=\"description\">\n"
@@ -1490,7 +1499,7 @@ def individual_post_as_html(signing_priv_key_pem: str,
         edits_json = load_json(edits_filename, 0, 1)
         if edits_json:
             edits_str = create_edits_html(edits_json, post_json_object,
-                                          translate, timezone)
+                                          translate, timezone, system_language)
 
     message_id_str = ''
     if message_id:
@@ -1936,7 +1945,8 @@ def individual_post_as_html(signing_priv_key_pem: str,
         container_class = 'container dm'
 
     # add any content warning from the cwlists directory
-    add_cw_from_lists(post_json_object, cw_lists, translate, lists_enabled)
+    add_cw_from_lists(post_json_object, cw_lists, translate, lists_enabled,
+                      system_language)
 
     post_is_sensitive = False
     if post_json_object['object'].get('sensitive'):
@@ -2029,8 +2039,6 @@ def individual_post_as_html(signing_priv_key_pem: str,
 
     if not is_pgp_encrypted(content_str):
         if not is_patch:
-            # append any edits
-            content_str += edits_str
             # Add bold text
             if bold_reading and \
                not displaying_ciphertext and \
@@ -2046,6 +2054,8 @@ def individual_post_as_html(signing_priv_key_pem: str,
                 switch_words(base_dir, nickname, domain, object_content)
             object_content = html_replace_email_quote(object_content)
             object_content = html_replace_quote_marks(object_content)
+            # append any edits
+            object_content += edits_str
         else:
             object_content = content_str
     else:
@@ -2330,7 +2340,8 @@ def html_individual_post(css_cache: {},
 
     instance_title = \
         get_config_param(base_dir, 'instanceTitle')
-    metadata_str = _html_post_metadata_open_graph(domain, original_post_json)
+    metadata_str = _html_post_metadata_open_graph(domain, original_post_json,
+                                                  system_language)
     header_str = html_header_with_external_style(css_filename,
                                                  instance_title, metadata_str)
     return header_str + post_str + html_footer()

website/EN/index.html

@@ -1194,7 +1194,7 @@
       <p>
       <p class="siteheader">An Internet of People, Not Corporate Agendas</p>
       <p class="intro">
-        Epicyon is written in Python with a HTML+CSS web interface and uses <i>no javascript</i> which makes display in a web browser very lightweight. It can run as a <a href="https://en.wikipedia.org/wiki/Progressive_web_application">Progressive Web App</a> on mobile. <i>Just say "no"</i> to boring social media sites packed with generic adverts and zombified corporate influencers.
+        Epicyon is written in Python with a HTML+CSS web interface and uses <a href="https://www.youtube.com/watch?v=Uo3cL4nrGOk">no javascript</a> which makes display in a web browser very lightweight. It can run as a <a href="https://en.wikipedia.org/wiki/Progressive_web_application">Progressive Web App</a> on mobile. <i>Just say "no"</i> to boring social media sites packed with generic adverts and zombified corporate influencers.
       </p>
 
       <table style="margin:0% 0%;width:100%" border="0">