From 44307b2f9f8d85cdaf4414cc1b67b90abb0aac46 Mon Sep 17 00:00:00 2001 From: Bob Mottram Date: Tue, 12 Apr 2022 10:50:55 +0100 Subject: [PATCH 01/10] Document importing emoji --- README_customizations.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/README_customizations.md b/README_customizations.md index a6f86ad7d..f18e94235 100644 --- a/README_customizations.md +++ b/README_customizations.md @@ -26,6 +26,20 @@ When a moderator report is created the message at the top of the screen can be c Extra emoji can be added to the *emoji* directory and you should then update the **emoji/emoji.json** file, which maps the name to the filename (without the .png extension). +Another way to import emoji is to create a text file where each line is the url of the emoji png file and the emoji name, separated by a comma. + +```bash +https://somesite/emoji1.png, :emojiname1: +https://somesite/emoji2.png, :emojiname2: +https://somesite/emoji3.png, :emojiname3: +``` + +Then this can be imported with: + +```bash +python3 epicyon.py --import-emoji [textfile] +``` + ## Themes If you want to create a new theme then the functions for that are within *theme.py*. These functions take the CSS templates and modify them. You will need to edit *themesDropdown* within *webinterface.py* and add the appropriate translations for the theme name. Themes are selectable from the profile screen of the administrator. From f627cc140160da775f7d62c0d95176089ad448c8 Mon Sep 17 00:00:00 2001 From: Bob Mottram Date: Tue, 12 Apr 2022 10:56:06 +0100 Subject: [PATCH 02/10] Changes for theme creation --- README_customizations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README_customizations.md b/README_customizations.md index f18e94235..462ee1cdd 100644 --- a/README_customizations.md +++ b/README_customizations.md @@ -42,4 +42,4 @@ python3 epicyon.py --import-emoji [textfile] ## Themes -If you want to create a new theme then the functions for that are within *theme.py*. These functions take the CSS templates and modify them. You will need to edit *themesDropdown* within *webinterface.py* and add the appropriate translations for the theme name. Themes are selectable from the profile screen of the administrator. +If you want to create a new theme then copy the *default* directory within the *theme* directory, rename it to your new theme name, then you can edit the colors and fonts within *theme.json*, and change the icons and banners. Themes are selectable from the graphic design section of the profile screen of the administrator, or of any accounts having the *artist* role. From 320a02d95fd01705f7f98719e04e4f8fedb40906 Mon Sep 17 00:00:00 2001 From: Bob Mottram Date: Tue, 12 Apr 2022 11:17:23 +0100 Subject: [PATCH 03/10] Architecture notes --- README_architecture.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/README_architecture.md b/README_architecture.md index 74c996d94..4dcb87bec 100644 --- a/README_architecture.md +++ b/README_architecture.md @@ -16,6 +16,10 @@ Although it can be single user, this is not strictly a single user system. The design of this system is opinionated, and to a large extent informed by years of past experience in the fediverse. There is no claim to neutrality of any sort. Automatic removal of hellthreads and other common griefing tactics is an example of this. +### Privacy Sensitive Defaults + +Follow approval should be required by default. This gives the user a chance to see who wants to follow them and make a decision. Also by default direct messages should not be permitted except with accounts that you are following. This helps to reduce spam and harrassment from random accounts in the wider fediverse. The aim is for the user to have a good experience by default, even if they have not yet built up any sort of block list. + ### Resisting Centralization Centralization is characterized by the typical fixation upon "scale" within the software industry. Systems which scale, in the way which is commonly understood, mean that a few individuals can control the social lives of many, and extract value from them in often cynical and manipulative ways. @@ -24,7 +28,7 @@ In general, methods have been preferred which do not vertically scale. This incl Being hostile towards the common notion of scaling means that this system will be of no interest to "big tech" and can't easily be used within extractive economic models without needing a substantial rewrite. This avoids the typical cooption strategies in which large companies eventually take over what was originally software developed by grassroots activists to address real community needs. -This system should however be able to scale rhizomatically with the deployment of many small instances federated together. Instead of scaling up, scale out. In a network of many small instances nobody has overall control and corporate capture is much more unlikely. Small instances also minimize the bureaucratic requirements for governance processes, which at medium to large scale eventually becomes tyrannical. +This system should however be able to scale rhizomatically with the deployment of many small instances federated together. Instead of scaling up, scale out. In a network of many small instances nobody has overall control and corporate capture is far less feasible. Small instances also minimize the bureaucratic requirements for governance processes, which at medium to large scale eventually becomes tyrannical. ### Roles @@ -32,11 +36,11 @@ The roles within an instance are comparable to the crew roles onboard a ship, wi ### No Javascript -This is so that the system can be accessed and used normally with javascript in the web browser turned off. If you want to have good security then this is useful, since lack of javascript greatly reduces the attack surface and constrains adversaries to a limited number of vectors. +This is so that the system can be accessed and used normally with javascript in the web browser turned off. If you want to have good security then this is useful, since lack of javascript greatly reduces the attack surface and constrains adversaries to a limited number of vectors. Not using javascript also makes this system usable in shell based browsers such as Lynx, or other less common browsers, which helps to avoid being locked in to a browser duopoly. ### Block Crawlers -Ordinarily web crawlers would not be a problem, but in the context of a social network even having crawlers index public posts can create ethical dilemmas in some circumstances. News instances may allow crawlers, but other types of instances should block them. +Ordinarily web crawlers would not be a problem, but in the context of a social network even having crawlers index public posts can create ethical dilemmas in some circumstances. News and blogging instances may allow crawlers, but other types of instances should block them. ### No Local or Federated Timelines @@ -64,7 +68,6 @@ Where Json linked data signatures are supported there should not be arbitrary sc In general avoid using web frameworks and instead use local modules which are prefixed with *webapp_*. Web frameworks are built for conventional software engineering by large companies who are designing for scale. They typically have database dependencies and contain a lot of hardcoded Google stuff or other things which will leak metadata or be incompatible with onion routing. Keeping up with web frameworks is a constant firefight. They also create a massive attack surface requiring constant vigilance. - ## High Level Architecture The main modules are *epicyon.py* and *daemon.py*. *epicyon.py* is the commandline interface and *daemon.py* is the http server. From eb5f561294eef9231f635e8b85ef0436fc896f54 Mon Sep 17 00:00:00 2001 From: Bob Mottram Date: Wed, 13 Apr 2022 15:20:38 +0100 Subject: [PATCH 04/10] Use contentMap, if available, when showing post diff --- content.py | 31 +++++++++++++++++++++++++++---- tests.py | 8 ++++++-- webapp_post.py | 2 +- 3 files changed, 34 insertions(+), 7 deletions(-) diff --git a/content.py b/content.py index f59112776..2adf09956 100644 --- a/content.py +++ b/content.py @@ -1463,7 +1463,8 @@ def content_diff(content: str, prev_content: str) -> str: def create_edits_html(edits_json: {}, post_json_object: {}, - translate: {}, timezone: str) -> str: + translate: {}, timezone: str, + system_language: str) -> str: """ Creates html showing historical edits made to a post """ if not edits_json: @@ -1471,20 +1472,42 @@ def create_edits_html(edits_json: {}, post_json_object: {}, if not has_object_dict(post_json_object): return '' if not post_json_object['object'].get('content'): - return '' + if not post_json_object['object'].get('contentMap'): + return '' edit_dates_list = [] for modified, item in edits_json.items(): edit_dates_list.append(modified) edit_dates_list.sort(reverse=True) edits_str = '' - content = remove_html(post_json_object['object']['content']) + content = None + if post_json_object['object'].get('contentMap'): + if post_json_object['object']['contentMap'].get(system_language): + content = \ + post_json_object['object']['contentMap'][system_language] + if not content: + if post_json_object['object'].get('content'): + content = post_json_object['object']['content'] + if not content: + return '' + content = remove_html(content) for modified in edit_dates_list: prev_json = edits_json[modified] if not has_object_dict(prev_json): continue + prev_content = None if not prev_json['object'].get('content'): + if not prev_json['object'].get('contentMap'): + continue + if prev_json['object'].get('contentMap'): + if prev_json['object']['contentMap'].get(system_language): + prev_content = \ + prev_json['object']['contentMap'][system_language] + if not prev_content: + if prev_json['object'].get('content'): + prev_content = prev_json['object']['content'] + if not prev_content: continue - prev_content = remove_html(prev_json['object']['content']) + prev_content = remove_html(prev_content) if content == prev_content: continue diff = content_diff(content, prev_content) diff --git a/tests.py b/tests.py index 9e8f9fdec..f8b3c768b 100644 --- a/tests.py +++ b/tests.py @@ -6945,6 +6945,7 @@ def _test_diff_content() -> None: '

' assert result == expected + system_language = "en" translate = { "SHOW EDITS": "SHOW EDITS" } @@ -6973,13 +6974,16 @@ def _test_diff_content() -> None: }, "2020-12-14T00:07:34Z": { "object": { - "content": content2, + "contentMap": { + "en": content2 + }, "published": "2020-12-14T00:07:34Z" } } } html_str = \ - create_edits_html(edits_json, post_json_object, translate, timezone) + create_edits_html(edits_json, post_json_object, translate, + timezone, system_language) assert html_str expected = \ '
SHOW EDITS' + \ diff --git a/webapp_post.py b/webapp_post.py index 448a73f85..e4e5b1c1b 100644 --- a/webapp_post.py +++ b/webapp_post.py @@ -1490,7 +1490,7 @@ def individual_post_as_html(signing_priv_key_pem: str, edits_json = load_json(edits_filename, 0, 1) if edits_json: edits_str = create_edits_html(edits_json, post_json_object, - translate, timezone) + translate, timezone, system_language) message_id_str = '' if message_id: From cd020565884d30f3897b00170993de758acde5e1 Mon Sep 17 00:00:00 2001 From: Bob Mottram Date: Wed, 13 Apr 2022 15:35:11 +0100 Subject: [PATCH 05/10] Use contentMap within cw list processing --- blocking.py | 16 +++++++++++++--- tests.py | 14 ++++++++++---- webapp_post.py | 3 ++- 3 files changed, 25 insertions(+), 8 deletions(-) diff --git a/blocking.py b/blocking.py index d3eefc4e4..4bce78539 100644 --- a/blocking.py +++ b/blocking.py @@ -1016,19 +1016,29 @@ def load_cw_lists(base_dir: str, verbose: bool) -> {}: def add_cw_from_lists(post_json_object: {}, cw_lists: {}, translate: {}, - lists_enabled: str) -> None: + lists_enabled: str, system_language: str) -> None: """Adds content warnings by matching the post content against domains or keywords """ if not lists_enabled: return if not post_json_object['object'].get('content'): - return + if not post_json_object['object'].get('contentMap'): + return cw_text = '' if post_json_object['object'].get('summary'): cw_text = post_json_object['object']['summary'] - content = post_json_object['object']['content'] + content = None + if post_json_object['object'].get('contentMap'): + if post_json_object['object']['contentMap'].get(system_language): + content = \ + post_json_object['object']['contentMap'][system_language] + if not content: + if post_json_object['object'].get('content'): + content = post_json_object['object']['content'] + if not content: + return for name, item in cw_lists.items(): if name not in lists_enabled: continue diff --git a/tests.py b/tests.py index f8b3c768b..35d2cfaa6 100644 --- a/tests.py +++ b/tests.py @@ -6504,6 +6504,7 @@ def _test_word_similarity() -> None: def _test_add_cw_lists(base_dir: str) -> None: print('test_add_CW_from_lists') translate = {} + system_language = "en" cw_lists = load_cw_lists(base_dir, True) assert cw_lists @@ -6514,7 +6515,8 @@ def _test_add_cw_lists(base_dir: str) -> None: "content": "" } } - add_cw_from_lists(post_json_object, cw_lists, translate, 'Murdoch press') + add_cw_from_lists(post_json_object, cw_lists, translate, 'Murdoch press', + system_language) assert post_json_object['object']['sensitive'] is False assert post_json_object['object']['summary'] is None @@ -6522,10 +6524,13 @@ def _test_add_cw_lists(base_dir: str) -> None: "object": { "sensitive": False, "summary": None, - "content": "Blah blah news.co.uk blah blah" + "contentMap": { + "en": "Blah blah news.co.uk blah blah" + } } } - add_cw_from_lists(post_json_object, cw_lists, translate, 'Murdoch press') + add_cw_from_lists(post_json_object, cw_lists, translate, 'Murdoch press', + system_language) assert post_json_object['object']['sensitive'] is True assert post_json_object['object']['summary'] == "Murdoch Press" @@ -6536,7 +6541,8 @@ def _test_add_cw_lists(base_dir: str) -> None: "content": "Blah blah news.co.uk blah blah" } } - add_cw_from_lists(post_json_object, cw_lists, translate, 'Murdoch press') + add_cw_from_lists(post_json_object, cw_lists, translate, 'Murdoch press', + system_language) assert post_json_object['object']['sensitive'] is True assert post_json_object['object']['summary'] == \ "Murdoch Press / Existing CW" diff --git a/webapp_post.py b/webapp_post.py index e4e5b1c1b..4e826686e 100644 --- a/webapp_post.py +++ b/webapp_post.py @@ -1936,7 +1936,8 @@ def individual_post_as_html(signing_priv_key_pem: str, container_class = 'container dm' # add any content warning from the cwlists directory - add_cw_from_lists(post_json_object, cw_lists, translate, lists_enabled) + add_cw_from_lists(post_json_object, cw_lists, translate, lists_enabled, + system_language) post_is_sensitive = False if post_json_object['object'].get('sensitive'): From 4d088eca0faba22cd04ee410434b36bbaa9351fd Mon Sep 17 00:00:00 2001 From: Bob Mottram Date: Wed, 13 Apr 2022 16:06:03 +0100 Subject: [PATCH 06/10] Use contentMap in calendar events --- daemon.py | 22 ++++++++++++++-------- happening.py | 26 ++++++++++++++++++-------- webapp_calendar.py | 7 ++++--- 3 files changed, 36 insertions(+), 19 deletions(-) diff --git a/daemon.py b/daemon.py index c9a4aa6f8..41610d720 100644 --- a/daemon.py +++ b/daemon.py @@ -16911,7 +16911,7 @@ class PubServer(BaseHTTPRequestHandler): self.server.domain_full, self.server.text_mode_banner, access_keys, - False) + False, self.server.system_language) if msg: msg = msg.encode('utf-8') msglen = len(msg) @@ -16952,12 +16952,17 @@ class PubServer(BaseHTTPRequestHandler): self.server.domain_full, self.server.text_mode_banner, access_keys, - True).encode('utf-8') - msglen = len(msg) - self._set_headers('text/calendar', - msglen, cookie, calling_domain, - False) - self._write(msg) + True, + self.server.system_language) + if msg: + msg = msg.encode('utf-8') + msglen = len(msg) + self._set_headers('text/calendar', + msglen, cookie, calling_domain, + False) + self._write(msg) + else: + self._404() fitness_performance(getreq_start_time, self.server.fitness, '_GET', 'icalendar shown', self.server.debug) @@ -18362,7 +18367,8 @@ class PubServer(BaseHTTPRequestHandler): self.server.http_prefix, curr_etag, self.server.recent_dav_etags, - self.server.domain_full) + self.server.domain_full, + self.server.system_language) elif endpoint_type == 'delete': response_str = \ dav_delete_response(self.server.base_dir, diff --git a/happening.py b/happening.py index 947a993f7..39bfd350c 100644 --- a/happening.py +++ b/happening.py @@ -228,7 +228,7 @@ def _event_text_match(content: str, text_match: str) -> bool: def get_todays_events(base_dir: str, nickname: str, domain: str, curr_year: int, curr_month_number: int, curr_day_of_month: int, - text_match: str) -> {}: + text_match: str, system_language: str) -> {}: """Retrieves calendar events for today Returns a dictionary of lists containing Event and Place activities """ @@ -268,8 +268,16 @@ def get_todays_events(base_dir: str, nickname: str, domain: str, continue if post_json_object.get('object'): - if post_json_object['object'].get('content'): - content = post_json_object['object']['content'] + content = None + if post_json_object['object'].get('contentMap'): + sys_lang = system_language + if post_json_object['object']['contentMap'].get(sys_lang): + content = \ + post_json_object['object']['contentMap'][sys_lang] + if not content: + if post_json_object['object'].get('content'): + content = post_json_object['object']['content'] + if content: if not _event_text_match(content, text_match): continue @@ -465,14 +473,14 @@ def get_todays_events_icalendar(base_dir: str, nickname: str, domain: str, year: int, month_number: int, day_number: int, person_cache: {}, http_prefix: str, - text_match: str) -> str: + text_match: str, system_language: str) -> str: """Returns today's events in icalendar format """ day_events = None events = \ get_todays_events(base_dir, nickname, domain, year, month_number, day_number, - text_match) + text_match, system_language) if events: if events.get(str(day_number)): day_events = events[str(day_number)] @@ -1010,7 +1018,7 @@ def dav_report_response(base_dir: str, nickname: str, domain: str, depth: int, xml_str: str, person_cache: {}, http_prefix: str, curr_etag: str, recent_dav_etags: {}, - domain_full: str) -> str: + domain_full: str, system_language: str) -> str: """Returns the response to caldav REPORT """ if ' int: def edited_post_filename(base_dir: str, nickname: str, domain: str, post_json_object: {}, debug: bool, - max_time_diff_seconds: int) -> (str, {}): + max_time_diff_seconds: int, + system_language: str) -> (str, {}): """Returns the filename of the edited post """ if not has_object_dict(post_json_object): @@ -5545,8 +5553,17 @@ def edited_post_filename(base_dir: str, nickname: str, domain: str, return '', None if debug: print(post_id + ' might be an edit of ' + lastpost_id) - if words_similarity(lastpost_json['object']['content'], - post_json_object['object']['content'], 10) < 70: + lastpost_content = lastpost_json['object']['content'] + if lastpost_json['object'].get('contentMap'): + if lastpost_json['object']['contentMap'].get(system_language): + lastpost_content = \ + lastpost_json['object']['contentMap'][system_language] + content = post_json_object['object']['content'] + if post_json_object['object'].get('contentMap'): + if post_json_object['object']['contentMap'].get(system_language): + content = \ + post_json_object['object']['contentMap'][system_language] + if words_similarity(lastpost_content, content, 10) < 70: return '', None print(post_id + ' is an edit of ' + lastpost_id) return lastpost_filename, lastpost_json From 078c7f6320f987bc77ed7768c7d67fc24cf2e230 Mon Sep 17 00:00:00 2001 From: Bob Mottram Date: Wed, 13 Apr 2022 17:32:17 +0100 Subject: [PATCH 08/10] Use contentMap --- video.py | 6 +++++- webapp_post.py | 18 ++++++++++++++---- 2 files changed, 19 insertions(+), 5 deletions(-) diff --git a/video.py b/video.py index 1fa03a72e..ae797c01c 100644 --- a/video.py +++ b/video.py @@ -87,7 +87,11 @@ def convert_video_to_note(base_dir: str, nickname: str, domain: str, post_json_object['license']['name']): return None content += '

' + post_json_object['license']['name'] + '

' - content += post_json_object['content'] + post_content = post_json_object['content'] + if post_json_object.get('contentMap'): + if post_json_object['contentMap'].get(system_language): + post_content = post_json_object['contentMap'][system_language] + content += post_content conversation_id = remove_id_ending(post_json_object['id']) diff --git a/webapp_post.py b/webapp_post.py index 4e826686e..66c169999 100644 --- a/webapp_post.py +++ b/webapp_post.py @@ -92,7 +92,8 @@ from blocking import add_cw_from_lists from reaction import html_emoji_reactions -def _html_post_metadata_open_graph(domain: str, post_json_object: {}) -> str: +def _html_post_metadata_open_graph(domain: str, post_json_object: {}, + system_language: str) -> str: """Returns html OpenGraph metadata for a post """ metadata = \ @@ -122,7 +123,11 @@ def _html_post_metadata_open_graph(domain: str, post_json_object: {}) -> str: "\" property=\"og:published_time\" />\n" if not obj_json.get('attachment') or obj_json.get('sensitive'): if obj_json.get('content') and not obj_json.get('sensitive'): - description = remove_html(obj_json['content']) + obj_content = obj_json['content'] + if obj_json.get('contentMap'): + if obj_json['contentMap'].get(system_language): + obj_content = obj_json['contentMap'][system_language] + description = remove_html(obj_content) metadata += \ " \n" @@ -150,7 +155,11 @@ def _html_post_metadata_open_graph(domain: str, post_json_object: {}) -> str: description = 'Attached: 1 audio' if description: if obj_json.get('content') and not obj_json.get('sensitive'): - description += '\n\n' + remove_html(obj_json['content']) + obj_content = obj_json['content'] + if obj_json.get('contentMap'): + if obj_json['contentMap'].get(system_language): + obj_content = obj_json['contentMap'][system_language] + description += '\n\n' + remove_html(obj_content) metadata += \ " \n" @@ -2331,7 +2340,8 @@ def html_individual_post(css_cache: {}, instance_title = \ get_config_param(base_dir, 'instanceTitle') - metadata_str = _html_post_metadata_open_graph(domain, original_post_json) + metadata_str = _html_post_metadata_open_graph(domain, original_post_json, + system_language) header_str = html_header_with_external_style(css_filename, instance_title, metadata_str) return header_str + post_str + html_footer() From 3f0f478315a741b8f8c358793c73a5510d4a47b9 Mon Sep 17 00:00:00 2001 From: Bob Mottram Date: Wed, 13 Apr 2022 18:20:01 +0100 Subject: [PATCH 09/10] Link --- website/EN/index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/EN/index.html b/website/EN/index.html index 6f34ccf6d..b93159608 100644 --- a/website/EN/index.html +++ b/website/EN/index.html @@ -1194,7 +1194,7 @@

An Internet of People, Not Corporate Agendas

- Epicyon is written in Python with a HTML+CSS web interface and uses no javascript which makes display in a web browser very lightweight. It can run as a Progressive Web App on mobile. Just say "no" to boring social media sites packed with generic adverts and zombified corporate influencers. + Epicyon is written in Python with a HTML+CSS web interface and uses no javascript which makes display in a web browser very lightweight. It can run as a Progressive Web App on mobile. Just say "no" to boring social media sites packed with generic adverts and zombified corporate influencers.

From 6a66be8fca0448741927e68544b2832bf17866f6 Mon Sep 17 00:00:00 2001 From: Bob Mottram Date: Wed, 13 Apr 2022 18:31:54 +0100 Subject: [PATCH 10/10] Append edits last --- webapp_post.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/webapp_post.py b/webapp_post.py index 66c169999..afafe1e4e 100644 --- a/webapp_post.py +++ b/webapp_post.py @@ -2039,8 +2039,6 @@ def individual_post_as_html(signing_priv_key_pem: str, if not is_pgp_encrypted(content_str): if not is_patch: - # append any edits - content_str += edits_str # Add bold text if bold_reading and \ not displaying_ciphertext and \ @@ -2056,6 +2054,8 @@ def individual_post_as_html(signing_priv_key_pem: str, switch_words(base_dir, nickname, domain, object_content) object_content = html_replace_email_quote(object_content) object_content = html_replace_quote_marks(object_content) + # append any edits + object_content += edits_str else: object_content = content_str else: