RSSTootalizer - Rate Limiting and Digest #5

New Issue

2018-04-20T14:07:10Z

mj-saunders commented

2018-04-20 14:07:10 +00:00

(Migrated from github.com)

Feel free to post additions via comments; we can always improve what follows.

Potential options to pursue:

Display every n^th RSS post - not ideal
Toot a digest of n posts per toot
Use a hashtag or keyword list - but who decides? And I guess the bot would have to somehow publish and be clear what tags it is searching for and posting.

Details

Option 2 - Toot Digest

This appears to be our best option for the time being.
For purposes here, n = 5

Method 1

Wait for 5 posts to accumulate, then toot
- Reset a 24hour timer
Toot a digest once every 24hours, even if 5 has not been reached
Allow more digests per day?
- How many?
- How to space these throughout the day? ¹
Allow for rollover of posts beyond the daily limit, so they have another chance to be tooted the following day
- How to stop this from creating massive delays eventually with high-volume bots? ¹

Method 2

Check RSS every hour
- Should this be less frequent? If so, do we risk too much delay?
If there are more than one post, toot a digest (otherwise do normal toot)
- up to a maximum of 5
If there are more than 5, rollover to next hourly check
- if next check yields again more than 5, how to prioritise?
This potentially leads to 24 toots per day for a bot
- Too much?
- How best to rate-limit?

¹ I dislike this as we are then altering when content is being published

I am sure there are more issues. Please comment and question so we can improve, decide and solve this quickly 😄

Feel free to post additions via comments; we can always improve what follows. ## Potential options to pursue: 1. Display every _n_th RSS post - not ideal 2. Toot a digest of _n_ posts per toot 3. Use a hashtag or keyword list - but who decides? And I guess the bot would have to somehow publish and be clear what tags it is searching for and posting. ## Details ### Option 2 - Toot Digest _This appears to be our best option for the time being._ For purposes here, _n_ = 5 **Method 1** - Wait for 5 posts to accumulate, then toot - Reset a 24hour timer - Toot a digest once every 24hours, even if 5 has not been reached - Allow more digests per day? - How many? - How to space these throughout the day? **1** - Allow for rollover of posts beyond the daily limit, so they have another chance to be tooted the following day - How to stop this from creating massive delays eventually with high-volume bots? **1** **Method 2** - Check RSS every hour - Should this be less frequent? If so, do we risk too much delay? - If there are more than one post, toot a digest (otherwise do normal toot) - up to a maximum of 5 - If there are more than 5, rollover to next hourly check - if next check yields again more than 5, how to prioritise? - This potentially leads to 24 toots per day for a bot - Too much? - How best to rate-limit? **1 I dislike this as we are then altering when content is being published** I am sure there are more issues. Please comment and question so we can improve, decide and solve this quickly 😄

Openmedianetwork commented

2018-04-20 16:38:57 +00:00

(Migrated from github.com)

OK the is no perfect option, from a technical point of view i prefer each link to be a separate toot. BUT this is not good from a user point of view.

POTP

not ideal
maybe the best option
we are doing a bigger project to make this idea work http://omn.openworlds.info but its not going to be in a usable state for a year or more, so no.

M1
Think M2 better.
Maybe have a configure setting for 3/5 toots for different toot flows

M2
Think maybe we should try this one?
Check every hour then toot as a group up to 5 or normal single toot

*** we mediate this by just not adding any feeds that are to fast flowing ***

OK the is no perfect option, from a technical point of view i prefer each link to be a separate toot. BUT this is not good from a user point of view. POTP 1) not ideal 2) maybe the best option 3) we are doing a bigger project to make this idea work http://omn.openworlds.info but its not going to be in a usable state for a year or more, so no. M1 Think M2 better. Maybe have a configure setting for 3/5 toots for different toot flows M2 Think maybe we should try this one? Check every hour then toot as a group up to 5 or normal single toot *** we mediate this by just not adding any feeds that are to fast flowing ***

mj-saunders commented

2018-04-21 03:43:50 +00:00

(Migrated from github.com)

OK. I agree - prefer M2.

I shall spend a little time going over the existing code, to see how best to implement this.

Will also allow for today, for any of us to come up with a better solution before I start hacking at code, and for the voice of a certain Vagabond to be added - I like consensus.

OK. I agree - prefer M2. I shall spend a little time going over the existing code, to see how best to implement this. Will also allow for today, for any of us to come up with a better solution before I start hacking at code, and for the voice of a certain Vagabond to be added - I like consensus.

mj-saunders commented

2018-05-03 12:05:56 +00:00

(Migrated from github.com)

Notes (mainly) to self - feel free to ignore this post:

Check RSS every hour

Already done with cronjob - could tweak frequency using crontab -e

If there are more than one post, toot a digest (otherwise do normal toot)

up to a maximum of 5

If there are more than 5, rollover to next hourly check

if next check yields again more than 5, how to prioritise?

Perl code edits. I believe this should all be achievable without altering the database structure.

This potentially leads to 24 toots per day for a bot

How best to rate-limit?

Even though cronjob checks feeds every hour, perhaps rate-limiting should be done via Perl.
Would allow for timely checking, but not necessarily such frequent posting.
RSSTootalizer originally intended to use the cronjob to determine posting frequency.

TODO:
Analyse how the db is utilising timecodes and decide how to proceed.

**Notes (mainly) to self** - feel free to ignore this post: >- Check RSS every hour Already done with cronjob - could tweak frequency using `crontab -e` >- If there are more than one post, toot a digest (otherwise do normal toot) > - up to a maximum of 5 >- If there are more than 5, rollover to next hourly check > - if next check yields again more than 5, how to prioritise? Perl code edits. I believe this should all be achievable without altering the database structure. >- This potentially leads to 24 toots per day for a bot > - How best to rate-limit? Even though cronjob checks feeds every hour, perhaps rate-limiting should be done via Perl. Would allow for timely checking, but not necessarily such frequent posting. RSSTootalizer originally intended to use the cronjob to determine posting frequency. **TODO:** Analyse how the db is utilising timecodes and decide how to proceed.

mj-saunders commented

2019-01-08 05:57:20 +00:00

(Migrated from github.com)

a functioning (prototype?) is now running utilising most of Method 2

Sign in to join this conversation.