PeerTube scales just fine as I understand it, thanks to WebTorrent and ActivityPub #1

Open
opened 2018-03-30 15:40:05 +00:00 by strypey · 7 comments
strypey commented 2018-03-30 15:40:05 +00:00 (Migrated from github.com)

[PeerTube] only use torrents for the downloading and webRTC for streaming so this make scaling more complex...

My understanding is that WebTorrent only uses WebRTC to handle the handshakes between the server hosting the video (webseeding it), and the browsers of the users viewing it. Otherwise, WebRTC gets out of the way and let's BitTorrent do the actual data streaming, with each viewer holding each part of the video they've watched in their browser cache, where it is available to be downloaded by other viewers (like a user downloading a normal torrent), reducing load on the hosting server.

The architecture is that anyone producing video can host their own video by running their own instance of PeerTube, or use the third party PeerTube host of their choice. aWebTorrent prevents their videos eating all their bandwidth or crashing their server, by spreading the streaming load across all the viewers of a given video. ActivityPub is used to make their videos discoverable and viewable through any other PeerTube instance, or any other federated app compatible with AP.

If I have this right, there's no reason you can't host all your visionONtv videos on one server (whether a leased server in a datacentre or a box in your closet). But instead of trying to make visionOntv the portal for all video-makers (Wikipedia for video), you'd encourage other video-makers to set up their own PeerTube instance to host their own videos, the same way people can currently set up their own WordPress instance to host their own blog.

> [PeerTube] only use torrents for the downloading and webRTC for streaming so this make scaling more complex... My understanding is that WebTorrent only uses WebRTC to handle the handshakes between the server hosting the video (webseeding it), and the browsers of the users viewing it. Otherwise, WebRTC gets out of the way and let's BitTorrent do the actual data streaming, with each viewer holding each part of the video they've watched in their browser cache, where it is available to be downloaded by other viewers (like a user downloading a normal torrent), reducing load on the hosting server. The architecture is that anyone producing video can host their own video by running their own instance of PeerTube, or use the third party PeerTube host of their choice. aWebTorrent prevents their videos eating all their bandwidth or crashing their server, by spreading the streaming load across all the viewers of a given video. ActivityPub is used to make their videos discoverable and viewable through any other PeerTube instance, or any other federated app compatible with AP. If I have this right, there's no reason you can't host all your visionONtv videos on one server (whether a leased server in a datacentre or a box in your closet). But instead of trying to make visionOntv the portal for all video-makers (Wikipedia for video), you'd encourage other video-makers to set up their own PeerTube instance to host their own videos, the same way people can currently set up their own WordPress instance to host their own blog.
Openmedianetwork commented 2018-03-30 16:22:45 +00:00 (Migrated from github.com)

As I understand it:

OK the issue is that seeding is only done outa the box in the browser and the core "seed" is stored on the central server so currently the has to be a copy of every video on the central server for the network to function.

The is an example of the #geekproblem as It dues not need to work like this, "users" could setup seed boxes to distribute the hosting of the seed videos. The torrent clients are starting to support this https://wiki.vuze.com/w/WebTorrent

We need a way to move the "seed" videos off the central server to the seedbox's so that all the video is redundantly backed up by the volunteer "users".

We have thousands of finished videos to upload, hundreds/thousands of hours of full resolution original tapes/files that could be "archived". Putting all this up on a single "magic box in the sky" is not a good outcome and a prime example of the #geekproblem

Redundantly seeding this across tens if not hundreds of "seeboxes" is the outcome we desire.

TO MAKE THIS WORK

  • Need implementation of RSS (torrent) by tag out of #peertube

  • webRTC by RSS in on seeboxes (this is already in place in vuze, though not tested)

  • Then we have a social project to outreach the seeboxes, which is a good networking opportunity, an inherent good in it self.

As I understand it: OK the issue is that seeding is only done outa the box in the browser and the core "seed" is stored on the central server so currently the has to be a copy of every video on the central server for the network to function. The is an example of the #geekproblem as It dues not need to work like this, "users" could setup seed boxes to distribute the hosting of the seed videos. The torrent clients are starting to support this https://wiki.vuze.com/w/WebTorrent We need a way to move the "seed" videos off the central server to the seedbox's so that all the video is redundantly backed up by the volunteer "users". We have thousands of finished videos to upload, hundreds/thousands of hours of full resolution original tapes/files that could be "archived". Putting all this up on a single "magic box in the sky" is not a good outcome and a prime example of the #geekproblem Redundantly seeding this across tens if not hundreds of "seeboxes" is the outcome we desire. TO MAKE THIS WORK * Need implementation of RSS (torrent) by tag out of #peertube * webRTC by RSS in on seeboxes (this is already in place in vuze, though not tested) * Then we have a social project to outreach the seeboxes, which is a good networking opportunity, an inherent good in it self.
strypey commented 2018-03-30 16:46:40 +00:00 (Migrated from github.com)

OK the issue is that seeding is only done outa the box in the browser and the core "seed" is stored on the central server so currently the has to be a copy of every video on the central server for the network to function.

Nope. There has to be a copy of the video on at least one PeerTube server somewhere with an internet connection. ActivityPub and WebTorrent do the rest.

"users" could setup seed boxes to distribute the hosting of the seed videos

Or they could just run instances of PeerTube. That way you don't need ...

implementation of RSS (torrent) by tag out of #peertube

OR

webRTC by RSS in on seeboxes (this is already in place in vuze, though not tested)

... because again, ActivityPub is doing all this for you. So as soon as PeerTube hits 1.0 (or you think the beta is stable enough to use for production), you can move on to ...

Then we have a social project to outreach the seeboxes, which is a good networking opportunity, an inherent good in it self.

BTW I'm not sure what you mean by #geekproblem, you seem to use it in a few different ways. Do you have it defined somewhere?

> OK the issue is that seeding is only done outa the box in the browser and the core "seed" is stored on the central server so currently the has to be a copy of every video on the central server for the network to function. Nope. There has to be a copy of the video on *at least one* PeerTube server somewhere with an internet connection. ActivityPub and WebTorrent do the rest. > "users" could setup seed boxes to distribute the hosting of the seed videos Or they could just run instances of PeerTube. That way you don't need ... > implementation of RSS (torrent) by tag out of #peertube OR > webRTC by RSS in on seeboxes (this is already in place in vuze, though not tested) ... because again, ActivityPub is doing all this for you. So as soon as PeerTube hits 1.0 (or you think the beta is stable enough to use for production), you can move on to ... > Then we have a social project to outreach the seeboxes, which is a good networking opportunity, an inherent good in it self. BTW I'm not sure what you mean by #geekproblem, you seem to use it in a few different ways. Do you have it defined somewhere?
Openmedianetwork commented 2018-03-30 18:36:07 +00:00 (Migrated from github.com)

As I understand it:

Check the peertube implementation it only webseeds from the server you post the original video to.

The idea is that seedboxs are just home computers not a whole server - this is a clear example of the #geekproblem ;)

Same #geekproblem normal people are not empowered to setup and run servers, but a scripted setup torrent client running on a old/spare computer with a big USB harddrive is doable DIY'ish. Or better buy a off the shelf box (raspberry pi) that geeks make at a hackspace and just plug it in at home.

This thread is about the current application not doing that.

The #geekproblem I illustrate it a bit in my replies above :)

As I understand it: Check the peertube implementation it only webseeds from the server you post the original video to. The idea is that seedboxs are just home computers not a whole server - this is a clear example of the #geekproblem ;) Same #geekproblem normal people are not empowered to setup and run servers, but a scripted setup torrent client running on a old/spare computer with a big USB harddrive is doable DIY'ish. Or better buy a off the shelf box (raspberry pi) that geeks make at a hackspace and just plug it in at home. This thread is about the current application not doing that. The #geekproblem I illustrate it a bit in my replies above :)
ghost commented 2018-03-30 19:15:17 +00:00 (Migrated from github.com)

The current early alpha version of peertube which I have on Freedombone works that way, but I know that there's plans to enable mirrored copies of videos.

There is always still a hurdle when setting up a server, in that you still need dynamic DNS and a domain name, which is sufficiently technical that most people will never attempt it. Peertube only runs on the clearnet (WebRTC won't work on Tor) and so this is unavoidable.

There are peer-to-peer media systems like ZeroTube on ZeroNet together with webtorrent-desktop which anyone can run from an old laptop. But having tried that myself I know it's very heavy on the CPU and probably not something which would work well on a Raspberry Pi.

https://github.com/webtorrent/webtorrent-desktop

The current early alpha version of peertube which I have on Freedombone works that way, but I know that there's plans to enable mirrored copies of videos. There is always still a hurdle when setting up a server, in that you still need dynamic DNS and a domain name, which is sufficiently technical that most people will never attempt it. Peertube only runs on the clearnet (WebRTC won't work on Tor) and so this is unavoidable. There are peer-to-peer media systems like ZeroTube on ZeroNet together with webtorrent-desktop which anyone can run from an old laptop. But having tried that myself I know it's very heavy on the CPU and probably not something which would work well on a Raspberry Pi. https://github.com/webtorrent/webtorrent-desktop
Openmedianetwork commented 2018-03-30 19:40:39 +00:00 (Migrated from github.com)

Yep, true, I was looking at vuze as it has RSS in plugin the BT-desktop dues not, asked BTD if they were implementing RSS in and got a soso reply.

Vuze is far from perfect but good to roll out a test, its java so has a footprint but its 10 year old tech... but probably added blot since then though.

Am being user centered in tech and organizing to try and get away from the centralizing #geekproblem so when some one reply the tech works my reply tends to be "for who".

strypey - am not getting at you when I say #geekproblem it just such a part of everything ones you start to get a bit geeky that its easy to point it out.

Yep, true, I was looking at vuze as it has RSS in plugin the BT-desktop dues not, asked BTD if they were implementing RSS in and got a soso reply. Vuze is far from perfect but good to roll out a test, its java so has a footprint but its 10 year old tech... but probably added blot since then though. Am being user centered in tech and organizing to try and get away from the centralizing #geekproblem so when some one reply the tech works my reply tends to be "for who". strypey - am not getting at you when I say #geekproblem it just such a part of everything ones you start to get a bit geeky that its easy to point it out.
strypey commented 2018-04-02 14:37:00 +00:00 (Migrated from github.com)

@Openmedianetwork I'm not here to argue with you for shits and giggles. I'm asking probing questions, and suggesting alternative tactics, so I can gain an understanding of why you want to do things the way you describe. I can't help if I don't understand your goals and why you believe your chosen tactics are the best way to achieve them.

My point is we definitely agree more than we disagree. We are in agreement that centralized data silos are bad, and that a decentralized network of computers sharing video is better. We are in agreement that this is best done in ways that involve educating and empowering average users, not having geeks running production and everyone else being webconsumers (is this what you mean by the #geekproblem?).

For example, rather than having an elite of geeks sending and receiving email for everyone else, it was better to teach everyone to use email themselves. For the same reason, rather than having an elite of geeks building all the servers (whether at corporate datacentres or at hackerspaces) and having everyone else just plug them in (and have no idea how to fix anything if it breaks), it's better to teach everyone how to set up and run simple servers, using common consumer hardware (old desktops and laptops) and household/ office internet connections. @bashrc has done lots of work in the FreedomBone project to make that as simple as possible for ordinary people to do.

I'll admit I haven't tried to install and run PeerTube yet. You may be right that it's more complicated than I imagine. I think the best solution for that is to work with Chocobozzz and Framasoft to work on ways of packaging PeerTube so that it becomes very simple to install and run.

That said, the two tactics are not mutually exclusive. We can work on setting up seedboxen using Vuze + RSS (or WT-desktop + RSS), and we can work with Chocobozzz and Framasoft to make PeerTube as simple as possible for ordinary people to use for publishing their own video, in parallel.

@Openmedianetwork I'm not here to argue with you for shits and giggles. I'm asking probing questions, and suggesting alternative tactics, so I can gain an understanding of why you want to do things the way you describe. I can't help if I don't understand your goals and why you believe your chosen tactics are the best way to achieve them. My point is we definitely agree more than we disagree. We are in agreement that centralized data silos are bad, and that a decentralized network of computers sharing video is better. We are in agreement that this is best done in ways that involve educating and empowering average users, not having geeks running production and everyone else being webconsumers (is this what you mean by the #geekproblem?). For example, rather than having an elite of geeks sending and receiving email for everyone else, it was better to teach everyone to use email themselves. For the same reason, rather than having an elite of geeks building all the servers (whether at corporate datacentres or at hackerspaces) and having everyone else just plug them in (and have no idea how to fix anything if it breaks), it's better to teach everyone how to set up and run simple servers, using common consumer hardware (old desktops and laptops) and household/ office internet connections. @bashrc has done lots of work in the FreedomBone project to make that as simple as possible for ordinary people to do. I'll admit I haven't tried to install and run PeerTube yet. You may be right that it's more complicated than I imagine. I think the best solution for that is to work with Chocobozzz and Framasoft to work on ways of packaging PeerTube so that it becomes very simple to install and run. That said, the two tactics are not mutually exclusive. We can work on setting up seedboxen using Vuze + RSS (or WT-desktop + RSS), *and* we can work with Chocobozzz and Framasoft to make PeerTube as simple as possible for ordinary people to use for publishing their own video, in parallel.
ghost commented 2018-04-02 16:02:28 +00:00 (Migrated from github.com)

One thing I've just been testing is some integration between Syncthing and PeerTube. You drop your videos into a desktop folder and then they appear on PeerTube. It's this type of integration which might make things as simple as possible for ordinary people.

One thing I've just been testing is some integration between Syncthing and PeerTube. You drop your videos into a desktop folder and then they appear on PeerTube. It's this type of integration which might make things as simple as possible for ordinary people.
Sign in to join this conversation.
No Milestone
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: Open-Media-Network/peertube-visionOntv#1
There is no content yet.