Skip to content

Commit 27f4902

Browse files
iameliiameli
authored
catalyst: add catalyst documentation (#455)
* catalyst: add catalyst documentation * Dockerfile * sharpless * catalyst: don't be the default page anymore * Makefile: add * catalyst: add MistServer documentation
1 parent 8f4906c commit 27f4902

File tree

6 files changed

+311
-0
lines changed

6 files changed

+311
-0
lines changed

‎Dockerfile

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
FROM node:current
2+
3+
RUN npm install -g mintlify
4+
5+
WORKDIR /app
6+
7+
ADD . .
8+
9+
CMD ["mintlify", "dev"]

‎Makefile

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
.PHONY: all
2+
all:
3+
docker buildx build --platform linux/amd64 --load -t livepeer/docs .

‎catalyst/developing-with-catalyst.mdx

+178
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,178 @@
1+
---
2+
title: Developing with Catalyst
3+
icon: gear
4+
---
5+
6+
Developing Catalyst is currently supported on macOS and Linux.
7+
8+
### Dependencies
9+
10+
You'll need the following things installed locally:
11+
12+
- Docker (or Podman 4.6.0+, which works even better)
13+
- Buildah (for `docker buildx`, included with Podman and Docker for Mac)
14+
- Go v1.20+
15+
- Node.js v18+ (Studio development only)
16+
- Make
17+
- git
18+
- llvm (`brew install llvm` for compiling MistServer and go-livepeer on MacOS
19+
only)
20+
21+
You'll first need to clone the Catalyst repo if you haven't already.
22+
23+
```shell
24+
git clone https://github.com/livepeer/catalyst.git
25+
cd catalyst
26+
```
27+
28+
Then you'll need to download/build all the binaries that you'll need.
29+
30+
```shell
31+
make # downloads all the external binaries & builds the local ones
32+
make box # builds the livepeer/in-a-box Docker image
33+
```
34+
35+
## Development
36+
37+
Now you're ready to boot up your development environment!
38+
39+
```shell
40+
make box-dev
41+
```
42+
43+
Lots and lots of logs will print out while all the dependencies finish booting
44+
up. After that, you'll now have a fully-functioning full-stack Livepeer Studio +
45+
Catalyst environment running locally! You can access it like so:
46+
47+
- URL: [http://localhost:8888](http://localhost:8888)
48+
- Email: `[email protected]`
49+
- Password: `livepeer`
50+
51+
### Customizing the Environment
52+
53+
If you need to add additional parameters to things running inside the box, you
54+
can create a `.env` file that contains key-value pairs:
55+
56+
```shell
57+
echo "CATALYST_API_CDN_REDIRECT_PREFIX=https://externalcdn.livepeer.com/mist/" >> .env
58+
echo "CATALYST_API_CDN_REDIRECT_PLAYBACK_IDS=222222222222" >> .env
59+
```
60+
61+
Note that this mechanism is limited to adding new configuration; existing
62+
configuration cannot be modified this way. If you need to modify existing
63+
configuration you can manually change the file in `config/full-stack.json`. A
64+
more robust configuration management system is under development.
65+
66+
### Making changes
67+
68+
TLDR: Use a command like this and the Makefile will take care of it for you:
69+
70+
```shell
71+
make livepeer-catalyst-api
72+
```
73+
74+
The general Livepeer in a Box development cycle works like this:
75+
76+
1. Make changes to code on your local filesystem
77+
2. Build a Linux binary from that code
78+
3. Move that Linux binary into the `bin` directory of `catalyst`, which is
79+
mounted by `make box-dev`
80+
4. Kill the old version of your binary and allow MistController to bring it back
81+
up.
82+
83+
Thankfully, this entire process has been automated. All you need to do is have
84+
the project you're working on cloned in a directory adjacent to `catalyst`. For
85+
example, if you're hacking on `task-runner`, you might have
86+
87+
```
88+
/home/user/code/catalyst
89+
/home/user/code/task-runner
90+
```
91+
92+
The catalyst Makefile is aware of the common paths for all of the other projects
93+
that go into the full stack. All that's necessary to build a new binary, package
94+
it in the container, and trigger a restart is a single command:
95+
96+
```shell
97+
make livepeer-task-runner
98+
```
99+
100+
Note that the names of all subprojects are prefixed with `livepeer`, just like
101+
the resulting binaries within the Catalyst container. This yields the following
102+
commands:
103+
104+
| Project | Command |
105+
| --------------------------- | --------------------------------- |
106+
| [catalyst-api] | `make livepeer-catalyst-api` |
107+
| [catalyst-uploader] | `make livepeer-catalyst-uploader` |
108+
| [task-runner] | `make livepeer-task-runner` |
109+
| [analyzer] | `make livepeer-analyzer` |
110+
| [Studio Node.js API Server] | `make livepeer-api` |
111+
| [MistServer] | `make mistserver` |
112+
113+
[catalyst-api]: https://github.com/livepeer/catalyst-api
114+
[catalyst-uploader]: https://github.com/livepeer/catalyst-uploader
115+
[task-runner]: https://github.com/livepeer/catalyst-uploader
116+
[analyzer]: https://github.com/livepeer/livepeer-data
117+
[Studio Node.js API Server]: https://github.com/livepeer/studio
118+
[MistServer]: https://github.com/livepeer/mistserver
119+
120+
### Connecting the Frontend
121+
122+
Livepeer in a Box comes with a [pkg](https://github.com/vercel/pkg)-bundled
123+
version of the Livepeer Studio API server and frontend, but does not include a
124+
full development environment for that frontend. If you are making changes to the
125+
frontend, you can boot it up as you usually would:
126+
127+
```
128+
cd studio/packages/www
129+
yarn run dev
130+
```
131+
132+
To connect it to the box; there's a hidden `localStorage` variable you can use
133+
to override the API server URL. Open your browser console and type in the
134+
following:
135+
136+
```javascript
137+
localStorage.setItem("LP_API_SERVER_OVERRIDE", "http://localhost:8888");
138+
```
139+
140+
Reload the page and your frontend should be connecting to the box as an API
141+
server.
142+
143+
Additional note: in the interest of build speed, `make livepeer-api` does not
144+
package the frontend within the `livepeer-api` binary that it builds, so if you
145+
experience your frontend suddenly 404ing after you run `make livepeer-api` you
146+
will have to use the above instructions to boot up the frontend on your host.
147+
148+
You can also build the full API server with a bundled frontend using
149+
`make livepeer-api-pkg`, but be aware this frequently takes 3-4 minutes to
150+
complete.
151+
152+
### Notes
153+
154+
- Your CockroachDB (Postgres) database and your Minio (S3) object store will be
155+
saved in the `data` subdirectory of your Catalyst installation. If you want to
156+
start from scratch again with the `[email protected]` database snapshot, shut
157+
down your box and `rm -rf data`.
158+
- You can press `Ctrl+C` to trigger a graceful shutdown of the container. If
159+
you're impatient, following it up with a `Ctrl+\` can uncleanly shut things
160+
down a bit more cleanly.
161+
- Sometimes the rate of logs produced by Catalyst somehow overwhelms Make and
162+
log output simply stops. You'll know if you get in this state because you'll
163+
press Ctrl+C and control will return immediately to your terminal instead of
164+
shutting down the Docker image. You can start everything back up with
165+
`docker rm -f catalyst` and `make box-dev`.
166+
167+
## Changelog
168+
169+
### 2023-08-12
170+
171+
- Changed the hardcoded streams in the database snapshots to have
172+
easy-to-remember stream keys like `2222-2222-2222-2222`
173+
- Changed the built-in streams to use the H264ConstrainedHigh profile so there
174+
are no B-Frames in the output
175+
- Moved all references from `127.0.0.1` to `localhost`; this is needed for
176+
WebRTC/Coturn to work properly
177+
- Removed outdated references to `GOOS=linux` and `KILL=true`; these are the
178+
defaults now

‎catalyst/introduction.mdx

+35
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
---
2+
title: What is Catalyst?
3+
icon: flask
4+
---
5+
6+
Catalyst is Livepeer's open-source, decentralized media server. It's built from
7+
the same software stack that backs the Livepeer Studio hosted platform.
8+
9+
**Self-hosted Catalyst is in Alpha**, but we're excited for the first group of
10+
hackers to start working on it!
11+
12+
### Current capabilities:
13+
14+
- Boots up a full-stack Livepeer experience on your laptop with a single
15+
command.
16+
- Facilitates easy development of various components of the Livepeer stack.
17+
- Allows for development of applications against the Livepeer Studio API locally
18+
that can then transfer to the hosted version at
19+
[livepeer.studio](https://livepeer.studio) when you're ready to go to
20+
production.
21+
- Bundles a fully-local offchain go-livepeer broadcaster and orchestrator, so
22+
that you may test transcoding with no external dependencie.s
23+
24+
### Current limitations:
25+
26+
- There is no way to allow for easy deployment to a server. There are presently
27+
many hardcoded references to `localhost`, and these things will break in any
28+
other environment.
29+
- No usage or billing data.
30+
- No GPU transcoding support. We recommend using very low-bitrate test files,
31+
especially if running the box using Docker for Mac or Docker for Windows. The
32+
built-in profiles for livestream transcoding use a single 240p low-quality
33+
rendition.
34+
- It is currently required to run the node as a Docker image. A single-binary
35+
fully-static build is in the works.

‎catalyst/running-catalyst.mdx

+73
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
---
2+
title: Running Catalyst
3+
icon: person-running
4+
---
5+
6+
First, select a directory for persisting your database and video content; in
7+
this example we will be using `$HOME/livepeer-catalyst`.
8+
9+
```shell
10+
CATALYST_DIR="$HOME/livepeer-catalyst"
11+
mkdir -p $CATALYST_DIR
12+
docker run \
13+
-v $CATALYST_DIR:/data \
14+
--rm \
15+
-it \
16+
--name catalyst \
17+
--shm-size=4gb \
18+
-p 8888:8888 \
19+
-p 5432:5432 \
20+
-p 1935:1935 \
21+
-p 4242:4242 \
22+
-p 3478:3478 \
23+
-p 3478:3478/udp \
24+
-p 5349:5349 \
25+
-p 40000-40100:40000-40100/udp \
26+
livepeer/catalyst
27+
```
28+
29+
You will be greeted with a very large amount of spam — give it a minute or so to
30+
boot up. You can then connect to your local box instance:
31+
32+
Address: [http://localhost:8888](http://localhost:8888)
33+
Email: `[email protected]`
34+
Password: `livepeer`
35+
36+
You can also access the MistServer dashboard to access some underlying
37+
livestreaming infrastructure:
38+
39+
Address: [http://localhost:4242](http://localhost:4242)
40+
Username: `test`
41+
Password: `test`
42+
43+
To get you started, the database snapshot includes a few predefined streams.
44+
45+
| Stream | Stream Key | Playback ID | Recording enabled? |
46+
| ---------------- | ------------------- | ------------ | ------------------ |
47+
| [tiny-transcode] | 2222-2222-2222-2222 | 222222222222 | No |
48+
| [tiny-recording] | 4444-4444-4444-4444 | 444444444444 | Yes |
49+
50+
[tiny-transcode]:
51+
http://localhost:8888/dashboard/streams/22222222-2222-2222-2222-222222222222
52+
[tiny-recording]:
53+
http://localhost:8888/dashboard/streams/44444444-4444-4444-4444-444444444444
54+
55+
For properly testing a livestream input comparable to OBS output, you will want
56+
a low-bitrate test file with no B-Frames and a short GOP length.
57+
[Here's a sample appropriately-formatted Big Buck Bunny file you can use](BBB).
58+
To stream in to your local box, you can use an `ffmpeg` command such as:
59+
60+
```shell
61+
curl -LO https://test-harness-gcp.livepeer.fish/Big_Buck_Bunny_360p_1sGOP_NoBFrames.mp4
62+
ffmpeg \
63+
-stream_loop \
64+
-1 \
65+
-re \
66+
-i Big_Buck_Bunny_360p_1sGOP_NoBFrames.mp4 \
67+
-c copy \
68+
-f flv \
69+
rtmp://localhost/live/2222-2222-2222-2222
70+
```
71+
72+
[BBB]:
73+
https://test-harness-gcp.livepeer.fish/Big_Buck_Bunny_360p_1sGOP_NoBFrames.mp4

‎mint.json

+13
Original file line numberDiff line numberDiff line change
@@ -349,6 +349,11 @@
349349
"icon": "brackets-curly",
350350
"url": "sdks"
351351
},
352+
{
353+
"name": "Catalyst",
354+
"icon": "flask",
355+
"url": "catalyst"
356+
},
352357
{
353358
"name": "What's New",
354359
"icon": "rocket",
@@ -479,6 +484,14 @@
479484
],
480485
"version": "Orchestrators"
481486
},
487+
{
488+
"group": "Catalyst",
489+
"pages": [
490+
"catalyst/introduction",
491+
"catalyst/running-catalyst",
492+
"catalyst/developing-with-catalyst"
493+
]
494+
},
482495
{
483496
"group": "References",
484497
"pages": [

0 commit comments

Comments
 (0)