DMCA is coming...?
- Download the live streaming or VOD from eplus and other websites via Streamlink or yt-dlp.
- Upload the video file to S3-compatible object storage via S3cmd or to Azure Storage container via Azure CLI.
For a 4-hour live event, the size of a MPEG-TS recording with the best quality is about 9.88 GB (9.2 GiB).
The yt-dlp support is experiment.
The output file is in ".ts" format. I believe that your media player is smart enough to get to know the actual codec.
The file is located in the "/SL-downloads" directory in the container. You are able to access those files by mounting a volume into that directory before creating the container (otherwise you may have to play with docker cp or anonymous volumes).
You will see some intermediate files. Those files will be renamed to "final files" finally:
# template:
${datetime}.${OUTPUT_FILENAME_BASE}.ts # full
${OUTPUT_FILENAME_BASE}.ts # NO_AUTO_PREFIX_DATETIME
# example:
'20220605T040302Z.name-1st-Otoyk-day0.ts' # full
'name-1st-Otoyk-day0.ts' # NO_AUTO_PREFIX_DATETIME
Final files:
# template:
${datetime}.${OUTPUT_FILENAME_BASE}.${size}.${md5}.ts # full
${OUTPUT_FILENAME_BASE}.${size}.${md5}.ts # NO_AUTO_PREFIX_DATETIME
${datetime}.${OUTPUT_FILENAME_BASE}.${md5}.ts # NO_AUTO_FILESIZE
${datetime}.${OUTPUT_FILENAME_BASE}.${size}.ts # NO_AUTO_MD5
# example:
'20220605T040302Z.name-1st-Otoyk-day0.123456789.0123456789abcdef0123456789abcdef.ts' # full
'name-1st-Otoyk-day0.123456789.0123456789abcdef0123456789abcdef.ts' # NO_AUTO_PREFIX_DATETIME
'20220605T040302Z.name-1st-Otoyk-day0.0123456789abcdef0123456789abcdef.ts' # NO_AUTO_FILESIZE
'20220605T040302Z.name-1st-Otoyk-day0.123456789.ts' # NO_AUTO_MD5
| Variable | Description |
|---|---|
| OUTPUT_FILENAME_BASE | base file name (env) |
| datetime | datetime at UTC in ISO 8601 format. strftime(${datetime}, 17, "%Y%m%dT%H%M%SZ", gmtime(&(time(0)))) |
| size | file size. du -b "$filepath" |
| md5 | file hash. md5sum "$filepath" |
Create your own object storage:
- AWS:
- DigitalOcean:
- Linode:
- Vultr:
- DreamObjects:
Environment variables:
| Name | Description |
|---|---|
| S3_BUCKET | URL in s3://bucket-name/dir-name/ style |
| S3_HOSTNAME | For example: s3-eu-west-1.amazonaws.com nyc3.digitaloceanspaces.com us-east-1.linodeobjects.com ewr1.vultrobjects.com objects-us-east-1.dream.io |
| AWS_ACCESS_KEY_ID | The access key |
| AWS_SECRET_ACCESS_KEY | The secret key |
Create a service principal on Azure.
- Create an Azure AD app and service principal in the portal - Microsoft identity platform | Microsoft Docs
- Create an Azure service principal – Azure CLI | Microsoft Docs
For azure-cli :
$ az login --use-device-code
$ az ad sp create-for-rbac --role 'Contributor' --name "${name}" --scopes "/subscriptions/${subscription}/resourceGroups/${resourceGroup}/providers/Microsoft.Storage/storageAccounts/${AZURE_STORAGE_ACCOUNT}"
Environment variables:
| Name | Description |
|---|---|
| AZ_SP_APPID | Application (client) ID |
| AZ_SP_PASSWORD | Client secret |
| AZ_SP_TENANT | Directory (tenant) ID |
| AZURE_STORAGE_ACCOUNT | Azure storage account |
| AZ_STORAGE_CONTAINER_NAME | Storage container name |
Install Docker Engine and use the docker compose command to manipulate Docker Compose.
Create a service:
# docker-compose.yml
services:
sl:
image: docker.io/pzhlkj6612/streamlink-eplus_jp-object_storage
volumes:
- ./SL-downloads:/SL-downloads:rw
- ./YTDLP:/YTDLP:rw # edit "cookies.txt" in it
environment:
# base file name; will use a random one if leaving empty.
- OUTPUT_FILENAME_BASE=
# output file name configuration
- NO_AUTO_PREFIX_DATETIME=
- NO_AUTO_FILESIZE=
- NO_AUTO_MD5=
# Input control
# only one input allowed; using file has the highest priority.
# file
# does NOT imply "NO_AUTO_PREFIX_DATETIME", "NO_AUTO_FILESIZE" and "NO_AUTO_MD5".
# does imply "NO_DOWNLOAD_TS".
- USE_EXISTING_MPEG_TS_VIDEO_FILE=
# proxy for streamlink and yt-dlp
- HTTPS_PROXY=http://127.0.0.1:1926 # empty by default.
# common
- DOWNLOAD_THREAD_NUM= # "--stream-segment-threads" for streamlink, "--concurrent-fragments" for yt-dlp
# streamlink
- EPLUS_JP_STREAM_URL= # enable streamlink.
- EPLUS_JP_STREAM_QUALITY= # "best" by default.
- STREAMLINK_RETRY_TOTAL_SECONDS= # 42 seconds between attempts, 0 second (no retry) by default.
- STREAMLINK_RINGBUFFER_SIZE= # "--ringbuffer-size", 200M by default.
- STREAMLINK_HLS_START_OFFSET= # "--hls-start-offset", 00:00:00 by default.
- STREAMLINK_OPTIONS= # options passed into streamlink before all others.
# yt-dlp
- YTDLP_STREAM_URL= # enable yt-dlp.
- YTDLP_QUALITY= # "--format", "bestvideo*+bestaudio/best" by default.
- YTDLP_WAIT_FOR_VIDEO= # "--wait-for-video", "19-26" by default.
- YTDLP_BUFFER_SIZE= # "--buffer-size", 200M by default.
- YTDLP_USERNAME= # "--username", empty by default.
- YTDLP_PASSWORD= # "--password", empty by default.
- YTDLP_OPTIONS= # options passed into yt-dlp before all others.
# direct download
- VIDEO_FILE_URL= # download a video file.
# ffmpeg
- GENERATE_STILL_IMAGE_MPEG_TS= # generate a still image MPEG-TS video.
# Output control
# multiple outputs supported.
# file
- NO_DOWNLOAD_TS= # do not save the video file. it may not be a MPEG-TS file, but a MKV one.
# rtmp
- RTMP_TARGET_URL= # enable RTMP streaming.
- RTMP_FFMPEG_USE_LIBX264_ENCODING= # enable video encoding, otherwise just copy the video stream.
- RTMP_FFMPEG_CRF= # CRF value, 23 by detault, see https://trac.ffmpeg.org/wiki/Encode/H.264#a1.ChooseaCRFvalue .
# uploading control
- ENABLE_S3= # enable s3cmd.
- ENABLE_AZURE= # enable azure-cli.
# s3cmd
- AWS_ACCESS_KEY_ID=
- AWS_SECRET_ACCESS_KEY=
- S3_BUCKET=
- S3_HOSTNAME=
- S3CMD_MULTIPART_CHUNK_SIZE_MB= # "--multipart-chunk-size-mb", 15 by default.
# azure-cli
- AZURE_STORAGE_ACCOUNT=
- AZ_SP_APPID=
- AZ_SP_PASSWORD=
- AZ_SP_TENANT=
- AZ_STORAGE_CONTAINER_NAME=
Run it:
$ docker compose up sl
For developers who want to build the image themselves:
$ docker build --tag ${tag} .
Install Podman. Create a pod and a "hostPath" volume:
# pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: sl
spec:
volumes:
- name: SL-downloads
hostPath:
path: ./SL-downloads
type: Directory
restartPolicy: Never
containers:
- name: sl
image: docker.io/pzhlkj6612/streamlink-eplus_jp-object_storage
resources: {}
volumeMounts:
- mountPath: /SL-downloads
name: SL-downloads
env:
# Please refer to the "Docker" section.
- name: # ...
value: # "..."
Finally, play it:
$ podman play kube ./pod.yaml # 開演！
For developers who want to build the image themselves:
$ podman build --tag ${tag} .
- Container Technologies:
- Open Container Initiative (OCI).
- Docker and Docker Compose.
- Podman and Kubernetes.
- Useful open-source programs and tools:
- s3tools/s3cmd.
- Azure/azure-cli.
- streamlink/streamlink and pmrowla/streamlink-plugins.
- yt-dlp/yt-dlp
- BtbN/FFmpeg-Builds.
- I'm using shell-format - Visual Studio Marketplace to format my bash shell script.
- Platforms:
- AWS, DigitalOcean, Linode, Vultr and DreamHost.
- Microsoft Azure.
- Stack Exchange website group.
- Linux and Ubuntu.