Packager fails to copy tmp files into place once its done when using a container volume + sshfs mount

[doughsay]

System info

Operating System: Alpine 3.19.1 container running through Podman 3.4.4 on Ubuntu 22.04.4 host
Shaka Packager Version: v3.0.4-ceeb378-release

Issue and steps to reproduce the problem

Packager Command:

shaka-packager in=e583a4d3-6ec2-48f8-b478-cfb20bc0f7ed.mp4,stream=audio,out=e583a4d3-6ec2-48f8-b478-cfb20bc0f7ed.mp4,playlist_name=e583a4d3-6ec2-48f8-b478-cfb20bc0f7ed_0.m3u8 --base_urls /uploads/media/ --hls_base_url /uploads/media/ --mpd_output e583a4d3-6ec2-48f8-b478-cfb20bc0f7ed.mpd --hls_master_playlist_output e583a4d3-6ec2-48f8-b478-cfb20bc0f7ed.m3u8 -quiet

Extra steps to reproduce the problem?

1. The container mounts a volume from the host system and uses that volume for the operations
2. The host system's mount is actually an sshfs mount from a remote NAS
   • fstab line: sshfs#user@x.x.x.x:/sshfs /mnt/nas fuse delay_connect,_netdev,idmap=user,allow_other,IdentityFile=/home/user/.ssh/id_rsa,reconnect 0 0

What is the expected result?

It works fine without the complicated container + volumes + sshfs setup

What happens instead?

Instead it seems to work, but fails to move the tmp files created during packaging to the current directory:

E0329 16:49:43.597311      79 file.cc:91] Failed to replace file 'e583a4d3-6ec2-48f8-b478-cfb20bc0f7ed.mpd' with '/tmp/packager-tempfile-4e-f69285b87c746f39-2', error: generic:18
E0329 16:49:43.597347      79 mpd_notifier_util.cc:28] Failed to write mpd to: e583a4d3-6ec2-48f8-b478-cfb20bc0f7ed.mpd
E0329 16:49:43.733331      78 file.cc:91] Failed to replace file 'e583a4d3-6ec2-48f8-b478-cfb20bc0f7ed_0.m3u8' with '/tmp/packager-tempfile-4e-98a771ca66729809-1', error: generic:18
E0329 16:49:43.733364      78 media_playlist.cc:505] Failed to write playlist to: e583a4d3-6ec2-48f8-b478-cfb20bc0f7ed_0.m3u8
E0329 16:49:43.733371      78 simple_hls_notifier.cc:255] Failed to write playlist e583a4d3-6ec2-48f8-b478-cfb20bc0f7ed_0.m3u8
E0329 16:49:43.733378      78 packager_main.cc:625] Packaging Error: 3 (INVALID_ARGUMENT): Failed to flush Hls.

without knowing what error: generic:18 means, I'm kinda stuck...

NOTE: It works with shaka-packager 2.6.1, for my use-case this appears to be a regression in 3.x

[Brainiarc7]

Please paste the full Docker command(s) used to execute shaka-packager.

[doughsay]

Please paste the full Docker command(s) used to execute shaka-packager.

I'm not using Docker to execute shaka-packager, I'm using podman to run a custom app where the image happens to contain the shaka-packager binary inside it. I gave the command I'm executing above (from within the running container). I could maybe try and reproduce with the official shaka-packager docker images, but I have not done so yet...

[joeyparrish]

@doughsay, please try with official images. We don't control whatever other image you're using and can't fix it if the problem is something inside that environment. If the issue exists with our official images, we will do what we can. But you'll need to provide detailed instructions for repro. Thanks!

[Brainiarc7]

This issue is also reproducible in the official Docker images..

…

On Mon, 15 Apr 2024, 21:18 Joey Parrish, ***@***.***> wrote: @doughsay <https://github.com/doughsay>, please try with official images. We don't control whatever other image you're using and can't fix it if the problem is something inside that environment. If the issue exists with our official images, we will do what we can. But you'll need to provide detailed instructions for repro. Thanks! — Reply to this email directly, view it on GitHub <#1383 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAR7NBZPSPCDSK7UWJAUBH3Y5QKX3AVCNFSM6AAAAABFOV62M6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJXGUZTINBRGU> . You are receiving this because you commented.Message ID: ***@***.***>

[doughsay]

@Brainiarc7 can you provide any details on how you were able to reproduce? I haven't had the time yet to try, but I can look into it soon unless you have something that'll make it easier/clearer

[mkcdot]

@doughsay I faced same issue today. I tried all past versions till 3.0.0. It has same issue. For versions before 3.0.0, this issue doesn't come.
The workaround to solve this issue is to give complete path of mpd file in the command as new code expects mpd file name has filepath mentioned along with file name.
Maybe this can be marked as bug as this change was not intentional.
As I see, you have also not mentioned filepath. Check if this works for you.

[Brainiarc7]

Correct.

…

On Thu, 25 Apr 2024 at 11:23, mkcdot ***@***.***> wrote: @doughsay <https://github.com/doughsay> I faced same issue today. I tried all past versions till 3.0.0. It has same issue. For versions before 3.0.0, this issue doesn't come. The workaround to solve this issue is to give complete path of mpd file in the command as new code expects mpd file name has filepath mentioned along with file name. Maybe this can be marked as bug as this change was not intentional. As I see, you have also not mentioned filepath. Check if this works for you. — Reply to this email directly, view it on GitHub <#1383 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAR7NB5X2H755CG2RLIISNTY7C4PPAVCNFSM6AAAAABFOV62M6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZWGYZTSMBYHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[doughsay]

Thanks @mkcdot, I've confirmed that using the full path for mpd file instead of a relative path doesn't cause the issue to occur.