Add env var for Companion Standalone to opt-in to using the previous "unsafe" behavior of getKey

[williamccrobinson]

Initial checklist

• I understand this is a feature request and questions should be posted in the Community Forum
• I searched issues and couldn’t find anything (or linked relevant results below)

Problem

Since issue #3869 , there is no way for standalone Companion to opt-in to using the older "unsafe" version of getKey that would just use the filename provided by uppy frontend.

The getKey used in standalone is the defaultGetKey, which is now appending a random uuid in front of the filename :

module.exports.defaultGetKey = (req, filename) => `${crypto.randomUUID()}-${filename}`

This new behavior in v4 means users of standalone Companion that need or require to be able to dictate the S3 object key from the uppy frontend, cannot do so anymore.

The only way to retain control over the naming of S3 objects in v4 is to discontinue using Companion standalone and setup their own server and provide their own custom getKey to Companion server.

Solution

Would the maintainer be opposed that we introduce a new environment variable (e.g. COMPANION_S3_GETKEY_UNSAFE_BEHAVIOR) that when set to true, will use the previous behavior of not prefixing the filename with a random uuid?

I propose to implement this in companion/src/standalone/helper.js as:

/**
 * Loads the config from environment variables
 *
 * @returns {object}
 */
const getConfigFromEnv = () => {  
  ...
  return {
    ...
    s3: {
      getKey: process.env.COMPANION_S3_GETKEY_UNSAFE_BEHAVIOR ? utils.unsafeGetKey : utils.defaultGetKey,
      ...
    }
    ...
  }

and add an alternate unsafe getKey which preserves previous behavior in companion/src/server/helpers/utils.js :

module.exports.unsafeGetKey = (req, filename) => `${filename}`

If this is an acceptable solution, I'll create a PR. Any alternate suggestions or alternate naming are welcome too. I'm just wanting to preserve the old s3 key behavior (no random prefix) while using Companion standalone.

Alternatives

Discontinue using Companion standalone and setup a custom server and provide a custom getKey to Companion server.

Feels drastic / cumbersome to put in this much work to retain an old behavior that is heavily integrated into an existing workflow.

[williamccrobinson]

Added proposed changes to PR #4051

[Murderlon]

Hi, I understand the issue, but I think it's an XY problem nonetheless. You are asking for feedback on the attempted solution of this environment variable, but the problem is actually that you want to access your files from S3 from the client.

In my opinion, the previously unsafe behavior should never be used. Sooner or later (very likely soon) you will run into data collision and data loss because people upload files with the same name.

@mifi do you know of another way to easily access randomly generated names for S3?

[mifi]

I think companion will report back to Uppy the full URL of the uploaded file. Can't that be used to determine the uploaded file's name?

[williamccrobinson]

Understood. In our use-case we need to have our files stored in s3 with a specific prefix/directory, which is determined ahead of time.

We are managing the objects on our end (setting filename to some uuid + random suffix) so that data collision do not occur.

We cannot just use the random prefix of the url reported back from companion as-is because we require specific objects to be grouped by specific key prefix/directory.

[williamccrobinson]

If the team doesn't want to preserve the old behavior on opt-in basis, that's completely understandable.

We've identified two alternatives we could implement if that's the case :

• create our own custom companion server with our custom getKey
• modify our existing workflow to move s3 objects to the desired location once Companion reports that the upload is completed

We just wanted to explore if it was possible to save our team some work by preserving the v3 behavior in v4

[mifi]

I don't think adding this environment variable is worth the added complexity, so I think I'll close this then. If this is a common problem that more devs are facing, we could reopen this later or find a better solution. Thanks!

[mojowill]

Just chiming in on this, all I want to do is set a prefix for a filename so that it goes into a "directory" on S3.

I can do everything I want to using the standalone companion server but am now having to mess around setting up an express app just so I can modify getKey.

Even an option like COMPANION_S3_PREFIX which could be inserted into the default getKey implementation would help?

[mifi]

Ah, so you're fine with a random ID in the filename, but you want a fixed prefix before it?

[mojowill]

Ah, so you're fine with a random ID in the filename, but you want a fixed prefix before it?

Yes currently we have our own companion implementation in PHP but I want to use the "proper" companion package so we can add support for other providers and make things a but easier to maintain, however having everything go into the bucket root makes me sad!

${CUSTOM_PREFIX}/${RANDOM_STUFF}-${FILENAME}

works for us

[mifi]

Ok, I agree that it makes sense to allow a prefix for uploads.

[mojowill]

@mifi Do you have any idea how quickly this might be implemented? Or perhaps you have a contributors guide and I can get one of my devs to take a look?

[mifi]

Yes look at uppy.io

[command-tab]

Just chiming in to say that I'd also find this particular feature very useful. Thanks for working on it! (I am in the process of migrating a number of apps from the flow.js uploader to Uppy and Uppy Companion.)

[icmaia90]

Following the same principles on this thread:

• companion as standalone service
• an S3 prefix before the filename

I will ask for some insights into a new use case: what if it has multiple "folders" in a bucket? Using the COMPANION_S3_PREFIX will only allow one "folder" but not many different folders.

To better illustrate:

• S3 bucket: user_avatars
• each user has its unique id: 1, 2, 3, 4 ...
• it should store the avatar image by using bucket_name/<user_id>/image.png

How this use case can be accomplished?

[Murderlon]

That use case is supported, but only if you don't use standalone: https://uppy.io/docs/companion/#s3getkeyreq-filename-metadata. Not sure how to allow such dynamic use in a safe way through env vars.

[icmaia90]

What if switching the random UUID to the end of the file key?

• Filename sent to Companion: image.png
• Companion getKey renames it to image-<random_uuid>.png

[movy]

related question: is there way to receive final upload filename (i.e. with UUID prefixed) before upload completes? Upon upload start we get an array with fileIDs, but fileID does not include UUID yet. Only upon 'complete' hook we can get an url for uploaded file with prefixed filename in it.