Expose additional volume mounts

[jbarlow-mcafee]

This PR, a replacement for #16, contains the following changes:

1. Enables volume mounts by default in the Dockerfile for the ssl cert/key and MISP web application config directories.

2. Moves data initialization for each of the volumes - ssl cert/key, MISP web application config, and database config - behind the execution of a new container startup script. This allows for the possibility of mounting empty external volumes at container creation time and having data be initialized one-time into the external volumes. This also would eliminate the need to have to mount an ephemeral container in order to populate the mysql database into an external volume before creating a full MISP container.

I tested an image built with these changes to see that:

• A new MISP container could be created with no external volumes mounted.

• A new MISP container could be created with empty external volumes mapped. After bringing up a container, I confirmed that data was populated onto the external volumes.

• A new MISP container could be created with external volumes having pre-initialized data. After bringing up a container, I confirmed that data from the pre-existing configuration was being used - certs/keys, user accounts/passwords, and some app plugin settings.

In each case above, I confirmed that the web app was working fine and persisting config changes across container restarts.

As these changes are a bit extensive, I could understand hesitance in adding these to default image. From a usability perspective, though, I think this might be helpful - especially for users just wanting to get a container up and running quickly (and optionally persisting external volume data) without needing to do too many customizations to the Dockerfile.

Thanks for considering this PR and for all of your work maintaining this project!

[ventz]

@jbarlow-mcafee Give me some time to go through this. I left a couple of comments on lines that stood out.

So for:
1.) - exposing the ports - no issues
2.) - volumes - see my inline comments. It basically comes down to working around this:

https://docs.docker.com/engine/reference/builder/#notes-about-specifying-volumes
specifically: "Changing the volume from within the Dockerfile: If any build steps change the data within the volume after it has been declared, those changes will be discarded."

3.) - My only concern so far about putting the .misp_config_default/init-misp-config is that it's moving away from the default MISP "supported" install. While that's not a problem per se, it will make it harder to merge in up stream changes (which has to be done manually). Maybe we can provide an option for people to "quick init" the container if they just want to play around with it?

I have noticed that a lot of people run this container in production, and it would be good to separate the one time DB init vs "run time" imo, because they can accidentally blow away all the data if they re-pull down the container and the /init is present and runs automatically.

But to your point - maybe we can bridge an option that's more of a "/quicktry" which builds the DB, starts it, etc.

[jbarlow-mcafee]

Thanks, @ventz for the initial comments.

For your comment about build step changes to the volume not being persisted in the image, the PR is basically working around the issue by not relying upon those changes being persisted in the image - and instead populating those changes at container launch time instead. I could have instead moved the volume command down toward the bottom of the Dockerfile and left lines that populate the data into the image. This would be fine for allowing the use of either the pre-initialized data in the image or pre-initialized data in external volumes at container start time. I had been thinking it would be convenient, though, to also have a way to have data automatically be generated into previously uninitialized external volumes - mostly to avoid having to have extra steps like what is done currently with creating a temporary container to generate the mysql database.

The logic currently avoids blowing away the contents of external volumes at startup if an appropriate dot intialized file is already present in the target directory - .db_initialized for the database, .ssl_initialized for the SSL files, .misp_config_initialized for the MISP app configuration - when the init scripts are run. If any of the dot initialized files were removed from the external volumes, I suppose, though, that there would be a risk of overwriting previously stored data.

I'm definitely open to other options which might be more maintainable long-term.

Thanks again for your thorough and quick feedback!

[jbarlow-mcafee]

@ventz Since it sounded like you were at least okay with the idea of exposing additional ports as part of the Dockerfile, I put up a separate PR with just that change, #19. If that is accepted, I can trim that off of this PR so that this one can just be about the possible volume mount enhancements. Thanks again!

[ventz]

Have not forgotten about this one. I was going over it thinking about how to provide some of the features, while potentially separating into a "get-going-with-an-all-in-one" branch.

[jbarlow-mcafee]

Since #19 was merged, I removed the redundant commit from this PR for exposing ports. Thanks again, @ventz, for your continued help looking into this use case.

[ventz]

Great - thanks!

[SHSauler]

This would be very interesting for our use case as well. Is there a workaround or are you planning on implementing it?

Thanks a bunch!

[ventz]

@SHSauler Sorry, it has been some time. Would you mind summarizing which parts/dirs/volumes you are interested in exposing?

[SHSauler]

Hi @ventz, thank you for your reply. I saw that the other docker project exposes the volumes. Please disregard my request. Thank you!

[ventz]

@SHSauler I am trying to understand the use case. We have a few thousand users, and so far other than @jbarlow-mcafee and yourself, no one else has mentioned this.

It is not a problem exposing some volumes -- as long as they are done earlier in the build process, so that if data is written there, it doesn't impact anything. But it would be helpful to understand the use case behind this.