Plugins with route pages and app configuration

[fooflare]

Hi Jess,

first of all, if you are busy stop reading here and leave it until you have some time (no hurry) :).

I have made some modifications in Plugin system to add some configuration and route template pages. This changes are in https://github.com/fooflare/nitrogen/tree/plugins_configuration but it is just experimental because I made some tests but I am sure that it is not enough. I created another repo with a plugin based on the nitro_login app https://github.com/fooflare/nitrogen_login. The idea is to install the plugin and make it work as if it was a normal app and make it as modular as possible (more ideas than another thing).

Now, the explanation of how it works right now: first of all, after adding the plugin to the rebar configuration and the dependencies of the plugin are installed, the plugin configuration changes are made, in the example two changes: one in vm.args (bcrypt is added to start and a config file that has nothing, it was added just to test) and another adding the config file (as I said, it is empty). If you run the plugin target in the makefile, it checks if the configuration file exists before adding it to the etc folder of the application and if the lines of vm.args already exists (I made a regular expression "trying" to avoid characters that can take to errors).

The next target was to find a way to add the route pages to the application. One thing that I thought was to change the extension of the erlang files (.erl to .route), maybe it is not the best way but... Why I did this? Well. the first reason, because it was a easy way to locate the files, but when this files are installed in site/src/plugin_name, they are renamed with the .erl extension automatically. The second was to avoid to be compiled code in the lib/plugin_folder that it is not necessary to be loaded. To know where install these files, I made another regular expression to check if it has a prefix (in the example, login_) and create the folder to put these pages inside. This solution has a problem and it is that it can take to errors if you want to create a file with '_' in the name and you want it in the root of the url (p.e. with create_account.erl), in my example there is no problem but... I have to think on it. As in the case of configuration files, it checks if they exist to not make modifications in files that already exist and could be modified before (p.e. when you make update-deps and then make plugins).

I am sure that I forget something, but this is more or less the most important. I don't know if you wanted to do something like this, but I did it because looked interesting for me and will be the way I will try to make modular my future developments. If you think that can be useful, I would like that you have a look at it and test it with the example of the login plugin.

Thanks for you patience and regards.

[choptastic]

Interesting ideas here.

Correct me if my summary is wrong, but basically, you want plugins to have certain .route files, which will be copied directly into the parent app's source tree in, e.g. site/src/plugin_name/plugin_name_module_name.erl, rather than being compiled as a standard dependency, and giving those modules a sort of "namespace" by prefixing them with the plugin's name.

Do I understand this correctly?

Also, I noticed the addition of the vm.args flags and etc files being copied -- that, in and of itself, is a fantastic addition, and solves #56

[fooflare]

You are almost right :) just one correction and I try to explain it with an example, maybe it is more clear. As you say, it generates a copy in the source tree, but just with an structure like this:

site/src/plugin_name/any_dir_name/any_dir_name_any_file_name.erl

e.g.: site/src/nitrogen_login/login/login_create_account.erl

Now, you can access to this page with the URL:

anydomain.com/login/create_account

Why did I choose this structure? Well, just for two reasons:

1.- As Nitrogen manages the routes, if the name of the directory is different to the prefix of the file, you have to access to the page from the root of the URL (anydomain.com/login_create_login in my example) and including the prefix of the file, but I didn't want that, because my intention is to create an structure of routes, and with the name_dir = file_prefix the route is restricted. If I am wrong in something of this, please correct me, I just made some tests and maybe I forgot something or didn't realize.

2.- I wanted to avoid using in the URL the name of the plugin, because my intention is not to create apps running under a bigger one, but just integrate modular functionality into one app.

Regarding to why I create this structure in the application source tree, it is because from the dependencies folder you can load modules with functionality as a library, but you can't manage routes (at least in my tests, I couldn't). And after that, I thought it is a good idea (even if you could manage routes from dependencies folder) to keep the sources from the plugin intact and just copy route files as templates in the application source, if you have more files than route files, e.g. in the example: db_login.erl, you can update these library files independently of route pages (in the source tree) if you customize them.

Another thing I thought is that I didn't want to compile the route files in the dependencies folder, just to keep it clean, and I needed one way to manage easily these route files during the installation process, and that is why I chose to change the .erl extension to .route in the plugin. I know that maybe it is not the most desirable way to manage it, but it is what came to mind in that moment.

Now, the TODO list I have in mind:

• Change the plugin folder structure for src: create the .route files inside folders and add the name of the folder to the files as prefix automatically. With this change: 1.- you can create route files in the root folder if you want with any name and 2.- the route restriction in the URL is made without prefix error.

e.g.:

Plugin directory structure:

├── src/
│      ├── db_logi.erl
│      ├── login/
│      │     ├── index.route
│      │     ├── login.route
│      │     ├── create_account.route

Application directory structure:

├── src/
│     ├── nitrogen_login/
│     │      ├── login/
│     │      │   ├── login_index.erl
│     │      │   ├── login_login.erl
│     │      │   ├── login_create_account.erl

This change avoids to generate not necessary folders in the structure, e.g.:

create_login.erl --> Generating a folder named create, when what we really want is to create this file in the root folder of the structure.

├── src/
│      ├── nitrogen_login/
│      │      ├── create_account.erl

• Windows compatibility.
• Update (something interactive to avoid overwriting files unintentionally) and remove (clean application structure) targets for the plugin system. For this task, I still have to think on it.
• Open to ideas.

Sorry for making the explanation so long, but I want to be as clear as possible and sometimes my English doesn't help too much :)

[choptastic]

Hey man, I've not forgotten about this. I just wanted to let you know, we
can talk about this for the gameplan for Nitrogen 2.4

On Wed, Nov 19, 2014 at 3:33 AM, fooflare notifications@github.com wrote:

You are almost right :) just one correction and I try to explain it with
an example, maybe it is more clear. As you say, it generates a copy in the
source tree, but just with an structure like this:

site/src/plugin_name/any_dir_name/any_dir_name_any_file_name.erl

e.g.: site/src/nitrogen_login/login/login_create_account.erl

Now, you can access to this page with the URL:

anydomain.com/login/create_account

Why did I choose this structure? Well, just for two reasons:

1.- As Nitrogen manages the routes, if the name of the directory is
different to the prefix of the file, you have to access to the page from
the root of the URL (anydomain.com/login_create_login in my example) and
including the prefix of the file, but I didn't want that, because my
intention is to create an structure of routes, and with the name_dir =
file_prefix the route is restricted. If I am wrong in something of this,
please correct me, I just made some tests and maybe I forgot something or
didn't realize.

2.- I wanted to avoid using in the URL the name of the plugin, because my
intention is not to create apps running under a bigger one, but just
integrate modular functionality into one app.

Regarding to why I create this structure in the application source tree,
it is because from the dependencies folder you can load modules with
functionality as a library, but you can't manage routes (at least in my
tests, I couldn't). And after that, I thought it is a good idea (even if
you could manage routes from dependencies folder) to keep the sources from
the plugin intact and just copy route files as templates in the application
source, if you have more files than route files, e.g. in the example:
db_login.erl, you can update these library files independently of route
pages (in the source tree) if you customize them.

Another thing I thought is that I didn't want to compile the route files
in the dependencies folder, just to keep it clean, and I needed one way to
manage easily these route files during the installation process, and that
is why I chose to change the .erl extension to .route in the plugin. I
know that maybe it is not the most desirable way to manage it, but it is
what came to mind in that moment.

Now, the TODO list I have in mind:

• Change the plugin folder structure for src: create the .route files
  inside folders and add the name of the folder to the files as prefix
  automatically. With this change: 1.- you can create route files in the root
  folder if you want with any name and 2.- the route restriction in the URL
  is made without prefix error.

e.g.:

Plugin directory structure:

├── src/
│ ├── db_logi.erl
│ ├── login/
│ │ ├── index.route
│ │ ├── login.route
│ │ ├── create_account.route

Application directory structure:

├── src/
│ ├── nitrogen_login/
│ │ ├── login/
│ │ │ ├── login_index.erl
│ │ │ ├── login_login.erl
│ │ │ ├── login_create_account.erl

This change avoids to generate not necessary folders in the structure,
e.g.:

create_login.erl --> Generating a folder named create, when what we
really want is to create this file in the root folder of the structure.

├── src/
│ ├── nitrogen_login/
│ │ ├── create_account.erl

• Update and remove targets for the plugin system. For this task, I
  still have to think on it.
• Open to ideas.

Sorry for making the explanation so long, but I want to be as clear as
possible and sometimes my English doesn't help too much :)

—
Reply to this email directly or view it on GitHub
#83 (comment).

Jesse Gumm
Owner, Sigma Star Systems
414.940.4866 || sigma-star.com || @jessegumm

[fooflare]

Hi Jesse, I've been a bit busy for a while and I couldn't work a lot on it, but I've not forgotten it either :) For me it would be great, I would like making something useful in which you can agree and of course, your experience can give me a very important point of view for the final solution

[fooflare]

Hi again Jesse, I have two questions for you about plugins structure.

First one, is there any special reason to use the 'priv' folder for static and templates? I ask this because I think that keeping the same structure for the plugin as in the 'site' folder would be maybe more clear.

Second one, why is being used 'plugins/myplugin' folders in 'static', instead of using directly 'myplugin'? I think it can simplify the url of the js, css and images (currently, '/plugins/myplugin/js/myplugin.js'). And second part of this question :) is, why not using inside 'templates', the 'myplugin' folder instead of 'plugins'? Plugins should be independent and right now you can overwrite an existing template if you use the same file name in different plugins.

I hope to have explained it clear, if not, tell me and I will try with an example.

Thank you.

[choptastic]

1. The 'priv' directory bit is totally optional and is there because priv is a common dir in erlang apps. I'm personally not a huge fan of the 'priv' dir way things are done, but I see why some use them that way. But using it is optional: https://github.com/nitrogen/nitrogen/blob/master/rel/overlay/common/do-plugins.escript#L141

2a) For the 'static/plugins/myplugin', I do it that way for two reasons: to simplify cleanup (deleting, updating plugins), and also removes the possibility of accidentally overwriting files that might be kept in the static directory that are unrelated to some plugin. Overall, it keeps them contained, and makes it clearer which subdirs in static are plugins and which aren't.

2b) For the plugins in site/templates, it should work the same as the 'static' where the template would go into site/templates/plugins/myplugin/something.html. The reasoning is the same there, just better organization/cleanup/whatnot.

Does that make sense?

[fooflare]

About the 'priv' directory, now I understand why you made it :) the answer was there, commented in the code, my fault. And I agree with you, it is the best decission because it is not strange for Erlang developers.

In relation with the 'plugins' directory, I understand your point of view too, I was obsessed with the idea of reducing the urls and not thinking in keep all plugins identified and secured in a separated directory. It was just logic, but sometimes one thinks in the most confusing solution instead of keeping it easy and clear.

Conclusion: I was thinking it in the wrong way and the best idea was to ask

Thanks for your answers and next time I will read/think it twice :)

[choptastic]

Don't feel bad for asking, they are perfectly legitimate questions, and ones I sympathize with. Indeed, my first version of the plugin script didn't use the priv dir, but after I released the first version, it was requested of me to support the priv dir. Given that I don't particularly care for it, I'd rather support both.

As for the other questions, I completely understand your viewpoint (shortening URLs a good thing), but it overall kept things cleaner and easier, even if it means we have to add "/plugins/" to our scripts to accommodate.

Anyway, seriously, don't ever feel bad for asking when you're confused.

[fooflare]

Hi Jesse, as you said you are refactoring code for the next version, I would like to ask if you are making something with the plugin system. I hadn't much time to finish the prototype I was working on, but in the last days I've been on it again. I prefer to ask to not overlap your work.

[choptastic]

Nope, I wasn't planning on doing anything with the plugin system for this
refactoring. This is more of an optimization thing, and wouldn't affect
anything as far as backwards compatibility for plugins are concerned.

-Jesse
On Jul 8, 2015 2:45 PM, "Amos Oviedo" notifications@github.com wrote:

Hi Jesse, as you said you are refactoring code for the next version, I
would like to ask if you are making something with the plugin system. I
hadn't much time to finish the prototype I was working on, but in the last
days I've been on it again. I prefer to ask to not overlap your work.

—
Reply to this email directly or view it on GitHub
#83 (comment).