Setup instructions revisions

[danielskeenan]

This is not a real PR! Just a place to discuss revisions to the download/install instructions.

Inside the repo are some baseline documents converted to Markdown (thanks pandoc!) to facilitate discussion.

[peternewman]

Would you mind moving your proposed files into a subfolder, just so I don't do something stupid!

I've also converted it to a draft given it's WIP, and again to protect mistakes.

I wonder if we should move with the times and look at something like this perhaps:
https://wordpress.org/plugins/documents-from-git/

[danielskeenan]

Good idea; moved stuff around.

Re. "move with the times", here's a proposal: since better docs seems to be the fix for my RPi woes, and plenty of the content needs to be re-written anyway, perhaps this is the time to ditch WordPress for something like Jekyll or Hugo. It would certainly ease the maintenance burden and make keeping docs up to date easier as external contributions (e.g. this one) can be reviewed and merged as with any other code change. Hosting static sites with GitHub Pages is also super easy, even with custom domains and such. (It's small, but I do exactly that for my nerdy video game tools here, and am in the process of moving my personal site over from GitLab pages). I was able to get the existing pages into markdown with a little Pandoc massaging (although all of the internal links broke); that might get easier if you've got the original pages before WordPress themes them to input to Pandoc.

If you feel it's worth it, I'm happy to put my efforts towards something like that.

[danielskeenan]

I've revised all of the download/install instructions to fit more in line with modern OS/best practices.

I switched the Windows instructions to using VirtualBox (as VMWare is generally out of favor these days, except in enterprise virtualization), but WSL2 is promising if a clean way of handling microsoft/WSL#4150 with UDP (i.e. all interesting lighting protocols) is developed. I left those instructions in but hidden in case anyone wants to revisit.

[kripton]

Very well written, thank you!

[peternewman]

A few comments from a user I pointed at these docs

[peternewman]

This seems to cause issue on some Ubuntu flavours (I think they end up in a different path and not found by configure), instead use:

Suggested change

	pip3 install --user \
	numpy \
	protobuf
	sudo apt-get install \
	python-numpy \
	python-protobuf

Or the python3 versions of them...

[danielskeenan]

Huh, I guess my local Ubuntu might be too new. I'll check other distros for this issue - I was trying to keep Python with Python, so to speak, by using pip instead of the system package manager.

[peternewman]

Possibly, there's also the Python 2/3 thing, you can check some of the Travis runs and config history for what I had to do to make it work. This failure also matched what a user on IRC experienced.

[danielskeenan]

Alright, it looks like the Python ClientWrapperTest is failing on Ubuntu when using pip, but passing when using system packages. It works either way on Fedora. I'm baffled. Anyway, updating the Python lib to note this problem.

[peternewman]

Alright, it looks like the Python ClientWrapperTest is failing on Ubuntu when using pip, but passing when using system packages. It works either way on Fedora. I'm baffled. Anyway, updating the Python lib to note this problem.

Can you open an issue for this in OLA @danielskeenan as it would be good to get to the bottom of it. Some OS version and test log failure stuff would be useful too.

[danielskeenan]

Will do next time I flip back to Linux, where all of my test VMs are.

[danielskeenan]

Opened OpenLightingProject/ola#1714. It turned out to be more interesting than I thought!

[peternewman]

This is now a bit out of date, some plugins are working on Windows:
https://github.com/OpenLightingProject/ola/blob/master/README.mingw32

List of plugins on the frontpage:
https://www.openlighting.org/ola/

[danielskeenan]

That's neat, I'll have a look tomorrow!

It seems the README.minigw32 disagrees with the openlighting.org page on working plugins - any idea which should be considered authoritative?

[peternewman]

Most likely the readme (or whichever lists the most stuff).

Every so often I go and do a test compilation on Windows and fix stuff if I can, or keep existing stuff going and then update the readme to match the compilation line I got working if I managed to add new stuff.

[peternewman]

Good idea; moved stuff around.

Thanks!

Re. "move with the times", here's a proposal: since better docs seems to be the fix for my RPi woes, and plenty of the content needs to be re-written anyway, perhaps this is the time to ditch WordPress for something like Jekyll or Hugo. It would certainly ease the maintenance burden and make keeping docs up to date easier as external contributions (e.g. this one) can be reviewed and merged as with any other code change. Hosting static sites with GitHub Pages is also super easy, even with custom domains and such.

Yeah, I've done some GitHub Pages stuff for other projects in the past. We also dynamically generate all the stuff under docs from git, although running stuff on the server side:
http://docs.openlighting.org/

The only bit I'm wary of, hence suggesting that interim option of pages from GitHub within Wikipedia, is that there are currently 73 pages on the site, covering everything from OLA to our RDM stuff to when we've been involved in GSOC. Which is a lot to migrate and either making some pages dynamic from code, or having an OLA manual (e.g. including the install instructions) like the OLE one http://docs.openlighting.org/ole/manual/latest/ (or even just part of the existing Doxygen developer docs) might be an alternative stop-gap, it could either be Doxygen or Jekyll etc.

We've also got a couple of truly dynamic pages, which we can't really port to Markdown or similar (I guess one of them could be auto-generated every few minutes or something).

I must admit your personal site looks a lot shinier than I'm used to seeing Markdown stuff, although I don't know if it's quite as glossy as our current WP, I'm not really a judge of, nor hugely interested in, such things.

I was able to get the existing pages into markdown with a little Pandoc massaging (although all of the internal links broke); that might get easier if you've got the original pages before WordPress themes them to input to Pandoc.

We could certainly share that, or get you a WP login.

If you feel it's worth it, I'm happy to put my efforts towards something like that.

I think there are definitely benefits and it's certainly something we need to do, docs is one of the things we're particularly bad at, well I am at least!

[peternewman]

I'll try and look at the doc changes/suggestions themselves soon.

[danielskeenan]

I'm working on your notes presently. I'm trying to test the compilation instructions (particularly the Python part) in different fresh VMs to see what happens.

Part of the reason I chose to use pip for the Python deps is to avoid possibly ending up with two different versions of protobuf and numpy if the user also develops other Python software; as these are very common packages this has bitten me in the past. So far, ./configure --enable-python-libs using protobuf/numpy from python 3 pip succeeds on a fresh Ubuntu 20.10 VM. Checking 20.04 and 18.04 once their VMs are up, and might as well check Fedora while I'm at it.

I'm not familiar with how autotools checks for Python packages, but I'd be curious to know how it could fail - something as simple as checking the return code of python3 -c 'import google.protobuf' should suffice. My only thought there is if both Python 2 and 3 are installed, configure may search Python 2 packages when the user installed them for Python 3.

[peternewman]

I'm working on your notes presently. I'm trying to test the compilation instructions (particularly the Python part) in different fresh VMs to see what happens.

Cool, that's very dedicated!

Part of the reason I chose to use pip for the Python deps is to avoid possibly ending up with two different versions of protobuf and numpy if the user also develops other Python software; as these are very common packages this has bitten me in the past. So far, ./configure --enable-python-libs using protobuf/numpy from python 3 pip succeeds on a fresh Ubuntu 20.10 VM. Checking 20.04 and 18.04 once their VMs are up, and might as well check Fedora while I'm at it.

Possibly, again same notes as above about Python versions.

If these are supposed to be easy instructions, shouldn't we stick with the package manager versions? They'll also guaranteed upgrade and uninstall cleanly and in the case of numpy already be a binary so not need lots of compilation. It also avoids the confusion of user or system installs etc.

To look at it another way, surely someone who develops other Python software will be more aware of the other options and can choose what's most appropriate to them. Also I think the libprotobuf and Python versions want to match for compatibility, which will be far easier if they've both come from the same package manager.

I'm not familiar with how autotools checks for Python packages, but I'd be curious to know how it could fail - something as simple as checking the return code of python3 -c 'import google.protobuf' should suffice.

Yeah that's what the check does:
https://github.com/OpenLightingProject/ola/blob/master/config/ax_python_module.m4#L28

My only thought there is if both Python 2 and 3 are installed, configure may search Python 2 packages when the user installed them for Python 3.

Yes, that may well be it, our Python detection is here:
https://github.com/OpenLightingProject/ola/blob/master/configure.ac#L824

[danielskeenan]

Part of the reason I dug out the VMs (besides being off today 😉) is to find the differences between using stuff from pip and stuff from package manager - I keep hearing about the issues with one approach vs. another and can't seem to reproduce it. I do like your point about upgrades using the system package manager though, I hadn't considered that.

I'm going to try to write something that covers both cases, since it seems there's good/bad points for both approaches. My hope is that, because these are compilation instructions, they can be a bit more technical than the basic "type these commands to download from apt" instructions on other pages. You're probably correct that Python developers probably know how to get Python libs already, so it may just be "if you don't have these libs already, get them this way" or similar.

[peternewman]

Thanks for this @danielskeenan . A few more comments.

Perhaps something for part 2, there are some steps to make some devices work on some OSes after install (via any means):
https://www.openlighting.org/ola/getting-started/device-specific-configuration/

Perhaps that and the how to connect/restart olad type stuff should be done as a common page after installation/compilation?

[peternewman]

As mentioned in the comments, mingw32 is an option too.

[peternewman]

@RenZ0 can also give details of *BSD packages which I think exist.

[peternewman]

Part of the reason I dug out the VMs (besides being off today wink) is to find the differences between using stuff from pip and stuff from package manager - I keep hearing about the issues with one approach vs. another and can't seem to reproduce it. I do like your point about upgrades using the system package manager though, I hadn't considered that.

Yeah I'd broadly think that unless there's strong reasons to not use the system packages (e.g. Python developer) we should be encouraging that. IMHO it's also more professional (e.g. it's how I deploy stuff at work, even building a package if one doesn't exist already), I think unifying things on one system should simplify too, they only need to upgrade via that one system. Plus hopefully the system packaging team will ensure upgrades of both versions work together still.

I'm going to try to write something that covers both cases, since it seems there's good/bad points for both approaches. My hope is that, because these are compilation instructions, they can be a bit more technical than the basic "type these commands to download from apt" instructions on other pages. You're probably correct that Python developers probably know how to get Python libs already, so it may just be "if you don't have these libs already, get them this way" or similar.

Yes and no unfortunately. Some people will be more technical, but others might be either just needing to compile master to get some supported plugin which hasn't been released (building nightly bleeding edge debs might solve some of these issues). Others might just be trying to test something like a new VID/PID pair for some of our stock plugins like FTDI before they open a PR (under instruction). I'm trying to push more towards that so we don't end up with so many things like OpenLightingProject/ola#400

I think as you alluded to, having the pip "warning" is probably sensible, so more "if you don't have these libs already (e.g. via pip), get them this way".

[danielskeenan]

I'm poking at Windows stuff right now, but I've just pushed some revisions that should address the other issues.

[peternewman]

A few quick bits I spotted with the recent changes.

Also generally I'd say sudo foo is better than

su
foo

Where it's available.

[danielskeenan]

sudo foo is definitely better, but base Debian doesn't come pre-configured with sudo (something I always have to re-remember when using Debian), requiring su. I'd use Ubuntu for this tutorial (which does come with sudo, among other comforts), but since we've fallen out of their repos Debian seemed the best option.

[danielskeenan]

Just checking in to say getting OLA to build on Windows is turning into a bit of a bear. MinGW as linked from the README is quite out of date (last updated October 2013), and more modern releases (I've been testing MSYS2, which distributes the MinGW-w64 fork alongside a handy package manager) haven't compiled at all for assorted reasons. Some solutions may be simple (pid_t is a long long int and some overloads aren't defined), some are less simple (sigset_t is not supported with MinGW-w64; that response is a few years old but represents the exact issue I ran in to).

It may be worth readdressing Windows support in a separate issue to avoid derailing this one; it doesn't feel right to ask users to use software that hasn't been touched in 7 years.