New disk management program

[wllacer]

Updated 21th of July
Code is ready to be tested and bugs and new features/checks are kindly accepted. There are slight changes at the UI from what is shown
the diskmanager script now fully interfaces with guided and all its features

Updated 8th of april
This is an ongoing effort to change the user interface of the disk management part of Archinstall.
We present it as a PR so you can check it, and propose as many changes/enhancements you feel like. For the time being it is a standalone program (a script called diskmanager.py). Expected to run at master with no additional changes.
Current version still has a lot of limitations (see below)

executing

sudo python -m archinstall --script diskmanager (--disk_layout json_file) (--long_form)
The arguments:

• --disk_layouts reads an existing disk layout file
• --dm_long_form generates screens with more data
• --dm_no_add_menu when adding a partition if set, the user will be asked a series of questions, otherwise a menu will be shown

and all the arguments relevant to archinstall, at least regarding disk management

If a layout is given it will go directly to the main screen

otherwise

If options 2 or 3 are selected, yo will be prompted for disk selection

And

if you select a disk

if you select a partition

if you select empty space, you will be asked for adding a partition, item by item, ending at the general partition screen

Development Phases

We have planned an iterative process, with five phases

• I User Interface with full navigation. Complete in the sense that all UI elements are in place and are navigable
• II Full UI implementation and dependencies in the workflow
• III Generation of json files which work. Only basic functionality (no -very little- "backoffice changes")
  Due to debugging needs, we have made this step earlier than expected. Now we generate files in the LOG_PATH directory under the name layout_diskmanager.json
• IV Full standalone implementation. (8th april) Without "backoffice" enhacements (a new only_hd )
• V General integration in the the tool.

Some new functionality may be left for a second iteration. The target is to be at least as functional as today's implementation
Up to the 3rd phase, there is no guarantee that any result (json file) from this program can be feed to Archinstall. Even at this level known incompatibilities and peculiarities may be present. See below
Up to the 4th phase, the development will be independent from the main application. Some bug solving and enhacements to the general code may be upstreamed, as long they're compatible with the actual implementation (f.i. subvolume handling has already been merged)

The order is merely a planning affair. Since the 5th, i'm working into real execution against disks, so it is the IV phase. And things go well ... Phase II, due to the need of collecting outside documentation has been put a bit on the backburner

As of 30 of March we feel the I phase is up to public display and critique.
As of 5 of April we steeped forward to Phase III. Now usable disk_layout json files are generated, and work (albeit idiosincratically). No support for delete partitions yet

Testing

As usual you can test it from my own repository
or better yet, using this PR as a local branch via the commands (those strings between <> are names you can/should update to your need)

git remote add <upstream> git://github.com/archlinux/archinstall.git
git fetch <upstream> pull/1053/head:<BRANCH_NAME>

Known incompatibilities

• Until there is a successful solution to the problems initially addressed in PRs Fix partition layout #794 and Rename Partition attribute "size" to "end" #854 (that size in disk_layouts really means end sector) our implementations are incompatible, as size has for us its semantic value, and we -still- don't use any end reference
  Updated We now generate a json which should be compatible to the current ones, only space is strictly in sector units
• We aren't still able to process a preexisting file, which doesn't imply a full disk wipe. We expect that it will be an algoritmically complex situation, and suspect it will be the last issue standing
• We are ready for/need real partition deletions, but it probably needs changes at Archinstall
• All our geometrical assumptions are for GPT drives. We need to gather more info for other cases
• [x ] All our physical variables are measured in sectors. The output needs to be more flexible (as the input is now).

• Updated* Everything is in place for this. Only the json is still not created with size other than sectors to simplify debugging

• We still struggle with entries with % at some places
  Updated We have managed to use percentages in the most important place. There are two kinds of percentages, one relative from the start of the partition till the end of the disk (which is the one will be written in the json file), and another strictly for the add partition task, which is form the start of the partition till the end of the free gap where it will be located. Translation between both is managed internally. Why? Only the first is transportable between different disks, but is unwieldly for actual space allocation.

[Torxed]

For lack of better words, this would be really amazing and a welcome change/upgrade!

[wllacer]

We have been working in the definition of space of the partitions. This is how it looks now

Note the different percentages. It's intentional

[wllacer]

Help needed !
I need a list of all possible restrictions you can think about partition management (size constrains for GPT/MBR disks, constrains due to UEFI/BIOS, what can and can not be encrypted, how many boot partitions in a disk, minimum partition size, ....
Let's see what can be implemented here.

[wllacer]

I've added the possibility of generating disk_layout files -before i had planned- Was needed in order to check compatibility.
It is generated at the usual directory with the name layout_diskmanager.json.
To be used (beware, wildy experimental, can eat your pets) you need a --harddrives argument in the usual scripts (only_hd is at your service)
Test have shown some minor anomalies, already changed. It looks promising

[wllacer]

I've decided to make it ready for review and integration, as i fell that; barring that some things i planned are still missing and some checks aren't implemented -even thought of- but have clear points of coding (search for _check_coherence methods) the code seems now in good shape and ready to be tested by others
Otherwise it will never be completed ...
The code as is now contains also following PRs (as they grew from here ;-))
#1346, #1355, #1373, #1376, #1383

[wllacer]

At least during testing purposes i've added an argument dm_add_menu which determines behaviour during add_partition. If set shows a PartitionMenu, if not shows a sequence of questions and then the menu.

[wllacer]

I changed the argument long_form to dm_long_form to have an unified "namespace" for the arguments needed for diskmanager
I changed the dm_add_menu to dm_no_add_menu as i switched the default behaviour to menu in add_partition. I've made a lot of changes in this function to ensure a more understandable interface, but i still have some problems with returns when i want to abort in the lineal version, and to get right the size of the last partition of the disk if i don't want to use a 100% size.

[wllacer]

I've added a perhaps controversial feature. When, at PartitionMenu requests location data for a partition (start position and size) if the user aborts the action (entering q) an user exception (LoopExit) will be triggered, which is consumed at PartitionList. It has been the only way with a minimum of clarity of code and simplicity i've found to implement a reasonable user workflow. I will explore alternate ways to reduce its exposure, but ... Even me is not specially fond of using exceptions for breaking deep nesting, but once in a while is this or extremely convoluted code or a big pile of code changes.
Yes, i thought about using something in the way of @svartkanin's solution to the Menu() case. Once you get the hang of it, is very logical (a function returns a status code and whatever by value data needs to pass back) but the whole PR is a "mess" enough without what would amount to changing most of the UI related code in this case ...

[Torxed]

I think breaking out using exceptions is fine in this case.
I agree that passing values and stuff is ideal, but not a deal breaker in this case :) We can come back to it later.
The functionality this would bring would add more value than making the code logic perfect hehe.