aptly role getting "ERROR: mkdir $HOME: permission denied" in aptly calls in tasks/repositories.yml

[timblaktu]

When running the aptly role I am getting "ERROR: mkdir $HOME: permission denied" in the aptly repo list and aptly repo create calls made in tasks/repositories.yml.

I am able to ssh into the controlled aptly server as the aptly user and run the same commands, without su, successfully. I believe the problem is the su commands used to execute aptly commands.

I am using manala_aptly_user: aptly and I get the same error with and without running the role with become: true.

There is no instruction in the role docs for what ansible user to run as, and there are runtime warnings about the su call:

[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running su

How am I supposed to get past this user issue? Here is how my playbook invokes your role:

- name: Install/Configure Aptly                                                                                                                                                     
  tags: ["install", "aptly"]                                                                                                                                                        
  # become: true                                                                                                                                                                    
  hosts: aptly_server                                                                                                                                                               
  vars:                                                                                                                                                                               
    manala_aptly_user: aptly                                                                                                                                                          
    manala_aptly_config_file: "/home/{{ manala_aptly_user }}/.aptly.conf"                                                                                                             
    manala_aptly_config_template: ../files/aptly_config_template.j2                                                                                                                   
    manala_aptly_repositories:                                                                                                                                                          
      - name: b-stretch  # Name for Aptly Local Repo                                                                                                                                  
        comment: \"Packages for Debian Stretch\"  # Description for Aptly Local Repo, must include quotes if contains spaces (role bug)                                             
        component: main  # e.g. main contrib non-free                                                                                                                                     
        distribution: stretch  # becomes a subdirectory in $ARCHIVE_ROOT/dists. Synonym for suite or codename in different contexts.                                                      
  roles:                                                                                                                                                                              
    - role: manala.aptly

I'm using role version 2.0.2 and ansible version 2.9.10:

ansible 2.9.10
config file = /home/tblack/src/cm/ansible/projects/aptly/ansible.cfg
configured module search path = ['/home/tblack/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /home/tblack/.local/lib/python3.8/site-packages/ansible
executable location = /home/tblack/.local/bin/ansible
python version = 3.8.3 (default, Jun 30 2020, 11:18:52) [GCC 6.3.0 20170516]

Thanks for your time!

[nervo]

Hello @timblaktu :)
As this role handle aptly package installation and configuration, it needs root access, and the become flag, if needed in your situation, is welcomed.
To run commands as the aptly user, we use su, i guess that's where your issue come from.
Could you try to log in as root, and type su aptly -c "aptly repo list --raw" ?

[timblaktu]

Same error, when running your suggested command as root:

root@aptly:/home/ansible# su aptly -c "aptly repo list --raw"
ERROR: mkdir $HOME: permission denied

I don't know what is trying to mkdir $HOME here. But note that $HOME var for root user is /root:

root@aptly:/home/ansible# echo $HOME
/root

which already exists and is owned by root:root, with 700 mode.

I'm running the playbook which invokes your role as a normal user with sudoer privileges, using become: true, and the default become method is sudo.

[nervo]

i suppose the permission denied is about aptly user home, not root one. Does it have a home ? What is its permissions ?

[timblaktu]

That's not the problem, aptly home exists and is writeable by aptly and root (of course):

ansible@aptly:~$ ls -la /home
total 16
drwxr-xr-x  4 root    root    4096 Oct 30 20:10 .
drwxr-xr-x 18 root    root    4096 Oct 30 11:23 ..
drwxr-xr-x  7 ansible ansible 4096 Nov  3 09:06 ansible
drwxr-xr-x  3 aptly   users   4096 Oct 30 20:35 aptly

I found something interesting in aptly home, looks like something created a directory named '$HOME':

ansible@aptly:~$ ls -la /home/aptly
total 32
drwxr-xr-x 3 aptly users 4096 Oct 30 20:35  .
drwxr-xr-x 4 root  root  4096 Oct 30 20:10  ..
drwxr-xr-x 3 aptly users 4096 Oct 30 20:35 '$HOME'
-rw-r--r-- 1 root  root   499 Oct 30 20:32  .aptly.conf
-rw------- 1 aptly users  160 Oct 30 20:36  .bash_history
-rw-r--r-- 1 aptly users  220 Apr 17  2019  .bash_logout
-rw-r--r-- 1 aptly users 3526 Apr 17  2019  .bashrc
-rw-r--r-- 1 aptly users  807 Apr 17  2019  .profile

[nervo]

Ok, that '$HOME' is weird... it looks like an escaping issue...
Oh, and what about this .aptly.conf owned by root ?

[timblaktu]

As you can see in the playbook snippet I posted above, I specify:

    manala_aptly_user: aptly                                                                                                                                                          
    manala_aptly_config_file: "/home/{{ manala_aptly_user }}/.aptly.conf"                                                                                                             

and although I have become: true commented out, I originally ran this with it uncommented at first. I don't know if that explains why that .aptly.conf is owned by root.

Let me know if you have any ideas. I will now reset the aptly VM to the pre-provisioning snapshot I have, and then rerun the ansible playbook against it with become: true and see how far it now gets.

[nervo]

Ah, yes, i did'nt realize you have changed the default config file path, that's why .aptly.conf is owned by root, you should not do that :)
Fyi, here is an exemple of a fully functionnal aptly provisionning:

  aptly_user: aptly
  aptly_config:
    - rootDir: /mnt/data/debian
    - architectures:
      - amd64
  aptly_repositories:
    - name: stretch
      comment: Stretch
      component: main
      distribution: stretch
      origin: Foo
      label: Bar
    - name: buster
      comment: Buster
      component: main
      distribution: buster
      origin: Foo
      label: Bar

And that's all :)

[timblaktu]

Yeah, I saw the example in your README. Are you saying that your documented role variable manala_aptly_config_file is not supposed to be used? Please confirm.

If so, I can experiment with using the default instead, but I strongly recommend you either:

1. Remove this role variable from your documentation or change it to say "Not user customizeable from the default"
2. Much preferable, change your "config > Template" task in tasks/config.yml to make the config file owned by aptly user instead of hard-coded to root.

BTW, I reran my playbook on fresh Debian buster machine (with become: true) and reproduced the same problem. I will now rerun again using default aptly config location.

[timblaktu]

Using you role's default for aptly config location, I still get the same error, so something else is going on:

TASK [manala.aptly : repositories > Create] **************************************************************************************************failed: [aptly] (item={u'comment': u'\\"Packages for Debian Stretch\\"', u'distribution': u'stretch', u'component': u'main', u'name': u'b-stretch'}) => changed=true
  ansible_loop_var: item
  cmd:
  - su
  - aptly                                                                                                                                       - -c
  - aptly repo create -comment="Packages for Debian Stretch" -component=main -distribution=stretch b-stretch                          delta: '0:00:00.013134'
  end: '2020-11-03 13:28:22.245441'
  item:                                                                                                                                           comment: \"Packages for Debian Stretch\"
    component: main
    distribution: stretch                                                                                                                         name: b-stretch
  msg: non-zero return code
  rc: 1                                                                                                                                         start: '2020-11-03 13:28:22.232307'
  stderr: 'ERROR: mkdir $HOME: permission denied'
  stderr_lines: <omitted>
  stdout: ''                                                                                                                                    stdout_lines: <omitted>

There is no longer a $HOME file in aptly home, and here is the other pertinent info:

aptly@aptly:~$ ls -la
total 28
drwxr-xr-x 4 aptly users 4096 Nov  3 13:31 .
drwxr-xr-x 4 root  root  4096 Nov  3 13:25 ..
-rw-r--r-- 1 aptly users  220 Apr 17  2019 .bash_logout
-rw-r--r-- 1 aptly users 3526 Apr 17  2019 .bashrc
drwx------ 3 aptly users 4096 Nov  3 13:31 .gnupg
-rw-r--r-- 1 aptly users  807 Apr 17  2019 .profile
drwx------ 2 aptly users 4096 Nov  3 13:31 .ssh
aptly@aptly:~$ ls -la /etc/aptly.conf
-rw-r--r-- 1 root root 499 Nov  3 13:28 /etc/aptly.conf
aptly@aptly:~$ ls -la /home
total 16
drwxr-xr-x  4 root    root    4096 Nov  3 13:25 .
drwxr-xr-x 18 root    root    4096 Oct 30 11:23 ..
drwxr-xr-x  5 ansible ansible 4096 Nov  3 13:31 ansible
drwxr-xr-x  4 aptly   users   4096 Nov  3 13:31 aptly

(Note that the aptly default location for this file is ~/.aptly.conf, per https://www.aptly.info/doc/configuration.)

While I have your attention (thank you, BTW!) is there any support in your role for setting up https access and http forwarding using ssl cert files and an nginx reverse proxy?

[nervo]

Well, what can i say... :)
This variable acts as a default value, if you know that you have to change it for any reasons, feel free to do it. If you're unsure, treat it like a default value, and don't change it.

[nervo]

Btw, this role is atomic, and have the responsability to install and configure, and only install and configure aptly. The rest is your own responsibility :)

[nervo]

The fact is there are some tests, and i can't see exactly what can differs from your own playbook...
Could you run them ? make test.debian.buster for instance

[timblaktu]

Below are the test results. It looks like apt is encountering some fetch errors. (i didn't see related errors when running my playbook, btw.)

tblack-stretch]cm/ansible/projects/aptly > find . -name Makefile
./roles/manala.aptly/.manala/make/Makefile
./roles/manala.aptly/Makefile
[tblack-stretch]cm/ansible/projects/aptly > cd roles/manala.aptly
[tblack-stretch]projects/aptly/roles/manala.aptly > make test.debian.buster
[22:26:38] [test@local] Test "manala.aptly" on "debian.buster"
[22:26:38] [test@local]     tests/0100_install.yml
Unable to find image 'manala/test-ansible:debian.buster' locally
debian.buster: Pulling from manala/test-ansible
31dd5ebca5ef: Pull complete
593a7cf29bc2: Pull complete
c54ca6e117a6: Pull complete
fdf7f86f3a36: Pull complete
bb74ef7c30e8: Pull complete
1f48375c11f3: Pull complete
3a5b2b0e40f3: Pull complete
2d2f9d7cab45: Pull complete
536c38f53bcf: Pull complete
1104aa07dcb7: Pull complete
97ef5d46315e: Pull complete
Digest: sha256:90dd040746bdb20d1806779261a107217d7e5ff30cb4cae2d390b2894ed97d6b
Status: Downloaded newer image for manala/test-ansible:debian.buster
/usr/local/lib/python2.7/dist-packages/cryptography/__init__.py:39: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
  CryptographyDeprecationWarning,

PLAY [0100_install] **************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************ok: [debian.buster]

TASK [Pre tasks > Aptly apt key] *************************************************************************************************************changed: [debian.buster]

TASK [Pre tasks > Aptly apt repository] ******************************************************************************************************An exception occurred during task execution. To see the full traceback, use -vvv. The error was: apt.cache.FetchFailedException
fatal: [debian.buster]: FAILED! => {
    "changed": false,
    "rc": 1
}

MSG:

MODULE FAILURE


MODULE_STDERR:

Traceback (most recent call last):
  File "/tmp/ansible_ueXBbK/ansible_module_apt_repository.py", line 550, in <module>
    main()
  File "/tmp/ansible_ueXBbK/ansible_module_apt_repository.py", line 542, in main
    cache.update()
  File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 591, in update
    raise FetchFailedException()
apt.cache.FetchFailedException



PLAY RECAP ***********************************************************************************************************************************debian.buster              : ok=2    changed=1    unreachable=0    failed=1

make: *** [.manala/make/Makefile:176: tests/0100_install.yml] Error 2
[22:28:23] [test@local]     tests/0200_config.yml
/usr/local/lib/python2.7/dist-packages/cryptography/__init__.py:39: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
  CryptographyDeprecationWarning,

PLAY [0200_config] ***************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************ok: [debian.buster]

TASK [Pre tasks > Aptly apt key] *************************************************************************************************************changed: [debian.buster]
TASK [Pre tasks > Aptly apt repository] ******************************************************************************************************An exception occurred during task execution. To see the full traceback, use -vvv. The error was: apt.cache.FetchFailedException
fatal: [debian.buster]: FAILED! => {
    "changed": false,
    "rc": 1
}

MSG:

MODULE FAILURE


MODULE_STDERR:

Traceback (most recent call last):
  File "/tmp/ansible_D9c0Bn/ansible_module_apt_repository.py", line 550, in <module>
    main()
  File "/tmp/ansible_D9c0Bn/ansible_module_apt_repository.py", line 542, in main
    cache.update()
  File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 591, in update
    raise FetchFailedException()
apt.cache.FetchFailedException



PLAY RECAP ***********************************************************************************************************************************debian.buster              : ok=2    changed=1    unreachable=0    failed=1

make: *** [.manala/make/Makefile:176: tests/0200_config.yml] Error 2
[22:29:35] [test@local]     tests/0300_repositories.yml
/usr/local/lib/python2.7/dist-packages/cryptography/__init__.py:39: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
  CryptographyDeprecationWarning,

PLAY [0300_repositories] *********************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************ok: [debian.buster]

TASK [apt] ***********************************************************************************************************************************fatal: [debian.buster]: FAILED! => {
    "changed": false
}

MSG:

No package matching 'gnupg1' is available


PLAY RECAP ***********************************************************************************************************************************debian.buster              : ok=1    changed=0    unreachable=0    failed=1

make: *** [.manala/make/Makefile:176: tests/0300_repositories.yml] Error 2
.manala/make/Makefile:193: recipe for target 'test@local' failed
make: *** [test@local] Error 1

[nervo]

Wtf...
Ok, forget about the tests.
The issue does not come from the role itself, but from the command su aptly -c "aptly repo list --raw". I guess aptly tries to write in its home folder, but get $HOME environement variable from root.
Could you try su aptly -c "env" ?

[timblaktu]

env and aptly help commands work via su aptly -c, but no aptly repo commands work (see below).

Remember, I'm running the playbook which invokes your role as a normal user named ansible that has passwordless sudoer privilege, and now I'm using become: true at the play level, and using the default become method sudo. When I run the playbook I pass it --extra-vars "ansible_user=ansible ansible_ssh_pass=*** ansible_become_pass=***".

How is this different from how you run the playbook successfully? What become_method is your role assuming?

ansible@aptly:~$ su aptly -c "env"
Password:
SHELL=/bin/bash
PWD=/home/ansible
LOGNAME=aptly
XDG_SESSION_TYPE=tty
_=/usr/bin/env
HOME=/home/aptly
LANG=en_US.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
SSH_CONNECTION=172.16.15.107 37400 172.16.22.24 22
XDG_SESSION_CLASS=user
TERM=screen-256color
USER=aptly
SHLVL=1
XDG_SESSION_ID=45
XDG_RUNTIME_DIR=/run/user/1000
SSH_CLIENT=172.16.15.107 37400 22
PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
MAIL=/var/mail/aptly
SSH_TTY=/dev/pts/0
ansible@aptly:~$ su aptly -c "aptly help"
Password:
aptly is a tool to create partial and full mirrors of remote
repositories, manage local repositories, filter them, merge,
upgrade individual packages, take snapshots and publish them
back as Debian repositories.

aptly's goal is to establish repeatability and controlled changes
in a package-centric environment. aptly allows one to fix a set of packages
in a repository, so that package installation and upgrade becomes
deterministic. At the same time aptly allows one to perform controlled,
fine-grained changes in repository contents to transition your
package environment to new version.

Options:
  -architectures="": list of architectures to consider during (comma-separated), default to all available
  -config="": location of configuration file (default locations are /etc/aptly.conf, ~/.aptly.conf)
  -db-open-attempts=10: number of attempts to open DB if it's locked by other instance
  -dep-follow-all-variants: when processing dependencies, follow a & b if dependency is 'a|b'
  -dep-follow-recommends: when processing dependencies, follow Recommends
  -dep-follow-source: when processing dependencies, follow from binary to Source packages
  -dep-follow-suggests: when processing dependencies, follow Suggests
  -dep-verbose-resolve: when processing dependencies, print detailed logs
  -gpg-provider="": PGP implementation ("gpg" for external gpg or "internal" for Go internal implementation)
ansible@aptly:~$ su aptly -c "aptly repo list --raw"
Password:
ERROR: mkdir $HOME: permission denied

[nervo]

The role (is supposed to) don't care about become method or become user.
As we can see, although $HOME is ok, the aptly repo list --raw seems to do something badly... But what ?
I think you should open an issue on aptly repository :)

[timblaktu]

Thanks. I filed an issue with aptly repo.

When creating that issue, I noticed that my aptly installation, done by this role, shows "unknown version":

ansible@aptly:~$ aptly version
aptly version: unknown
ansible@aptly:~$ su aptly -c "aptly version"
Password:
aptly version: unknown
ansible@aptly:~$ which aptly
/usr/bin/aptly

I'm using all the defaults in your role. I see in your install task, that this is just installing aptly package. On my Debian Buster system, this version apt shows is: 1.3.0+ds1-2.2~deb10u1.

Looks like 1.4.0 is the latest tag on the aptly git project. The 1.4.0 changelog doesn't seem to address this issue though.

[nervo]

Well, you're installing aptly package from debian repositories, and not aptly ones.
Let's see how aptly maintainers address your issue :)

[timblaktu]

OK thanks. I also wanted to check with you if I am supposed to be creating the aptly user before invoking your role. Because, I am, using this play run before your role:

- name: Configure aptly user and groups
  tags: ["user"]
  become: true
  hosts: aptly_server
  tasks:
    - name: Create groups for aptly user
      group:
        name: "{{ item.name }}"
      with_items:
        - { name: aptly }
    - name: >
        Create aptly user if it doesn't exist, with default home dir and specified groups.
        This is a Linux system user that aptly processes will run as.
      user:
        name: "{{ aptly_linux_user_name }}"
        password: $6$T4ZTzdc.wu/sm$zoYfpWS41hrv0dXWFy/J9eN/cEAsnSMC3kPfmMgCLu17XEJXjt.TpMFYd9ucpZ8tXtbf3Ai9v/.x06FjOwMN0
        groups:
          - aptly
          - wheel  
        shell: /bin/bash

Regarding the dir named "$HOME" was created in /home/aptly:

ansible@aptly:~$ ls -la ~aptly
total 36
drwxr-xr-x 5 aptly users 4096 Nov  4 13:43  .
drwxr-xr-x 4 root  root  4096 Nov  3 13:25  ..
drwxr-xr-x 3 aptly users 4096 Nov  4 13:43 '$HOME'
-rw------- 1 aptly users  717 Nov  9 09:59  .bash_history
-rw-r--r-- 1 aptly users  220 Apr 17  2019  .bash_logout
-rw-r--r-- 1 aptly users 3526 Apr 17  2019  .bashrc
drwx------ 3 aptly users 4096 Nov  9 10:00  .gnupg
-rw-r--r-- 1 aptly users  807 Apr 17  2019  .profile
drwx------ 2 aptly users 4096 Nov  3 13:31  .ssh

I noticed that it appears to contain the expected contents in aptly user's home dir, i.e.

ansible@aptly:~$ tree -a ~aptly/\$HOME/
/home/aptly/$HOME/
└── .aptly
    ├── db
    │   ├── 000006.ldb
    │   ├── 000009.log
    │   ├── CURRENT
    │   ├── LOCK
    │   ├── LOG
    │   └── MANIFEST-000010
    └── public
        ├── dists
        │   └── stretch
        │       ├── main
        │       │   └── binary-amd64
        │       │       ├── Packages
        │       │       ├── Packages.bz2
        │       │       ├── Packages.gz
        │       │       └── Release
        │       └── Release
        └── pool

8 directories, 11 files