Merge 230e5f2a98 into 2f699307c7

[CI] Refactoring CI tests for both remote and container tests

SUMMARY
Refactored CI tests:

Remove tests for Ansible Core 2.10 and 2.11 that already reached EOL.
Remote test target of ansible.posix will be the latest version of RHEL8,9 only.
The target OS of container tests has been modified, and a few OS have been removed
Add Ansible Core 2.16 and new devel branch to container and remote test target.
#506

For CI testing, other platforms can be added as needed.
ISSUE TYPE

CI tests Pull Request

COMPONENT NAME

ansible.posix

ADDITIONAL INFORMATION
None

.azure-pipelines/azure-pipelines.yml

@@ -51,16 +51,29 @@ stages:
         parameters:
           testFormat: devel/linux/{0}/1
           targets:
-            - name: CentOS 7
-              test: centos7
             - name: Fedora 38
               test: fedora38
-            - name: openSUSE 15 py3
-              test: opensuse15
             - name: Ubuntu 20.04
               test: ubuntu2004
             - name: Ubuntu 22.04
               test: ubuntu2204
+  - stage: Docker_2_16
+    displayName: Docker 2.16
+    dependsOn: []
+    jobs:
+      - template: templates/matrix.yml
+        parameters:
+          testFormat: 2.16/linux/{0}/1
+          targets:
+            - name: CentOS 7
+              test: centos7
+            - name: Fedora 38
+              test: fedora38
+            - name: Ubuntu 20.04
+              test: ubuntu2004
+            - name: Ubuntu 22.04
+              test: ubuntu2204
+
   - stage: Docker_2_15
     displayName: Docker 2.15
     dependsOn: []
@@ -141,44 +154,6 @@ stages:
               test: ubuntu1804
             - name: Ubuntu 20.04
               test: ubuntu2004
-  - stage: Docker_2_11
-    displayName: Docker 2.11
-    dependsOn: []
-    jobs:
-      - template: templates/matrix.yml
-        parameters:
-          testFormat: 2.11/linux/{0}/1
-          targets:
-            - name: CentOS 6
-              test: centos6
-            - name: CentOS 7
-              test: centos7
-            - name: openSUSE 15 py2
-              test: opensuse15py2
-            - name: openSUSE 15 py3
-              test: opensuse15
-            - name: Ubuntu 18.04
-              test: ubuntu1804
-  - stage: Docker_2_10
-    displayName: Docker 2.10
-    dependsOn: []
-    jobs:
-      - template: templates/matrix.yml
-        parameters:
-          testFormat: 2.10/linux/{0}/1
-          targets:
-            - name: CentOS 6
-              test: centos6
-            - name: CentOS 7
-              test: centos7
-            - name: openSUSE 15 py2
-              test: opensuse15py2
-            - name: openSUSE 15 py3
-              test: opensuse15
-            - name: Ubuntu 16.04
-              test: ubuntu1604
-            - name: Ubuntu 18.04
-              test: ubuntu1804
   - stage: Docker_2_9
     displayName: Docker 2.9
     dependsOn: []
@@ -209,16 +184,21 @@ stages:
         parameters:
           testFormat: devel/{0}/1
           targets:
-            - name: MacOS 13.2
+            - name: RHEL 9.3
-              test: macos/13.2
+              test: rhel/9.3
-            - name: RHEL 7.9
+  - stage: Remote_2_16
-              test: rhel/7.9
+    displayName: Remote 2.16
+    dependsOn: []
+    jobs:
+      - template: templates/matrix.yml
+        parameters:
+          testFormat: 2.16/{0}/1
+          targets:
             - name: RHEL 8.8
               test: rhel/8.8
             - name: RHEL 9.2
               test: rhel/9.2
-            - name: FreeBSD 13.2
+
-              test: freebsd/13.2
   - stage: Remote_2_15
     displayName: Remote 2.15
     dependsOn: []
@@ -227,18 +207,12 @@ stages:
         parameters:
           testFormat: 2.15/{0}/1
           targets:
-            - name: MacOS 13.2
-              test: macos/13.2
             - name: RHEL 7.9
               test: rhel/7.9
             - name: RHEL 8.7
               test: rhel/8.7
             - name: RHEL 9.1
               test: rhel/9.1
-            - name: FreeBSD 12.4
-              test: freebsd/12.4
-            - name: FreeBSD 13.1
-              test: freebsd/13.1
   - stage: Remote_2_14
     displayName: Remote 2.14
     dependsOn: []
@@ -247,18 +221,10 @@ stages:
         parameters:
           testFormat: 2.14/{0}/1
           targets:
-            - name: MacOS 12.0
-              test: macos/12.0
             - name: RHEL 7.9
               test: rhel/7.9
             - name: RHEL 8.6
               test: rhel/8.6
-            - name: RHEL 9.0
-              test: rhel/9.0
-            - name: FreeBSD 12.3
-              test: freebsd/12.3
-            - name: FreeBSD 13.1
-              test: freebsd/13.1
   - stage: Remote_2_13
     displayName: Remote 2.13
     dependsOn: []
@@ -267,16 +233,10 @@ stages:
         parameters:
           testFormat: 2.13/{0}/1
           targets:
-            - name: MacOS 12.0
-              test: macos/12.0
             - name: RHEL 7.9
               test: rhel/7.9
             - name: RHEL 8.5
               test: rhel/8.5
-            - name: FreeBSD 12.3
-              test: freebsd/12.3
-            - name: FreeBSD 13.0
-              test: freebsd/13.0
   - stage: Remote_2_12
     displayName: Remote 2.12
     dependsOn: []
@@ -285,40 +245,10 @@ stages:
         parameters:
           testFormat: 2.12/{0}/1
           targets:
-            - name: MacOS 11.1
-              test: macos/11.1
             - name: RHEL 7.9
               test: rhel/7.9
             - name: RHEL 8.4
               test: rhel/8.4
-            - name: FreeBSD 13.0
-              test: freebsd/13.0
-  - stage: Remote_2_11
-    displayName: Remote 2.11
-    dependsOn: []
-    jobs:
-      - template: templates/matrix.yml
-        parameters:
-          testFormat: 2.11/{0}/1
-          targets:
-            - name: MacOS 11.1
-              test: macos/11.1
-            - name: RHEL 7.9
-              test: rhel/7.9
-            - name: RHEL 8.3
-              test: rhel/8.3
-  - stage: Remote_2_10
-    displayName: Remote 2.10
-    dependsOn: []
-    jobs:
-      - template: templates/matrix.yml
-        parameters:
-          testFormat: 2.10/{0}/1
-          targets:
-            - name: RHEL 7.9
-              test: rhel/7.9
-            - name: RHEL 8.2
-              test: rhel/8.2
   - stage: Remote_2_9
     displayName: Remote 2.9
     dependsOn: []
@@ -339,10 +269,6 @@ stages:
     dependsOn:
       - Remote_2_9
       - Docker_2_9
-      - Remote_2_10
-      - Docker_2_10
-      - Remote_2_11
-      - Docker_2_11
       - Remote_2_12
       - Docker_2_12
       - Remote_2_13
@@ -351,6 +277,8 @@ stages:
       - Docker_2_14
       - Remote_2_15
       - Docker_2_15
+      - Remote_2_16
+      - Docker_2_16
       - Remote_devel
       - Docker_devel
     jobs:

changelogs/fragments/334_synchronize_using_become_pass.yml

@@ -0,0 +1,3 @@
+---
+minor_changes:
+- synchronize - elevating privileges now works even when `sudo` requires entering the `become_pass`

changelogs/fragments/508_ci_update.yml

@@ -0,0 +1,3 @@
+---
+trivial:
+  - "Refactoring remote CI targets."

docs/ansible.posix.synchronize_module.rst

@@ -580,7 +580,7 @@ Notes
 
    - The user and permissions for the synchronize `dest` are those of the `remote_user` on the destination host or the `become_user` if `become=yes` is active.
    - In Ansible 2.0 a bug in the synchronize module made become occur on the "local host".  This was fixed in Ansible 2.0.1.
-   - Currently, synchronize is limited to elevating permissions via passwordless sudo.  This is because rsync itself is connecting to the remote machine and rsync doesn't give us a way to pass sudo credentials in.
+   - Currently, synchronize is limited to elevating permissions via sudo.  This now even works when password entry is required.
    - Currently there are only a few connection types which support synchronize (ssh, paramiko, local, and docker) because a sync strategy has been determined for those connection types.  Note that the connection for these must not need a password as rsync itself is making the connection and rsync does not provide us a way to pass a password to the connection.
    - Expect that dest=~/x will be ~<remote_user>/x even if using sudo.
    - Inspect the verbose output to validate the destination user/host/path are what was expected.

plugins/action/synchronize.py

@@ -390,10 +390,24 @@ class ActionModule(ActionBase):
                 # If no rsync_path is set, become was originally set, and dest is
                 # remote then add privilege escalation here.
                 if self._play_context.become_method == 'sudo':
-                    if self._play_context.become_user:
+                    
-                        rsync_path = 'sudo -u %s rsync' % self._play_context.become_user
+                    # if become is set, we can either rely on passwordless sudo or pass the password
+                    if self._play_context.become_pass is None:
+                        rsync_path = 'sudo '
                     else:
-                        rsync_path = 'sudo rsync'
+                        # pass the become password using the environment so that the synchronize module
+                        # can wrap ssh on the host with a shell script that injects the password into
+                        # stdin, allowing for `sudo -S` on the target machine to retrieve the password
+                        if hasattr(self._task, 'environment'):
+                            self._task.environment = []
+                        self._task.environment.append({'BECOME_PASS': self._play_context.become_pass})
+                        _tmp_args['_ssh_wrapper'] = True
+                        rsync_path = 'sudo -S '
+                    
+                    if self._play_context.become_user:
+                        rsync_path += '-u %s rsync' % self._play_context.become_user
+                    else:
+                        rsync_path += 'rsync'
                 # TODO: have to add in the rest of the become methods here
 
             # We cannot use privilege escalation on the machine running the

plugins/modules/synchronize.py

@@ -198,7 +198,7 @@ notes:
      delegate_to host when delegate_to is used).
    - The user and permissions for the synchronize `dest` are those of the `remote_user` on the destination host or the `become_user` if `become=yes` is active.
    - In Ansible 2.0 a bug in the synchronize module made become occur on the "local host".  This was fixed in Ansible 2.0.1.
-   - Currently, synchronize is limited to elevating permissions via passwordless sudo.  This is because rsync itself is connecting to the remote machine
+   - Currently, synchronize is limited to elevating permissions via sudo.  This now even works when password entry is required.
      and rsync doesn't give us a way to pass sudo credentials in.
    - Currently there are only a few connection types which support synchronize (ssh, paramiko, local, and docker) because a sync strategy has been
      determined for those connection types.  Note that the connection for these must not need a password as rsync itself is making the connection and
@@ -414,6 +414,7 @@ def main():
             rsync_opts=dict(type='list', default=[], elements='str'),
             ssh_args=dict(type='str'),
             ssh_connection_multiplexing=dict(type='bool', default=False),
+            _ssh_wrapper=dict(type='bool', default=False),
             partial=dict(type='bool', default=False),
             verify_host=dict(type='bool', default=False),
             delay_updates=dict(type='bool', default=True),
@@ -456,6 +457,7 @@ def main():
     rsync_opts = module.params['rsync_opts']
     ssh_args = module.params['ssh_args']
     ssh_connection_multiplexing = module.params['ssh_connection_multiplexing']
+    ssh_wrapper = module.params['_ssh_wrapper']
     verify_host = module.params['verify_host']
     link_dest = module.params['link_dest']
     delay_updates = module.params['delay_updates']
@@ -550,6 +552,13 @@ def main():
             ssh_cmd_str = ' '.join(shlex_quote(arg) for arg in ssh_cmd)
             if ssh_args:
                 ssh_cmd_str += ' %s' % ssh_args
+            # When `become: yes` is set but the account on the target requires a password for sudo, we have to supply
+            # it from the host side by wrapping the remote shell and inserting the password into stdin.
+            # In the ActionPlugin, the password is assigned to the BECOME_PASS environment variable, so we will not have
+            # to make it visible if anyone logs the command issued by ansible.
+            # Adapted from https://askubuntu.com/a/1263657
+            if ssh_wrapper:
+                ssh_cmd_str = '/bin/sh -c "{ echo $BECOME_PASS; cat - ; } | ' + ssh_cmd_str + ' $0 $* &"'
             cmd.append('--rsh=%s' % shlex_quote(ssh_cmd_str))
 
     if rsync_path: