How to include all host keys from all hosts in group

Question

In trying to learn Ansible, I created a scenario that I'm having trouble implementing. Say I want to use the known_hosts module to add all SSH host keys to each member's /etc/ssh/ssh_known_hosts files in the group. I'm using 2.7 and trying to stay away from the with_ depreciated loop functions. I can get all the keys listed in debug, but I can't figure a way do it in a loop with the module. To understand how to do this generically I'm assuming that not all hosts will have all types of keys, so it needs to be dynamic. Here is what I have so far:

- name: SSH Host Keys Debug                                                     
  debug:                                                                        
    msg: "Hostname: {{ hostvars[item].ansible_hostname }},                      
          IP Address: {{ hostvars[item].ansible_eth0.ipv4.address }},           
          Keys: {{ hostvars[item] | select('match', '^ansible_ssh_host_key_.+_public') | map('extract', hostvars[item]) | list }}"
  loop: "{{ groups['haproxy'] }}" 

And then create a loop function that calls the module multiple times per host based on the number of keys it has.

- name: SSH Host Keys                                                           
  become: true                                                                  
  known_hosts:                                                                  
    dest: /etc/ssh/ssh_known_hosts                                              
    name: "{{ item.ansible_hostname }}"                                         
    key: "{{ item.ansible_hostname }},{{ item.ansible_eth0.ipv4.address }} {{ item.<public_key> }}"
  loop: <nested loop that provides hostname, ipv4_address and public_keys>

The facts include this interesting section:

...
        "ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDhGYjJS18MSCojIDLA9MTxITHpy+IOBFCHR+ZSZMyr5ek0r4RCK+zo6D1WZNs0dWcUB7IUJMThKcPpxdbrC0rk=",
        "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1lZDI1NTE5AAAAIBZjI7AJ5SHU31V6Vs9WTxLss/5gU/3pJJlTTzpJxcyr",                                                                                                                                                         
        "ansible_ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDH3OunSMQfaksqVxlLhKUyDUegaw6QuOiemgBWwwJypiDRshF0N2xQ6/RqHYA/gY+oDieIHzbs6OtxNt7JbSOwkjnKrYBkqzVQqtCtjpJ+pbzAcxdYwjfEYNlV/Fq41XrsFaWQgbQB57yS3dJVneheXskSc/mIwX3a2143X1CFLSz9krhwcNIWaAhZFMlV0ZqRSvH
DoiDZ4rQ4qQ4riaTm/NXjJzJjQqSiwUZUQdBtv88Ik1trQJUwKsYq2WZKiuv6yp/XVLL1/LLYQQJeH2GRqy8EI1TYRunrfHEo/D3T5QPsaJ1up/YNPtRP+H3dA68Ybwowb8m5A9IoAtHbHdEr",
...

The output from the debug is:

TASK [base_conf : SSH Host Keys Debug] ************************************************************************************************
task path: /home/rleblanc/code/ansible_learn/base_conf/tasks/main.yml:30
ok: [ansible-a] => (item=ansible-a) => {
    "msg": "Hostname: ansible-a, IP Address: 192.168.99.48, Keys: ['AAAAB3NzaC1yc2EAAAADAQABAAABAQDH3OunSMQfaksqVxlLhKUyDUegaw6QuOiemgBWwwJypiDRshF0N2xQ6/RqHYA/gY+oDieIHzbs6OtxNt7JbSOwkjnKrYBkqzVQqtCtjpJ+pbzAcxdYwjfEYNlV/Fq41XrsFaWQgbQB57yS3dJVneheXskSc/mIwX3a2143X1CFLSz9krhwcNIWaAhZFMlV0ZqRSvHDoiDZ4rQ4qQ4riaTm/NXjJzJjQqSiwUZUQdBtv88Ik1trQJUwKsYq2WZKiuv6yp/XVLL1/LLYQQJeH2GRqy8EI1TYRunrfHEo/D3T5QPsaJ1up/YNPtRP+H3dA68Ybwowb8m5A9IoAtHbHdEr', 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDhGYjJS18MSCojIDLA9MTxITHpy+IOBFCHR+ZSZMyr5ek0r4RCK+zo6D1WZNs0dWcUB7IUJMThKcPpxdbrC0rk=', 'AAAAC3NzaC1lZDI1NTE5AAAAIBZjI7AJ5SHU31V6Vs9WTxLss/5gU/3pJJlTTzpJxcyr']"
}

Edit 1

Some pseudo code that accomplishes what I'd like for each host:

for host in groups['haproxy']:
   for key in host.ansible_ssh_host_key_*_public:
     <pass key to known_host module>

Possible Solution for Roles

As I was trying to do this in a role, I needed to adjust the accepted answer a bit. I'm putting it here for posterity.

- name: Generate Host key list per host                                         
  set_fact:                                                                     
    myhostkeys: >-                                                              
      {{ (myhostkeys | default([]) + [{                                         
        'hostname': inventory_hostname,                                         
        'ip_addr': hostvars[inventory_hostname]['ansible_default_ipv4']['address'],
        'type': item | regex_replace('ansible_ssh_host_key_([^_]+)_public', '\1'),
        'key': hostvars[inventory_hostname][item]                               
      }]) }}                                                                    
  loop: >-                                                                      
    {{ hostvars[inventory_hostname] | select('match', '^ansible_ssh_host_key_.*_public') | list }}
                                                                                
- name: Combine all host keys                                                   
  set_fact:                                                                     
    hostkeys: >-                                                                
      {{ (hostkeys | default([])) + hostvars[item]['myhostkeys'] }}             
  loop: >-                                                                      
      {{ ansible_play_hosts_all }}                                              
                                                                                
- name: Add all host keys                                                       
  become: true                                                                  
  known_hosts:                                                                  
    path: /etc/ssh/ssh_known_hosts                                              
    name: "{{ item.hostname }}"                                                 
    key: "{{ item.hostname }},{{ item.ip_addr }} {{ keytype }} {{ item.key }}"  
  vars:                                                                         
    keytype: >-                                                                 
      {{ (item.type == 'ecdsa') |                                               
        ternary('ecdsa-sha2-nistp256', 'ssh-' ~ item.type) }}                   
  loop: >-                                                                      
    {{ hostvars[inventory_hostname]['hostkeys'] }}                              
  loop_control:                                                                 
    label: "Key: \"{{ item.hostname }},{{ item.ip_addr }} {{ keytype }} {{ item.key[:20] }}...\""

Answer 1

Ansible isn't really good at nested loops. We can work around that, but it's a bit ugly. Our lives are complicated by ecdsa public keys, because while for most keys the type is specified as ssh-<type> in the public key files, for ecdsa key it is apparently ecdsa-sha2-nistp256.

Here's what I came up with:

---
- hosts: all
  tasks:

    # In this first task, we construct for each host in the `all` group
    # a list of dictionaries that contain information about the ssh
    # hostkeys, extracted from the `ansible_ssh_host_key_*` variables.
    #
    # We are looping over all variables (and values) for this host for which
    # the variable name starts with `ansible_ssh_host_key`.
    - set_fact:
        hostkeys: >-
          {{ (hostkeys|default([])) + [{
            'hostname': inventory_hostname,
            'type': item.key|regex_replace('ansible_ssh_host_key_([^_]+)_public', '\1'),
            'key': item.value
          }] }}
      loop: >-
        {{ hostvars[inventory_hostname]|
           dict2items|
           selectattr('key', 'match', '^ansible_ssh_host_key_')|list }}

- hosts: localhost
  gather_facts: false
  tasks:

    # Now we take those per-host lists and construct one combined list
    # with all the information.
    - set_fact:
        hostkeys: "{{ (hostkeys|default([])) + hostvars[item].hostkeys }}"
      loop: "{{ groups.all }}"

- hosts: all
  gather_facts: false
  tasks:

    # Finally, on each host, we write a known_hosts file containing all the
    # host keys. I'm using an alternate path here because I didn't want
    # to actually write to /etc/ssh/known_hosts on my system.
    - known_hosts:
        path: "/tmp/hosts-{{ inventory_hostname }}"
        name: "{{ item.hostname }}"
        key: "{{ item.hostname }} {{ keytype }} {{ item.key }}"
      vars:
        keytype: >-
          {{ (item.type == 'ecdsa')|
             ternary('ecdsa-sha2-nistp256', 'ssh-' ~ item.type) }}
      loop: "{{ hostvars.localhost.hostkeys }}"
      loop_control:
        label: "{{ item.hostname }} {{ keytype }} {{ item.key[:20] }}..."

I haven't bothered with ip addresses here, but that shouldn't be too difficult to add (include that information in the hostkeys dictionary we generate in the first step, and then use it in the arguments to the known_hosts module).

Answer 2

I created a small Ansible look-up module host_ssh_keys to achieve exactly this. Adding all hosts' public ssh keys to /etc/ssh/ssh_known_hosts is then as simple as this, thanks to Ansible's integration of loops with look-up plugins:

- name: Add public keys of all inventory hosts to known_hosts
  ansible.builtin.known_hosts:
    path: /etc/ssh/ssh_known_hosts
    name: "{{ item.host }}"
    key: "{{ item.known_hosts }}"
  with_host_ssh_keys: "{{ ansible_play_hosts }}"

You can place it in one of the look-up plugin search location paths, or add it to a local directory and configure it in ansible.cfg as in the example below, which reads the plugin from local subdirectory plugins/lookup/.

[defaults]
lookup_plugins = plugins/lookup

---

Code:

#!/usr/bin/python
# Copyright 2022 Google LLC.
# SPDX-License-Identifier: Apache-2.0
# -*- coding: utf-8 -*-

from __future__ import (absolute_import, annotations, division, print_function)

__metaclass__ = type

DOCUMENTATION = r"""
  name: host_ssh_keys
  author: Petr (@ppetr)
  short_description: Looks up gathered public SSH keys of given hosts
  description: For each given hostname scans its gathered facts
    'ansible_ssh_host_key_...' and produces a dictionary of these keys as well
    as a full 'known_hosts' snippet.
  seealso:
  - name: Use ansible_facts in module code
    link: https://stackoverflow.com/q/45074728/1333025
  notes:
  - Handling of custom SSH ports hasn't been tested at all.
"""
RETURN = r"""
  _list:
    description:
      List composed of dictonaries containing the available public SSH keys and
      the corresponding known_hosts snippet.
    type: list
    elements: complex
    contains:
      host:
        description: The original input host name.
        type: str
      keys:
        description: Available public SSH keys for the host indexed by the key
          type.
        type: dict
      known_hosts:
        description: A string with all the keys suitable for appending to
          .ssh/known_hosts.
        type: str
"""
EXAMPLES = r"""
- name: Add public keys of all inventory hosts to known_hosts
  ansible.builtin.known_hosts:
    path: /etc/ssh/ssh_known_hosts
    name: "{{ item.host }}"
    key: "{{ item.known_hosts }}"
  with_host_ssh_keys: "{{ ansible_play_hosts }}"
"""

import collections
from typing import (Dict, Generator, Tuple)
import re

from ansible.errors import AnsibleError
from ansible.module_utils.six import string_types
from ansible.plugins.lookup import LookupBase
from ansible.utils.display import Display

_display = Display()


class LookupModule(LookupBase):

    _PUBLIC_KEY_RE = re.compile(r"""ansible_ssh_host_key_(.*)_public""")

    @staticmethod
    def _extract_public_keys(
            host: str,
            hostvars: dict) -> Generator[Tuple[str, str], None, None]:
        for var, value in hostvars.items():
            _display.vvvvv("Examining variable '%s' for host '%s'" %
                           (var, host))
            match = LookupModule._PUBLIC_KEY_RE.fullmatch(var)
            if match:
                key_type = hostvars[var + "_keytype"]
                _display.vvvv(
                    "Found ssh public key for host '%s' of type '%s': %s" %
                    (host, key_type, value))
                yield (key_type, value)

    def run(self, terms, variables=frozenset(), **kwargs):
        ret = []
        for host in terms:
            if not isinstance(host, string_types):
                raise AnsibleError(
                    'Invalid setting identifier, "%s" is not a string, its a %s'
                    % (host, type(host)))
            _display.debug("Looking up ssh public keys of host '%s'" % host)
            keys: Dict[str, str] = collections.OrderedDict(
                LookupModule._extract_public_keys(host,
                                                  variables['hostvars'][host]))
            known_hosts: str = ''
            for key_type, key in keys.items():
                known_hosts += '%s %s %s\n' % (host, key_type, key)
            ret.append(dict(host=host, keys=keys, known_hosts=known_hosts))

        return ret

_{Feel free to share/(re)use!}