Fix OS grain inconsistencies if lsb-release is installed or not

Most Linux distributions ship an os-release file by default. Some do not
ship lsb-release information, but they can be installed afterwards.
Installing/Removing lsb-release can lead to different OS grain values.

| OS               | grain      | without lsb-release              | with lsb-release |
|------------------|------------|----------------------------------|------------------|
| AlmaLinux 8      | oscodename | AlmaLinux 8.5 (Arctic Sphynx)    | ArcticSphynx     |
| Astra CE         | os         | Astra (Orel)                     | AstraLinuxCE     |
| Astra CE         | os_family  | Astra (Orel)                     | Debian           |
| Astra CE         | osfullname | Astra Linux (Orel)               | AstraLinuxCE     |
| Astra CE 2.12.40 | osfinger   | Astra Linux (Orel)-2             | AstraLinuxCE-2   |
| Debian           | osfullname | Debian GNU/Linux                 | Debian           |
| Mendel           | osfullname | Mendel GNU/Linux                 | Mendel           |
| Mendel 10        | osfinger   | Mendel GNU/Linux-10              | Mendel-10        |
| Mint             | osfullname | Linux Mint                       | Linuxmint        |
| Mint 20.3        | osfinger   | Linuxmint-20                     | Linux Mint-20    |
| Pop              | osfullname | Pop!_OS                          | Pop              |
| Pop 20.04        | osfinger   | Pop!_OS-20                       | Pop-20           |
| Rocky            | osfullname | Rocky Linux                      | Rocky            |
| Rocky 8          | osfinger   | Rocky Linux-8                    | Rocky-8          |
| Rocky 8          | oscodename | Rocky Linux 8.5 (Green Obsidian) | GreenObsidian    |

The current code that determines the OS grains on Linux is a mess: First
lsb-release is queried. If that fails, fall back to read os-release and
parse some `/etc/*-release` files. Then query `_linux_distribution` and
use a mixtures of those for the OS grains. `_linux_distribution` queries
the Python `distro` library. `distro` queries the os-release file,
lsb-release, and then `/etc/*-release`.

Rewrite the code that determines the OS grains on Linux. Solely rely on
the data provided by the os-release file. To not cause regressions, only
switch the distribution that has been tested. All other distributions
will use the legacy code (which was moved to
`_legacy_linux_distribution_data`).

The new code derives the `os_family` grain from the `ID_LIKE` field from
os-release (see https://github.com/saltstack/salt/issues/59061 for this
feature request). To enable this feature, the new code needs to be used
by default (and not just for selected distributions).

This commit introduces a few changes to the OS grains:

* AlmaLinux and Rocky Linux extract the codename from the `VERSION` field
  now instead of using the full `PRETTY_NAME`.
* Mendel uses now `Mendel GNU/Linux` as `osfullname` and correctly
  extracts the `osrelease` from `PRETTY_NAME`.
* Pop!_OS changes the `osfullname` from `Pop` to `Pop!_OS`.
* Astra Linux changes the `osfullname` from `AstraLinuxCE` to
  `Astra Linux (Orel)` and `AstraLinuxSE` to `Astra Linux (Smolensk)`
  respectively.

Fixes https://github.com/saltstack/salt/issues/61618
Signed-off-by: Benjamin Drung <benjamin.drung@ionos.com>

changelog/61618.fixed

@@ -0,0 +1,4 @@
+Some Linux distributions (like AlmaLinux, Astra Linux, Debian, Mendel, Linux
+Mint, Pop!_OS, Rocky Linux) report different `oscodename`, `osfullname`,
+`osfinger` grains if lsb-release is installed or not. They have been changed to
+only derive these OS grains from `/etc/os-release`.

salt/grains/core.py

@@ -1685,6 +1685,11 @@ def id_():
     return {"id": __opts__.get("id", "")}
 
 
+# Pattern for os-release PRETTY_NAME containing "name version (codename)"
+_PRETTY_NAME_RE = re.compile(r"[^\d]+ (?P<version>\d[\d.+\-a-z]*) \((?P<codename>.+)\)")
+# Pattern for os-release VERSION containing "version (codename)"
+_VERSION_RE = re.compile(r"\d[\d.+\-a-z]* \((?P<codename>.+)\)")
+
 _REPLACE_LINUX_RE = re.compile(r"\W(?:gnu/)?linux", re.IGNORECASE)
 
 # This maps (at most) the first ten characters (no spaces, lowercased) of
@@ -1731,14 +1736,31 @@ _OS_NAME_MAP = {
     "mendel": "Mendel",
 }
 
+# This dictionary maps the pair of os-release ID and NAME to the 'os' grain
+# that Salt traditionally uses, and is used by the os_data() function to
+# create the "os" grain.
+#
+# Add entries to this dictionary to retain historic values of the "os" grain.
+_ID_AND_NAME_TO_OS_NAME_MAP = {
+    ("astra", "Astra Linux (Orel)"): "AstraLinuxCE",
+    ("astra", "Astra Linux (Smolensk)"): "AstraLinuxSE",
+    ("pop", "Pop!_OS"): "Pop",
+}
 
-def _derive_os_grain(osfullname):
+
+def _derive_os_grain(osfullname, os_id=None):
     """
     Derive the 'os' grain from the 'osfullname' grain
 
+    For deriving the 'os' grain from the os-release data,
+    pass NAME as 'osfullname' and ID as 'os_id'.
+
     The 'os' grain that Salt traditionally uses is a shortened
     version of the 'osfullname' grain.
     """
+    if (os_id, osfullname) in _ID_AND_NAME_TO_OS_NAME_MAP:
+        return _ID_AND_NAME_TO_OS_NAME_MAP[(os_id, osfullname)]
+
     distroname = _REPLACE_LINUX_RE.sub("", osfullname).strip()
     # return the first ten characters with no spaces, lowercased
     shortname = distroname.replace(" ", "").lower()[:10]
@@ -1820,6 +1842,25 @@ _OS_FAMILY_MAP = {
     "OSMC": "Debian",
 }
 
+
+# Map the 'family_id' (from os-release) to the 'os_family' grain. If your
+# system is having trouble with detection, please make sure that the
+# 'family_id' is determined correctly first (in case multiple ID_LIKE entries
+# are specified).
+_OS_FAMILY_ID_MAP = {
+    # Red Hat Enterprise Linux (RHEL) is based on Fedora
+    # and Fedora is the successor of Red Hat Linux (RHL).
+    "fedora": "RedHat"
+}
+
+
+def _prettify_os_family(family_id):
+    if family_id in _OS_FAMILY_ID_MAP:
+        return _OS_FAMILY_ID_MAP[family_id]
+    # Fall back to use the os_id with an capital starting letter.
+    return family_id.capitalize()
+
+
 # Matches any possible format:
 #     DISTRIB_ID="Ubuntu"
 #     DISTRIB_ID='Mageia'
@@ -2020,6 +2061,91 @@ def _linux_lsb_distrib_data():
     return grains, has_error
 
 
+def _family_id(os_id, id_like):
+    """
+    Return the family ID which is the oldest distribution ancestor.
+    """
+    if not id_like:
+        # If ID_LIKE is not specified, the distribution has no derivative.
+        return os_id
+
+    ids_like = [os_id] + id_like.split()
+
+    # Linux Mint 20.3 does not declare to be a derivative of Debian.
+    if "debian" in ids_like or "ubuntu" in ids_like:
+        return "debian"
+
+    # The IDs are ordered from closest to farthest.
+    return ids_like[-1]
+
+
+def _os_release_quirks_for_oscodename(os_release):
+    """
+    Apply quirks for 'oscodename' grain for faulty os-release files
+
+    Some distributions do not (fully) follow the os-release
+    specification. This function bundles all required quirks
+    for the 'oscodename' grain. To be on the safe side, only
+    apply the quirks for allow-listed distributions. Better
+    not set the codename instead of setting it wrong.
+    """
+    if os_release["ID"] in ("astra",):
+        # Astra Linux has no version codename, but Salt used
+        # to report the variant ID as oscodename.
+        return os_release.get("VARIANT_ID")
+    if os_release["ID"] in ("almalinux", "rocky"):
+        # VERSION_CODENAME is not set, but the codename is
+        # mentioned in PRETTY_NAME and VERSION.
+        match = _VERSION_RE.match(os_release.get("VERSION", ""))
+        if match:
+            return match.group("codename")
+    return None
+
+
+def _os_release_quirks_for_osrelease(os_release):
+    """
+    Apply quirks for 'osrelease' grain for faulty os-release files
+
+    Some distributions do not (fully) follow the os-release
+    specification. This function bundles all required quirks
+    for the 'osrelease' grain. To be on the safe side, only
+    apply the quirks for allow-listed distributions. Better
+    not set the release instead of setting it wrong.
+    """
+    if os_release["ID"] in ("mendel",):
+        # Mendel sets VERSION_CODENAME but not VERSION_ID.
+        # Only PRETTY_NAME mentions the version number.
+        match = _PRETTY_NAME_RE.match(os_release["PRETTY_NAME"])
+        if match:
+            return match.group("version")
+    return None
+
+
+def _os_release_to_grains(os_release):
+    """
+    Transform the given os-release data to grains.
+
+    The os-release file is a freedesktop.org standard:
+    https://www.freedesktop.org/software/systemd/man/os-release.html
+
+    The keys NAME, ID, and PRETTY_NAME are expected to exist. All
+    other keys are optional.
+    """
+    family_id = _family_id(os_release["ID"], os_release.get("ID_LIKE"))
+    grains = {
+        "os": _derive_os_grain(os_release["NAME"], os_release["ID"]),
+        "os_family": _prettify_os_family(family_id),
+        "oscodename": os_release.get("VERSION_CODENAME")
+        or _os_release_quirks_for_oscodename(os_release),
+        "osfullname": os_release["NAME"].strip(),
+        "osrelease": os_release.get("VERSION_ID")
+        or _os_release_quirks_for_osrelease(os_release),
+    }
+
+    # oscodename and osrelease could be empty or None. Remove those.
+    return {key: value for key, value in grains.items() if key}
+
+
 def _linux_distribution_data():
     """
     Determine distribution information like OS name and version.
@@ -2033,9 +2159,53 @@ def _linux_distribution_data():
 
     This function might also return lsb_distrib_* grains
     from _linux_lsb_distrib_data().
+
+    Most Linux distributions should ship a os-release file
+    and this file should be the sole source for deriving the
+    OS grains. To not cause regressions, only switch the
+    distribution that has been tested.
     """
     grains, lsb_has_error = _linux_lsb_distrib_data()
 
+    log.trace("Getting OS name, release, and codename from freedesktop_os_release")
+    try:
+        os_release = _freedesktop_os_release()
+        grains.update(_os_release_to_grains(os_release))
+
+        # To prevent regressions, only let distributions solely
+        # use os-release after testing.
+        if os_release["ID"] in (
+            "almalinux",
+            "astra",
+            "debian",
+            "linuxmint",
+            "mendel",
+            "pop",
+            "rocky",
+            "ubuntu",
+        ):
+            # Solely use os-release data. See description of the function.
+            return grains
+
+    except OSError:
+        os_release = {}
+
+    # Warning: The remaining code is legacy code. Please solely rely
+    # on os-release data. See description of the function.
+    if "osrelease" in grains:
+        # Let the legacy code define osrelease to avoid discrepancies.
+        del grains["osrelease"]
+    return _legacy_linux_distribution_data(grains, os_release, lsb_has_error)
+
+
+def _legacy_linux_distribution_data(grains, os_release, lsb_has_error):
+    """
+    Legacy heuristics to determine distribution information.
+
+    Most Linux distributions should ship a os-release file
+    and this file should be the sole source for deriving the
+    OS grains. See _linux_distribution_data.
+    """
     if lsb_has_error:
         if grains.get("lsb_distrib_description", "").lower().startswith("antergos"):
             # Antergos incorrectly configures their /etc/lsb-release,
@@ -2044,10 +2214,6 @@ def _linux_distribution_data():
             grains["osfullname"] = "Antergos Linux"
         elif "lsb_distrib_id" not in grains:
             log.trace("Failed to get lsb_distrib_id, trying to parse os-release")
-            try:
-                os_release = _freedesktop_os_release()
-            except OSError:
-                os_release = {}
             if os_release:
                 if "NAME" in os_release:
                     grains["lsb_distrib_id"] = os_release["NAME"].strip()
@@ -2056,13 +2222,7 @@ def _linux_distribution_data():
                 if "VERSION_CODENAME" in os_release:
                     grains["lsb_distrib_codename"] = os_release["VERSION_CODENAME"]
                 elif "PRETTY_NAME" in os_release:
-                    codename = os_release["PRETTY_NAME"]
+                    grains["lsb_distrib_codename"] = os_release["PRETTY_NAME"]
-                    # https://github.com/saltstack/salt/issues/44108
-                    if os_release["ID"] == "debian":
-                        codename_match = re.search(r"\((\w+)\)$", codename)
-                        if codename_match:
-                            codename = codename_match.group(1)
-                    grains["lsb_distrib_codename"] = codename
                 if "CPE_NAME" in os_release:
                     cpe = _parse_cpe_name(os_release["CPE_NAME"])
                     if not cpe:

tests/pytests/unit/grains/test_core.py

@@ -1017,7 +1017,7 @@ def test_rocky_8_os_grains():
     expectation = {
         "os": "Rocky",
         "os_family": "RedHat",
-        "oscodename": "Rocky Linux 8.5 (Green Obsidian)",
+        "oscodename": "Green Obsidian",
         "osfullname": "Rocky Linux",
         "osrelease": "8.5",
         "osrelease_info": (8, 5),
@@ -1052,8 +1052,22 @@ def test_osmc_os_grains():
 @pytest.mark.skip_unless_on_linux
 def test_mendel_os_grains():
     """
-    Test if OS grains are parsed correctly in Mendel Linux
+    Test if OS grains are parsed correctly in Mendel Linux 5.3 Eagle (Nov 2021)
     """
+    # From https://coral.ai/software/
+    # downloaded enterprise-eagle-flashcard-20211117215217.zip
+    # -> flashcard_arm64.img -> rootfs.img -> /etc/os-release
+    _os_release_data = {
+        "PRETTY_NAME": "Mendel GNU/Linux 5 (Eagle)",
+        "NAME": "Mendel GNU/Linux",
+        "ID": "mendel",
+        "ID_LIKE": "debian",
+        "HOME_URL": "https://coral.ai/",
+        "SUPPORT_URL": "https://coral.ai/",
+        "BUG_REPORT_URL": "https://coral.ai/",
+        "VERSION_CODENAME": "eagle",
+    }
+    # Note: "lsb_release -a" falsely reports the version to be 10.0
     _os_release_map = {
         "_linux_distribution": ("Mendel", "10.0", "eagle"),
     }
@@ -1062,13 +1076,13 @@ def test_mendel_os_grains():
         "os": "Mendel",
         "os_family": "Debian",
         "oscodename": "eagle",
-        "osfullname": "Mendel",
+        "osfullname": "Mendel GNU/Linux",
-        "osrelease": "10.0",
+        "osrelease": "5",
-        "osrelease_info": (10, 0),
+        "osrelease_info": (5,),
-        "osmajorrelease": 10,
+        "osmajorrelease": 5,
-        "osfinger": "Mendel-10",
+        "osfinger": "Mendel GNU/Linux-5",
     }
-    _run_os_grains_tests(None, _os_release_map, expectation)
+    _run_os_grains_tests(_os_release_data, _os_release_map, expectation)
 
 
 @pytest.mark.skip_unless_on_linux
@@ -1100,7 +1114,7 @@ def test_almalinux_8_os_grains():
     expectation = {
         "os": "AlmaLinux",
         "os_family": "RedHat",
-        "oscodename": "AlmaLinux 8.5 (Arctic Sphynx)",
+        "oscodename": "Arctic Sphynx",
         "osfullname": "AlmaLinux",
         "osrelease": "8.5",
         "osrelease_info": (8, 5),
@@ -1273,20 +1287,37 @@ def test_pop_focal_os_grains():
     """
     Test if OS grains are parsed correctly in Pop!_OS 20.04 "Focal Fossa"
     """
+    # /etc/pop-os/os-release data taken from
+    # pop-default-settings 4.0.6~1642047816~20.04~932caee
+    _os_release_data = {
+        "NAME": "Pop!_OS",
+        "VERSION": "20.04 LTS",
+        "ID": "pop",
+        "ID_LIKE": "ubuntu debian",
+        "PRETTY_NAME": "Pop!_OS 20.04 LTS",
+        "VERSION_ID": "20.04",
+        "HOME_URL": "https://pop.system76.com",
+        "SUPPORT_URL": "https://support.system76.com",
+        "BUG_REPORT_URL": "https://github.com/pop-os/pop/issues",
+        "PRIVACY_POLICY_URL": "https://system76.com/privacy",
+        "VERSION_CODENAME": "focal",
+        "UBUNTU_CODENAME": "focal",
+        "LOGO": "distributor-logo-pop-os",
+    }
     _os_release_map = {
-        "_linux_distribution": ("Pop", "20.04", "focal"),
+        "_linux_distribution": ("pop", "20.04", "focal"),
     }
     expectation = {
         "os": "Pop",
         "os_family": "Debian",
         "oscodename": "focal",
-        "osfullname": "Pop",
+        "osfullname": "Pop!_OS",
         "osrelease": "20.04",
         "osrelease_info": (20, 4),
         "osmajorrelease": 20,
-        "osfinger": "Pop-20.04",
+        "osfinger": "Pop!_OS-20.04",
     }
-    _run_os_grains_tests(None, _os_release_map, expectation)
+    _run_os_grains_tests(_os_release_data, _os_release_map, expectation)
 
 
 @pytest.mark.skip_unless_on_linux
@@ -1294,20 +1325,37 @@ def test_pop_impish_os_grains():
     """
     Test if OS grains are parsed correctly in Pop!_OS 21.10 "Impish Indri"
     """
+    # /etc/pop-os/os-release data taken from
+    # pop-default-settings 5.1.0~1640204937~21.10~3f0be51
+    _os_release_data = {
+        "NAME": "Pop!_OS",
+        "VERSION": "21.10",
+        "ID": "pop",
+        "ID_LIKE": "ubuntu debian",
+        "PRETTY_NAME": "Pop!_OS 21.10",
+        "VERSION_ID": "21.10",
+        "HOME_URL": "https://pop.system76.com",
+        "SUPPORT_URL": "https://support.system76.com",
+        "BUG_REPORT_URL": "https://github.com/pop-os/pop/issues",
+        "PRIVACY_POLICY_URL": "https://system76.com/privacy",
+        "VERSION_CODENAME": "impish",
+        "UBUNTU_CODENAME": "impish",
+        "LOGO": "distributor-logo-pop-os",
+    }
     _os_release_map = {
-        "_linux_distribution": ("Pop", "21.10", "impish"),
+        "_linux_distribution": ("pop", "21.10", "impish"),
     }
     expectation = {
         "os": "Pop",
         "os_family": "Debian",
         "oscodename": "impish",
-        "osfullname": "Pop",
+        "osfullname": "Pop!_OS",
         "osrelease": "21.10",
         "osrelease_info": (21, 10),
         "osmajorrelease": 21,
-        "osfinger": "Pop-21.10",
+        "osfinger": "Pop!_OS-21.10",
     }
-    _run_os_grains_tests(None, _os_release_map, expectation)
+    _run_os_grains_tests(_os_release_data, _os_release_map, expectation)
 
 
 @pytest.mark.skip_unless_on_linux
@@ -1315,20 +1363,37 @@ def test_astralinuxce_os_grains():
     """
     Test that OS grains are parsed correctly for Astra Linux Orel
     """
+    # os-release data taken from astra-version 8.1.24+v2.12.43.6
+    # found in pool on installer ISO downloaded from
+    # https://mirrors.edge.kernel.org/astra/stable/orel/iso/orel-current.iso
+    _os_release_data = {
+        "PRETTY_NAME": "Astra Linux (Orel 2.12.43)",
+        "NAME": "Astra Linux (Orel)",
+        "ID": "astra",
+        "ID_LIKE": "debian",
+        "ANSI_COLOR": "1;31",
+        "HOME_URL": "http://astralinux.ru",
+        "SUPPORT_URL": "http://astralinux.ru/support",
+        "VARIANT_ID": "orel",
+        "VARIANT": "Orel",
+        "LOGO": "astra",
+        "VERSION_ID": "2.12.43",
+        "VERSION_CODENAME": "orel",
+    }
     _os_release_map = {
-        "_linux_distribution": ("AstraLinuxCE", "2.12.40", "orel"),
+        "_linux_distribution": ("astra", "2.12.43", "orel"),
     }
     expectation = {
         "os": "AstraLinuxCE",
         "os_family": "Debian",
         "oscodename": "orel",
-        "osfullname": "AstraLinuxCE",
+        "osfullname": "Astra Linux (Orel)",
-        "osrelease": "2.12.40",
+        "osrelease": "2.12.43",
-        "osrelease_info": (2, 12, 40),
+        "osrelease_info": (2, 12, 43),
         "osmajorrelease": 2,
-        "osfinger": "AstraLinuxCE-2",
+        "osfinger": "Astra Linux (Orel)-2",
     }
-    _run_os_grains_tests(None, _os_release_map, expectation)
+    _run_os_grains_tests(_os_release_data, _os_release_map, expectation)
 
 
 @pytest.mark.skip_unless_on_linux
@@ -1336,20 +1401,34 @@ def test_astralinuxse_os_grains():
     """
     Test that OS grains are parsed correctly for Astra Linux Smolensk
     """
+    # /etc/os-release data taken from base-files 7.2astra2
+    # from Docker image crbrka/astra16se:latest
+    _os_release_data = {
+        "PRETTY_NAME": "Astra Linux (Smolensk 1.6)",
+        "NAME": "Astra Linux (Smolensk)",
+        "ID": "astra",
+        "ID_LIKE": "debian",
+        "ANSI_COLOR": "1;31",
+        "HOME_URL": "http://astralinux.ru",
+        "SUPPORT_URL": "http://astralinux.ru/support",
+        "VARIANT_ID": "smolensk",
+        "VARIANT": "Smolensk",
+        "VERSION_ID": "1.6",
+    }
     _os_release_map = {
-        "_linux_distribution": ("AstraLinuxSE", "1.6", "smolensk"),
+        "_linux_distribution": ("astra", "1.6", "smolensk"),
     }
     expectation = {
         "os": "AstraLinuxSE",
         "os_family": "Debian",
         "oscodename": "smolensk",
-        "osfullname": "AstraLinuxSE",
+        "osfullname": "Astra Linux (Smolensk)",
         "osrelease": "1.6",
         "osrelease_info": (1, 6),
         "osmajorrelease": 1,
-        "osfinger": "AstraLinuxSE-1",
+        "osfinger": "Astra Linux (Smolensk)-1",
     }
-    _run_os_grains_tests(None, _os_release_map, expectation)
+    _run_os_grains_tests(_os_release_data, _os_release_map, expectation)
 
 
 @pytest.mark.skip_unless_on_windows