* Rewrite vault core, orchestrate AppRoles for minions
This commit represents a fundamental rewrite in how Salt interacts with
Vault. The master should still be compatible with minions running the
old code. There should be no breaking changes to public interfaces and
the old configuration format should still apply.
Core:
- Issue AppRoles to minions
- Manage entities with templatable metadata for minions
- Use inbuilt Salt cache
- Separate config cache from token cache
- Cache: introduce connection-scope vs global scope
Utility module:
- Support being imported (__utils__ deprecation)
- Raise exceptions on queries to simplify response handling
- Add classes to wrap complexity, especially regarding KV v2
- Lay some groundwork for renewing tokens
Execution module:
- Add patch_secret
- Add version support to delete_secret
- Allow returning listed keys only in list_secret
- Add policy_[fetch/write/delete] and policies_list
- Add query for arbitrary API queries
State module:
- Make use of execution module
- Change output format
Docs:
- Update for new configuration format
- Correct examples
- Add configuration examples
- Add required policies
* Fix linting for rewritten vault integration
* Add pytest unit tests for utils.vault, fix found issues
* Fix old vault runner tests
* Rewrite vault sdb tests, migrate to pytests
* Adapt vault ext_pillar tests
* Adapt vault execution module tests, migrate to pytests
* Add more vault execution module unit tests
* Support python <3.7 (vault util), time-independent tests
* Add/migrate vault runner unit tests (pytest)
* Add vault state module pytests
* Fix tests lint
* Refactor Vault container fixture, move to session scope
* Fix for existing vault execution/sdb module integration tests
* Improve existing vault runner integration tests
* Fix vault test support, add list policies
* Add more functional execution module tests, fix deprecated warning
* Refactor vault pytest support
* Add integration tests, improve/fix caching/issue_params
* Improve caching behavior, fix tests
* Always use session cache as well
* Also flush session cache when requested
* Make KV metadata caching behavior configurable
* Update tests to account for changes from prev commit
* Allow to autodiscover platform default CA bundle
* Remove runner approle param overrides
There is no simple way to ensure they are kept.
* Add clear_cache runner function
* Also manage token metadata for issued secret IDs
* Cleanup tests
* Cleanup code, pylint logging suggestions
* Do not always invalidate config when verify=default
* Ensure concatted metadata lists are sorted
* Add changelog (partly)
* Work with legacy peer_run configuration as well
* Consume a token use regardless of status code
* Correct verify semantics
* Refine token uses handling, add changelog/tests for old issues
* Add changelog for main features
* Add test for issue 58580
* Fix vault docs
* Provide all old make_request functionality, add tests
* Allow token use override, add docstrings to query funcs
* Simplify config_location merge
* Cleanup
* Fix make_request warning
* Attempt to fix memory issues during CI test run
* Increase documented version
* Improve lease handling
* Refine lease ttl handling/add token lifecycle management
* Fix docs build
* Adapt formatting
* assert what you get against what you expect
* drop empty parentheses after wrapper
* use `is` to compare against strictly boolean vars
* Fix issue param overrides
* during pillar rendering, they were always reset by the master (for
AppRoles)
* overrides were only respected for some settings (AppRoles)
* old config syntax was using the old syntax internally (tech debt)
* Introduce session-scoped cache
* Tokens with a single use left are unrenewable
* Allow override of flushing of cached leases during lookup
* Refactor cache classes, save lease data
* Rename session token cache key
* Add lease management utility
* Fix runner integration tests
after renaming the token cache key
* Do not overwrite data of cached leases after renewal
* Pass token_lifecycle to minions
* Do not fail syncing multiple approles/entities with pillar templates
* Ensure config cache expiration can be disabled
* Rename changelog files (.md)
* Declare vaultpolicylexer as parallel read safe
* Correct meta[data] payload key
For tokens it is `meta`, but for secret IDs, `metadata`.
* Reuse TCP connection
* Refactor utils module
* Ensure client is recreated after clearing cache
* Always use unwrap_client config as expected server
This should fix the test failure in the runner integration test
TestAppRoleIssuance::test_server_switch_does_not_break_minion_auth
* Ensure client is recreated after clearing cache 2
* Simulate patch for KV v1 or missing `patch` capability
* Add `patch` option to Vault SDB driver
* Reduce lease validity when revocation fails
* Extract AppRole/Identity API from runner into utils
* Revoke tokens, fire events, improve cache/exception handling
* Tokens (and therefore associated leases) are revoked when cleared by default
* It's possible to disable clearing cache when a perfectly valid token
is available, but a PermissionDeniedError is encountered.
* UnwrapExceptions always cause an event to be fired
* It's possible to enable sending of events when
a) cache is cleared
b) a lease is requested from cache, but it is/will be invalid
* A VaultAuthException does not immediately lead to clearing
the connection cache
* get_authd_client and others: multiple small enhancements and fixes
* Allow updating cached config w/o closing session
* Homogenize funcs, update docs, cleanup
* Minor internal fixes
`is_valid_for` is present on all lease-like objects, while `is_valid`
specifically should account for more, e.g. the number of uses.
The Vault API does not return 404 when a lookup fails.
* Add release note
* Address review remarks
* Fix release notes
* Remove loading minion_mods from factory
* Address other review remarks
* Add inline specification of trusted CA root cert
* Small QoL additions
* Fix lint
* Fix lint for Python >=3.8 support
* Add missing fixes
* Fix unit tests
In some cases, the `spec` calls were failing because the underlying
object was already patched
---------
Co-authored-by: Thomas Phipps <tphipps@vmware.com>
