Having our middleware client hit a socket timeout usually means that
the test VM is seriously broken. We should swiftly abort test run
in this case.

(cherry picked from commit d79f31ada028f3592fe6037c923343c3b05eea5b)

(cherry picked from commit 46fff0048ae6037e326cba1fab9f4edb103065c1)
Co-authored-by: Caleb <yocalebo@gmail.com>

* simplify to 1 nvme function

(cherry picked from commit 43ef8403c723cd6587902855e8398f8885ffcb5e)

* simplify and clean-up jbof/crud.py

(cherry picked from commit 27755a4c136fe1c96e11e59c404614cb42f97703)

---------
Co-authored-by: Caleb <yocalebo@gmail.com>

If networking is not fully configured, kinit attempt may fail with
EAGAIN. Retry several times before giving up. Since the actual kinit
has a timeout set for 30 seconds, this loop may end up being quite
long-running and so convert directoryservices.initialize into a
middleware job. The only direct caller for this private endpoint
is smb.configure which is itself a job and so we don't have to worry
about any potential timeouts here.

(cherry picked from commit 06002cbf9271bfafd7506cb700d4bba0c51e8f8a)
Co-authored-by: Andrew Walker <awalker@ixsystems.com>

Allow API keys to toggle services. We only need to check whether
this is a user session since API keys don't support roles. This
means that once an API key is within the permissions check function
we already know that it has been explicitly granted access to this
endpoint via its allowlist.

(cherry picked from commit a54645566d755e79ce4a2e10bb5059b71c2fb986)
Co-authored-by: Andrew Walker <awalker@ixsystems.com>

This particular check never quite worked correctly in that our
filter for whether the dataset was in boot pool was a de-facto
no op (since we were not populating the pool name before filtering).

This issue was not detected due to a compensating bug in filter_list
where missing keys were not being handled correctly either.

(cherry picked from commit 9126570b1662acaa40c47f43e2e730ba4c3b5573)
Co-authored-by: Andrew Walker <awalker@ixsystems.com>

This validation is perhaps overly restrictive as it prevents us
from automatically registering IPv6 and public IPv4 addresses in
Active Directory.

(cherry picked from commit 46e3e94b88907d33ca92d9bd66fcc64abd04a9cf)
Co-authored-by: Andrew Walker <awalker@ixsystems.com>

NAS-127457 / 24.04-RC.1 / Improve speed of jbof.create and jbof.delete APIs (by bmeagherix) (#13204)

* Improve speed of jbof.create and jbof.delete APIs

Improve the speed of these APIs, generally by performing certain
operations in parallel rather than in series.

Also add some additional logging.

(cherry picked from commit 691c0ad2db12319c19dd09e9a68acbb169996b7e)

* Address review: remove superfluous debug

(cherry picked from commit b9b105e38f4075f45adbc265ab33ac77e4c364dc)

---------
Co-authored-by: Brian Meagher <brian.meagher@ixsystems.com>

(cherry picked from commit 11c7c99784555b53cf3b8e5fcce6898fc1c56823)
Co-authored-by: Andrew Walker <awalker@ixsystems.com>

(cherry picked from commit 84ba7109e76632d1965fef505e77daca0abd6433)
Co-authored-by: Andrew Walker <awalker@ixsystems.com>

(cherry picked from commit 7052d3e02c818f9ae78ed6e4597d57c5e8071709)
Co-authored-by: M. Rehan <mrehanlm93@gmail.com>

(cherry picked from commit 73707fda666904e67798ef47e78f1314a90ed985)
Co-authored-by: Caleb <yocalebo@gmail.com>

NAS-127151 / 24.04-RC.1 / Make sure there is no sensitive information in apps/tc/vm events plugins (by sonicaj) (#13189)

* Make sure there is no sensitive information in apps/vm events plugin

This commit adds changes to make sure we don't expose any private info in apps/vm plugin events.

(cherry picked from commit ee9711b7a4ce2750d44d0424a3322285fc200e96)

* Add a private endpoint to retrieve event config for tc

(cherry picked from commit d187c3d05c615a743d23ca3686015b3ad7e8480a)

* Redact apit key from tc plugin events

(cherry picked from commit b86e138518708a76bfdc6b2b6ad9aad9f093ac28)

---------
Co-authored-by: Waqar Ahmed <waqarahmedjoyia@live.com>

(cherry picked from commit 4e885ef92fc1633102bd3c413867eb9c2ecafcc2)
Co-authored-by: themylogin <themylogin@gmail.com>

* Add truecommand read/write roles

(cherry picked from commit 98c485e7460cdefc2421a88a90a012dcfd6a6095)

* Specify roles for truecommand plugin

(cherry picked from commit 1d955359617a7ad700e54f5bba3d243727a6bc83)

* Add integration test for truecommand roles

(cherry picked from commit 5e86165dd1519897c1e4e5764808a97f5b3d4e3a)

* Fix no auth req endpoint roles

(cherry picked from commit fed8014f2ca0604c48faf76509ef10885a97382e)

---------
Co-authored-by: Waqar Ahmed <waqarahmedjoyia@live.com>

* stop the vrrp service in vrrp_backup

(cherry picked from commit 354ddb213182cbb403662ff91106291cb277802d)

* add global lock for vrrp_master/backup

(cherry picked from commit 9e5b3cbfeca13882278f139d5a1226026775456b)

---------
Co-authored-by: Caleb <yocalebo@gmail.com>

(cherry picked from commit 6323c828fa99beba82ab146e0907fae7e78c59d2)
Co-authored-by: Caleb <yocalebo@gmail.com>

(cherry picked from commit 714b719c609269d3db8be60cb5518ac695558174)
Co-authored-by: Brian Meagher <brian.meagher@ixsystems.com>

(cherry picked from commit 776302b0dfa00c001cf49e631512f95e03e141ca)
Co-authored-by: Brian Meagher <brian.meagher@ixsystems.com>

* add failover.vrrp.get_priority

(cherry picked from commit 98949bb5b0533f9d10feb006a2a69fd3d1616499)

* change keepalived.conf

(cherry picked from commit e43470707ba5c1e37453e227fb5859f8c337736a)

* reconfigure keepalived on master event

(cherry picked from commit 0ee7b239d814d38c066a58319337b4c48c58a9ee)

* fix flake8 while im here

(cherry picked from commit 45a874eb6035bf99b2e112d7a1528bc2ff79bdb9)

* short-circuit if failover.status is MASTER

(cherry picked from commit f180cbb3132f20b18912f910fb420dca324d3aec)

* missing comma

(cherry picked from commit 18b22e4ae605e4e88c27edae713abf2fe5d19535)

---------
Co-authored-by: Caleb <yocalebo@gmail.com>

(cherry picked from commit 493aa113cc6980fd6929c4833bf583eff27966a2)
Co-authored-by: M. Rehan <mrehanlm93@gmail.com>

(cherry picked from commit d075cff4b8af53af8f14433c1a45a0da4941b853)
Co-authored-by: Caleb <yocalebo@gmail.com>

This commit adds changes marking app's config as private as it can contain potentially passwords/secrets of user's app configuration.

(cherry picked from commit ceb1a940793e986a97fa3e1028ee3879a8fb31f1)
Co-authored-by: Waqar Ahmed <waqarahmedjoyia@live.com>

(cherry picked from commit 580c36ccb6687ced2780d8226ffa7fbac92af789)
Co-authored-by: M. Rehan <mrehanlm93@gmail.com>

We should raise a ValidationError on duplicate DNS names
because there shouldn't be multiple idmap backends configured
for a single domain.

(cherry picked from commit e2bfa8004d119d86c3236fbddeeded680f6f8859)
Co-authored-by: Andrew Walker <awalker@ixsystems.com>

(cherry picked from commit f4e94c765f0867d37dd93030d474a13a70f70332)
Co-authored-by: themylogin <themylogin@gmail.com>

entry may be None type if user has queried a non-existent user
directly via filters (for example ['name', '=', 'bob']).

(cherry picked from commit f22ce52e890e1754e5081081595d85870daeeeec)
Co-authored-by: Andrew Walker <awalker@ixsystems.com>

(cherry picked from commit d0e2088d

)
Co-authored-by: Andrew Walker <awalker@ixsystems.com>

NAS-126731 / 24.04-RC.1 / Improve failover handling of a large number of targets (by bmeagherix) (#13127)

- On boot do not start with cluster_mode enabled on target extents.

- When the STANDBY node boots, it will initiate a job that resettles the DLM
  (on both nodes) and then steps thru the extents (20 at a time), configuring
  cluster_mode

- Define the fileio/blockio based extents in /etc/scst.conf for BOTH nodes,
  albeit with active 0 parameter set on the STANDBY node.

- On failover on the ACTIVE node, become_active iscsitarget rather than restart it.
  This avoids running scstadmin, instead will make precise changes to the scst /sys
  interface. We only do the bare minimum that we need (makes extents active, and
  replace the HA target based LUNs with real extent based ones).

- On failover on the STANDBY node, start iscsitarget but with minimal config.
  This will be corrected by the iscsi.alua.standby_after_start job.

- For inter-node HA targets tweak the recovery_tmo parameter to 10 seconds.
  Otherwise we can expect stalls on a newly elected ACTIVE node, before it logs
  out the HA targets it needed when it was STANDBY.

- When adding a target-extent mapping, configure the ACTIVE node first, then
  the STANDBY node.  The STANDBY node will initiate the bring up of
  cluster_mode.

- When removing a target-extent mapping call removed_target_extent on the
  STANDBY node to more cleanly tear down associated DLM config.

local and ds groups are written via group ID, but are returned
as full group entries.

(cherry picked from commit 08e90245

)
Co-authored-by: Andrew Walker <awalker@ixsystems.com>

We need to allow webui access if at least one of user roles grant
webui access.

(cherry picked from commit 1540cfcaf72daec87cab45292c6ebebd7a7cc0d3)
Co-authored-by: Andrew Walker <awalker@ixsystems.com>

(cherry picked from commit c0c45c56364e1133e40b7ae2949612dee843e10f)
Co-authored-by: Caleb <yocalebo@gmail.com>

(cherry picked from commit 03c0551081d00bd83071375a4405b5034f5f3c47)
Co-authored-by: M. Rehan <mrehanlm93@gmail.com>