Fix syslog_only parameter for SMB config (#7489)

NAS-112656 / 21.08-BETA.1 / Run midclt as non-privileged user in API key tests (by themylogin)

(cherry picked from commit a65cda470a6e2b7ca6495c48f90a8eb69e28bbe1)

(cherry picked from commit 74cc57c50e5566f3290a784dc7a9b85fc01fd721)

NAS-112081 / 21.08-BETA.1 / Ignore deletion of hidden dataset clones such as `%recv` dataset created by replication (#7434)

(cherry picked from commit bf60ef1c389fee853230b3c6ff12ab6e39242146)
Co-authored-by: themylogin <themylogin@gmail.com>

(cherry picked from commit 76402bc8776397b5d880129c0e6468baef3238e3)
Co-authored-by: Waqar Ahmed <waqarahmedjoyia@live.com>

NAS-111997 / 21.08-BETA.1 / Properly retrieve registry config and ACLs in SMB debug (by anodos325) (#7401)

* Properly retrieve registry config and ACLs in SMB debug

`getfacl` is not ACLtype-agnostic. Parse sharing.smb.query output
and use `filesystem.getacl` to retrieve share ACL.

(cherry picked from commit be3721622f5c27d3e51abe4b1324ba48430679e6)

* use smb.groupmap_list to list group mapping file

Returns empty list if LDAP enabled.

(cherry picked from commit 689b7749aa7546cd3cbc32bbd7b5d2e60fe83ea0)

* Bypass smb.groupmap_list if not using tdbsam passdb backend

(cherry picked from commit 90d0f08d8d0030f67cacf3f7281cbf3460901882)
Co-authored-by: Andrew Walker <awalker@ixsystems.com>

(cherry picked from commit ed8b2099c5bae76874b3e63e653472d6a8ec0624)
Co-authored-by: Waqar Ahmed <waqarahmedjoyia@live.com>

* improve IoThreadPoolExecutor

(cherry picked from commit 9efec7e51a24b2ffa30b65625713d30e5bc4aeb4)

* <= 1 for semaphore value check

(cherry picked from commit 407ba6e63d8f97f74c4f62bf7a4cd8fd0bbf3c78)
Co-authored-by: caleb <yocalebo@gmail.com>

(cherry picked from commit a639860ef418fc5690da8c64eeb644c979a8f9e4)
Co-authored-by: themylogin <themylogin@gmail.com>

NAS-111936 / 21.08-BETA.1 / Use libvirt to determine how much memory a VM is consuming (by sonicaj) (#7373)

* Use libvirt to retrieve memory usage of a vm

(cherry picked from commit 02d58e8fb2b2301b541dfd7a0087743b8e05042d)

* Use new implementation of libvirt to retrieve vm memory usage

(cherry picked from commit 9aa6613dda79530688a371397d6856b0bee37619)
Co-authored-by: Waqar Ahmed <waqarahmedjoyia@live.com>

Avoid touching winbindd unless it's absolutely necessary.
NetBIOS names and workgroups cannot be changed while AD is
enabled and so we should be protected from having inconsistent
config on winbindd and smbd.

Row proxy object's values cannot be changed, so we should convert it to a dict for temporary changes instead.

There are behavior nuances related to when we grant IPC$ access
for share enumeration. Update test to account for these so that
we have more complete coverage of this feature.

Since we are now using registry backend for global parameters
we need to disallow "lock directory" and "config backend"
parameters as auxiliary parameters. If we don't, then
attempts to set GLOBAL section will fail with
SBC_ERR_INVALID_PARAM.

Rules are in smbconf_reg_parameter_is_valid() in
source3/lib/smbconf/smbconf_reg.c

This commit adds changes to make sure that we force sqlite3 to give up space which has accumulated over time by deletions of entries. What sqlite3 does is that each data which is removed from the database, sqlite3 does not use the space consumed by that payload and instead moves it to it's list of free pages and whenever more data is added it uses these free pages first and then asks for more space. However with time these free pages can consume lots of space like for a user his db size was 15 mb approx and after vacumming it got to 706K. This change uses vacuum functionality of sqlite3 to make sure that we remove such free unused pages which are in hold by sqlite3.

Right now if no nfs shares are configured and if user attempts to start NFS, it does not start as the configuration file is missing for ganesha. This leads to violation of POLA where consumer adding shares later thinks/assumes that NFS would be running but that's not the case and it needs to be started again after configuring a share.

Changes to share guest access require alteration of global SMB
server configuration. Specifically, we need to ensure that
"map to guest" is always set to "Bad User" in this case. This
parameter is generally set to "Never" because modern windows
versions will deny access to SMB share if user it received does
not match one that it requested, which has the effect of causing
unexpected denial of access to the SMB server in general as this
also applies to the server's IPC$ share.

This change fixes this behavior in SCALE, where the transition
to using the registry for storing SMB configuration details
caused a regression. The specific change is to expand the
situations where sharing.smb.create and sharing.smb.update
will re-initialize the global SMB parameters and restart the
SMB server. The server restart in this situation is required
because the settings may impact desired access to existing shares.

This make non-recursive operation do the same thing as our
recursive operations. There are some edge cases where xattr
removal and setfacl -b for POSIX1E ACLs yields different
results.

Entries wtih tags associated with file User, Group, or Other /Everyone
may not have a numeric id set. For backwards compatibility, the
following values are permitted for these special entries: -1, null/None.

This was broken by acef89b7

(cherry picked from commit 4730b3ff68398fc3ee74c4ba1b5900996bcd91f7)

Retrieve acltype through maximum of two getxattr calls. Use this
to inform how we read ACLs. Add proper returns for case where
ZFS acltype is off.

There's no need to generate this in middleware and we must
ensure that it's configured before middleware starts.

The passdb path is now hard-coded and guaranteed to exist (doesn't
differ between HA and non-HA.

* Add tests for directory services user/group cache

During AD join process we generate a list of IDs known to us from
Samba's gencache.tdb, and then iterate the list, converting their
respective pwd and grp entries into our typical user / group query
result structures. These get inserted into key-value stores in
/root/tdb/persistent on the local filesystem. The local cache
gets refilled once per day. Directory service users / groups
are included in query results when the filter-option
`{"extra": {"search_dscache": True}}` is specifid. LDAP server
is not queried directly if it can be avoided so that unnecessary
load is not generated (especially in large or poorly designed
environments). This was historically a major issue in legacy
versions of FreeNAS (11.1 and earlier). This does create an
issue for cases where recently added users may not be present
in the cache. It is common for API users to query a single
user during user / group validation with a single filter
`[["username", "=", <name>]]`. In this case the directory services
name cache will look up the user directly through nss (pwd or grp module).
If user is found, cache entry will be inserted into the cache
backend so that it will appear in future query results.

This series of tests validates the following:
- Cache fill job completes successfully
- Cache refresh job completes successfully
- Direct lookup of user / group by id or name works correctly
- Cache entries are inserted after direct lookup

* Improve docstrings for test

* Clarify description of first test in AD users/groups

Co-authored-by: caleb <yocalebo@gmail.com>

This commit adds changes to make sure that the gateway specified is accurate and is actually accessible by the system, as we have had cases with users trying to use 8.8.8.8 for the gateway which is obviously going to fail.