Flush out samba's gencache before starting test, also
chown datasets so that they're owned by our SMB user.

those are not for HA only CORE

* Added propper timeout with exit 1 in boot.exp and install.exp

changed the install timout to 20 min instead of 30 minutes
changed the boot timout to 12 min instaed of 30 min

* fixed the boot timeout time

If group_mapping.tdb contains a duplicate gid entries,
then append second entry to `invalid` list to be deleted
during batch OPs. Also ensure that we always have entry
for S-1-5-32-545.

Samba server will check ZFS_PROP_WRITTEN on snapshots to determine
whether to present them the VSS clients.

At present this is limited to verifying that shadow copies are
presented over SMB1 / SMB2 and that handles can be opened on
data and streams.

Future improvement will be validation of contents of previous
versions of files and streams, but this will require some
enhancement of py-libsmb.
Co-authored-by: Andrew Walker <awalker@ixsystems.com>

* add nfsd.scope and nfsd.owner_major on Core HA

* add comment about setting 2 sysctls to same value

* Make netbiosname tests ha-aware

* Rewrite SMB SID on virtual hostname change

* Fix ups reporting when running in slave mode

This commit fixes an issue when we would not be showing reporting for UPS when it was running in slave mode because collectd stores ups logs in a different directory when it's running in slave mode.

* Make _base_path a property

NAS-112041 / 12.0 / Fix disk.identifier_to_device not working with {uuid} and {label} identifiers (#7522)

NAS-112038 / 12.0 / Rename non-builtin user/group if a builtin user/group with a same name should exist (#7523)

Changing these permissions through our API should not be permitted
since it has a high likelihood of breaking all jails.

Minor difference in return for `idmap.domain_info` in 12 and
SCALE. 12 key is `SID` and SCALE it is `sid`.

Also fix case where we weren't generating a default group mapping
for BUILTIN\Users, and we weren't removing stale mappings.

* Fix AD group unix token generation (#6950)

*Background*
In a Windows / AD environment, all objects are identified by
SID. SIDs for user / group objects take the form
`S-1-5-21-<domain>-<rid>`

The <domain> component for a SID will vary depending on
whether the account is a local one or one from an external domain.
A domain sid is of the form `S-1-5-21-<domain>`, and every
TrueNAS server has a unique domain sid, which is randomly
generated by samba libraries when it is first needed.

Each relative id (rid) uniquely identifies an object in the domain.
Certain rids are present in every domain. Of particular note
for this commit are the following:

```
S-1-5-21-<domain>-501 - Guest
S-1-5-21-<domain>-512 - Domain Admins
S-1-5-21-<domain>-513 - Domain Users
S-1-5-21-<domain>-514 - Domain Guests
```

In addition to domain (S-1-5-21) sids, every Windows computer
and Samba server has sids that are identical on every machine
that are prefixed with S-1-5-32 (built-in groups). Of particular note
for this commit are the following:
```
S-1-5-32-544 - Adminstrators
S-1-5-32-545 - Users
S-1-5-32-546 - Guests
```

Since Unix-like servers use uids / gids (xids) to identify users and
groups rather than sids, sids must be mapped into xids and vice-versa.
This task falls on Samba's passdb, groupdb, and winbindd's idmapping
facilities.

During samba startup, if samba's groupdb lacks entries for
Administrators, Users, and Guests, then they will be automatically
added by allocating new gids for each of them from winbindd's idmap
backend that has been configured to provide mappings for built-in
sids. This allocation increments the xid high-water mark in
winbindd_idmap.tdb, (but does not write the explicit mapping in
the key-value store) and then writes the explicit mapping in
group_mapping.tdb.

Windows has the concept of nested groups. Groups in Windows may
have members that are either users or groups. Accordingly, each
groupmap entry in group_mapping.tdb may have zero or more
foreign memberships in it. The following is a sample tdb entry:

```
{
key(23) = "UNIXGROUP/S-1-5-32-546\00"
data(32) = "\83J]\05\04\00\00\00Guests\00Local Unix group\00"
}
{
key(54) = "MEMBEROF/S-1-5-21-944110568-1438105595-1944063070-514\00"
data(13) = "S-1-5-32-546\00"
}
```

In this case, S-1-5-32-546 is mapped to gid 90,000,002 and has
a foreign member of S-1-5-21-944110568-1438105595-1944063070-514.

During the domain-join process, libads adds domain sids as members
of the above built-in groups:
```
S-1-5-21-<new domain>-512 --> S-1-5-32-544
S-1-5-21-<new domain>-513 --> S-1-5-32-545
S-1-5-21-<new domain>-514 --> S-1-5-32-546
```

Which means that when nss_winbind generates a passwd struct for
a domain user, BUILTIN\Users is added to the grouplist with the
gid listed in the group_mapping.tdb.

*Problem*
There are various situations that can occur where original
mapping for builtins is lost or remapped to different ids
foreign memberships are lost, or id collisions are generated
with other groups allocated in winbindd_idmap.tdb.

Although these built-in groups are not exposed via middleware and
the webui, they are exposed via the SMB protocol and Samba's
RPC endpoints. The most common reason for them to be used is
when robocopy is used to migrate data from a Windows share on
Windows server where the system administrator is using
built-in groups rather than AD groups to share data.

*Impact*
Impact of potential indetermenancy with the mapping potentially
profound. User tokens may be generated with incorrect ids,
and filesystem ACLs may cease to grant expected access.
If foreign group membership is dropped, then built-in groups
will not appear in passwd entries for AD and local users.

*Resolution*
When built-in groups are handled by idmap_tdb (default),
ensure that Administrators, Users, and Guests are mapped
explicitly to the lowest three gids in the range that is
specified for the default domain (*). Use newly-added
json-based batch operations for groupmap to achieve this.
net_groupmap text variant does not allow direct manipulation
of gids in the groupmap file, which creates a chicken-and-egg
problem for mapping built-ins explicitly when there is no
corresponding winbindd_idmap.tdb entry.

Adjust high-water-mark in idmap_tdb to never allocate gids
in this reserved area.

Ensure that local builtin_users, builtin_admins, builtin_guests
groups are mapped to the respective local domain (TrueNAS)
domain users, domain admins, domain guests SIDs.

Ensure that foreign mappings for Administrators, Users, and Guests
always exist.

* Fix foreign groupmap alias removal (#7258)

* NAS-111710 / 21.08 / Fix groupmap tests (#7278)

Do not background groupmap synchronization while adding groups.
This slightly increases risk of middleware timeouts, but we now optimized
groupmap ops by adding support for batched changes to samba's
net groupmap command. This means the odds of getting blocked behind
long-running iteration of `net groupmap` commands is minimal.

Add support for a "modify" queue during synchronization. This is
an issue in cases where we want to modify the `nt_name` of
a groupmap without altering the `sid` to `gid` mapping.

* Fix groupmap synchronization (#7397)

Migration to storing global parameters in registry config resulted
in normalizing how idmap parameters were written to SMB configuration.
This commit updates checks when we set up S-1-5-32 in groupmapping.tdb
so that tdb insertion / alias creation happens as required.

* NAS-112257 / 21.10 / Convert builtin_users mapping to ordinary local group (#7508)

Mapping to RID 513 can cause us to override the default user
primary group on file creation. Convert the builtin_users
account to a normal local users account with auto-generated RID.

(cherry picked from commit 706dfe94e6a02f557c6e4ed146cdf97753d088ce)
Co-authored-by: caleb <yocalebo@gmail.com>

Co-authored-by: caleb <yocalebo@gmail.com>

This commit adds changes to wait for lagg interface to actually become active before we try moving on starting services which might be relying on the lagg interface. We add a grace period of 10 seconds for that to happen and if it doesn't till then, we move on logging that it wasn't active as there can be cases where that never happens.

When run without TERM set, top fails to correctly configure its line
width, causing the process list lines to be blank.

Invoke in non-interactive mode to force line_width to 1024.

Jira: NAS-110666

Raising an exception at this point will cause systemdataset
setup to fail, which is generally a bad thing. During boot
there will be no active SMB sessions and so potential impact
is minimal.

Prune branches inside snapdirs so that we don't fail while
trying to recursively set the ACL.

Only attempt to get acl for dataset mountpoint if dataset is
mounted and has a non-legacy mountpoint.

* Added tests to make sure robocopy timestamp is kept

Fixed send_file and get_file functions in functions.py
moved test 004 in test 003

* fixed definition text in api2/smb.py

(cherry picked from commit f8b845917cc9fefbcd913a88ef4c6cd2428651a6)
Co-authored-by: themylogin <themylogin@gmail.com>

* Add adv_syslog_tls_certificate_authority_id migration

* Add validation ensuring that specified CA for remote syslog is valid

* Correctly generate syslog-ng configuration for remote syslog

* Do not alow deleting CA in use by syslog tls settings

Since asigra is an important plugins we should test the installation of
asigra instead of plexmediaserver.

I also move the wait_on_job in the job the id is taken from.

NAS-111808 / 12.0 / Add endpoint to retrieve serial port choices with serial port names in freebsd (#7390)

(cherry picked from commit d12d000bd5056b09f38ee109e7c7b9074e2d5558)
Co-authored-by: Waqar Ahmed <waqarahmedjoyia@live.com>

removed api2/pool_acltype.py since it is for scale
backport webdav test from master branch and removed api2/webdav_bsd.py
and api2/webdav_osx.py
improved api2/smb_protocol.py
cleaned some code

(cherry picked from commit f171bb57e6f2e7982667feadc33d9114bc532f67)
Co-authored-by: Waqar Ahmed <waqarahmedjoyia@live.com>

* improve IoThreadPoolExecutor

(cherry picked from commit 9efec7e51a24b2ffa30b65625713d30e5bc4aeb4)

* <= 1 for semaphore value check

(cherry picked from commit 407ba6e63d8f97f74c4f62bf7a4cd8fd0bbf3c78)
Co-authored-by: caleb <yocalebo@gmail.com>