GitLab

The kernel modules are now available in the Arch User Repository
(AUR) via zfs.  Since their packaging is maintained and superior
to ours it is being removed from the tree.

  https://wiki.archlinux.org/index.php/ZFS



Now that various distributions are picking up the packages we
should eventually be able to remove most of this infrastructure.
Packaging belongs with the distributions not upstream.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>

The standard dracut directory has moved from /usr/share/dracut to
/usr/lib/dracut.  To ensure the dracut modules get installed in
the correct location provide a --with-dracutdir configure option
to set the path.

The default install location has been updated to /usr/lib/dracut
which is used by more current versions of Fedora.  However, this
default is overriden by the RPM packaging for consistency.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>

Provide a mechanism to control the directory name the modules
are installed in.  The kernel privdes INSTALL_MOD_DIR for
this but it was hardcoded to be 'addon/zfs'.

Add a KMODDIR variable which can be passed to 'make install'
to override the default directory name.  While we're here
change the default from 'addon/zfs' to 'extra' which is the
kernel.org default.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>

The default permissions used by install are 755.  Since this
file isn't executable 644 is more appropriate.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>

According to the FHS.  Testing scripts and examples which are all
architecture independent should be installed in a subdirectory
under /usr/share.

  http://www.pathname.com/fhs/2.2/fhs-4.11.html

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>

The new snapdev dataset property may be set to control the
visibility of zvol snapshot devices.  By default this value
is set to 'hidden' which will prevent zvol snapshots from
appearing under /dev/zvol/ and /dev/<dataset>/.  When set to
'visible' all zvol snapshots for the dataset will be visible.

This functionality was largely added because when automatic
snapshoting is enabled large numbers of read-only zvol snapshots
will be created.  When creating these devices the kernel will
attempt to read their partition tables, and blkid will attempt
to identify any filesystems on those partitions.  This leads
to a variety of issues:

1) The zvol partition tables will be read in the context of
   the `modprobe zfs` for automatically imported pools.  This
   is undesirable and should be done asynchronously, but for
   now reducing the number of visible devices helps.

2) Udev expects to be able to complete its work for a new
   block devices fairly quickly.  When many zvol devices are
   added at the same time this is no longer be true.  It can
   lead to udev timeouts and missing /dev/zvol links.

3) Simply having lots of devices in /dev/ can be aukward from
   a management standpoint.  Hidding the devices your unlikely
   to ever use helps with this.  Any snapshot device which is
   needed can be made visible by changing the snapdev property.

NOTE: This patch changes the default behavior for zvols which
      was effectively 'snapdev=visible'.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1235
Closes #945
Issue #956
Issue #756

The changes to zvol.c were never merged from the last onnv_147
bulk update.  This was because zvol.c was largely rewritten
for Linux making it fairly easy to miss these sorts of changes.

This causes a regression when importing a zpool with zvols
read-only.  This does not impact pool which only contain
filesystem datasets.

References:
  illumos/illumos-gate@f9af39b
Signed-off-by: Richard Yao <ryao@cs.stonybrook.edu>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1332
Closes #1333

The PaX team modified the kernel's modpost to report writeable function
pointers as section mismatches because they are potential exploit
targets. We could ignore the warnings, but their presence can obscure
actual issues. Proper const correctness can also catch programming
mistakes.

Building the kernel modules against a PaX/GrSecurity patched Linux 3.4.2
kernel reports 133 section mismatches prior to this patch. This patch
eliminates 130 of them. The quantity of writeable function pointers
eliminated by constifying each structure is as follows:

vdev_opts_t             52
zil_replay_func_t       24
zio_compress_info_t     24
zio_checksum_info_t     9
space_map_ops_t         7
arc_byteswap_func_t     5

The remaining 3 writeable function pointers cannot be addressed by this
patch. 2 of them are in zpl_fs_type. The kernel's sget function requires
that this be non-const. The final writeable function pointer is created
by SPL_SHRINKER_DECLARE. The kernel's set_shrinker() and
remove_shrinker() functions also require that this be non-const.
Signed-off-by: Richard Yao <ryao@cs.stonybrook.edu>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1300

PaX/GrSecurity patched kernels implement a dialect of C that relies on a
GCC plugin for enforcement. A basic idea in this dialect is that
function pointers in structures should not change during runtime.
This causes code that modifies function pointers at runtime to fail to
compile in many instances. The autotools checks rely on whether or
not small test cases compile against a given kernel. Some
autotools checks assume some default case if other cases fail. When one
of these autotools checks tests a PaX/GrSecurity patched kernel by
modifying a function pointer at runtime, the default case will be used.

Early detection of such situations is possible by relying on compiler
warnings, which are compiler errors when --enable-debug is used.
Unfortunately, very few people build ZFS with --enable-debug. The more
common situation is that these issues manifest themselves as runtime
failures in the form of NULL pointer exceptions.

Previous patches that addressed such issues with PaX/GrSecurity
compatibility largely relied on rewriting autotools checks to avoid
runtime function pointer modification or the addition of PaX/GrSecurity
specific checks. This patch takes the previous work to its logical
conclusion by eliminating the use of runtime function pointer
modification. This permits the removal of PaX-specific autotools checks
in favor of ones that work across all supported kernels.

This should resolve issues that were reported to occur with
PaX/GrSecurity-patched Linux 3.7.5 kernels on Gentoo Linux.

https://bugs.gentoo.org/show_bug.cgi?id=457176



We should be able to prevent future regressions in PaX/GrSecurity
compatibility by ensuring that all changes to ZFSOnLinux avoid runtime
function pointer modification. At the same time, this does not solve the
issue of silent failures triggering default cases in the autotools
check, which is what permitted these regressions to become runtime
failures in the first place. This will need to be addressed in a future
patch.
Reported-by: Marcin Mirosław <bug@mejor.pl>
Signed-off-by: Richard Yao <ryao@cs.stonybrook.edu>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1300

The issue with hot spares in ZoL is because it opens all leaf
vdevs exclusively (O_EXCL).  On Linux, exclusive opens cause
subsequent exclusive opens to fail with EBUSY.

This could be resolved by not opening any of the devices
exclusively, which is what Illumos does, but the additional
protection offered by exclusive opens is desirable.  It cleanly
prevents you from accidentally adding an in-use non-ZFS device
to your pool.

To fix this we very slightly relaxed the usage of O_EXCL in
the following ways.

1) Functions which open the device but only read had the
   O_EXCL flag removed and were updated to use O_RDONLY.

2) A common holder was added to the vdev disk code.  This
   allow the ZFS code to internally open the device multiple
   times but non-ZFS callers may not.

3) An exception was added to make_disks() for hot spare when
   creating partition tables.  For hot spare devices which
   are already opened exclusively we skip creating the partition
   table because this must already have been done when the disk
   was originally added as a hot spare.

Additional minor changes include fixing check_in_use() to use
a partition instead of a slice suffix.  And is_spare() was moved
above make_disks() to avoid adding a forward reference.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #250

As described by the comment and enforced the by assertion the
v->vdev_wholedisk will never be -1.  The wholedisk handling
is performed by the user space utilities.  To prevent confusion
this dead code is being removed.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>

When vdev_disk.c was implemented for Linux we failed to handle the
reopen case.  According to the vdev_reopen() comment leaf vdevs should
not be closed or opened when v->vdev_reopening is set.  Under Linux
we would always close and open the device.

This issue was only noticed when a 'zpool scrub' command was run while
the leaf vdev device names in /dev/disk/by-vdev were missing.  The
scrub command calls vdev_reopen() which caused the vdevs to be closed
but they couldn't be reopened due to the missing links.  The result
was that all the vdevs were marked unavailable and the pool was
halted due to failmode=wait.

This patch adds the missing functionality in a similiar fashion to
to the Illumos code.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>

`zpool status -x` should only flag errors or where the pool is
unavailable.  If it imported fine but isn't using the latest features
available in the code, that's not an error.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1319

To determine whether the kernel is capable of handling empty barrier
BIOs, we check for the presence of the bio_empty_barrier() macro,
which was introduced in 2.6.24. If this macro is defined, then we can
flush disk vdevs; if it isn't, then flushing is disabled.

Unfortunately, the bio_empty_barrier() macro was removed in 2.6.37,
even though the kernel is still capable of handling empty barrier BIOs.

As a result, flushing is effectively disabled on kernels >= 2.6.37,
meaning that starting from this kernel version, zfs doesn't use
barriers to guarantee on-disk data consistency. This is quite bad and
can lead to potential data corruption on power failures.

This patch fixes the issue by removing the configure check for
bio_empty_barrier(), as we don't support kernels <= 2.6.24 anymore.

Thanks to Richard Kojedzinszky for catching this nasty bug.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1318

As a matter of fact, we're already using -Werror for most tests because
of a bug in kernel-bio-empty-barrier.m4 which sets -Werror without
reverting it afterwards. This meant that all tests which ran after this
one was using -Werror.

This patch simply makes it clear that we're using -Werror and makes
the code more readable and more predictable.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1317

The zfs_arc_memory_throttle_disable module option was introduced
by commit 0c5493d4

 to resolve a
memory miscalculation which could result in the txg_sync thread
spinning.

When this was first introduced the default behavior was left
unchanged until enough real world usage confirmed there were no
unexpected issues.  We've now reached that point.  Linux's
direct reclaim is working as expected so we're enabling this
behavior by default.

This helps pave the way to retire the spl_kmem_availrmem()
functionality in the SPL layer.  This was the only caller.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #938

Rather then setting _prefix=/ and having to override all the
default install locations.  It's cleaner, and more understandable,
to leave prefix=/usr and only override _sbindir and _libdir.  This
fixes three issues:

* The commands no longer get built with an incorrect rpath for
  the libraries.   This is good because fixing this sort of
  thing is required by the Fedora packaging guidelines.

    http://fedoraproject.org/wiki/Packaging:Guidelines#Beware_of_Rpath



* The various AUTHORS, COPYRIGHT, etc files are now correctly
  installed under /usr/share/doc instead of /share/doc.

* _libexecdir is now handled properly for each distribution.
  Fedora/RHEL=/usr/libexec, OpenSUSE/SLES=/usr/lib, Debian=/usr/lib/rpm
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1058

A couple of assertions in spa.c were designed to prevent the use of
invalid pool versions. They were written under the assumption
that all valid pools are less than SPA_VERSION. Since feature flags
jumped from 28 to 5000, any numbers in the range 28 to 5000
non-inclusive will fail to trigger them.  We switch to the new
SPA_VERSION_IS_SUPPORTED macro to correct this.
Signed-off-by: Richard Yao <ryao@cs.stonybrook.edu>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1282

It turns out that the Linux VFS doesn't strictly handle all cases
where a component path name exceeds MAXNAMELEN.  It does however
appear to correctly handle MAXPATHLEN for us.

The right way to handle this appears to be to add an explicit
check to the zpl_lookup() function.  Several in-tree filesystems
handle this case the same way.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1279

As of 0.6.0-rc11 using ZFS volumes as Linux swap devices is
supported.  Swapping to files in ZFS filesystems is not.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1189

Two more locations where KM_SLEEP was used in a call which must
use KM_PUSHPAGE were found while using the zpool upgrade command.
See commit b8d06fca

 for additional details.

Also make a small correction to the comment block above
dsl_dir_open_spa().
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1268

The machelf.h header is never included by anything in the zfs
build process.  It is all effectively dead code which can be
safely removed.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1265

binutils 2.23.1 fails in situations that generate function relocations
on PowerPC and possibly other architectures. This causes linking of
libzpool to fail because it depends on libnvpair. We add a dependency on
libnvpair to lib/libzpool/Makefile.am to correct that.
Signed-off-by: Richard Yao <ryao@cs.stonybrook.edu>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1267

Explicitly case this value to an unsigned long long for 32-bit
systems to inform the compiler that a long type should not be
used.  Otherwise we get the following compiler error:

  dmu_send.c:376: error: integer constant is too large for
  ‘long’ type
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>

The zpool-features(5) man page should reference the Linux zfs(8)
and zpool(8) man pages.  The 1M convention isn't used on Linux.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1184

The zpool-features(5) man page was accidentally omitted from the
build target when feature flags was merged.  As a result it doesn't
get installed as part of 'make install' so none of the packages
include this man page.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1262

The way in which virtual box ab(uses) memory can throw off the
free memory calculation in arc_memory_throttle().  The result is
the txg_sync thread will effectively spin waiting for memory to
be released even though there's lots of memory on the system.

To handle this case I'm adding a zfs_arc_memory_throttle_disable
module option largely for virtual box users.  Setting this option
disables free memory checks which allows the txg_sync thread to
make progress.

By default this option is disabled to preserve the current
behavior.  However, because Linux supports direct memory reclaim
it's doubtful throttling due to perceived memory pressure is ever
a good idea.  We should enable this option by default once we've
done enough real world testing to convince ourselve there aren't
any unexpected side effects.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #938

Commit 1eb5bfa3

 introduced a new zfs_disable_dup_eviction tunable.
It should have been made available as a module option in the
original patch but was overlooked.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>

This is a minor nit, but the second line of the 'action' message
when you need to upgrade your pool to support feature flags exceeds
the standard 80 character limit.  Fix it by moving the word
'feature' on to the third line.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>

When a system attribute layout is created an inconsistency may occur
between the system attribute header (sa_hdr_phys_t) size and the
variable-sized attribute count stored in the layout.  The inconsistency
results in the following failed assertion when SA_HDR_SIZE_MATCH_LAYOUT
returns false:

SPLError: 11315:0:(sa.c:1541:sa_find_idx_tab())
ASSERTION((IS_SA_BONUSTYPE(bonustype) && SA_HDR_SIZE_MATCH_LAYOUT(hdr,
tb)) || !IS_SA_BONUSTYPE(bonustype) || (IS_SA_BONUSTYPE(bonustype) &&
hdr->sa_layout_info == 0)) failed

The bug originates in this snippet from sa_find_sizes().

    if (is_var_sz && var_size > 1) {
            if (P2ROUNDUP(hdrsize + sizeof (uint16_t),
                *total < full_space) {
                    hdrsize += sizeof (uint16_t);

This assumes that the current variable-sized attribute will be stored in
the current buffer and accounts for the space needed to store its size
in the sa_hdr_phys_t. However if the next attribute spills over we need
to store a blkptr_t at the end of the bonus buffer to point to the spill
block. If the current attribute is in the way of the blkptr_t then it
too will be relocated into the spill block. But since we've already
accounted for it in the header size we get the inconsistency described
above.

To avoid this, record the index of the last variable-sized attribute
that prompted a hdrsize increase, and reverse the increase if we later
determine that that attribute will be relocated to the spill block.
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1250

A rounding discrepancy exists between how sa_build_layouts() and
sa_find_sizes() calculate when the spill block needs to be kicked in.
This results in a narrow size range where sa_build_layouts() believes
there must be a spill block allocated but due to the discrepancy there
isn't.  A panic then occurs when the hdl->sa_spill NULL pointer is
dereferenced.

The following reproducer for this bug was isolated:

    truncate -s 128m /tmp/tank
    zpool create tank /tmp/tank
    zfs create -o xattr=sa tank/fish
    ln -s `perl -e 'print "z" x 41'` /tank/fish/z
    setfattr -hn trusted.foo -v`perl -e 'print "z"x45'` /tank/fish/z

This test results in roughly the following system attribute (SA)
layout:

  176 bytes - "standard" SA's
   41 bytes - name of symbolic link target
  100 bytes - XDR encoded nvlist for xattr
  ---
  317 bytes - total

Because 317 is less than DN_MAX_BONUSLEN (320), sa_find_sizes()
decides no spill block is needed. But sa_build_layouts() rounds 41 up
to 48 when computing the space requirements so it tries to switch to
the spill block.

Note that we were only able to reproduce this bug using a combination
of symbolic links and the Linux-specific xattr=sa dataset property.
So while this issue is not technically Linux-specific, it may be
difficult or impossible to hit the narrow size range needed to
reproduce it on other platforms.

To fix the discrepancy, round the running total in sa_find_sizes() up
to an 8-byte boundary before accounting for each SA, since this is how
they will be stored in the bonus and (possibly) spill buffers.

To make the intent of the code more clear, explicitly assert key
assumptions about expected alignment of data and whether spill-over
will occur.

Signed-off-by: Matthew Ahrens <mahrens@delphix.com
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1240

3447 improve the comment in txg.c

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Richard Lowe <richlowe@richlowe.net>
Reviewed by: Garrett D'Amore <garrett@damore.org>
Reviewed by: Richard Elling <richard.elling@dey-sys.com>
Approved by: Dan McDonald <danmcd@nexenta.com>

References:
  illumos/illumos-gate@adbbcfface63b3a71922d5a25d34a2018c0435de
  https://www.illumos.org/issues/3447

Ported-by: Brian Behlendorf <behlendorf1@llnl.gov>

In the interest of maintaining only one udev helper to give vdevs
user friendly names, the zpool_id and zpool_layout infrastructure
is being retired.  They are superseded by vdev_id which incorporates
all the previous functionality.

Documentation for the new vdev_id(8) helper and its configuration
file, vdev_id.conf(5), can be found in their respective man pages.
Several useful example files are installed under /etc/zfs/.

  /etc/zfs/vdev_id.conf.alias.example
  /etc/zfs/vdev_id.conf.multipath.example
  /etc/zfs/vdev_id.conf.sas_direct.example
  /etc/zfs/vdev_id.conf.sas_switch.example
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #981

Commit 4b2f65b2

 increased the user
space stack by 4x to resolve certain stack overflows.  As such it
no longer makes sense to worry about a single extra page which
might or might not be part of the process stack.  There is now
ample headroom for normal usage.

By eliminating this configure check we are also resolving the
following segfault which intentionally occurs at configure time
and may be logged in dmesg.

  conftest[22156]: segfault at 7fbf18a47e48 ip 00000000004007fe
  sp 00007fbf18a4be50 error 6 in conftest[400000+1000]
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>

3035 LZ4 compression support in ZFS and GRUB

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Approved by: Christopher Siden <csiden@delphix.com>

References:
  illumos/illumos-gate@a6f561b4aee75d0d028e7b36b151c8ed8a86bc76
  https://www.illumos.org/issues/3035
  http://wiki.illumos.org/display/illumos/LZ4+Compression+In+ZFS



This patch has been slightly modified from the upstream Illumos
version to be compatible with Linux.  Due to the very limited
stack space in the kernel a lz4 workspace kmem cache is used.
Since we are using gcc we are also able to take advantage of the
gcc optimized __builtin_ctz functions.

Support for GRUB has been dropped from this patch.  That code
is available but those changes will need to made to the upstream
GRUB package.

Lastly, several hunks of dead code were dropped for clarity.  They
include the functions real_LZ4_uncompress(), LZ4_compressBound()
and the Visual Studio specific hunks wrapped in _MSC_VER.
Ported-by: Eric Dillmann <eric@jave.fr>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1217

The -q option should quiet the mkfs.ext2 output but certain
versions of e2fsprogs appear to ignore it.  This can result in
an extra 'done' message in the test output.  To keep this noise
from distracting just direct stdout to /dev/null.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>

It's doubtful many people were impacted by this but commit 6c285672


accidentally broke ZFS builds for 2.6.26 and earlier kernels.  This
commit depends on the lookup_bdev() function which exists in 2.6.26
but wasn't exported until 2.6.27.

The availability of the function isn't critical so a wrapper is
introduced which returns ERR_PTR(-ENOTSUP) when the function isn't
defined.  This will have the effect of causing zvol_is_zvol() to
always fail for 2.6.26 kernels.  This in turn means vdevs will
always get opened concurrently which is good for normal usage.
This will only become an issue if your using a zvol as a vdev in
another pool.  In which case you really should be using a newer
kernel anyway.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1205

Test 5, 6, 7, and 7 in zconfig.sh use /bin/ as a source of random
directories and files for their test.  This has lead to unexpected
tests failures because the total size of /bin/ on the test system
isn't checked and it is entirely possible for it to be larger than
the target filesystem.

To resolve this issue we create a somewhat random collection of
files and directories in /var/tmp to use.  On average we expect
about 5MB of data with the worst case being 20MB.  This is large
enough to be interesting and small enough to always fit in the
default test datasets.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1113

The differ() function used strerror_r() instead of strerror() because
it allowed the error message to be directly copied in to a buffer.
This causes two issues under Linux.

* There are two versions of strerror_r() available an XSI-compliant
  version which returns an 'int' error code.  And a GNU-specific
  version which return a 'char *' to the resulting error string.

    int strerror_r(int errnum, char *buf, size_t buflen);   /* XSI */
    char *strerror_r(int errnum, char *buf, size_t buflen); /* GNU */

* The most recent versions of strerror_r() are annotated with the
  warn_unused_result attribute.  This causes the following warning
  since the upstream implementation casts the result to void.

    warning: ignoring return value of 'strerror_r', declared with
    attribute warn_unused_result [-Wunused-result]

The cleanest way to resolve both of these problems is just to use
strerror() and make a copy of the result in to the buffer.  This
resolves both issues and this is the only instance of strerror_r()
in the code base.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1231