nbdkit-lzip-filter - Man Page

nbdkit lzip filter

Synopsis

 nbdkit --filter=lzip file FILENAME.lz

 nbdkit --filter=lzip curl https://example.com/FILENAME.lz

nbdkit-lzip-filter is a filter for nbdkit(1) which transparently decompresses an lzip(1)-compressed file. You can place this filter on top of nbdkit-file-plugin(1) to decompress a local .lz file, or on top of other plugins such as nbdkit-curl-plugin(1):

 nbdkit curl --filter=lzip https://example.com/disk.lz

With nbdkit-partition-filter(1) it can be used to mount a specific partition of a compressed disk image:

 nbdkit curl --filter=partition --filter=lzip \
        https://example.com/disk.img.lz partition=2

The filter only supports read-only connections.

Note that this filter was made primarily for multi-member files. Single-member files are currently only supported up to a size of lzip-max-block and their contents are kept in memory.

Single-member files and multi-member files

An lzip-compressed file consists of one or more members, with each member containing an LZMA stream and some metadata. Most lzip files consist of only a single member holding all of the compressed data. You can find out the number of members in an lzip file by doing:

 $ lzip --list --verbose lfs.img.lz
   dict   memb  trail   uncompressed   compressed   saved  name
   8 MiB     1      0    10737418240   1808781538  83.15%  lfs.img.lz
           ↑↑↑
          members

Seeking an lzip file is done by seeking directly to the lower member boundary, then decompressing data until the precise byte is reached. Because of this, single-member files are not seekable and must be decompressed entirely.

Getting the best random access performance

Use the plzip(1) parallel compressor with the --data-size option set to a small-ish block size. plzip first splits the input file into fixed-size data blocks before compressing each block as a separate member. For example this is the same image as above compressed with plzip on default settings, resulting in a block size of 16 MiB:

 $ lzip --list --verbose lfs.img.lz
   dict   memb  trail   uncompressed   compressed   saved  name
   8 MiB   640      0    10737418240   1839569039  82.87%  lfs.img.lz
           ↑↑↑
          members

This file allows random access in constant time. At most 16 MiB will have to be decompressed to seek to any byte. The position of the member that needs to be decompressed can be computed directly.

If you don't have access to plzip, you can achieve the same result using the split(1) utility together with regular lzip.

 $ split -b 16m disk.img disk.img.part
 $ lzip disk.img.part*
 $ cat disk.img.part*.lz > disk.img.lz

Do not just use lzip with the --member-size option, since that option refers to member size, not data block size.

Multi-member files with variable-size data blocks

Some ways of creating multi-member lzip files, such as the --member-size option or the tarlz(1) utility, result in data blocks of varying sizes. In this case, seeking requires a binary search on the member index and random access takes logarithmic time.

Parameters

lzip-max-block=SIZE

The maximum block size that the filter will read. The filter will refuse to read lzip files that contain any data block larger than this size. This value refers to size of the uncompressed block, not the size of the member holding it.

This parameter is optional. If not specified it defaults to 512M.

lzip-max-depth=N

Maximum number of blocks stored in the LRU block cache.

This parameter is optional. If not specified it defaults to 8.

The filter may allocate up to maximum block size in file × maxdepth bytes of memory per connection.

Files

$filterdir/nbdkit-lzip-filter.so

The filter.

Use nbdkit --dump-config to find the location of $filterdir.

Version

nbdkit-lzip-filter first appeared in nbdkit 1.42. It is derived from nbdkit-xz-filter which first appeared in nbdkit 1.10.

Authors

Jan Felix Langenbach

Richard W.M. Jones

Copyright

License

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
Neither the name of Red Hat nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Referenced By

nbdkit(1), nbdkit-bzip2-filter(1), nbdkit-filter(3), nbdkit-gzip-filter(1), nbdkit-tar-filter(1), nbdkit-xz-filter(1).

2024-10-14 nbdkit-1.41.8