slimdata - Man Page

-p,  --preserve

Preserve the original raw file when compressing or the original slim file when expanding.  The default behavior is to delete the input file(s) as each output file is successfully compressed or expanded (except when the option --stdout is used).  If this option is selected when expanding files, then the new output file is named by replacing the ".slm" extension with ".raw".

-X,  --compress

The file or files given as arguments are raw files to be compressed. The default is to assume raw files, except where the file names end with the .slm or .SLM suffix.  This option is the only way to force compression of a file named something.slm.

-x,  --expand

The file or files given as arguments are slim files to be expanded. The default is to assume raw files, except where the file names end with the .slm or  .SLM suffix.

-k,  --force

Force compression or expansion to overwrite any existing file of the same name.  Without this option, slim will refuse to overwrite any files.

-V,  --version

Print the slim file version and exit.

-q,  --quiet

Don't print any output unless there are errors.

-?,  --help

Print a usage message and exit.

These options are ignored when expanding files.

-m,  --method=method

Use encoding method method (default = 2).  The methods are:

   * -m2 The reduced binary encoder (default)
   * -m5 Runlength coder.  Good for data where the values are
         strictly identical for long periods.

All other values are reserved.  Methods numbered 1, 3, and 4 were once implemented.  They all proved wrong for the job and have been removed. Method 1 was a slight variation on the reduced binary system, differing only in how the parameters were computed.  For more on methods 2 through 5, see Encoding Algorithms.

-d,  --deltas

Encoders operate on the differences between successive data values (the deltas).  Default is to operate on the data itself.  Deltas are slightly slower to encode and decode, but they compress more effectively in almost all situations.  In principle, raw data could compress to 0.5 fewer bits per data value, but this would be expected only for extremely stable channels that are not typical of physics data.

-b,  --permit-bitrotation

Bit-rotation is an encoding feature useful in some data streams where the lowest n bits are always the same.  When using this option, slim will check each channel for the existence of some number of constant lowest bits.  If any are found, then the data are compressed by "rotating" the lowest bits to the top of the number and then proceeding with the usual methods.  The author finds this useful with some analog-to-digital schemes that have several low-order bits fixed at zero.  For more, see Bit rotation below.

-n,  --filename

Save the original filename in the slim file header.  By default, slim does not include the filename in the slim file.

-S,  --rawsize

Save the original file's size (in bytes) in the slim file header. (Currently, this is on by default and cannot be turned off.)  Note that when expanding, the raw file size is not tested against the size stored in the file header.

-C,  --compute-crc32

On compression, compute a CRC-32 checksum on the raw data in each section and include its value in the slim file.  Default is not to compute the CRC.  (On expansion, slim will compute and test the checksum if it is present in the slim file, with or without this option set.  However, also see --ignore-crc32 )

These options are ignored when compressing files.

-o,  --stdout

Write the results of expansion to stdout instead of to a file or files.  This option implies "--preserve" and is equivalent to running slimcat

-0,  --ignore-crc32

On expansion, do not compute and test the CRC-32 checksum, even if one is included in the slim file.

The current version of slim requires that all data channels be the same (i.e. of the same raw type, having the same number of repeats per frame, and so forth). Therefore, when given, all of the options in this section apply to all channels.  These requirements are not limitations inherent to the slim file specification, but only to the current implementation of slim. In the future, it could be possible to slim a more heterogeneous mix of data channels, if a configuration file parser were to be added.  For more on the differences between the current restrictions on compression and the full generality of the expander, see More general slim data structures.

Data types are assumed to be 1-, 2- or 4-byte integers with little-endian byte order.  (Single-byte integer types cannot currently be specified at the command line.)  There are no immediate plans to add facilities for handling 8-byte integers or floating-point numbers to slim (although command-line arguments have been reserved for the floating-points).

See Raw Data Structure for more information about how the raw data is assumed to be structured, and for the definitions of the terms channels, sections, frames, and repeats.

All data description options apply to compression only; they are all ignored when expanding a slim file.

-c,  --num-chan=nc

The data consist of nc separate data channels.

-r,  --repeats=reps

One frame contains reps consecutive repeated data values for each channel.

-F,  --frames=nf

There are nf frames in each data section.  Note that slim is allowed to break up these frames into multiple sections as needed to keep the section size below the hard limit of 16 MB, or for any other reason.  However, slim will also enforce a change to a new section at the end of every nf frames with this option.

-i,  --int

-u,  --unsigned

All data are signed or unsigned 32-bit integers (the default is signed 32-bit).

-s,  --short

-v,  --ushort

All data are signed or unsigned 16-bit integers.  Some aspects of slim are not implemented for short integers, such as bit rotation.  See BUGS.

-y,  --char

All data are signed 8-bit integers.

-f,  --float

-g,  --double

All data are 32-bit or 64-bit floating point numbers using the IEEE-754 standard.  Currently no floating-point compression schemes are implemented, and this option is merely reserved for future use. Instead, 32-bit floating point data are treated as signed 32-bit integers, while 64-bit data are stored without compression.

The main encoder system (the reduced binary method) works by sampling only some fraction of the raw data and assuming that the statistical properties of the sample are adequate to predict the behavior of the entire set.  The data sampling options govern what fraction of the data are used in the sampling.  These control the trade-off between speed and compression ratio.

Experience suggests that sampling a greater fraction of the data offers very little benefit; we suggest sampling no more than 10% of all values (the default), unless you plan to do (or have done) a careful test of speed and compression ratio.

Note that the encoders are all universal, in the sense that all possible values can be encoded (though of course most values will not be compressed to a smaller size).  This means that the occasional unusual data value can be encoded, and sparse sampling is not fatal to the resulting slim file.

Sampling can be done on a maximum of 20,000 data values.  All sample options refer to percentages of either this maximum or the full number of available values, whichever is smaller.  Regardless of the options given, a minimum of 20 data values or all values will be sampled, whichever is smaller.

-G,  --sample-pct=pct

Sample pct percent of the data (between 2 and 100%) up to a maximum of 200*pct samples when determining the encoding parameters.  Default is 10% up to 2000 samples.

-9,  --18-pct,  --best

Sample 18% of the data (up to 3600 samples) to determine encoding parameters.

-8,  --16-pct

Sample 16% of the data (up to 3200 samples) to determine encoding parameters.

-7,  --14-pct

Sample 14% of the data (up to 2800 samples) to determine encoding parameters.

-6,  --12-pct

Sample 12% of the data (up to 2400 samples) to determine encoding parameters.

-5,  --10-pct

Sample 10% of the data (up to 2000 samples) to determine encoding parameters.  (This is the default.)

-4,  --8-pct

Sample 8% of the data (up to 1600 samples) to determine encoding parameters.

-3,  --6-pct

Sample 6% of the data (up to 1200 samples) to determine encoding parameters.

-2,  --4-pct

Sample 4% of the data (up to 800 samples) to determine encoding parameters.

-1,  --2-pct,  --fast

Sample 2% of the data (up to 400 samples) to determine encoding parameters.

-B,  --debug-buffer=bufsize

read or write raw data from a debugging buffer of size bufsize bytes.  This is not the optimal way to compress or expand files (though the price is small).  This option is only intended to exercise and debug parts of the code that would be used in a planned slimlib library of functions.

The slim executable produces slim files with several restrictions that are not inherent to the definition of the slim file format itself.  The expander can expand files meeting the more general specification, but there is no way at this time to construct these more general files.  

Here we list some ways that the structure could be less restrictive than the current compression executable permits.  For one thing, the data type of different channels need not be the same, and the number of repeats can also be different.  The list of channels could change between one section and the next.  Also, the slim file specification allows all sections--not only the last--to be of arbitrary size. However, the current implementation of the slim executable does not offer a way to break up the raw data into sections at arbitrary places in the raw file.

Currently, slim enforces a limit that sections be no longer than 16 MB (2^24 bytes), because of the implementation detail that sections must be held in memory when compressing or expanding them.  If data comprising what is conceptually one section exceed this limit, then slim silently divides it into two or more sections of nearly equal length and writes them one after another.  This fact is not generally relevant to the user, but it does mean that compression will not be very effective if there are channels that repeat only a few times per 16 MB of raw data.

Three other algorithms were implemented into slim and later removed.  Each of them improves on the compression ratios of the reduced-binary code, but the improvements are small and come at a price in compression and expansion speed (not to mention program complexity).  Development on these algorithms was dropped for this reason, and we mention them in case you are also pondering ideas for improved encoding algorithms in slim.

Codes A and B

Codes A and B are minor variations on the reduced-binary code.  The former changes how the presence of overflows is signaled, and the latter also changes how their overflowing values are stored.

Code A arises from the observation that overflow values might be rare, but they are still more common than any other single value.  The idea is to encode most normal values as an N-bit number but the overflow as an m-bit number, where m<N. The result is to make each overflow take up (N-m) fewer bits, while the allowed range of values contains only 2^(N)-2^(N-m) values, slightly fewer than it would have had otherwise.  This results in savings of approximately 0.15 to 0.20 bits per channel for normally-distributed data.  However, the method was removed from slim because of its CPU cost: decoding each normal value requires first reading m bits, testing whether they are the overflow code, and (if not) then reading in an additional (N-m) bits to find the full N-bit number.

Code B expands on Code A.  It starts with the observation that overflow values are not generally taken randomly from the full range of possible values, but instead are most likely found near the allowed range of non-overflowing values.  It is therefore beneficial to write the overflowed value not as a full 32-bit number (assuming the raw data are 32-bit numbers), but instead as an N-bit, or (N+1)-bit number.  Thus, overflows are encoded as in Code A, followed by a (prefix, value), where the prefix is an exponential Golomb code for the value's size (actually, the size with N subtracted, since the size of an overflowing value is guaranteed to be no less than N).  As with Code A, it improves compression by some fraction of one bit per value, but the savings were judged not to be worth the performance penalty.

Huffman coding

Huffman coding of the raw data clearly accomplished the best compression ratios of all methods.  However, it was also the slowest and so was removed from the program.  

The approach is to take the data and split it into "upper" and "lower" bits.  The upper bits are Huffman-coded, while the lower bits are assumed to be uniformly-distributed random values and are repeated verbatim into the compressed data stream.  The split into upper and lower is made such that the data sample contains no more than 127 distinct values for the upper bits.  The Huffman tree is built on the sampled frequencies of the values of the upper bits (with an extra symbol to signal overflows).  The choice of 127 symbols was motivated by the possibility of storing the code tree with N symbols in only (N+1) bytes, and by the observation that even as few as 32 symbols work just as well on truly Gaussian-distributed data.  

The price of decoding Huffman-encoded symbols is steep: we found that decoding took twice as long when the Huffman encoders were used.  The problem is that the prefix-free Huffman codes must be read one bit at a time, for there is no way to know their length in advance.

Bit-rotation is an advanced feature that helps in compressing channels where the lowest B bits are constant.  This sort of channel violates the usual assumption that data follow a smooth distribution function, and that the lowest few bits are likely to be distributed almost uniformly. The author of slim has certainly found channels in real experiments where (for example) the six least significant bits are always zero.  This might happen a case where an IEEE 754 floating-point number with 24 bits in its sign-plus-mantissa was converted to and recorded as an integer.

If such a channel were encoded by the reduced-binary encoder, then the unchanging B lowest bits would be faithfully but wastefully reproduced over and over again.  Bit rotation offers a way around this redundancy.

Bit rotation has slim encode such a channel by bit-shifting the raw value by B bits to the right and then copying what were the B lowest bits to the top (most significant bits) of the shifted result. This makes, for example, the value 0xbeef9900 into 0x00beef99 for the case of B=8. The reduced binary encoder can then act on the bit-rotated result, in which the lowest bits of the new value are presumably less repetitive than they were before bit rotation was applied.

The user selects only whether to try bit rotation with the --permit-bitrotation option.  The program then decides whether to choose a non-zero value of B based on the sampled data set.  The choice will be the number of lowest-order bits that are strictly constant throughout the sample set.

The penalty for trying bit rotation on every channel is modest when compressing--causing roughly a 1.5% increase in instructions executed, in one test.  There is of course no penalty in expanding unless bit rotation was actually used.