LOGARCHIVE - Man Page

The volume and .meta files are divided into self-identifying records.

All three types of files begin with an “archive label” header, which identifies the host name, starting timestamp and timezone information; all referring to the host that was the source of the performance data (which may be different to the host where pmlogger(1) was running).

The “archive label” format differs between Version 2 and Version 3, with the latter providing enhanced timestamps (64-bit encoding of the seconds part and nanosecond precision) and some additional fields.

The “archive feature bits” are intended to encode possible future extensions or differences to the on-disk structure or the the archive semantics. At this stage there are no such features, but if they are introduced at some point in the future, there will be associated PM_LOG_FEATURE_XXX macros added to the <pcp/pmapi.h> header file.

All fields, except for the “current archive volume number”, match for all files in a single PCP archive.

After the archive label record, an archive volume file contains one or more records, each providing metric values corresponding to the pmResult from one pmFetch(3) operation. The record size may vary according to number of metrics being fetched and the number of instances in the associated instance domains.

For Version 2 the file size is limited to 2GiB, due to storage of 32-bit byte offsets within the temporal index. For Version 3 the file size is limited to 8191PiB, due to storage of 62-bit byte offsets within the temporal index.

The pmResult format differs between Version 2 and Version 3, with the latter providing enhanced timestamps (64-bit encoding of the seconds part and nanosecond precision).

Records with a “number of metrics” equal to zero are “mark records”, and represent interruptions, missing data, or time discontinuities in logging.

This subrecord represents the values for one metric at one point in time.

The metadata describing metrics is found in the .meta file where the entries are not timestamped, as the metadata is assumed to be unchanging throughout an archive.

This subrecord represents one value for one instance of a metric at one point in time. It is a variant type, depending on the parent pmValueSet's value format field. This allows small numbers to be encoded compactly, but retain flexibility for larger or variable length data to be stored later in the pmResult record in a pmValueBlock subrecord.

The metadata describing the instance domain for metrics is found in the .meta file. Since the numeric mappings may change during the lifetime of the logging session, it is important to match up the timestamp of the measurement record with the corresponding instance domain record. That is, the instance domain corresponding to a measurement at time T is the instance domain observation for the metric's instance domain with largest timestamp T' <= T.

Instances of this subrecord are placed at the end of the pmValueSet, after all the pmValue subrecords. If (and only if) needed, they are padded at the end to the next 32-bit boundary.

Note that for PM_TYPE_STRING, the length includes an explicit NULL terminator byte. For PM_TYPE_EVENT, the value byte string is further structured. Refer to PMDAEVENTARRAY(3) for more information about how arrays of event records are packed inside a pmResult container.

Instances of this (pmDesc) record provide the description or metadata for each metric appearing in the PCP archive. This metadata includes the metric's PMID, data type, data semantics, instance domain identifier (or PM_INDOM_NULL for singular metrics with only one value) and a set of (1 or more) names.

A set-valued metric is defined over an instance domain, which consists of an instance domain identifier (will have already been mentioned in a prior pmDesc record), a count of the number of instances and a map that defines the association between internal instance identifiers (integers) and external instance names (strings).

Because instance domains can change over time, the instance domain also requires a timestamp, and the same instance domain can occur multiple times within the .meta file. The timestamps are used to search for the temporally correct instance domain when decoding pmResult records from the archive data volumes, or answering metadata queries against the instance domain.

The instance domain format differs markedly between Version 2 and Version 3. Version 3 provides enhanced timestamps (64-bit encoding of the seconds part and nanosecond precision) and introduces a new “delta” instance domain format that encodes differences between the previous observation of the instance domain and the current state of the instance domain.

The “delta” instance domain record in Version 3 uses the same physical structure as the “full” instance domain above with the following differences:

• The tag is TYPE_INDOM_DELTA=6.
• The “number of instances in domain” field becomes the sum of the number of instances added and the number of instances deleted.
• Deleted instances are encoded with the string offset set to -1 and there is no corresponding string table entry.
• Added instances are encoded exactly the same way.

The “delta” instance domain format is used to provide a more compact on-disk encoding for instance domains that have a large number of instances and are subject to frequent small changes, e.g. the instance domain of process ids, as exported by pmdaproc(1).

For “full” instance domain records the instance domain replace the previous instance domain: prior records are not searched for instance domain metadata queries after this timestamp.

Each instance domain in a Version 3 archive must have an initial “full” instance domain record. Subsequent records for the same instance domain can be the `full'' or the “delta” variant. Any instance mentioned in the prior observation of an instance domain that is not mentioned in the “delta” instance domain record is assumed to continue to exist for the current observation of the instance domain.

Instances of this (pmLogLabelSet) record provide sets of label-name:label-value pairs associated with labels of the context, instance domains and individual performance metrics - refer to pmLookupLabels(3) for further details.

Any instance domain identifier will have already been mentioned in a prior pmDesc record.

As new labels can appear during an archiving session, these records are timestamped and must be searched when decoding pmResult records from the archive data volumes. The pmLogLabelSet format differs between Version 2 and Version 3, with the latter providing enhanced timestamps (64-bit encoding of the seconds part and nanosecond precision).

Records of this form replace the existing labels for a given label type: prior records are not searched for resolving that class of label in measurements after this timestamp.

The individual labelset array entries are variable length, depending on the number of labels present within that set. These entries contain the instance identifiers (in the case of type PM_LABEL_INSTANCES labels), lengths and offsets of each label name and value, and also any flags set for each label.

This (pmLogText) record stores help text associated with a metric or an instance domain - as provided by pmLookupText(3) and pmLookupInDomText(3).

The metric identifier and instance domain identifier will have already been mentioned in a prior pmDesc record.