fi_cntr - Man Page

fi_cntr_open allocates a new fabric counter. The properties and behavior of the counter are defined by struct fi_cntr_attr.

struct fi_cntr_attr {
    enum fi_cntr_events  events;    /* type of events to count */
    enum fi_wait_obj     wait_obj;  /* requested wait object */
    struct fid_wait     *wait_set;  /* optional wait set */
    uint64_t             flags;     /* operation flags */
};

events

A counter captures different types of events. The specific type which is to counted are one of the following:

- FI_CNTR_EVENTS_COMP

The counter increments for every successful completion that occurs on an associated bound endpoint. The type of completions – sends and/or receives – which are counted may be restricted using control flags when binding the counter and the endpoint. Counters increment on all successful completions, separately from whether the operation generates an entry in an event queue.

- FI_CNTR_EVENTS_BYTES

The counter is incremented by the number of user bytes, excluding any CQ data, transferred in a transport message upon reaching the specified completion semantic. For initiator side counters, the count reflects the size of the requested transfer and is updated after the message reaches the desired completion level (FI_INJECT_COMPLETE, FI_TRANSMIT_COMPLETE, etc.). For send and write operations, the count reflects the number of bytes transferred to the peer. For read operations, the count reflects the number of bytes returned in a read response. Operations which may both write and read data, such as atomics, behave as read operations at the initiator, but writes at the target. For target side counters, the count reflects the size of received user data and is incremented subject to target side completion semantics. In most cases, this indicates FI_DELIVERY_COMPLETE, but may differ when accessing device memory (HMEM). On error, the tranfer size is not applied to the error field, that field is increment by 1. The FI_COLLECTIVE transfer type is not supported.

wait_obj

Counters may be associated with a specific wait object. Wait objects allow applications to block until the wait object is signaled, indicating that a counter has reached a specific threshold. Users may use fi_control to retrieve the underlying wait object associated with a counter, in order to use it in other system calls. The following values may be used to specify the type of wait object associated with a counter: FI_WAIT_NONE, FI_WAIT_UNSPEC, FI_WAIT_SET, FI_WAIT_FD, FI_WAIT_MUTEX_COND, and FI_WAIT_YIELD. The default is FI_WAIT_NONE.

- FI_WAIT_NONE

Used to indicate that the user will not block (wait) for events on the counter.

- FI_WAIT_UNSPEC

Specifies that the user will only wait on the counter using fabric interface calls, such as fi_cntr_wait. In this case, the underlying provider may select the most appropriate or highest performing wait object available, including custom wait mechanisms. Applications that select FI_WAIT_UNSPEC are not guaranteed to retrieve the underlying wait object.

- FI_WAIT_SET

Indicates that the event counter should use a wait set object to wait for events. If specified, the wait_set field must reference an existing wait set object.

- FI_WAIT_FD

Indicates that the counter should use a file descriptor as its wait mechanism. A file descriptor wait object must be usable in select, poll, and epoll routines. However, a provider may signal an FD wait object by marking it as readable, writable, or with an error.

- FI_WAIT_MUTEX_COND

Specifies that the counter should use a pthread mutex and cond variable as a wait object.

- FI_WAIT_YIELD

Indicates that the counter will wait without a wait object but instead yield on every wait. Allows usage of fi_cntr_wait through a spin.

wait_set

If wait_obj is FI_WAIT_SET, this field references a wait object to which the event counter should attach. When an event is added to the event counter, the corresponding wait set will be signaled if all necessary conditions are met. The use of a wait_set enables an optimized method of waiting for events across multiple event counters. This field is ignored if wait_obj is not FI_WAIT_SET.

flags

Flags are reserved for future use, and must be set to 0.

The fi_close call releases all resources associated with a counter. When closing the counter, there must be no opened endpoints, transmit contexts, receive contexts or memory regions associated with the counter. If resources are still associated with the counter when attempting to close, the call will return -FI_EBUSY.

The fi_cntr_control call is used to access provider or implementation specific details of the counter. Access to the counter should be serialized across all calls when fi_cntr_control is invoked, as it may redirect the implementation of counter operations. The following control commands are usable with a counter:

FI_GETOPSFLAG (uint64_t *)

Returns the current default operational flags associated with the counter.

FI_SETOPSFLAG (uint64_t *)

Modifies the current default operational flags associated with the counter.

FI_GETWAIT (void **)

This command allows the user to retrieve the low-level wait object associated with the counter. The format of the wait-object is specified during counter creation, through the counter attributes. See fi_eq.3 for addition details using control with FI_GETWAIT.

The fi_cntr_read call returns the current value of the counter.

The read error call returns the number of operations that completed in error and were unable to update the counter.

This adds the user-specified value to the counter.

This adds the user-specified value to the error value of the counter.

This sets the counter to the specified value.

This sets the error value of the counter to the specified value.

This call may be used to wait until the counter reaches the specified threshold, or until an error or timeout occurs. Upon successful return from this call, the counter will be greater than or equal to the input threshold value.

If an operation associated with the counter encounters an error, it will increment the error value associated with the counter. Any change in a counter’s error value will unblock any thread inside fi_cntr_wait.

If the call returns due to timeout, -FI_ETIMEDOUT will be returned. The error value associated with the counter remains unchanged.

It is invalid for applications to call this function if the counter has been configured with a wait object of FI_WAIT_NONE or FI_WAIT_SET.