scout - Man Page

scout.ser [OPTION] (ANCHOR_FILE | EXPERIMENT_DIR)
scout.omp [OPTION] (ANCHOR_FILE | EXPERIMENT_DIR)
scout.mpi [OPTION] (ANCHOR_FILE | EXPERIMENT_DIR)
scout.hyb [OPTION] (ANCHOR_FILE | EXPERIMENT_DIR)

scout is the scalable automatic trace-analysis component of the Scalasca Trace Tools. In particular, it provides the ability to

• identify wait states in communication and synchronization operations that occur, for example, as a result of unevenly distributed workloads,
• pinpoint the root causes of those wait states (i.e., delays), and
• identify the activities on the critical path of the target application, highlighting those routines which determine the length of the program execution and therefore constitute the best candidates for optimization.

Usually, scout is launched automatically by the Scalasca measurement collection and analysis nexus scan(1) after a successful measurement if event tracing is configured. However, it can also be run manually on an existing event trace measurement.

scout currently supports trace experiments in two different event trace formats: OTF2 traces generated by the Score-P instrumentation and measurement system, and legacy traces in EPILOG format generated by the measurement system of the Scalasca 1.x release series. For OTF2 event traces, scout has to be provided with the corresponding ANCHOR_FILE (e.g., 'traces.otf2'), for EPILOG traces with the experiment directory name EXPERIMENT_DIR.

Depending on the build configuration and the capabilities of the target platform, the scout analyzer may be available in up to four forms:

scout.ser

is always built. It is used to analyze event traces generated by serial applications. It can also be used to analyze event traces from multi-threaded applications, however, it will then only provide information about the master thread.

scout.omp

is built whenever the Scalasca Trace Tools are configured with OpenMP support. It is used to analyze event traces generated by pure multi-threaded applications (e.g., using OpenMP or POSIX threads). It can also be used to analyze event traces from serial applications, though analysis incurs a higher overhead than using scout.ser.

scout.mpi

is built whenever the Scalasca Trace Tools are configured with MPI support. It is used to analyze event traces generated by pure MPI applications. It can also be used on traces from multi-threaded MPI applications, however, it will then only provide information about the master thread of each process and its MPI activities.

scout.hyb

is built if the Scalasca Trace Tools are configured with both MPI and OpenMP support. It is used to analyze event traces generated by multi-threaded MPI applications (e.g., MPI+OpenMP or MPI+Pthreads), providing information about all OpenMP/POSIX threads of each MPI process. It can also be used on traces from pure MPI applications, though analysis incurs a slightly higher overhead than using scout.mpi.  

Note that scout.mpi and scout.hyb are implemented as MPI programs, and therefore have to be executed using appropriate MPI launch commands and flags. Also, the number of MPI processes for scout must be identical to the number of MPI processes used for the target application execution.

If successful, scout produces the following output files in the measurement archive directory:

• scout.cubex: the trace analysis result
• trace.stat: trace analysis pattern statistics

scout accepts a number of command-line options to enable/disable particular analysis features. When scout is launched automatically from the Scalasca measurement collection and analysis nexus scan(1), these options can be passed to the analyzer via the SCAN_ANALYZE_OPTS environment variable.

scout exits with status 0 if automatic trace analysis was successful, and greater than 0 if errors occur.

scout poses a number of requirements on the input event trace data, which are documented in the OPEN_ISSUES file installed as part of the Scalasca Trace Tools documentation. It is also available online at https://www.scalasca.org. If those requirements are not met, scout may abort, deadlock, or crash.

If scout crashes or deadlocks even though the documented requirements are met (which usually indicates a bug), restricting the scope of the analysis by disabling certain features (e.g., critical-path and/or root-cause analysis) may help as a workaround. In any case, please report such issues for further investigation (see Bugs below).

Please report bugs to scalasca@fz-juelich.de. Make sure to include at least the following information in your bug report:

• The Scalasca Trace Tools version reported by 'scalasca -V'.
• The Scalasca Trace Tools configuration reported by 'scalasca -c'.
• The Score-P version reported by 'scorep --version'.
• The Score-P configuration reported by 'scorep-info config-summary'.
• The exact command line of the failing command.
• The exact failure/error message.

Also, please archive a copy of the entire experiment archive directory including the event trace data, as this may be required to aid in debugging. However, ONLY PROVIDE TRACE DATA IF EXPLICITLY REQUESTED, as the data volume may be excessive.

scout.omp scorep_foo_Ox4_trace/traces.otf2
Perform the Scalasca OpenMP event trace analysis on the OTF2 event trace with anchor file scorep_foo_Ox4_trace/traces.otf2.

mpiexec -n 16 scout.mpi --time-correct scorep_bar_16_trace/traces.otf2
Apply the enhanced timestamp correction and perform the Scalasca MPI event trace analysis on the OTF2 event trace with anchor file scorep_bar_16_trace/traces.otf.

mpiexec -n 4 scout.hyb epik_foobar_4x4_trace
Perform the hybrid Scalasca MPI+OpenMP event trace analysis on the EPILOG event trace in the experiment archive epik_foobar_4x4_trace generated by the Scalasca 1.x release series.

scalasca(1), scan(1), square(1)

The full Scalasca Trace Tools documentation is available online at https://www.scalasca.org.

The man pages scout.hyb(1), scout.mpi(1), scout.omp(1) and scout.ser(1) are aliases of scout(1).