uberftp - Man Page

uberftp [options] [host options] [host]

uberftp [options] [host options] host “cmd”

uberftp [options] srcurl dsturl

uberftp [options] -f urlfile

uberftp [options] -cmd url

uberftp is a GridFTP-enabled client that supports both interactive use and  FTP commands on the uberftp command line to transfer files between two  computers. It is intended for use with computers that have a GridFTP  server installed. Uberftp supports GSI authentication, parallel data channels and striping. For more information about GridFTP, see the GridFTP URL in the "See Also" section below.

Only the first usage shown above will create an interactive session. If host is specified, uberftp immediately attempts to establish a connection to the GridFTP server on  host,  enters its command interpreter and awaits commands from the user. If host is not specified, uberftp immediately drops into the command interpreter without connecting to any GridFTP server.

The second usage option denotes a non interactive session in which “cmd” is a series of one or more commands as described in the Commands section below to run once the control connection is established. These  commands are run exactly as if they had been entered from the interactive prompt. This list must be enclosed in quotes. Multiple commands are semicolon or comma delimited. uberftp will execute these commands and then exit. Uberftp will exit upon the first error encountered.

The third and forth usage statements use the URL style format for specifying the source and destination for the files to transfer. The third usage statement places these URLs on the command line. The forth usage allows the user to  specify multiple URL pairs in a separate file for Uberftp to transfer one at a time. The supported URL syntaxes are gsiftp://[user@]host[:port]/file, ftp://[user[:pass]@]host[:port]/file and file://path.

The fifth usage statement allows for commands that take pathnames to accept URLs instead. The allowable commands are listed in the -cmds section below.

-P port

Connect to port instead of the default. The default for GSI  authentication is 2811. The default for password authentication is 21.

-u user

Connect as user. This is useful for both password and GSI authentication mechanisms.

-p pass

Use pass when authenticating. If pass equals X, UberFTP will prompt for the password with character echoing turned off.

-active

Use ACTIVE mode for data transfers.

-ascii

Use ASCII mode for data transfers.

-binary

Use BINARY mode for data transfers.

-blksize n

Set the internal buffer size to n.

-cksum [on|off]

Enable/Disable CRC checks after file transfers.

-cos name

Set the storage class of service to name. Used with HPSS installations. Use the class of service name default to allow the remote server to decide which class of service to use.

-d

Enable debugging. Same as '-debug 3'. Deprecated.

-debug n

Set the debug level to n.

-family name

Set the storage family to name. Use the family name default to allow the remote server to decide which family to use.

-glob [on|off]

Enable/Disable filename expansion.

-hash

Enable printing of hash marks during transfers.

-keepalive n

Send control channel keepalive messages every n seconds during data transfers.

-mode [E|S]

Switch the transfer mode to extended block (E) or streams mode (S).

-parallel n

Use n parallel data channels during extended block transfers.

-passive

Use PASSIVE mode for data transfers.

-pbsz n

Set the data protection buffer size to n n bytes.

-prot [C|S|E|P]

Set the data protection lelvel to clear (C), safe (S), confidential (E) or private (P).

-retry n

Retry commands that fail with transient errors n times.

-resume path

Retry the recursive transfer starting at path.

-tcpbuf n

Set the TCP read/write buffers to n bytes.

-wait

This will cause the client to wait for remote files to stage before attempting to transfer them.

-v

Print UberFTP version information and exit.

-version

Print UberFTP version information and exit.

-versions

Print version information about all used globus modules and exit.

-cat url

Print to stdout the contents of the remote file. See Cat Behaviour below for details about the behaviour of this functionality.

-chgrp [-r] group url

Set the group ownership of the remote object(s).

-chmod [-r] perms url

Set the permissions of the remote object(s).

-dir [-r] url

List the contents of the remote object.

-link url path

Create a hardlink named <path> to the remote object.

-ls [-r] url

List the contents of the remote object.

-mkdir url

Create the remote directory.

-rename url path

Rename the remote object to the given path.

-rm [-r] url

Remove the remote object(s).

-rmdir url

Remove the remote directory.

-size url

Return the size of the remote object.

-stage [-r] seconds url

Attempt to stage the remote object(s) over the time period given in seconds.

-symlink url path

Create a symlink named <path> to the remote object.

By default, without any special environment variables, command line options or commands, uberftp will transfer files in PASSIVE STREAMS mode. PASSIVE means that the client will initiate the data connection which is useful for users behind firewalls. STREAMS mode implies that GRIDFTP features including striping and parallel data connections are not used. In order to take advantage of these features with GridFTP capable servers, you must either change the mode directly using -m command line switch or the mode interactive command, or you can change the mode indirectly by specifying  more than one parallel data connection using the -c command line switch or by using the parallel interactive command.

By default, uberftp requires a GSI certificate. If you do not already have a certificate, see the following web page to learn how to get one:

http://www.ncsa.uiuc.edu/UserInfo/Grid/Security/GetUserCert.html

Once you have a certificate, use the grid-proxy-init command to get a valid proxy.

! [command]

Run the command using a shell on the local machine. If no command is given, invoke an interactive shell.

? [command]

If command is given, print a (hopefully) helpful blurb about it. Otherwise, list all commands.

active

Change to ACTIVE mode which causes the server to initiate the data connection. The default is PASSIVE mode unless the variable UBERFTP_ACTIVE_MODE is set in the environment. If you are behind a firewall you must use PASSIVE mode.

ascii

Change the data transfer type to ASCII which causes the server to do some simple transformations to the file being transferred. This is mostly useful for changing EOL (end of line) in text files when moving between platforms. This option is almost never necessary today. The default is BINARY mode also known as IMAGE mode.

binary

Change the data transfer type to BINARY (aka IMAGE) which causes the server to not perform transformations to the file being transferred. This is the default and is faster than an ASCII transfer.

blksize size

Change the size of the memory buffer used to read and write data to disks to size bytes. The default block size is 1024*1024 (1048576) bytes but it can be changed at compile time. The block size can be increased to improve file transfer performance. This is not related to the extended block mode block size used to determine the ratio of data to header for data transferred on the data channel.

bugs

Prints information regarding bug reporting and feature requests.

bye

Close all control and data connections and exit.

cat file1 [file2 ... filen]

Print the contents of the remote file(s) to stdout. See Cat Behaviour below for details about the behaviour of this functionality.

cdup

Change the remote working directory up one level.

cd [dir]

Change the remote working directory to dir. If dir is not given, the client will make every attempt to change to the user's home directory.

chgrp [-r] group object [object2 ... objectn]

Change group ownership on the remote object(s).
-r   Recursively chgrp everything in the given directory.

chmod [-r] perms object [object2 ... objectn]

Change permissions on the remote object(s).
-r   Recursively chmod everything in the given directory.

close

Close the control connection to the remote host.

cksum [on|off]

Enable file cksum comparison after each file transfer. This only works with NCSA's mass storage system.
on    Enable checksum comparison
off   Disable checksum comparison

cos name

Sets the HPSS class of service to name on the FTP service if the service supports it. If name is omitted, the current class of service is printed. Use the class of service name default to allow the remote server to decide which class of service to use.

dcau [N|A|S subject]

Change the data channel authentication settings. If the service does not support DCAU, these settings are ignored.
N  Disabled dcau.
A  Expect the remote identity to be mine. (Default)
S subject Expect the remote identity to be subject.

debug [0-3]

Turn debug statements on/off. If no value is given, this command will toggle between debug(2) and non debug(1) mode. Otherwise the debug level is set to the given level.
0  Only errors are printed
1  Default. Errors and some helpful messages are printed
2  Print useful control channel information
3  Print all information

family name

Sets the tape family to name on the FTP service if the service supports it. If name is omitted, the current family is printed. Use the family name default to allow the remote server to decide which family to use.

glob [on|off]

Enable or disable filename globbing. If no option is given, this command will toggle the current setting.
on    Enable filename globbing
off   Disable filename globbing

dir [-r] [target]

List the contents of the remote target directory. If target is not given, then the current working directory is used.
-r      Recursively list target.
target  Directory or file to list. '.' is used by default.

get [-r] source [destination]

Retrieve file(s) from the remote service. If source implies multiple transfers, either through regular expressions or by using the recursive feature, then destination must be a directory. If destination is not specified, source is used.
-r   Recursively transfer the given directory.

hash

Print hash marks during data transfers. This does not work during third party transfers.

help [command]

If command is given, print a helpful blurb about command. Otherwise, list all commands.

keepalive [seconds]

Attempts to keep the control channel from being blocked by firewalls during long data channel operations. UberFTP sends a NOOP command to the service at intervals equal to the specified number of seconds. Setting it to zero will disable keepalive. If seconds are not given, the current timeout is displayed. This feature is disabled by default.
seconds  number of seconds between NOOPs. Disabled if zero.

lcat file1 [file2 ... filen]

Print the contents of the local file(s) to stdout. See Cat Behaviour below for details about the behaviour of this functionality.

lcd [dir]

Change the local working directory to dir. If dir is not given, the client will make every attempt to change to the user's home directory.

lcdup

Change the local working directory up one level.

lchgrp [-r] group object [object2 ... objectn]

Change group ownership on the local object(s).
-r   Recursively chgrp everything in the given directory.

lchmod [-r] perms object [object2 ... objectn]

Change permissions on the local object(s).
-r   Recursively chmod everything in the given directory.

lclose

Close the control connection to the local host.

ldir [-r] [target]

List the contents of the local target directory. If target is not given, then the current working directory is used.
-r      Recursively list target.
target  Directory or file to list. '.' is used by default.

link [oldfile] [newfile]

Create a hardlink to oldfile named newfile on the remote service.

llink [oldfile] [newfile]

Create a hardlink to oldfile named newfile on the local service.

lls [-r] [target]

List the contents of the local target directory. If target is not given, then the current working directory is used.
-r      Recursively list target.
target  Directory or file to list. '.' is used by default.

llscos

List the available class of services on the local server (HPSS only).

llsfam

List the available tape families on the local server (HPSS only).

lmkdir dir1 [dir2 ... dirn]

Create the local directory(ies).

lopen [-P port] [-u user] [-p pass | X] host

Opens a control channel to host and that host becomes the 'local' machine. After using lopen, all local (l*) commands perform their respective operations on host rather than the local machine. This is how third party transfers are accomplished. GSI authentication is used unless the -p option is used.
-P port   Connect to port (Default 2811 for GSI, 21 for password).
-u user   Connect as alternate user.
-p pass | X
         Use password pass when authenticating with host.
         If pass equals X, read the password from STDIN with
         character echoing turned off.
host      Connect to host.

lpwd

Prints the current local working directory.

lrename src dst

Rename the local object src to dst.

lrm [-r] object1 [object1...objectn]

Removes the local file system object(s).
-r   Recursively remove the given directory.

lrmdir dir1 [dir2...dirn]

Removes the given directories from the local service.

lquote cmd

Pass cmd to the local FTP service. This allows the user to use server-specific commands that are not available through the uberftp interface.

ls [-r] [target]

List the contents of the remote target directory. If [target] is not given, then the current working directory is used.
-r      Recursively list target.
target  Directory or file to list. '.' is used by default.

lscos

List the available class of services on the remote server (HPSS only).

lsfam

List the available tape families on the remote server (HPSS only).

lsize file1 [file2...filen]

Prints the size of the given object(s).

lstage [-r] seconds object1 [object2...objectn]

Attempt to stage all matching files within the given number of seconds on the local service.
seconds  number of seconds to attempt staging
-r       Recursively stage all files in the given subdirectory.

lsymlink [oldfile] [newfile]

Create a symlink to oldfile named newfile on the local service.

mput [-r] object1 [object2...objectn]

Retrieve file(s) from the remote service. This is similiar to making multiple calls to get without specifying a destination.
-r   Recursively transfer the given directory.

mkdir dir

Create the remote directory.

mode [E|S]

Toggle the data transfer mode between Streams mode and Extended Block mode. The default is Streams mode. If no option is given, it will display the current mode.
E   Extended block mode
S   Streams mode

mput [-r] object1 [object2...objectn]

Store file(s) to the remote service. This is similiar to making multiple calls to put without specifying a destination.
-r   Recursively transfer the given directory.

open [-P port] [-u user] [-p pass | X] host

Opens a control channel to host and that host becomes the 'remote' machine. GSI authentication is used unless the -p option is used.
-P port   Connect to port (Default 2811 for GSI, 21 for password).
-u user   Connect as user.
-p pass | X
         Use password pass when authenticating with host.
         If pass equals X, read the password from STDIN with
         character echoing turned off.
host      Connect to host.

order [type]

Changes the order of lists returned from ls and lls to the given scheme. If type is not given, the current order is displayed.
type    Ordering scheme to use. Value options are:
          none  Do not order listings
          name  Order listings by name
          size  Order listings by size
          type  Order listings by type

parallel [number]

Set the number of parallel data connections to number. This is only useful for extended block mode transfers. The default number of data connections is one. If no number is given, the current setting for the number of parallel connects is printed.

passive

Change to PASSIVE mode which causes the client to initiate the data connection. This is the default mode unless the variable UBERFTP_ACTIVE_MODE is set in the environment. If you are behind a firewall you must use PASSIVE mode.

pbsz [size]

Change the length of the protection buffer. The protection buffer is used to encrypt data on the data channel. The length of the protection buffer represents the largest encoded message that is allowed on the data channel. By default, the protection buffer is grown to match the internal buffer used. For efficient transfers, pbsz should be sufficiently larger than blksize so that the wrapped buffer fits within the protection buffer. Otherwise, the blksize buffer is broken into multiple pieces so that each write is less than pbsz when wrapped. If pbsz is not given, the current size is displayed.
size   length of protection buffer. 0 will set it to its default.

pget offset size srcfile [destfile]

Retrieve only the specified portion of the file(s). If srcfile is a regular expression and expands to multiple files, and destination is given, destination must refer to a directory.
offset   Offset within the file
size     Amount of data to retrieve
srcfile  Name of remote file
destfile Name of local file. srcfile is used if destfile
is not specified

pput offset size srcfile [destfile]

Store only the specified portion of the file(s). If srcfile is a regular expression and expands to multiple files, and destination is given, destination must refer to a directory.
offset   Offset within the file
size     Amount of data to retrieve
srcfile  Name of local file
destfile Name of remote file. srcfile is used if destfile
        is not specified

prot [C|S|E|P]

This command configures the level of security on the data channel after data channel authentication has completed. Clear means that the data will not be protected. Safe means that the data will be integrity protected meaning that altered data will be detected. Confidential means that the data will be unreadable to third parties. Private mode means the data will be confidential and safe.
C  Set protection level to clear.
S  Set protection level to safe.
E  Set protection level to confidential.
P  Set protection level to private.

put [-r] source [destination]

Store file(s) to the remote service. If source implies multiple transfers, either through regular expressions or by using the recursive feature, then destination must be a directory. If destination is not specified, source is used.
-r   Recursively transfer the given directory.

pwd

Prints the current working directory.

quit

Close all control and data connections and exit.

quote cmd

Pass cmd to the remote FTP service. This allows the user to use server-specific commands that are not available through the uberftp interface.

rename src dst

Rename the remote object src to dst.

retry [cnt]

Configures retry on failed commands that have transient errors. cnt represents the number of times a failed command is retried. A value of zero effectively disables retry. Zero is the default. If no value is given the current setting is displayed.
cnt    Number of times a failed command is retried.

resume [-d] path

Sets a restart point for recursive transfers. If a long recursive transfer fails, you can set resume to the path that failed and UberFTP will skip all file and directory creations up to the given path.
path   Path to resume transfer at. If path is not given, print the current
      resume target.
-d     Remove the current resume path.

rm [-r] object1 [object1...objectn]

Removes the remote file system object(s).
-r   Recursively remove the given directory.

rmdir dir1 [dir2...dirn]

Removes the given directories from the remote service.

runique

Toggles the client to store files using unique names during put operations.

size file1 [file2...filen]

Prints the size of the given object(s).

stage [-r] seconds object1 [object2...objectn]

Attempt to stage all matching files within the given number of seconds on the remote service.
seconds  number of seconds to attempt staging
-r       Recursively stage all files in the given subdirectory.

sunique

Toggles the client to store files using unique names during get operations.

symlink [oldfile] [newfile]

Create a symlink to oldfile named newfile on the remote service.

tcpbuf [size]

Set the data channel TCP buffer size to size bytes. If size is not given, the current TCP buffer size will be printed.

versions

Prints the versions of all Globus modules being used.

wait

Toggles whether the client should wait for files to stage before attempting to retrieve them.

Use the active command to enable active mode FTP when  using NCSA's UniTree mass storage system if possible since it  will give much better file transfer performance. When tranferring files over long distances, use a large value (for example,  16777216) for tcpbuf. When there is high network traffic, you may be able to improve  performance using the parallel command to increase the number of parallel data connections to 2-4.

In order to perform a third-party transfer, you must log into two  FTP servers. Typically, you connect to a single FTP service to  "get" files to the local machine and "put" files to the remote service.  For third-party transfers, you must connect to a second service  thereby replacing the former local machine. In UberFTP terminology,  it is referred to as "opening a new local service" since, from  the perspective of the user, the new local service will appear  as though the user initiated the FTP session from that machine.

All remote service commands have "l*" counterparts that allow you  to specify that the command is to be performed on the local service,  whether that service is the local machine or a new local service.  So to open a new local service, use the "l*" version of the open command:

 UberFTP> lopen mss.ncsa.teragrid.org
 UberFTP> lclose
      Once you have connected to both services, files can be transferred as  before with the change that files you "get" go to the new local service  and files you "put" are sent from the new local service.

By default, local port selection is managed by the operating system. However, you may wish to specify which ports UberFTP should use for incoming and  out going connections. This is useful when dealing with firewalls.

Setting UBERFTP_TCP_PORT_RANGE in your environment will cause all inbound connections to use the specified port range. Likewise, setting UBERFTP_TCP_SOURCE_RANGE in your environment will cause all outbound connections to use the specified port range.

The environment variables GLOBUS_TCP_PORT_RANGE and GLOBUS_TCP_SOURCE_RANGE will also control the ephemeral port selection. These variables behave exactly as their UBERFTP counterparts and are available for backwards compatibility with older versions. The UBERFTP variables take precedence over the GLOBUS variables.

The values of the variables specify a port range, a minimum port number and a maximum port number, separated by either a comma or a space. For example, to set the inbound port range, you would set:

 UBERFTP_TCP_PORT_RANGE=40000,50000

Using the space delimiter, this format is also acceptable:

 UBERFTP_TCP_PORT_RANGE="40000 50000"

See your shell documentation for the proper syntax for settings variables within your environment.

Setting the ephemeral port range to an unusable range will cause UberFTP connections to fail. For instance, setting a port range from 10 to 100 with a non root process will fail on most operating systems.

UberFTP 2.9.1 comes with an additional environment variable:

 UBERFTP_CAT_CORRECT

If this environment variable is present UberFTP omits the odd newline that is otherwise printed out in addition to the file contents by the cat functionality of previous versions of UberFTP for non-empty files. If it is not present, the old behaviour is used for compatibility reasons. The content of this environment variable is not evaluated, just its presence in your environment. Also see:

 https://github.com/gridcf/UberFTP/issues/22

UberFTP will exit with a value of 0 if no errors occurred during the session, otherwise it will exit with a value of 1. In non interactive, commandline mode, it will exit after the first error occurs.

Set the environment variable to set active mode FTP  (improves file transfer performance to the mass storage system). Connect to NCSA's UniTree mass storage system interactively from  NCSA's TeraGrid cluster:

 setenv UBERFTP_ACTIVE_MODE on
 % uberftp mss.ncsa.teragrid.org
 ...
 220 UNIX Archive FTP server ready.
 230 User consult logged in.
 UberFTP>

Use the command-line interface to copy a file from NCSA's TeraGrid cluster  to the UniTree mass storage system. (There is no need to set tcpbuf since it is over a LAN but active mode is turned on to improve file transfer performance to the mass storage system.):

 uberftp mss.ncsa.teragrid.org \
    "active; cd work; get file.tar"

Copy a file from SDSC's TeraGrid cluster to NCSA's TeraGrid cluster. (Note that tcpbuf is set to 16777216 since there is a long network latency between NCSA and SDSC):

 uberftp tg-gridftp.sdsc.teragrid.org \
    "tcpbuf 16777216; cd scr; put file.tar"

mssftp(1), msscmd(1), ftp(1),
GridFTP:
 https://gridcf.org/gct-docs/latest/gridftp/
TCP Window Size:
 http://www.vonwelch.com/report/tcp_windows/
 http://www.psc.edu/tcp-tune
Active vs. Passive FTP:
 http://slacksite.com/other/ftp.html

Note: The links above are not under the GridCF's control so they may become obsolete.