mogtool - Man Page

mogtool -- Inject/extract data to/from a MogileFS installation

WARNING: this utility is deprecated! See MogileFS::Utils

    $ mogtool [general-opts] <command> [command-opts] <command-args>

    $ mogtool --trackers=127.0.0.1:6001 --domain=foo --class=bar ...
    $ mogtool --conf=foo.conf ...

    $ mogtool inject thefile.tgz thefilekey
    $ mogtool inject --bigfile thebigfile.tgz thefilekey
    $ mogtool inject --bigfile --gzip thebigfile.tar thefilekey
    $ mogtool inject --bigfile --gzip mydirectory thedirkey
    $ mogtool inject --bigfile --gzip /dev/hda4 thedevkey
    $ mogtool inject --nobigfile bigcontiguousfile bigcfilekey

    $ mogtool inject --bigfile --gzip --verify \
                     --description="Description" \
                     --receipt="foo@bar.com, baz@bar.com" \
                     --concurrent=5 --chunksize=32M \
                     somehugefile thefilekey

    $ mogtool extract thefilekey thenewfile.tgz
    $ mogtool extract thefilekey -
    $ mogtool extract --bigfile thedirkey .
    $ mogtool extract --bigfile --asfile thedirkey thefile.tgz
    $ mogtool extract --bigfile thedevkey /dev/hda4

    $ mogtool delete thekey
    
    $ mogtool locate --noverify thekey
    $ mogtool locate --bigfile thekey 

    $ mogtool list
    $ mogtool listkey key

--debug

Turn on MogileFS debug output.

--trackers=<[preferred_ip/]ip:port>[,<[preferred_ip/]ip:port>]*

Specify one or more trackers for your MogileFS installation.  Note that you can specify preferred IPs to override the default IPs with.  So it would look something like 10.10.0.1/10.0.0.1:8081.

--domain=<domain>

Set the MogileFS domain to use.

--class=<class>

Set the class within the domain to use.  Defaults to _default.

--conf=<file>

Specify a configuration file to load from.

--lib=<directory>

Specify a directory to use as a library path.  Right now, this should be the directory where you expect to find the MogileFS.pm file, if it's not actually installed.

inject|i

Insert a resource into MogileFS.  See "Inject Options" and "Inject Arguments" for the rest of how to use the inject mode.

extract|x

Extract a resource from MogileFS.  See "Extract Options" and "Extract Arguments" for how to use extract.

delete|rm

Delete a resource.  See "Delete Options" and "Delete Arguments".

locate|lo key

List the paths to the file identified by the given key.

list|ls

List all big files contained in MogileFS.  No options, no arguments.

listkey|lsk key

List all files which match the key. Key is just a prefix, and this will list all keys which match the prefix. So if you specify key as "ABC1" then you'll get all keys which start with the characters "ABC1"

The following options are used to control the behavior of the injector.

--bigfile|-b

If specified, use chunking to break the resource into manageable pieces.

--chunksize=<size>[B|K|M|G]

When instructed to break files into chunks, the injector will use the specified chunk size as the maximum chunk size.  Defaults to 64M.  You can specify the chunk size manually and specify the units--defaults to bytes.

--gzip|-z

If specified, mogtool will gzip the data as it's going into MogileFS.  The resource will be marked as compressed.

Note that you do not need to specify this if the resource is already gzipped, but it doesn't hurt.  (We automatically detect that and mark it as compressed.)

--overwrite

If you previously were working on injecting a big file as chunks and the process died, normally mogtool refuses to do it again.  Specify this option to force the overwrite of that file.

NOTE: Other than in the above case (partial failure), mogtool will not prompt before overwriting an existing file.

--verify

If on, we do a full MD5 verification of every chunk after it is replicated.  This can take a while on large files!

--description=<text>

Specifies a description for this file.  Optional, but assists in reporting and listing the large files in MogileFS.  (This is also displayed in any receipts that are created.)

--receipt=<email address>[, <email address>]*

If specified, emails a copy of the receipt file to the specified comma-separated email addresses.  Also creates a local filesystem copy of the receipt file.

--concurrent=<number>

Specifies the number of concurrent processes to run for MogileFS insertion.  If you are noticing mogtool spend most of it's time waiting for children and not actually buffering data, you may wish to raise this number.  The default is 1 but we've found 3 or 4 work well.

resource

What you actually want to inject.  This can be a file, directory, or a raw partition in the format /dev/X.

Please see "Usage Examples" for more information on how to inject these different types of resources and the differences thereof.

key

Specifies the key to save this file to.  For big files, the key is actually "_big_N:key" and "key,#" where N is one of a bunch of things we use and # is the chunk number.

Generally, you want this to be descriptive so you remember what it is later and can identify the file just by looking at the key.

--bigfile|-b

If specified, indicates that this resource was chunked on injection and should be reassembled for extraction.

--gzip|-z

Specifies to mogtool that it should ungzip the output if and only if it was compressed when inserted into the MogileFS system.  So, if you're extracting a file that wasn't gzipped to begin with, this doesn't do anything.

--asfile

Useful when extracting something previously inserted as a directory--this option instructs mogtool to treat the resource as a file and not actually run it through tar for decompression.

key

Specifies the key to get the file from.

destination

What destination means varies depending on what type of resource you're extracting. However, no matter what, you can specify a single dash (-) to mean STDOUT.

Please see the usage examples for more information on how extract works.

--bigfile|-b

The resource is a "big file" and all chunks should be deleted.

key

Specifies the key of the file to delete.

--verify

Verify that the returned paths actually contain the file.  The locate commands defaults to verify, you can disable it with --noverify.

--bigfile|-b

The resource is a "big file" and the locations of the information key should be printed.

key

Specifies the key of the file to locate

0

Success during operation.

1

During the locate, list, or listkey operation, the key was not found.

2

Some fatal error occurred.

Please note that all examples assume you have a default config file that contains the tracker and domain to use.  Saves us from having to clutter up the command line.

    $ mogtool inject foo.dbm foo.dbm.2004.12

    $ mogtool inject --gzip foo.dbm foo.dbm.2004.12

    $ mogtool extract foo.dbm.2004.12 newfoo.dbm

    $ mogtool extract --gzip foo.dbm.2004.12 newfoo.dbm

    $ mogtool extract foo.dbm.2004.12 -

    $ mogtool inject --bigfile largefile.dat largefile.dat

    $ mogtool inject --bigfile --chunksize=16M largefile.dat largefile.dat

    $ mogtool inject --bigfile --chunksize=1000K --gzip largefile.dat somekey

    $ mogtool inject --bigfile --concurrent=5 --gzip largefile.dat somekey

    $ mogtool inject --bigfile --chunksize=32M --concurrent=3 --gzip \
              --receipt="foo@bar.com" --verify --description="A large file" \
              largefile.dat somekey

    $ mogtool extract --bigfile somekey newfile.dat

    $ mogtool extract --bigfile --gzip somekey newfile.dat

    $ mogtool inject --bigfile mydir mykey

    $ mogtool inject --bigfile --gzip --concurrent=5 mydir mykey

    $ mogtool extract --bigfile mykey .

    $ mogtool extract --bigfile --asfile mykey foo.tar

    $ mogtool inject --bigfile /dev/hda3 hda3.backup

    $ mogtool inject --bigfile --gzip /dev/hda3 hda3.backup

    $ mogtool extract --bigfile hda3.backup /dev/hda4

    $ mogtool delete thekey

    $ mogtool delete --bigfile thekey

    $ mogtool list backup

    $ mogtool listkey abc1

Instead of adding a ton of options to the command line every time, mogtool enables you to create a default configuration file that it will read all of the options from. It searches two locations for a default configuration file: ~/.mogtool and /etc/mogilefs/mogtool.conf.  (Alternately, you can specify --conf=whatever as an option on the command line.)

The file can consist of any number of the following items:

    trackers = 10.0.0.3:7001, 10.10.0.5/10.0.0.5:7001
    domain = mogiledomain
    class = fileclass
    lib = /home/foo/lib
    gzip = 1
    big = 1
    overwrite = 1
    chunksize = 32M
    receipt = foo@bar.com, baz@bar.com
    verify = 1
    concurrent = 3

None?  Send me any you find!  :)

--concurrent for extract

It would be nice to have concurrent extraction going on.

recover mode

If the receipt file is ever corrupt in MogileFS it would be useful to recover a file given just a receipt.  It would have the same arguments as the extract mode, except use a receipt file as the data source.

partition size verification

We can easily get the partition size when we save one to MogileFS, so we should use that information to determine during extraction if a target partition is going to be big enough.

on the fly gzip extraction

Right now we can gzip on an injection, but we should support doing decompression on the fly coming out of MogileFS.

make list take a prefix

If you can specify a prefix, that makes things easier for finding small files that are stored in MogileFS.

more information on list

Have list load up the info file and parse it for information about each of the big files being stored.  Maybe have this as an option (-l).  (This means the reading and parsing of info files should be abstracted into a function.)

Mark Smith <junior@danga.com> - most of the implementation and maintenance.

Brad Fitzpatrick <brad@danga.com> - concepts and rough draft.

Robin H. Johnson <robbat2@orbis-terrarum.net> - locate function.

Copyright (c) 2002-2004 Danga Interactive. All rights reserved.

Hey! The above document had some coding errors, which are explained below:

Around line 289:

Expected text after =item, not a number

Around line 293:

Expected text after =item, not a number