gabbi - Man Page

gabbi — Gabbi

Gabbi tests are expressed in YAML as a series of HTTP requests with their expected response:

tests: — name: retrieve root
     GET: /
     status: 200

This will trigger a GET request to / on the configured Target Host. The test will pass if the response's status code is 200.

The top-level tests category contains an ordered sequence of test declarations, each describing the expected response to a given request:

GET: /index

method: GET
url: /index

The top-level fixtures category contains a sequence of named Fixtures.

response_* keys are examples of Response Handlers. Custom handlers may be created by test authors for specific use cases. See Content Handlers for more information.

There are a number of magical variables that can be used to make reference to the state of a current test, the one just prior or any test prior to the current one. The variables are replaced with real values during test processing.

$RESPONSE:int['$.some_string_value']

The data key has some special handing to allow for a bit more flexibility when doing a POST or PUT:

• If the value is not a string (that is, it is a sequence or structure) it is treated as a data structure that will be turned into a string by the dumps method on the relevant content handler. For example if the content-type of the body is application/json the data structure will be turned into a JSON string.
• If the value is a string that begins with <@ then the rest of the string is treated as a filepath to be loaded. The path is relative to the test directory and may not traverse up into parent directories.
• If the value is an undecorated string, that's the value.

NOTE:

When reading from a file care should be taken to ensure that a reasonable content-type is set for the data as this will control if any encoding is done of the resulting string value. If it is text, json, xml or javascript it will be encoded to UTF-8.

To run gabbi tests with a test harness they must be generated in some fashion and then run. This is accomplished by a test loader. Initially gabbi only supported those test harnesses that supported the load_tests protocol in UnitTest. It now possible to also build and run tests with pytest with some limitations described below.

NOTE:

It is also possible to run gabbi tests from the command line. See YAML Runner.

NOTE:

By default gabbi will load YAML files using the safe_load function. This means only basic YAML types are allowed in the file. For most use cases this is fine. If you need custom types (for example, to match NaN) it is possible to set the safe_yaml parameter of build_tests() to False. If custom types are used, please keep in mind that this can limit the portability of the YAML files to other contexts.

WARNING:

If test are being run with a runner that supports concurrency (such as testrepository) it is critical that the test runner is informed of how to group the tests into their respective suites. The usual way to do this is to use a regular expression that groups based on the name of the yaml files. For example, when using testrepository the .testr.conf file needs an entry similar to the following:

group_regex=gabbi\.suitemaker\.(test_[^_]+_[^_]+)

To run the tests with a load_tests style loader a test file containing a load_tests method is required. That will look a bit like:

"""A sample test module."""

# For pathname munging
import os

# The module that build_tests comes from.
from gabbi import driver

# We need access to the WSGI application that hosts our service
from myapp import wsgiapp


# We're using fixtures in the YAML files, we need to know where to
# load them from.
from myapp.test import fixtures

# By convention the YAML files are put in a directory named
# "gabbits" that is in the same directory as the Python test file.
TESTS_DIR = 'gabbits'


def load_tests(loader, tests, pattern):
    """Provide a TestSuite to the discovery process."""
    test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
    # Pass "require_ssl=True" as an argument to force all tests
    # to use SSL in requests.
    return driver.build_tests(test_dir, loader,
                              intercept=wsgiapp.app,
                              fixture_module=fixtures)

For details on the arguments available when building tests see build_tests().

Once the test loader has been created, it needs to be run. There are many options. Which is appropriate depends very much on your environment. Here are some examples using unittest or testtools that require minimal knowledge to get started.

By file:

python -m testtools.run -v test/test_loader.py

By module:

python -m testttols.run -v test.test_loader

python -m unittest -v test.test_loader

Using test discovery to locate all tests in a directory tree:

python -m testtools.run discover

python -m unittest discover test

See the source distribution and the tutorial repo for more advanced options, including using testrepository and subunit.

Since pytest does not support the load_tests system, a different way of generating tests is required. Two techniques are supported.

The original method (described below) used yield statements to generate tests which pytest would collect. This style of tests is deprecated as of pytest>=3.0 so a new style using pytest fixtures has been developed.

WARNING:

"""A sample pytest module for pytest >= 3.0."""

# For pathname munging
import os

# The module that py_test_generator comes from.
from gabbi import driver

# We need test_pytest so that pytest test collection works properly.
# Without this, the pytest_generate_tests method below will not be
# called.
from gabbi.driver import test_pytest  # noqa

# We need access to the WSGI application that hosts our service
from myapp import wsgiapp

# We're using fixtures in the YAML files, we need to know where to
# load them from.
from myapp.test import fixtures

# By convention the YAML files are put in a directory named
# "gabbits" that is in the same directory as the Python test file.
TESTS_DIR = 'gabbits'


def pytest_generate_tests(metafunc):
    test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
    driver.py_test_generator(
        test_dir, intercept=wsgiapp.app,
        test_loader_name=__name__,
        fixture_module=fixtures, metafunc=metafunc)

py.test -svx pytest3.0-example.py

"""A sample pytest module."""

# For pathname munging
import os

# The module that build_tests comes from.
from gabbi import driver

# We need access to the WSGI application that hosts our service
from myapp import wsgiapp

# We're using fixtures in the YAML files, we need to know where to
# load them from.
from myapp.test import fixtures

# By convention the YAML files are put in a directory named
# "gabbits" that is in the same directory as the Python test file.
TESTS_DIR = 'gabbits'


def test_gabbits():
    test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
    # Pass "require_ssl=True" as an argument to force all tests
    # to use SSL in requests.
    test_generator = driver.py_test_generator(
        test_loader_name=__name__,
        test_dir, intercept=wsgiapp.app,
        fixture_module=fixtures)

    for test in test_generator:
        yield test

py.test -svx pytest-example.py

# Fixtures can be used to set any necessary configuration, such as a
# persistence layer, and establish sample data. They operate per
# file. They are context managers, each one wrapping the next in the
# sequence.

fixtures:
    - ConfigFixture
    - SampleDataFixture

# There is an included fixture named "SkipAllFixture" which can be
# used to declare that all the tests in the given file are to be
# skipped.

# Each test file can specify a set of defaults that will be used for
# every request. This is useful for always specifying a particular
# header or always requiring SSL. These values will be used on every
# test in the file unless overriden. Lists and dicts are merged one
# level deep, except for "data" which is copied verbatim whether it
# is a string, list or dict (it can be all three).

defaults:
    ssl: True
    request_headers:
        x-my-token: zoom

# The tests themselves are a list under a "tests" key. It's useful
# to use plenty of whitespace to help readability.

tests:

# Each request *must* have a name which is unique to the file. When it
# becomes a TestCase the name will be lowercased and spaces will
# become "_". Use that generated name when limiting test runs.

    - name: a test for root
      desc: Some explanatory text that could be used by other tooling

# The URL can either be relative to a host specified elsewhere or
# be a fully qualified "http" or "https" URL. *You* are responsible
# for url-encoding the URL.

      url: /
      method: GET

# If no status or method are provided they default to "200" and
# "GET".

# Instead of explicitly stating "url" and "method" you can join
# those two keys into one key representing the method. The method
# *must* be uppercase.

    - name: another test for root
      desc: Same test as above but with GET key
      GET: /

# A single test can override settings in defaults (set above).

    - name: root without ssl redirects
      ssl: False
      GET: /
      status: 302

# When evaluating response headers it is possible to use a regular
# expression to not have to test the whole value. Regular expressions match
# anywhere in the output, not just at the beginning.

      response_headers:
          location: /^https/

# By default redirects will not be followed. This can be changed.

    - name: follow root without ssl redirect
      ssl: False
      redirects: True
      GET: /
      status: 200 # This is the response code after the redirect.

# URLs can express query parameters in two ways: either in the url
# value directly, or as query_parameters. If both are used then
# query_parameters are appended. In this example the resulting URL
# will be equivalient to
# /foo?section=news&article=1&article=2&date=yesterday
# but not necessarily in that order.

    - name: create a url with parameters
      GET: /foo?section=news
      query_parameters:
          article:
              - 1
              - 2
          date: yesterday

# Request headers can be used to declare media-type choices and
# experiment with authorization handling (amongst other things).
# Response headers allow evaluating headers in the response. These
# two together form the core value of gabbi.

    - name: test accept
      GET: /resource
      request_headers:
          accept: application/json
      response_headers:
          content-type: /application/json/

# If a header must not be present in a response at all that can be
# expressed in a test as follows.

    - name: test forbidden headers
      GET: /resource
      response_forbidden_headers:
          - x-special-header

# All of the above requests have defaulted to a "GET" method. When
# using "POST", "PUT" or "PATCH", the "data" key provides the
# request body.

    - name: post some text
      POST: /text_repo
      request_headers:
          content-type: text/plain
      data: "I'm storing this"
      status: 201

# If the data is not a string, it will be transformed into JSON.
# You must supply an appropriate content-type request header.

    - name: post some json
      POST: /json_repo
      request_headers:
          content-type: application/json
      data:
          name: smith
          abode: castle
      status: 201

# If the data is a string prepended with "<@" the value will be
# treated as the name of a file in the same directory as the YAML
# file. Again, you must supply an appropriate content-type. If the
# content-type is one of several "text-like" types, the content will
# be assumed to be UTF-8 encoded.

    - name: post an image
      POST: /image_repo
      request_headers:
          content-type: image/png
      data: <@kittens.png

# A single request can be marked to be skipped.

    - name: patch an image
      skip: patching images not yet implemented
      PATCH: /image_repo/12d96fb8-e78c-11e4-8c03-685b35afa334

# Or a single request can be marked that it is expected to fail.

    - name: check allow headers
      desc: the framework doesn't do allow yet
      xfail: True
      PUT: /post_only_url
      status: 405
      response_headers:
          allow: POST

# The body of a response can be evaluated with response handlers.
# The most simple checks for a set of strings anywhere in the
# response. Note that the strings are members of a list.

    - name: check for css file
      GET: /blog/posts/12
      response_strings:
          - normalize.css

# For JSON responses, JSONPath rules can be used.

    - name: post some json get back json
      POST: /json_repo
      request_headers:
          content-type: application/json
      data:
          name: smith
          abode: castle
      status: 201
      response_json_paths:
          $.name: smith
          $.abode: castle

# Requests run in sequence. One test can make reference to the test
# immediately prior using some special variables.
# "$LOCATION" contains the "location" header in the previous
# response.
# "$HEADERS" is a pseudo dictionary containing all the headers of
# the previous response.
# "$ENVIRON" is a pseudo dictionary providing access to the current
# environment.
# "$RESPONSE" provides access to the JSON in the prior response, via
# JSONPath. See http://jsonpath-rw.readthedocs.io/ for
# jsonpath-rw formatting.
# $SCHEME and $NETLOC provide access to the current protocol and
# location (host and port).

    - name: get the thing we just posted
      GET: $LOCATION
      request_headers:
          x-magic-exchange: $HEADERS['x-magic-exchange']
          x-token: $ENVIRON['OS_TOKEN']
      response_json_paths:
          $.name: $RESPONSE['$.name']
          $.abode: $RESPONSE['$.abode']
      response_headers:
          content-location: /$SCHEME://$NETLOC/

# For APIs where resource creation is asynchronous it can be
# necessary to poll for the resulting resource. First we create the
# resource in one test. The next test uses the "poll" key to loop
# with a delay for a set number of times.

    - name: create asynch
      POST: /async_creator
      request_headers:
          content-type: application/json
      data:
          name: jones
          abode: bungalow
      status: 202

    - name: poll for created resource
      GET: $LOCATION
      poll:
          count: 10 # try up to ten times
          delay: .5 # wait .5 seconds between each try
      response_json_paths:
          $.name: $RESPONSE['$.name']
          $.abode: $RESPONSE['$.abode']

{
    "pets": [
        {"type": "cat", "sound": "meow"},
        {"type": "dog", "sound": "woof"}
    ]
}

response_json_paths:
    # length of list is two
    $.pets.`len`: 2
    # sound of second item in list is woof
    $.pets[1].sound: woof

response_json_paths:
    # sort by type
    $.pets[/type][0].sound: meow
    # sort by type, reversed
    $.pets[\type][0].sound: woof
    # all the sounds
    $.pets[/type]..sound: ['meow', 'woof']
    # filter by type = dog
    $.pets[?type = "dog"].sound: woof

response_json_paths:
    $:
        pets:
            - type: cat
              sound: meow
            - type: dog
              sound: woof

response_json_paths:
    $: @<data.json

response_json_paths:
    $.pets: <@pets.json
    $.pets[0]: <@cat.json

{
    "values": [{
        "pets": [{
            "type": "cat",
            "sound": "meow"
        }, {
            "type": "dog",
            "sound": "woof"
        }]
    }, {
        "people": [{
            "name": "chris",
            "id": 1
        }, {
            "name": "justin",
            "id": 2
        }]
    }]
}

response_json_paths:
    $.pets: <@pets.json
    $.pets[?type = "cat"].sound: <@values.json:$.values[0].pets[?type = "cat"].sound

Substitutions can be made in both the left (query) and right (expected) hand sides of the json path expression. When subtitutions are used in the query, care must be taken to ensure proper quoting of the resulting value. For example if there is a uuid (with hyphens) at $RESPONSE['$.id'] then this expression may fail:

$.nested.structure.$RESPONSE['$.id'].name: foobar

as it will evaluate to something like:

$.nested.structure.ADC8AAFC-D564-40D1-9724-7680D3C010C2.name: foobar

which may be treated as an arithemtic expression by the json path parser. The test author should write:

$.nested.structure["$RESPONSE['$.id']"].name: foobar

to quote the result of the substitution.

The target host is the host on which the API to be tested can be found. Gabbi intends to preserve the flow and semantics of HTTP interactions as much as possible, and every HTTP request needs to be directed at a host of some form. Gabbi provides three ways to control this:

• Using wsgi-intercept to provide a fake socket and WSGI environment on an arbitrary host and port attached to a WSGI application (see intercept examples).
• Using fully qualified url values in the YAML defined tests (see full examples).
• Using a host and (optionally) port defined at test build time (see live examples).

The intercept and live methods are mutually exclusive per test builder, but either kind of test can freely intermix fully qualified URLs into the sequence of tests in a YAML file.

For test driven development and local tests the intercept style of testing lowers test requirements (no web server required) and is fast. Interception is performed as part of Fixtures processing as the most deeply nested fixture. This allows any configuration or database setup to be performed prior to the WSGI application being created.

For the implementation of the above see build_tests().

Each suite of tests is represented by a single YAML file, and may optionally use one or more fixtures to provide the necessary environment required by the tests in that file.

Fixtures are implemented as nested context managers. Subclasses of GabbiFixture must implement start_fixture and stop_fixture methods for creating and destroying, respectively, any resources managed by the fixture. While the subclass may choose to implement __init__ it is important that no exceptions are thrown in that method, otherwise the stack of context managers will fail in unexpected ways. Instead initialization of real resources should happen in start_fixture.

At this time there is no mechanism for the individual tests to have any direct awareness of the fixtures. The fixtures exist, conceptually, on the server side of the API being tested.

Fixtures may do whatever is required by the testing environment, however there are two common scenarios:

• Establishing (and then resetting when a test suite has finished) any baseline configuration settings and persistence systems required for the tests.
• Creating sample data for use by the tests.

If a fixture raises unittest.case.SkipTest during start_fixture all the tests in the current file will be skipped. This makes it possible to skip the tests if some optional configuration (such as a particular type of database) is not available.

If an exception is raised while a fixture is being used, information about the exception will be stored on the fixture so that the stop_fixture method can decide if the exception should change how the fixture should clean up. The exception information can be found on exc_type, exc_value and traceback method attributes.

If an exception is raised when a fixture is started (in start_fixture) the first test in the suite using the fixture will be marked with an error using the traceback from the exception and all the tests in the suite will be skipped. This ensures that fixture failure is adequately captured and reported by test runners.

In some contexts (for example CI environments with a large number of tests being run in a broadly concurrent environment where output is logged to a single file) it can be important to capture and consolidate stray output that is produced during the tests and display it associated with an individual test. This can help debugging and avoids unusable output that is the result of multiple streams being interleaved.

Inner fixtures have been added to support this. These are fixtures more in line with the tradtional unittest concept of fixtures: a class on which setUp and cleanUp is automatically called.

build_tests() accepts a named parameter arguments of inner_fixtures. The value of that argument may be an ordered list of fixtures.Fixture classes that will be called when each individual test is set up.

An example fixture that could be useful is the FakeLogger.

NOTE:

Content handlers are responsible for preparing request data and evaluating response data based on the content-type of the request and response. A content handler operates as follows:

• Structured YAML data provided via the data attribute is converted to a string or bytes sequence and used as request body.

• The response body (a string or sequence of bytes) is transformed into a content-type dependent structure and stored in an internal attribute named response_data that is:

  • used when evaluating the response body
  • used in $RESPONSE[] substitutions

By default, gabbi provides content handlers for JSON. In that content handler the data test key is converted from structured YAML into a JSON string. Response bodies are converted from a JSON string into a data structure in response_data that is used when evaluating response_json_paths entries in a test or doing JSONPath-based $RESPONSE[] substitutions.

A YAMLDiskLoadingJSONHandler has been added to extend the JSON handler. It works the same way as the JSON handler except for when evaluating the response_json_paths handle, data that is read from disk can be either in JSON or YAML format. The YAMLDiskLoadingJSONHandler is not enabled by default and must be added as shown in the Extensions section in order to be used in the tests.

Further content handlers can be added as extensions. Test authors may need these extensions for their own suites, or enterprising developers may wish to create and distribute extensions for others to use.

NOTE:

Content handlers are an evolution of the response handler concept in earlier versions gabbi. To preserve backwards compatibility with existing response handlers, old style response handlers are still allowed, but new handlers should implement the content handler interface (described below).

Registering additional custom handlers is done by passing a subclass of ContentHandler to build_tests():

driver.build_tests(test_dir, loader, host=None,
                   intercept=simple_wsgi.SimpleWsgi,
                   content_handlers=[MyContentHandler])

If pytest is being used:

driver.py_test_generator(test_dir, intercept=simple_wsgi.SimpleWsgi,
                         test_loader_name=__name__,
                         content_handlers=[MyContenHandler])

Gabbi provides an additional custom handler named YAMLDiskLoadingJSONHandler. This can be used for loading JSON and YAML files from disk when evaluating the response_json_paths handle.

WARNING:

With gabbi-run, custom handlers can be loaded via the --response-handler option -- see load_response_handlers() for details.

NOTE:

gabbi-run [host[:port]] < /my/test.yaml

gabbi-run http://host:port < /my/test.yaml

gabbi-run http://host:port -- /my/test.yaml /my/other.yaml

gabbi-run host[:port] [prefix] < /my/test.yaml

gabbi-run http://host:port/prefix < /my/test.yaml

• Add support for regular expressions in response_strings values.

This is being done as a major release because it has a chance of breaking existing tests that use response_strings values that look like regular expressions but are not.

• Corrected support for Python 3.12

Remove use of six. Thanks to Alexandre Detiste.

• Pin to urllib3 <2.0.0 to avoid issues with changes in constructors.

• Add timeout parameter per test, defaulting to 30 seconds.
• Ensure stestr > 4.0.0 in test-requirements.txt

• Handle SNI when host request header is any case.

• Clear the PoolManager after each test to avoid running out of socket file descriptors.

• Add -l argument to gabbi-run to allow loading content and response handlers relative to the local directory.

• Fix naming of tests when using pytest. This changes means that setting test_loader_name is now required when using pytest.

• Support SNI in https requests when host request header is set.

• Publish a docker container that runs gabbi-run with each tagged release.

• For the $ENVIRON and $RESPONSE substitutions it is now possible to cast the value to a type of int, float, str, or bool.
• The JSONHandler is now more strict about how it detects that a body content is JSON, avoiding some errors where the content-type header suggests JSON but the content cannot be decoded as such.
• Better error message when content cannot be decoded.
• Addition of the disable_response_handler test setting for those cases when the test author has no control over the content-type header and it is wrong.

• Allow substitutions in skip for more flexible skipping, notablly with $ENVIRON.
• Better test name output when running with unittest and verbose.

• Remove support for Python 3.5. Ensure testing support for 3.8, 3.9, and pypy3.
• Adapt to new behavior in urllib.parse.
• Correct fixture start when using pytest.

• If no content-type is provided with a response and verbose is on for responses, display the response body as if it were text.

• Properly declare that gabbi 2.x is Python 3 only.

• Correct management of response handler default fields.

• Drop support for Python 2. If you need Python 2 support, use an older version.
• Stop using testtools and fixtures. These two modules present several difficulties and their maintenance situation suggests those difficulties will not be resolved. Since Python 2 support is being removed, the need for the modules can be removed as well without losing functionality. "Inner fixtures" that use the fixtures.Fixture interface should continue to work.

• Add support for not validating certificates in https requests. Controlled by the cert_validate attribute in individual tests and build_tests() and the -k or --insecure argument to gabbi-run.

• Support pytest 5.0.0 in Python >=3.5. For earlier versions of Python, pytest<5.0.0 will be used; the pytest project is dropping support for older versions of Python.

• Use pytest<5.0.0 until gabbi has solutions for the changes in 5.0.0.

• A -q argument is added to gabbi-run to suppress output from the test runner.

• Adjust loading of YAML to be ready for new release of PyYAML.

• Provide the YAMLDiskLoadingJSONHandler class that allows test result data for response_json_path checks to be loaded from YAML-on-disk.

• Use JSONPath to select a portion of data-on-disk in response_json_path checks.
• Restrict PyYAML to <4.0.

• Allow listing of tests with no host configured. When host is an empty string, tests can be listed (for discovery), but will be skipped on run.

• JSON $RESPONSE substitutions in the data field may be complex types (lists and dicts), not solely strings.

• When the HTTP response begins with a bad status line, have BadStatusLine be raised from urllib3.

• Allow substitutions in the key portion of request and response headers, not just the value.

• Remove support for Python 3.3.
• Make handling of fixture-level skips in pytest actually work.

• Add safe_yaml parameter to build_tests().

• use_prior_test is added to test Metadata.
• Extensive cleanups in regular expression handling when constructing tests from YAML.

JSONPath handling gets two improvements:

• The value side of a response_json_paths entry can be loaded from a file using the <@file.json syntax also used in Data.
• The key side of a response_json_paths entry can use substitutions. This was already true for the value side.

Substitutions in $RESPONSE handling now preserve numeric types instead of casting to a string. This is useful when servers are expecting strong types and tests want to send response data back to the server.

count and delay test keys allow substitutions. gabbi.driver.build_tests() accepts a verbose parameter to set test verbosity for an entire session.

Better failure reporting when using gabbi-run with multiple files. Test names are based on the files and a summary of failed files is provided at the end of the report.

Effectively capture a failure in a fixture and report the traceback. Without this some test runners swallow the error and discovering problems when developing fixtures can be quite challenging.

Thanks to Samuel Fekete, tests can use the $HISTORY dictionary to refer to any prior test in the same file, not just the one immediately prior, when doing substitutions.

Filenames used to read data into tests using the <@ syntax may now use pathnames relative to the YAML file. See Data.

gabbi-run gains a --verbose parameter to force all tests run in a session to run with verbose set.

When using pytest to load tests, a new mechanism is available which avoids warnings produced in when using a version of pytest greater than 3.0.

When verbosely displaying request and response bodies that are JSON, pretty print for improved readability.

Allow gabbi-run to accept multiple filenames as command line arguments instead of reading tests from stdin.

Switch from response handlers to Content Handlers to allow more flexible processing of both response _and_ request bodies.

Add inner fixtures for per test fixtures, useful for output capturing.

Allow the test_loader_name arg to gabbi.driver.build_tests() to override the prefix of the pretty printed name of generated tests.

String values in JSONPath matches may be wrapped in /.../` to be treated as regular expressions.

Better documentation of how to run gabbi in a concurrent environment. Improved handling of pytest fixtures and test counts.

Add url to gabbi.driver.build_tests() to use instead of host, port and prefix.

Add require_ssl to gabbi.driver.build_tests() to force use of SSL.

Add $COOKIE substitution.

Correctly support IPV6 hosts.

Add $LAST_URL substitution.

Introduce support for loading and running tests with pytest.

Use urllib3 instead of httplib2 for driving HTTP requests.

Add sorting and filtering to JSONPath handling.

Add the response_forbidden_headers to response expectations.

Instead of:

tests:
- name: a simple get
  url: /some/path
  method: get

1.7.0 also makes it possible to:

tests:
- name: a simple get
  GET: /some/path

Any upper case key is treated as a method.

Enhanced flexibility and colorization when setting tests to be verbose.

Adds the query_parameters key to request parameters.

The start of improvements and extensions to JSONPath handling. In this case the addition of the len function.

Vastly improved output and behavior in gabbi-run.

Version 1 was the first release with a commitment to a stable Test Format. Since then new fields have been added but have not been taken away.

The following people have contributed code to gabbi. Thanks to them. Thanks also to all the people who have made gabbi better by reporting issues and their successes and failures with using gabbi.

• Chris Dent
• FND
• Mehdi Abaakouk
• Tom Viner
• Jason Myers
• Josh Leeb-du Toit
• Duc Truong
• Zane Bitter
• Ryan Spencer
• Kim Raymoure
• Travis Truman
• Samuel Fekete
• Michael McCune
• Imran Hayder
• Julien Danjou
• Trevor McCasland
• Danek Duvall
• Marc Abramowitz
• Scott Wallace
• Alexandre Detiste

NOTE:

This section provides a collection of questions with answers that don't otherwise fit in the rest of the documentation. If something is missing, please create an issue.

As this document grows it will gain a more refined structure.

A single HTTP request represented as a subclass of unittest.TestCase

The test case encapsulates the request headers and body and expected response headers and body. When the test is run an HTTP request is made using urllib3. Assertions are made against the response.

class gabbi.case.HTTPTestCase(methodName='runTest')

Bases: TestCase

Encapsulate a single HTTP request as a TestCase.

If the test is a member of a sequence of requests, ensure that prior tests are run.

To keep the test harness happy we need to make sure the setUp and tearDown are only run once.

assert_in_or_print_output(expected, iterable)

Assert the iterable contains expected or print some output.

If the output is long, it is limited by either GABBI_MAX_CHARS_OUTPUT in the environment or the MAX_CHARS_OUTPUT constant.

base_test = {'cert_validate': True, 'data': '', 'desc': '', 'disable_response_handler': False, 'method': 'GET', 'name': '', 'poll': {}, 'query_parameters': {}, 'redirects': False, 'request_headers': {}, 'skip': '', 'ssl': False, 'status': '200', 'timeout': 30, 'url': '', 'use_prior_test': True, 'verbose': False, 'xfail': False}

get_content_handler(content_type)

Determine the content handler for this media type.

load_data_file(filename)

Read a file from the current test directory.

replace_template(message, escape_regex=False)

Replace magic strings in message.

run(result=None)

Store the current result handler on this test.

setUp()

Hook method for setting up the test fixture before exercising it.

shortDescription()

Returns a one-line description of the test, or None if no description has been provided.

The default implementation of this method returns the first line of the specified test method's docstring.

tearDown()

Hook method for deconstructing the test fixture after testing it.

test_request()

Run this request if it has not yet run.

If there is a prior test in the sequence, run it first.

gabbi.case.potentialFailure(func)

Decorate a test method that is expected to fail if 'xfail' is true.

Generate HTTP tests from YAML files

Each HTTP request is its own TestCase and can be requested to be run in isolation from other tests. If it is a member of a sequence of requests, prior requests will be run.

A sequence is represented by an ordered list in a single YAML file.

Each sequence becomes a TestSuite.

An entire directory of YAML files is a TestSuite of TestSuites.

gabbi.driver.build_tests(path, loader, host=None, port=8001, intercept=None, test_loader_name=None, fixture_module=None, response_handlers=None, content_handlers=None, prefix='', require_ssl=False, cert_validate=True, url=None, inner_fixtures=None, verbose=False, use_prior_test=True, safe_yaml=True)

Read YAML files from a directory to create tests.

Each YAML file represents a list of HTTP requests.

Parameters

• path -- The directory where yaml files are located.
• loader -- The TestLoader.
• host -- The host to test against. Do not use with intercept.
• port -- The port to test against. Used with host.
• intercept -- WSGI app factory for wsgi-intercept.
• test_loader_name -- Base name for test classes. Use this to align the naming of the tests with other tests in a system.
• fixture_module -- Python module containing fixture classes.
• response_handers -- ResponseHandler classes.
• content_handlers (List of ContentHandler classes.) -- ContentHandler classes.
• prefix -- A URL prefix for all URLs that are not fully qualified.
• url -- A full URL to test against. Replaces host, port and prefix.
• require_ssl -- If True, make all tests default to using SSL.
• inner_fixtures (List of classes with setUp and cleanUp methods to be used as fixtures.) -- A list of Fixtures to use with each individual test request.
• verbose -- If True or 'all', make tests verbose by default 'headers' and 'body' are also accepted.
• use_prior_test -- If True, uses prior test to create ordered sequence of tests
• safe_yaml -- If True, recognizes only standard YAML tags and not Python object
• cert_validate -- If False ssl server certificate will be ignored, further it will not be validated if provided (set cert_reqs=CERT_NONE to the Http object)

Return type

TestSuite containing multiple TestSuites (one for each YAML file).

gabbi.driver.py_test_generator(test_dir, host=None, port=8001, intercept=None, prefix=None, test_loader_name=None, fixture_module=None, response_handlers=None, content_handlers=None, require_ssl=False, url=None, metafunc=None, use_prior_test=True, inner_fixtures=None, safe_yaml=True, cert_validate=True)

Generate tests cases for py.test

This uses build_tests to create TestCases and then yields them in a way that pytest can handle.

test_loader_name is required!

gabbi.driver.test_pytest(test, result)

gabbi.driver.test_suite_from_yaml(loader, test_base_name, test_yaml, test_directory, host, port, fixture_module, intercept, prefix='')

Legacy wrapper retained for backwards compatibility.

The code that creates a suite of tests.

The key piece of code is test_suite_from_dict(). It produces a gabbi.suite.GabbiSuite containing one or more gabbi.case.HTTPTestCase.

class gabbi.suitemaker.TestBuilder(name, bases, attributes)

Bases: type

Metaclass to munge a dynamically created test.

required_attributes = {'has_run': False}

class gabbi.suitemaker.TestMaker(test_base_name, test_defaults, test_directory, fixture_classes, loader, host, port, intercept, prefix, response_handlers, content_handlers, test_loader_name=None, inner_fixtures=None)

Bases: object

A class for encapsulating test invariants.

All of the tests in a single gabbi file have invariants which are provided when creating each HTTPTestCase. It is not useful to pass these around when making each test case. So they are wrapped in this class which then has make_one_test called multiple times to generate all the tests in the suite.

make_one_test(test_dict, prior_test)

Create one single HTTPTestCase.

The returned HTTPTestCase is added to the TestSuite currently being built (one per YAML file).

gabbi.suitemaker.test_suite_from_dict(loader, test_base_name, suite_dict, test_directory, host, port, fixture_module, intercept, prefix='', handlers=None, test_loader_name=None, inner_fixtures=None)

Generate a GabbiSuite from a dict represent a list of tests.

The dict takes the form:

Parameters

• fixtures -- An optional list of fixture classes that this suite can use.
• defaults -- An optional dictionary of default values to be used in each test.
• tests -- A list of individual tests, themselves each being a dictionary. See gabbi.case.BASE_TEST.

gabbi.suitemaker.test_update(orig_dict, new_dict)

Modify test in place to update with new data.

Manage fixtures for gabbi at the test suite level.

class gabbi.fixture.GabbiFixture

Bases: object

A context manager that operates as a fixture.

Subclasses must implement start_fixture and stop_fixture, each of which contain the logic for stopping and starting whatever the fixture is. What a fixture is is left as an exercise for the implementor.

These context managers will be nested so any actual work needs to happen in start_fixture and stop_fixture and not in __init__. Otherwise exception handling will not work properly.

start_fixture()

Implement the actual workings of starting the fixture here.

stop_fixture()

Implement the actual workings of stopping the fixture here.

exception gabbi.fixture.GabbiFixtureError

Bases: Exception

Generic exception for GabbiFixture.

class gabbi.fixture.SkipAllFixture

Bases: GabbiFixture

A fixture that skips all the tests in the current suite.

start_fixture()

Implement the actual workings of starting the fixture here.

gabbi.fixture.nest(fixtures)

Nest a series of fixtures.

This is duplicated from nested in the stdlib, which has been deprecated because of issues with how exceptions are difficult to handle during __init__. Gabbi needs to nest an unknown number of fixtures dynamically, so the with syntax that replaces nested will not work.

Package for response and content handlers that process the body of a response in various ways.

A TestSuite for containing gabbi tests.

This suite has two features: the contained tests are ordered and there are suite-level fixtures that operate as context managers.

class gabbi.suite.GabbiSuite(tests=())

Bases: TestSuite

A TestSuite with fixtures.

The suite wraps the tests with a set of nested context managers that operate as fixtures.

If a fixture raises unittest.case.SkipTest during setup, all the tests in this suite will be skipped.

run(result, debug=False)

Override TestSuite run to start suite-level fixtures.

To avoid exception confusion, use a null Fixture when there are no fixtures.

start(result, tests=None)

Start fixtures when using pytest.

stop()

Stop fixtures when using pytest.

gabbi.suite.noop(*args)

A noop method used to disable collected tests.

Implementation of a command-line runner for gabbi files (AKA suites).

gabbi.runner.extract_file_paths(argv)

Extract file paths from the command-line.

File path arguments follow a -- end-of-options delimiter, if any.

gabbi.runner.initialize_handlers(response_handlers, local_handlers)

gabbi.runner.load_response_handlers(import_path)

Load and return custom response handlers from the import path.

The import path references either a specific response handler class ("package.module:class") or a module that contains one or more response handler classes ("package.module").

For the latter, the module is expected to contain a gabbi_response_handlers object, which is either a list of response handler classes or a function returning such a list.

gabbi.runner.run()

Run simple tests from STDIN.

This command provides a way to run a set of tests encoded in YAML that is provided on STDIN. No fixtures are supported, so this is primarily designed for use with real running services.

Host and port information may be provided in three different ways:

• In the URL value of the tests.
• In a host or host:port argument on the command line.
• In a URL on the command line.

An example run might looks like this:

gabbi-run example.com:9999 < mytest.yaml

or:

gabbi-run http://example.com:999 < mytest.yaml

It is also possible to provide a URL prefix which can be useful if the target application might be mounted in different locations. An example:

gabbi-run example.com:9999 /mountpoint < mytest.yaml

or:

gabbi-run http://example.com:9999/mountpoint < mytest.yaml

Use -x or --failfast to abort after the first error or failure:

gabbi-run -x example.com:9999 /mountpoint < mytest.yaml

Use -v or --verbose with a value of all, headers or body to turn on verbosity for all tests being run.

Multiple files may be named as arguments, separated from other arguments by a --. Each file will be run as a separate test suite:

gabbi-run http://example.com -- /path/to/x.yaml /path/to/y.yaml

Output is formatted as unittest summary information. Use -q or --quiet to silence that output.

Use -r or --response-handler to load a custom response or content handler for use with tests.

Use -l to load response handlers relative to the current working directory.

For example, to load a handler named HTMLHandler from the handlers.html module relative to the current directory:

gabbi-run -l -r handlers.html:HTMLHandler http://example.com < my.yaml

gabbi.runner.run_suite(handle, handler_objects, host, port, prefix, force_ssl=False, failfast=False, data_dir='.', verbosity=False, name='input', safe_yaml=True, quiet=False, cert_validate=True)

Run the tests from the YAML in handle.

TestRunner and TestResult for gabbi-run.

class gabbi.reporter.ConciseTestResult(stream, descriptions, verbosity)

Bases: TextTestResult

A TextTestResult with simple but useful output.

If the output is a tty or GABBI_FORCE_COLOR is set in the environment, output will be colorized.

addError(test, err)

Called when an error has occurred. 'err' is a tuple of values as returned by sys.exc_info().

addExpectedFailure(test, err)

Called when an expected failure/error occurred.

addFailure(test, err)

Called when an error has occurred. 'err' is a tuple of values as returned by sys.exc_info().

addSkip(test, reason)

Called when a test is skipped.

addSuccess(test)

Called when a test has completed successfully

addUnexpectedSuccess(test)

Called when a test was expected to fail, but succeed.

getDescription(test)

printErrorList(flavor, errors)

startTest(test)

Called when the given test is about to be run

class gabbi.reporter.ConciseTestRunner(stream=None, descriptions=True, verbosity=1, failfast=False, buffer=False, resultclass=None, warnings=None, *, tb_locals=False, durations=None)

Bases: TextTestRunner

A TextTestRunner that uses ConciseTestResult for reporting results.

resultclass

alias of ConciseTestResult

class gabbi.reporter.PyTestResult(stream=None, descriptions=None, verbosity=None)

Bases: TestResult

Wrap a test result to allow it to work with pytest.

The main behaviors here are:

• to turn what had been exceptions back into exceptions
• use pytest's skip and xfail methods

addError(test, err)

Called when an error has occurred. 'err' is a tuple of values as returned by sys.exc_info().

addExpectedFailure(test, err)

Called when an expected failure/error occurred.

addFailure(test, err)

Called when an error has occurred. 'err' is a tuple of values as returned by sys.exc_info().

addSkip(test, reason)

Called when a test is skipped.

Utility functions grab bag.

gabbi.utils.create_url(base_url, host, port=None, prefix='', ssl=False)

Given pieces of a path-based url, return a fully qualified url.

gabbi.utils.decode_response_content(header_dict, content)

Decode content to a proper string.

gabbi.utils.extract_content_type(header_dict, default='application/binary')

Extract parsed content-type from headers.

gabbi.utils.get_colorizer(stream)

Return a function to colorize a string.

Only if stream is a tty .

gabbi.utils.host_info_from_target(target, prefix=None)

Turn url or host:port and target into test destination.

gabbi.utils.load_yaml(handle=None, yaml_file=None, safe=True)

Read and parse any YAML file or filehandle.

Let exceptions flow where they may.

If no file or handle is provided, read from STDIN.

gabbi.utils.not_binary(content_type)

Decide if something is content we'd like to treat as a string.

gabbi.utils.parse_content_type(content_type, default_charset='utf-8')

Parse content type value for media type and charset.

Gabbi specific exceptions.

exception gabbi.exception.GabbiDataLoadError

Bases: ValueError

An exception to alert when data streams cannot be loaded.

exception gabbi.exception.GabbiFormatError

Bases: ValueError

An exception to encapsulate poorly formed test data.

exception gabbi.exception.GabbiSyntaxWarning

Bases: SyntaxWarning

A warning about syntax that is not desirable.

class gabbi.httpclient.Http(num_pools: int = 10, headers: Mapping[str, str] | None = None, **connection_pool_kw: Any)

Bases: PoolManager

A subclass of the urllib3.PoolManager to munge the data.

This transforms the response to look more like what httplib2 provided when it was used as the HTTP client.

request(absolute_uri, method, body, headers, redirect, timeout)

Make a request using urlopen() with the appropriate encoding of fields based on the method used.

This is a convenience method that requires the least amount of manual effort. It can be used in most situations, while still having the option to drop down to more specific methods when necessary, such as request_encode_url(), request_encode_body(), or even the lowest level urlopen().

Parameters

• method -- HTTP request method (such as GET, POST, PUT, etc.)
• url -- The URL to perform the request on.
• body -- Data to send in the request body, either str, bytes, an iterable of str/bytes, or a file-like object.
• fields -- Data to encode and send in the URL or request body, depending on method.
• headers -- Dictionary of custom headers to send, such as User-Agent, If-None-Match, etc. If None, pool headers are used. If provided, these headers completely replace any pool-specific headers.
• json -- Data to encode and send as JSON with UTF-encoded in the request body. The "Content-Type" header will be set to "application/json" unless specified otherwise.

class gabbi.httpclient.VerboseHttp(**kwargs)

Bases: Http

A subclass of Http that verbosely reports on activity.

If the output is a tty or GABBI_FORCE_COLOR is set in the environment, then output will be colorized according to COLORMAP.

Output can include request and response headers, request and response body content (if of a printable content type), or both.

The color of the output has reasonable defaults. These may be overridden by setting the following environment variables

• GABBI_CAPTION_COLOR
• GABBI_HEADER_COLOR
• GABBI_REQUEST_COLOR
• GABBI_STATUS_COLOR

to any of: BLACK RED GREEN YELLOW BLUE MAGENTA CYAN WHITE

COLORMAP = {'caption': 'BLUE', 'header': 'YELLOW', 'request': 'CYAN', 'status': 'CYAN'}

HEADER_BLACKLIST = ['status', 'reason']

REQUEST_PREFIX = '>'

RESPONSE_PREFIX = '<'

request(absolute_uri, method, body, headers, redirect, timeout)

Display request parameters before requesting.

gabbi.httpclient.get_http(verbose=False, caption='', cert_validate=True, hostname=None, timeout=30)

Return an Http class for making requests.

Keep one single global jsonpath parser.

gabbi.json_parser.parse(path)

Parse a JSONPath expression use the global parser.

Gabbi is a tool for running HTTP tests where requests and responses are expressed as declarations in YAML files:

tests:
- name: retrieve items
  GET: /items

See the rest of these docs for more details on the many features and formats for setting request headers and bodies and evaluating responses.

Tests can be run from the command line with gabbi-run or programmatically using either py.test or unittest-style test runners. See installation instructions below.

The name is derived from "gabby": excessively talkative. In a test environment having visibility of what a test is actually doing is a good thing. This is especially true when the goal of a test is to test the HTTP, not the testing infrastructure. Gabbi tries to put the HTTP interaction in the foreground of testing.

If you want to get straight to creating tests look at Example Tests, the test files in the source distribution and Test Format. A gabbi-demo repository provides a tutorial of using gabbi to build an API, via the commit history of the repo.

Gabbi works to bridge the gap between human readable YAML files (see Test Format for details) that represent HTTP requests and expected responses and the rather complex world of automated testing.

Each YAML file represents an ordered list of HTTP requests along with the expected responses. This allows a single file to represent a process in the API being tested. For example:

• Create a resource.
• Retrieve a resource.
• Delete a resource.
• Retrieve a resource again to confirm it is gone.

At the same time it is still possible to ask gabbi to run just one request. If it is in a sequence of tests, those tests prior to it in the YAML file will be run (in order). In any single process any test will only be run once. Concurrency is handled such that one file runs in one process.

These features mean that it is possible to create tests that are useful for both humans (as tools for learning, improving and developing APIs) and automated CI systems.

Significant flexibility and power is available in the Test Format to make it relatively straightforward to test existing complex APIs. This extended functionality includes the use of JSONPath to query response bodies and templating of test data to allow access to the prior HTTP response in the current request. For APIs which do not use JSON additional Content Handlers can be created.

Care should be taken when using this functionality when you are creating a new API. If your API is so complex that it needs complex test files then you may wish to take that as a sign that your API itself too complex. One goal of gabbi is to encourage transparent and comprehensible APIs.

Though gabbi is written in Python and under the covers uses unittest data structures and processes, there is no requirement that the Target Host be a Python-based service. Anything talking HTTP can be tested. A YAML Runner makes it possible to simply create YAML files and point them at a running server.

As a Python package, gabbi is typically installed via pip:

pip install gabbi

You might want to create a virtual environment; an isolated context for Python packages, keeping gabbi cleany separated from the rest of your system.

Python 3 comes with a built-in tool to create virtual environments:

python3 -m venv venv
. venv/bin/activate

pip install gabbi

This way we can later use deactivate and safely remove the venv directory, thus erasing any trace of gabbi from the system.

If you prefer to not install gabbi, or perhaps want to use it in a dynamic fashion in a CI setting, there is an official container image hosted at docker hub as cdent/gabbi. It allows running gabbi-run with any arguments you might need, providing tests on STDIN of via a mounted volume.

unknown