testutil

The testutil package is a test framework that provides facilities for specifying test cases and recording test results.

You would use it to build regression tests for your library.

Description

A package may optionally contain a set of test cases. Test cases are not normally compiled and linked when a package is built; they are only included when the “test” identity is specified. All of a package’s test code goes in its src/test directory. For example, the nffs package’s test code is located in the following directory:

* fs/nffs/src/test/

This directory contains the source and header files that implement the nffs test code.

The test code has access to all the header files in the following directories:

* src
* src/arch/<target-arch>
* include
* src/test
* src/test/arch/<target-arch>
* include directories of all package dependencies

Package test code typically depends on the testutil package, described later in this document.

Some test cases or test initialization code may be platform-specific. In such cases, the platform-specific function definitions are placed in arch subdirectories within the package test directory.

While building the test code (i.e., when the test identity is specified), the newt tool defines the TEST macro. This macro is defined during compilation of all C source files in all projects and packages.

Tests are structured according to the following hierarchy:

           [test]
          /      \
   [suite]        [suite]
  /       \      /       \
[case] [case]  [case] [case]

I.e., a test consists of test suites, and a test suite consists of test cases.

The test code uses testutil to define test suites and test cases.

Regression test can then be executed using ‘newt target test’ command, or by including a call to your test suite from project/test/src/test.c.

Example

This Tutorial shows how to create a test suite for a Mynewt package.

Data structures

struct ts_config {
    int ts_print_results;
    int ts_system_assert;

    const char *ts_suite_name;

    /*
     * Called prior to the first test in the suite
     */
    tu_init_test_fn_t *ts_suite_init_cb;
    void *ts_suite_init_arg;

    /*
     * Called after the last test in the suite
     */
    tu_init_test_fn_t *ts_suite_complete_cb;
    void *ts_suite_complete_arg;

    /*
     * Called before every test in the suite
     */
    tu_pre_test_fn_t *ts_case_pre_test_cb;
    void *ts_case_pre_arg;

    /*
     * Called after every test in the suite
     */
    tu_post_test_fn_t *ts_case_post_test_cb;
    void *ts_case_post_arg;

    /*
     * Called after test returns success
     */
    tu_case_report_fn_t *ts_case_pass_cb;
    void *ts_case_pass_arg;

    /*
     * Called after test fails (typically thoough a failed test assert)
     */
    tu_case_report_fn_t *ts_case_fail_cb;
    void *ts_case_fail_arg;

    /*
     * restart after running the test suite - self-test only
     */
    tu_suite_restart_fn_t *ts_restart_cb;
    void *ts_restart_arg;
};

The global ts_config struct contains all the testutil package’s settings.

API

typedef void tu_case_report_fn_t(const char *msg, void *arg)

typedef void tu_pre_test_fn_t(void *arg)

typedef void tu_post_test_fn_t(void *arg)

typedef void tu_testsuite_fn_t(void)

struct ts_testsuite_list g_ts_suites

struct tu_config tu_config

const char *tu_suite_name

const char *tu_case_name

int tu_any_failed

int tu_suite_failed

int tu_case_reported

int tu_case_failed

int tu_case_idx

jmp_buf tu_case_jb

void tu_set_pass_cb(tu_case_report_fn_t *cb, void *cb_arg)

void tu_set_fail_cb(tu_case_report_fn_t *cb, void *cb_arg)

void tu_suite_init(const char *name)

void tu_suite_pre_test(void)

void tu_suite_complete(void)

int tu_suite_register(tu_testsuite_fn_t *ts, const char *name)

SLIST_HEAD (ts_testsuite_list, ts_suite)

void tu_restart(void)

void tu_start_os(const char *test_task_name, os_task_func_t test_task_handler)

void tu_suite_set_pre_test_cb(tu_pre_test_fn_t *cb, void *cb_arg)

void tu_case_set_post_test_cb(tu_post_test_fn_t *cb, void *cb_arg)

void tu_case_init(const char *name)

void tu_case_complete(void)

void tu_case_pass(void)

void tu_case_fail(void)

void tu_case_fail_assert(int fatal, const char *file, int line, const char *expr, const char *format, ...)

void tu_case_write_pass_auto(void)

void tu_case_pass_manual(const char *file, int line, const char *format, ...)

void tu_case_post_test(void)

TEST_SUITE_DECL(suite_name)

TEST_SUITE_REGISTER(suite_name)

TEST_SUITE(suite_name)

TEST_CASE_DECL(case_name)

TEST_CASE_DEFN(case_name, do_sysinit, body)

TEST_CASE(case_name)

Defines a test case suitable for running in an application.

The TEST_CASE() macro should not be used for self-tests (i.e., tests that are run with newt test). Instead, TEST_CASE_SELF() or TEST_CASE_TASK() should be preferred; those macros perform system clean up before the test runs.

TEST_CASE_SELF_EMIT_(case_name)

TEST_CASE_TASK_EMIT_(case_name)

TEST_CASE_SELF(case_name)

Defines a test case for self-test mode (i.e., suitable for newt test).

Test cases defined with TEST_CASE_SELF() execute sysinit() before the test body.

TEST_CASE_TASK(case_name)

Defines a test case that runs inside a temporary task.

Most tests don’t utilize the OS scheduler; they simply run in main(), outside the context of a task. However, sometimes the OS is required to fully test a package, e.g., to verify timeouts and other timed events.

The TEST_CASE_TASK() macro simplifies the implementation of test cases that require the OS. This macro is identical in usage to TEST_CASE_SELF(), except the test case it defines performs some additional preliminary work:

1. Creates the default task.

2. Creates the “test task” (the task where the test itself runs).

3. Starts the OS.

The body following the macro invocation is what actually runs in the test task. The test task has a priority of OS_MAIN_TASK_PRIO + 1, so it yields to the main task. Thus, tests using this macro typically have the following form:

TEST_CASE_TASK(my_test)
{
    enqueue_event_to_main_task();
    // Event immediately runs to completion.
    TEST_ASSERT(expected_event_result);

    // ...
}

The TEST_CASE_TASK() macro is only usable in self-tests (i.e., tests that are run with newt test).

FIRST_AUX(first, ...)

FIRST(...)

NUM(...)

ARG10(a1, a2, a3, a4, a5, a6, a7, a8, a9, a10, ...)

REST_OR_0(...)

REST_OR_0_AUX(qty, ...)

REST_OR_0_AUX_INNER(qty, ...)

REST_OR_0_AUX_1(first)

REST_OR_0_AUX_N(first, ...)

XSTR(s)

STR(s)

TEST_ASSERT_FULL(fatal, expr, ...)

TEST_ASSERT(...)

TEST_ASSERT_FATAL(...)

TEST_PASS(...)

ASSERT_IF_TEST(expr)

struct ts_suite

#include <testutil.h>

struct tu_config

#include <testutil.h>