View Source ct_suite behaviour (common_test v1.27.3)

The following section describes the mandatory and optional test suite functions that Common Test calls during test execution. For more details, see section Writing Test Suites in the User's Guide.

Summary

Types

ct_config()

The configuration data that can be modified.

ct_group_def()

The test group definition, as returned by Module:groups/0.

ct_group_props()

ct_group_props_ref()

ct_group_ref()

ct_group_repeat_type()

ct_groupname()

The name of the test group.

ct_hooks()

ct_info()

The test suite information, as returned by Module:suite/0, Module:group/1 and Module:Testcase/0.

ct_info_required()

ct_info_required_subkeys()

ct_info_timetrap()

ct_info_timetrap_fun()

ct_status()

The status value for a nested subgroup.

ct_subgroups_def()

ct_test_def()

The test suite definition, as returned by Module:all/0.

ct_test_repeat()

ct_testcase_ref()

ct_testcase_repeat_prop()

ct_testname()

The name of the testcase function.

Callbacks: Callback Functions

all()

Returns the list of all test cases and test case groups in the test suite module to be executed.

end_per_group(GroupName, Config)

This function is called after the execution of a test case group is finished. It is meant to be used for cleaning up after Module:init_per_group/2.

end_per_suite(Config)

This function is called as the last test case in the suite. It is meant to be used for cleaning up after Module:init_per_suite/1.

end_per_testcase(TestCase, Config)

This function is called after each test case, and can be used to clean up after Module:init_per_testcase/2 and the test case.

group(GroupName)

The test case group information function. It is supposed to return a list of tagged tuples that specify various properties related to the execution of a test case group (that is, its test cases and subgroups). Properties set by Module:group/1 override properties with the same key that have been set previously by Module:suite/0.

groups()

Defines test case groups. For details, see section Test Case Groups in the User's Guide.

init_per_group(GroupName, Config)

This configuration function is called before execution of a test case group. It typically contains initializations that are common for all test cases and subgroups in the group, and that must only be performed once.

init_per_suite(Config)

This configuration function is called as the first function in the suite. It typically contains initializations that are common for all test cases in the suite, and that must only be done once.

init_per_testcase(TestCase, Config)

This function is called before each test case.

suite()

The test suite information function. Returns a list of tagged tuples specifying various properties related to the execution of this test suite (common for all test cases in the suite).

'Testcase'()

The test case information function. It is supposed to return a list of tagged tuples that specify various properties related to the execution of this particular test case. Properties set by Module:Testcase/0 override properties set previously for the test case by Module:group/1 or Module:suite/0.

'Testcase'(Config)

The implementation of a test case. Call the functions to test and check the result. If something fails, ensure the function causes a runtime error or call ct:fail/1,2 (which also causes the test case process to terminate).

Types

ct_config()

-type ct_config() :: [{Key :: atom(), Value :: term()}].

The configuration data that can be modified.

ct_group_def()

-type ct_group_def() ::
          {ct_groupname(),
           ct_group_props(),
           [ct_testname() | ct_group_def() | {group, ct_groupname()} | ct_testcase_ref()]}.

The test group definition, as returned by Module:groups/0.

ct_group_props()

(not exported)

-type ct_group_props() ::
          [parallel | sequence | shuffle |
           {shuffle, Seed :: {integer(), integer(), integer()}} |
           {ct_group_repeat_type(), ct_test_repeat()}].

ct_group_props_ref()

(not exported)

-type ct_group_props_ref() :: ct_group_props() | default.

ct_group_ref()

(not exported)

-type ct_group_ref() ::
          {group, ct_groupname()} |
          {group, ct_groupname(), ct_group_props_ref()} |
          {group, ct_groupname(), ct_group_props_ref(), ct_subgroups_def()}.

ct_group_repeat_type()

(not exported)

-type ct_group_repeat_type() ::
          repeat | repeat_until_all_ok | repeat_until_all_fail | repeat_until_any_ok |
          repeat_until_any_fail.

ct_groupname()

-type ct_groupname() :: atom().

The name of the test group.

ct_hooks()

(not exported)

-type ct_hooks() ::
          [CTHModule ::
               atom() |
               {CTHModule :: atom(), CTHInitArgs :: term()} |
               {CTHModule :: atom(), CTHInitArgs :: term(), CTHPriority :: integer()}].

ct_info()

-type ct_info() ::
          {timetrap, ct_info_timetrap()} |
          {require, ct_info_required()} |
          {require, Name :: atom(), ct_info_required()} |
          {userdata, UserData :: term()} |
          {silent_connections, Conns :: [atom()]} |
          {stylesheet, CSSFile :: string()} |
          {ct_hooks, CTHs :: ct_hooks()}.

The test suite information, as returned by Module:suite/0, Module:group/1 and Module:Testcase/0.

ct_info_required()

(not exported)

-type ct_info_required() ::
          Key ::
              atom() |
              {Key :: atom(), SubKeys :: ct_info_required_subkeys()} |
              {Key :: atom(), SubKey :: atom()} |
              {Key :: atom(), SubKey :: atom(), SubKeys :: ct_info_required_subkeys()}.

ct_info_required_subkeys()

(not exported)

-type ct_info_required_subkeys() :: SubKey :: atom() | [SubKey :: atom()].

ct_info_timetrap()

(not exported)

-type ct_info_timetrap() ::
          MilliSec ::
              integer() |
              {seconds, integer()} |
              {minutes, integer()} |
              {hours, integer()} |
              {Mod :: atom(), Func :: atom(), Args :: list()} |
              ct_info_timetrap_fun().

ct_info_timetrap_fun()

(not exported)

-type ct_info_timetrap_fun() :: fun().

ct_status()

-type ct_status() :: ok | skipped | failed.

The status value for a nested subgroup.

ct_subgroups_def()

(not exported)

-type ct_subgroups_def() ::
          {ct_groupname(), ct_group_props_ref()} |
          {ct_groupname(), ct_group_props_ref(), ct_subgroups_def()}.

ct_test_def()

-type ct_test_def() :: ct_testname() | ct_group_ref() | ct_testcase_ref().

The test suite definition, as returned by Module:all/0.

ct_test_repeat()

(not exported)

-type ct_test_repeat() :: integer() | forever.

ct_testcase_ref()

(not exported)

-type ct_testcase_ref() :: {testcase, ct_testname(), ct_testcase_repeat_prop()}.

ct_testcase_repeat_prop()

(not exported)

-type ct_testcase_repeat_prop() ::
          [{repeat, ct_test_repeat()} |
           {repeat_until_ok, ct_test_repeat()} |
           {repeat_until_fail, ct_test_repeat()}].

ct_testname()

-type ct_testname() :: atom().

The name of the testcase function.

Callbacks: Callback Functions

all()

-callback all() -> [TestDef :: ct_test_def()] | {skip, Reason :: term()}.

Returns the list of all test cases and test case groups in the test suite module to be executed.

This list also specifies the order the cases and groups are executed by Common Test. A test case is represented by an atom, the name of the test case function, or a testcase tuple indicating that the test case shall be repeated. A test case group is represented by a group tuple, where GroupName, an atom, is the name of the group (defined in Module:groups/0). Execution properties for groups can also be specified, both for a top-level group and for any of its subgroups. Group execution properties specified here override properties in the group definition (see Module:groups/0). (With value default, the group definition properties are used).

If {skip, Reason} is returned, all test cases in the module are skipped and Reason is printed on the HTML result page.

For details on groups, see section Test Case Groups in the User's Guide.

end_per_group(GroupName, Config)

(optional)

-callback end_per_group(GroupName :: ct_groupname(), Config :: ct_config()) ->
                           term() | {return_group_result, Status :: ct_status()}.

This function is called after the execution of a test case group is finished. It is meant to be used for cleaning up after Module:init_per_group/2.

A status value for a nested subgroup can be returned with {return_group_result, Status}. The status can be retrieved in Module:end_per_group/2 for the group on the level above. The status is also used by Common Test for deciding if execution of a group is to proceed if property sequence or repeat_until_* is set.

For details about test case groups, see section Test Case Groups in the User's Guide.

If this function is defined, then Module:init_per_group/2 must also be defined.

end_per_suite(Config)

(optional)

-callback end_per_suite(Config :: ct_config()) -> term() | {save_config, SaveConfig :: ct_config()}.

This function is called as the last test case in the suite. It is meant to be used for cleaning up after Module:init_per_suite/1.

For information on save_config, see section Saving Configuration Data in the User's Guide.

If this function is defined, then Module:init_per_suite/1 must also be defined.

end_per_testcase(TestCase, Config)

(optional)

-callback end_per_testcase(TestCase :: ct_testname(), Config :: ct_config()) ->
                              term() | {fail, Reason :: term()} | {save_config, SaveConfig :: ct_config()}.

This function is called after each test case, and can be used to clean up after Module:init_per_testcase/2 and the test case.

Any return value (besides {fail, Reason} and {save_config, SaveConfig}) is ignored. By returning {fail, Reason}, TestCase is marked as faulty (even though it was successful in the sense that it returned a value instead of terminating).

For information on save_config, see section Saving Configuration Data in the User's Guide.

If this function is defined, then Module:init_per_testcase/2 must also be defined.

group(GroupName)

(optional) (since OTP R15B)

-callback group(GroupName :: ct_groupname()) -> [Info :: ct_info()].

Tag timetrap sets the maximum time that each test case is allowed to execute (including Module:init_per_testcase/2 and Module:end_per_testcase/2). If the timetrap time is exceeded, the test case fails with reason timetrap_timeout. A TimeFunc function can be used to set a new timetrap by returning a TimeVal. It can also be used to trigger a timetrap time-out by, at some point, returning a value other than a TimeVal. For details, see section Timetrap Time-Outs in the User's Guide.

Tag require specifies configuration variables required by test cases (or configuration functions) in the suite. If the required configuration variables are not found in any of the configuration files, all test cases in this group are skipped. For details about the require functionality, see function ct:require/1,2.

With userdata, the user can specify any test case group related information that can be read by calling ct:userdata/2.

Tag ct_hooks specifies the Common Test Hooks to be run with this suite.

Other tuples than the ones defined are ignored.

For details about the test case group information function, see section Group Information Function in the User's Guide.

groups()

(optional)

-callback groups() -> [GroupDef :: ct_group_def()].

Defines test case groups. For details, see section Test Case Groups in the User's Guide.

init_per_group(GroupName, Config)

(optional)

-callback init_per_group(GroupName :: ct_groupname(), Config :: ct_config()) ->
                            NewConfig :: ct_config() | {skip, Reason :: term()}.

GroupName is the name of the group, as specified in the group definition (see Module:groups/0). Parameter Config is the configuration data that can be modified. The return value of this function is given as Config to all test cases and subgroups in the group.

If {skip, Reason} is returned, all test cases in the group are skipped and Reason is printed in the overview log for the group.

For information about test case groups, see section Test Case Groups in the User's Guide.

If this function is defined, then Module:end_per_group/2 must also be defined.

init_per_suite(Config)

(optional)

-callback init_per_suite(Config :: ct_config()) ->
                            NewConfig ::
                                ct_config() |
                                {skip, Reason :: term()} |
                                {skip_and_save, Reason :: term(), SaveConfig :: ct_config()}.

This configuration function is called as the first function in the suite. It typically contains initializations that are common for all test cases in the suite, and that must only be done once.

Parameter Config is the configuration data that can be modified. Whatever is returned from this function is specified as Config to all configuration functions and test cases in the suite.

If {skip, Reason} is returned, all test cases in the suite are skipped and Reason is printed in the overview log for the suite.

For information on save_config and skip_and_save, see section Saving Configuration Data in the User's Guide.

If this function is defined, then Module:end_per_suite/1 must also be defined.

init_per_testcase(TestCase, Config)

(optional)

-callback init_per_testcase(TestCase :: ct_testname(), Config :: ct_config()) ->
                               NewConfig ::
                                   ct_config() | {fail, Reason :: term()} | {skip, Reason :: term()}.

This function is called before each test case.

Argument TestCase is the test case name, and Config (list of key-value tuples) is the configuration data that can be modified. The NewConfig list returned from this function is given as Config to the test case.

If {fail, Reason} is returned, the test case is marked as failed without being executed.

If {skip, Reason} is returned, the test case is skipped and Reason is printed in the overview log for the suite.

If this function is defined, then Module:end_per_testcase/2 must also be defined.

suite()

(optional)

-callback suite() -> [Info :: ct_info()].

The test suite information function. Returns a list of tagged tuples specifying various properties related to the execution of this test suite (common for all test cases in the suite).

Tag require specifies configuration variables required by test cases (or configuration functions) in the suite. If the required configuration variables are not found in any of the configuration files, all test cases are skipped. For details about the require functionality, see function ct:require/1,2.

With userdata, the user can specify any test suite-related information, which can be read by calling ct:userdata/2.

Tag ct_hooks specifies the Common Test Hooks to be run with this suite.

Other tuples than the ones defined are ignored.

For details about the test suite information function, see section Test Suite Information Function in the User's Guide.

'Testcase'()

(optional) (since OTP R14B)

-callback 'Testcase'() -> [ct_info()].

Tag timetrap sets the maximum time that the test case is allowed to execute. If the timetrap time is exceeded, the test case fails with reason timetrap_timeout. Module:init_per_testcase/2 and Module:end_per_testcase/2 are included in the timetrap time. A TimeFunc function can be used to set a new timetrap by returning a TimeVal. It can also be used to trigger a timetrap time-out by, at some point, returning a value other than a TimeVal. For details, see section Timetrap Time-Outs in the User's Guide.

Tag require specifies configuration variables that are required by the test case (or init_per_testcase/2 or end_per_testcase/2). If the required configuration variables are not found in any of the configuration files, the test case is skipped. For details about the require functionality, see function ct:require/1,2.

If timetrap or require is not set, the default values specified by Module:suite/0 (or Module:group/1) are used.

With userdata, the user can specify any test case-related information that can be read by calling ct:userdata/3.

Other tuples than the ones defined are ignored.

For details about the test case information function, see section Test Case Information Function in the User's Guide.

'Testcase'(Config)

(optional) (since OTP R14B)

-callback 'Testcase'(Config) ->
                        term() |
                        {skip, Reason} |
                        {fail, Reason} |
                        {comment, Comment} |
                        {save_config, SaveConfig} |
                        {skip_and_save, Reason, SaveConfig}
                        when
                            Config :: ct_config(),
                            SaveConfig :: ct_config(),
                            Reason :: term(),
                            Comment :: string().

Elements from the Config list can, for example, be read with proplists:get_value/2 in STDLIB.

Possible return values are:

{fail, Reason} - The test case is considered failed, and the Reason will be logged.
{skip, Reason} - The test case is considered skipped, and the Reason will be logged.
{comment, Comment} - The test case is considered successful, and the Comment will be logged.
{save_config, SaveConfig} - The test case is considered successful, and the SaveConfig will be stored in the Config (see section Saving Configuration Data in the User's Guide).
{skip_and_save, Reason, SaveConfig} - The test case is considered skipped, the Reason will be logged, and the SaveConfig will be stored in the Config (see section Saving Configuration Data in the User's Guide).

If the function returns any other term, the test case is considered successful.

If the test case function crashes, it is considered failed.

For details about test case implementation, see section Test Cases in the User's Guide.