OpenHTF and Spintop-OpenHTF offers constructs to sequence test cases and control the execution flow.
PhaseOutcome: Test Phase Outcome
The outcome of a test phase can be one of the following values, which is determined by the PhaseResult described below:
- PhaseOutcome.PASS: The phase result was CONTINUE and all measurements validators passed.
- PhaseOutcome.FAIL: The phase result was CONTINUE and one or more measurements outcome failed, or the phase result was FAIL_AND_CONTINUE or STOP.
- PhaseOutcome.SKIP: The phase result was SKIP or REPEAT.
- PhaseOutcome.ERROR: An uncatched exception was raised, and that exception is not part of the
failure_exceptionslist. The test will stop in such a case.
Although you should not need to import this enum for test sequencing, you can import it using
from openhtf.core.test_record import PhaseOutcome. Usage would be to analyze results.
PhaseResult: Test Phase Flow Control
The result of a test phase is controlled by the value it returns. It can return None (or no return statement) for the standard CONTINUE. If an uncatched exception occurs during the test, the phase will be marked as ERROR. OpenHTF will infer the outcome based on the test phase result.
- PhaseResult.CONTINUE:Causes the framework to process the phase measurement outcomes and execute the next phase.
- PhaseResult.FAIL_AND_CONTINUE: Causes the framework to mark the phase with a fail outcome and execute the next phase.
- PhaseResult.REPEAT: Causes the framework to execute the same phase again, ignoring the measurement outcomes for this instance.
If returned more than the phase's
repeat_limitoption, this will be treated as a STOP.
- PhaseResult.SKIP: Causes the framework to ignore the measurement outcomes and execute the next phase.
- PhaseResult.STOP: Causes the framework to stop executing, indicating a failure. The next phases will not be executed.
You can import this enum using
from openhtf import PhaseResult.
To identify certain exceptions as FAIL instead of errors, you can add failure exceptions to the test plan:
1 2 3 4 5 6
test_plan = TestPlan() test_plan.failure_exceptions += (Exception,) @test_plan.testcase('test1') def my_test(test): raise Exception('!!') # Will mark phase as FAIL instead of ERROR.
Let's say you wish to mark a phase as fail but to continue executing the test. You simply need to return the FAIL_AND_CONTINUE phase result:
1 2 3 4 5 6 7 8
from openhtf import PhaseResult test_plan = TestPlan() @test_plan.testcase('test1') def my_test(test): # (do stuff) return PhaseResult.FAIL_AND_CONTINUE
Phase groups allow to add setup and teardown functions around the test phases. A phase group is a phase in itself, which means that they can be infinitely nested.
Setup and Teardown
Setups and teardown functions can be added to a test sequence using the
1 2 3 4
plan = TestPlan() @plan.teardown('teardown') def teardown(test): # (teardown stuff)
A setup function is very similar to a normal test phase except that it's outcome can skip entirely the group it is part of, including the teardown. If the setup phase
Test Nesting Example
To illustrate how a test can be gradually nested, let's use an incrementally built simple example:
(1) One test case
1 2 3 4 5
plan = TestPlan() @plan.testcase('test1') def test1(test): test.logger.info('test1')
(2) + An empty sub group
sub_group = plan.sub_sequence('sub-group')
sub-group, an empty phase group
(3) + Sub setup, test case and teardown
1 2 3 4 5 6 7 8 9 10 11 12 13 14
@sub_group.setup('sub setup') def sub_setup(test): """Says Sub setup.""" test.logger.info('Sub setup') @sub_group.testcase('sub Hello') def sub_hello(test): """Says Sub hello.""" test.logger.info('Sub hello') @sub_group.teardown('sub cleanup') def sub_cleanup(test): """Says Sub cleanup.""" test.logger.info('Sub cleanup')
(4) + Final teardown
1 2 3 4
@plan.teardown('cleanup') def cleanup(test): """Says Cleaned up.""" test.logger.info('Cleaned up.')
|First noticeable result||test1||sub_setup||sub_hello||sub_cleanup||cleanup|
Things to note:
- Teardown is always executed as soon as one the group test phases is executed. If
cleanupwill be. If
- Setup failure cancels the execution of the group, including the teardown.