00b67f09dd
Also known as ISC bind. This import adds utilities such as host(1), dig(1), and nslookup(1), as well as many other tools and libraries. Change-Id: I035ca46e64f1965d57019e773f4ff0ef035e4aa3
185 lines
6.9 KiB
Plaintext
185 lines
6.9 KiB
Plaintext
Things to do Automated Testing Framework
|
|
===========================================================================
|
|
|
|
|
|
Last revised: November 30th, 2010
|
|
|
|
|
|
This document includes the list of things that need to be done in ATF that
|
|
are most requested by the users. This information used to be available in
|
|
an ad-hoc bug tracker but that proved to be a bad idea. I have collected
|
|
all worthy comments in here.
|
|
|
|
Please note that most work these days is going into Kyua (see
|
|
http://code.google.com/p/kyua/). The ideas listed here apply to the
|
|
components of ATF that have *not* been migrated to the new codebase yet.
|
|
For bug reports or ideas that apply to the components that already have
|
|
been migrated, please use the bug tracker in the URL above. Similarly,
|
|
whenever a component is migrated, the ideas in this file should be revised
|
|
and migrated to the new bug tracker where appropriate.
|
|
|
|
|
|
---------------------------------------------------------------------------
|
|
Add build-time checks to atf-sh
|
|
|
|
The 0.7 release introduced build-time tests to atf-c and atf-c++, but not
|
|
to atf-sh. Expose the functionality to the shell interface.
|
|
|
|
This will probably require writing an atf-build utility that exposes the C
|
|
code and can be called from the shell.
|
|
|
|
---------------------------------------------------------------------------
|
|
Revisit what to do when an Atffile lists a non-existent file
|
|
|
|
---------------------------------------------------------------------------
|
|
Add ATF_CHECK* versions to atf-c++ to support non-fatal tests
|
|
|
|
---------------------------------------------------------------------------
|
|
Implement race-condition tests
|
|
|
|
gcooper:
|
|
|
|
I would think that stress/negative tests would be of more value than race
|
|
condition tests (they're similar, but not exactly the same in my mind).
|
|
|
|
In particular,
|
|
|
|
1. Feed through as much data as possible to determine where reporting
|
|
breaks down.
|
|
2. Feed through data quickly and terminate ASAP. The data should be
|
|
captured. Terminate child applications with unexpected exit codes and
|
|
signals (in particular, SIGCHLD, SIGPIPE, exit codes that terminate,
|
|
etc).
|
|
3. Open up a file descriptor in the test application, don't close the file
|
|
descriptor.
|
|
4. fork(2) a process; don't wait(2) for the application to complete.
|
|
|
|
There are other scenarios that could be exercised, but these are the ones
|
|
I could think of off the topic of my head.
|
|
|
|
--
|
|
|
|
jmmv:
|
|
|
|
1. The thing is: how do you express any of this in a portable/abstract
|
|
interface? How do you express that a test case "receives data"? What
|
|
does that exactly mean? I don't think the framework should care about
|
|
this: each test should be free to decide where its data is and how to
|
|
deal with it.
|
|
|
|
2. Ditto.
|
|
|
|
3. Not sure I understand your request, but testing for "unexpected exit
|
|
codes" is already supported. See wiki:DesignXFail for the feature
|
|
design details.
|
|
|
|
4. What's the problem with this case? The test case exits right away after
|
|
terminating the execution of its body; any open file descriptors,
|
|
leaked memory, etc. die with it.
|
|
|
|
5. forking and not waiting for a subprocess was a problem already
|
|
addressed.
|
|
|
|
I kinda have an idea of what Antti means with "race condition tests", but
|
|
every time I have tried to describe my understanding of matters I seem to
|
|
be wrong. Would be nice to have a clear description of what this involves;
|
|
in particular, what are the expectations from the framework and how should
|
|
the feature be exposed.
|
|
|
|
As of now, what I understand by "race condition test" is: a test case that
|
|
exercises a race condition. The test case may finish without triggering
|
|
the race, in which case it just exists with a successful status.
|
|
Otherwise, if the race triggers, the test case gets stuck and times out.
|
|
The result should be reported as an "expected failure" different from
|
|
timeout.
|
|
|
|
--
|
|
|
|
pooka:
|
|
|
|
Yup. Plus some atf-wide mechanism for the operator to supply some kind of
|
|
guideline if the test should try to trigger the race for a second or for
|
|
an hour.
|
|
|
|
--
|
|
|
|
jmmv:
|
|
|
|
Alright. While mocking up some code for this, I think that your two
|
|
requests are complementary.
|
|
|
|
On the one hand, when you are talking about a "race condition" test you
|
|
really mean an "expected race condition" test. Correct? If so, we need to
|
|
extend the xfail mechanism to add one more case, which is to report any
|
|
failures as a race condition error and, if there is no failure, report the
|
|
test as successful.
|
|
|
|
On the other hand, the atf-wide mechanism to support how long the test
|
|
should run for can be thought as a "stress test" mechanism. I.e. run this
|
|
test for X time / iterations and report its results regularly without
|
|
involving xfail at all.
|
|
|
|
So, with this in mind:
|
|
|
|
* For a test that triggers an unfixed race condition, you set xfail to
|
|
race mode and define the test as a stress test. Any failures are
|
|
reported as expected failures.
|
|
|
|
* For a test that verifies a supposedly-fixed race condition, you do *not*
|
|
set xfail to race mode, and only set the test to stress test. Any
|
|
failures are reported as real failures.
|
|
|
|
These stress test cases implement a single iteration of the test and
|
|
atf-run is in charge of running the test several times, stopping on the
|
|
first failure.
|
|
|
|
Does that make sense?
|
|
|
|
---------------------------------------------------------------------------
|
|
Implement ATF_REQUIRE_ERRNO
|
|
|
|
pooka:
|
|
|
|
Most of the lines in tests against system functionality are:
|
|
|
|
if (syscall(args) == -1)
|
|
atf_tc_fail_errno("flop")
|
|
|
|
Some shorthand would be helpful, like ATF_REQUIRE_ERRNO(syscall(args))
|
|
Also, a variant which allows arbitrary return value checks (e.g. "!= 0" or
|
|
"< 124" or "!= size") would be nice.
|
|
|
|
--
|
|
|
|
gcooper:
|
|
|
|
There's a problem with this request; not all functions fail in the same
|
|
way ... in particular compare the pthread family of functions (which
|
|
return errno) vs many native syscalls. Furthermore, compare some
|
|
fcntl-like syscalls vs other syscalls. One size fits all solutions may not
|
|
be a wise idea in this case, so I think that the problem statement needs
|
|
to be better defined, because the above request is too loose.
|
|
|
|
FWIW, there's also a TEST macro in LTP, which tests for non-zero status,
|
|
and sets an appropriate set of global variables for errnos and return
|
|
codes, respectively. It was a good idea, but has been mostly abandoned
|
|
because it's too difficult to define a success and failure in a universal
|
|
manner, so I think that we need to be careful with what's implemented in
|
|
ATF to not repeat the mistakes that others have made.
|
|
|
|
--
|
|
|
|
jmmv:
|
|
|
|
I think you've got a good point.
|
|
|
|
This was mostly intended to simplify the handling of the stupid errno
|
|
global variable. I think this is valuable to have, but maybe the
|
|
macro/function name should be different because _ERRNO can be confusing.
|
|
Probably something like an ATF_CHECK_LIBC / ATF_CHECK_PTHREAD approach
|
|
would be more flexible and simple.
|
|
|
|
|
|
===========================================================================
|
|
vim: filetype=text:textwidth=75:expandtab:shiftwidth=2:softtabstop=2
|