11be35a165
To do so, a few dependencies have been imported: * external/bsd/lutok * external/mit/lua * external/public-domain/sqlite * external/public-domain/xz The Kyua framework is the new generation of ATF (Automated Test Framework), it is composed of: * external/bsd/atf * external/bsd/kyua-atf-compat * external/bsd/kyua-cli * external/bsd/kyua-tester * tests Kyua/ATF being written in C++, it depends on libstdc++ which is provided by GCC. As this is not part of the sources, Kyua is only compiled when the native GCC utils are installed. To install Kyua do the following: * In a cross-build enviromnent, add the following to the build.sh commandline: -V MKBINUTILS=yes -V MKGCCCMDS=yes WARNING: At this point the import is still experimental, and not supported on native builds (a.k.a make build). Change-Id: I26aee23c5bbd2d64adcb7c1beb98fe0d479d7ada
232 lines
7.8 KiB
Groff
232 lines
7.8 KiB
Groff
.\"
|
|
.\" Automated Testing Framework (atf)
|
|
.\"
|
|
.\" Copyright (c) 2007 The NetBSD Foundation, Inc.
|
|
.\" All rights reserved.
|
|
.\"
|
|
.\" Redistribution and use in source and binary forms, with or without
|
|
.\" modification, are permitted provided that the following conditions
|
|
.\" are met:
|
|
.\" 1. Redistributions of source code must retain the above copyright
|
|
.\" notice, this list of conditions and the following disclaimer.
|
|
.\" 2. Redistributions in binary form must reproduce the above copyright
|
|
.\" notice, this list of conditions and the following disclaimer in the
|
|
.\" documentation and/or other materials provided with the distribution.
|
|
.\"
|
|
.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND
|
|
.\" CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
|
|
.\" INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
|
|
.\" MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
|
|
.\" IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY
|
|
.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
|
|
.\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
|
|
.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
|
|
.\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
|
|
.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
|
|
.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
.\"
|
|
.Dd December 20, 2011
|
|
.Dt ATF-FORMATS 5
|
|
.Os
|
|
.Sh NAME
|
|
.Nm atf-formats
|
|
.Nd machine-parseable data formats used by ATF
|
|
.Sh DESCRIPTION
|
|
This manual page describes the multiple data formats used in ATF.
|
|
These formats affect configuration files, control files and any data that
|
|
is externalized or internalized by the tools.
|
|
.Pp
|
|
Data files are always organized as follows:
|
|
.Bd -literal -offset indent
|
|
Header1: Value1 \\
|
|
... | head
|
|
HeaderN: ValueN /
|
|
mandatory blank line
|
|
Free-form text contents \\
|
|
... | body
|
|
... /
|
|
.Ed
|
|
.Pp
|
|
A file must always contain a
|
|
.Sq Content-Type
|
|
header and must always separate that header from the body with a blank
|
|
line, even if the body is empty.
|
|
.Pp
|
|
The
|
|
.Sq Content-Type
|
|
is always of the form:
|
|
.Bd -literal -offset indent
|
|
Content-Type: application/X-atf-<subtype>; version="<version>"
|
|
.Ed
|
|
.Pp
|
|
where
|
|
.Sq subtype
|
|
indicates the specific file format and
|
|
.Sq version
|
|
its format version.
|
|
This header must be the first one of the file.
|
|
.Pp
|
|
The main purpose of the
|
|
.Sq Content-Type
|
|
header, aside from determining the format used in the file, is to allow
|
|
future changes to a given format.
|
|
Whenever an incompatible change is made, the version is bumped by one.
|
|
By keeping the header in the first line, future versions may even remove
|
|
the need for such a header -- e.g. if some format was replaced by XML
|
|
files, which have their own mandatory header.
|
|
.Pp
|
|
The rest of this document details the different format types.
|
|
.Ss Format: application/X-atf-atffile, version: 1
|
|
Atffiles are logically divided into three sections:
|
|
.Bl -bullet
|
|
.It
|
|
Test programs: the list of test programs that define the test suite
|
|
described by the Atffile.
|
|
.It
|
|
Meta-data properties: these define some constant values applicable to
|
|
all the test programs defined in the file.
|
|
In some sense they define the properties that describe the test suite.
|
|
.It
|
|
Configuration variables: defaults for configuration variables that
|
|
can be overridden through configuration files or the command line.
|
|
.El
|
|
.Pp
|
|
The grammar for Atffiles is the following:
|
|
.Bd -literal -offset indent
|
|
DATA ::= ( ( CONF | PROP | TP )? COMMENT? NEWLINE )* EOF
|
|
CONF ::= 'conf:' WORD EQUAL STRING
|
|
PROP ::= 'prop:' WORD EQUAL STRING
|
|
TP ::= TPFILE | TPGLOB
|
|
TPFILE ::= 'tp: ' STRING
|
|
TPGLOB ::= 'tp-glob: ' STRING
|
|
STRING ::= WORD | '"' WORD* '"'
|
|
.Ed
|
|
.Pp
|
|
The meaning of the constructions above is:
|
|
.Bl -tag -width TPGLOBXX
|
|
.It CONF
|
|
Definition of a configuration variable.
|
|
.It PROP
|
|
Definition of a meta-data property.
|
|
.It TPFILE
|
|
Addition of a test program into the test suite.
|
|
The string is taken literally as the program's name, and this program
|
|
must exist.
|
|
.It TPGLOB
|
|
Addition of multiple test programs into the test suite.
|
|
The string is taken as a glob pattern, which may have or not have any
|
|
matches in the current directory.
|
|
.El
|
|
.Pp
|
|
An example:
|
|
.Bd -literal -offset indent
|
|
prop: test-suite = utilities
|
|
|
|
conf: unprivileged-user = nobody
|
|
|
|
tp: t_cp
|
|
tp: t_mv
|
|
tp: t_df
|
|
tp-glob: t_dir_*
|
|
.Ed
|
|
.Ss Format: application/X-atf-config, version: 1
|
|
Configuration files are very simple: they only contain a list of variable
|
|
name/variable value pairs.
|
|
Their grammar is:
|
|
.Bd -literal -offset indent
|
|
DATA ::= ( VAR? COMMENT? NEWLINE )* EOF
|
|
VAR ::= WORD EQUAL STRING
|
|
COMMENT ::= HASH WORD*
|
|
STRING ::= WORD | '"' WORD* '"'
|
|
.Ed
|
|
.Pp
|
|
An example:
|
|
.Bd -literal -offset indent
|
|
# This is the system-wide configuration file for ATF.
|
|
# The above and this line are comments placed on their own line.
|
|
|
|
var1 = this is a variable value
|
|
var2 = this is another one # Optional comment at the end.
|
|
.Ed
|
|
.Ss Format: application/X-atf-tps, version: 3
|
|
The
|
|
.Sq application/X-atf-tps
|
|
format multiplexes the standard output, standard error and results output
|
|
streams from multiple test programs into a single data file.
|
|
This format is used by
|
|
.Xr atf-run 1
|
|
to report the execution of several test programs and is later parsed by
|
|
.Xr atf-report 1
|
|
to inform the user of this process.
|
|
It has the following grammar:
|
|
.Bd -literal -offset indent
|
|
DATA ::= INFO* TPS-COUNT TP-STANZA* INFO* EOF
|
|
INFO ::= 'info' COLON STRING COMMA STRING NEWLINE
|
|
TPS-COUNT ::= 'tps-count' COLON POSITIVE-NUMBER NEWLINE
|
|
TP-STANZA ::= TP-START TC-STANZA* TC-END
|
|
TP-START ::= 'tp-start' COLON TIMESTAMP COMMA STRING COMMA
|
|
POSITIVE-NUMBER NEWLINE
|
|
TP-END ::= 'tc-end' COLON TIMESTAMP COMMA STRING (COMMA STRING)?
|
|
TC-STANZA ::= TC-START (TC-SO | TC-SE)* TC-END
|
|
TC-START ::= 'tc-start' COLON TIMESTAMP COMMA STRING NEWLINE
|
|
TC-SO ::= 'tc-so' COLON STRING NEWLINE
|
|
TC-SE ::= 'tc-se' COLON STRING NEWLINE
|
|
TC-END ::= 'tc-end' COLON TIMESTAMP COMMA STRING COMMA TCR NEWLINE
|
|
TCR ::= 'passed' | ('failed' | 'skipped') COMMA STRING
|
|
TIMESTAMP ::= [0-9]+.[0-9]+
|
|
.Ed
|
|
.Pp
|
|
The meaning of the constructions above is:
|
|
.Bl -tag -width TPSXCOUNTXX
|
|
.It TPS-COUNT
|
|
Indicates the number of test programs that will be executed.
|
|
There will be this exact amount of
|
|
.Sq TP-STANZA
|
|
constructions following it.
|
|
.It TP-START
|
|
Indicates the beginning of a test program.
|
|
This includes the program's name and the amount of test cases that
|
|
will follow.
|
|
.It TP-END
|
|
Indicates the completion of a test program.
|
|
This is followed by the program's name and, if the program ended
|
|
prematurely, an error message indicating the reason of its failure.
|
|
A successful execution of a test program (regardless of the status of its
|
|
test cases) must not be accompanied by any reason.
|
|
.It TC-START
|
|
Indicates the beginning of a test case.
|
|
This is accompanied by the test case's name.
|
|
.It TC-SO
|
|
Contains a text line sent to the standard output stream during the
|
|
execution of the test case.
|
|
Leading and trailing space is preserved.
|
|
.It TC-SE
|
|
Contains a text line sent to the standard error stream during the
|
|
execution of the test case.
|
|
Leading and trailing space is preserved.
|
|
.It TC-END
|
|
Indicates the completion of a test case.
|
|
This is accompanied by the test case's name, its result and the reason
|
|
associated with this result (if applicable).
|
|
.El
|
|
.Pp
|
|
An example:
|
|
.Bd -literal -offset indent
|
|
tps-count: 2
|
|
tp-start: calculator, 1324318951.838923, 2
|
|
tc-start: add, 1324318951.839101
|
|
tc-end: add, 1324318951.839500, passed
|
|
tc-start: subtract, 1324318951.840001
|
|
tc-so: 3-2 expected to return 1 but got 0
|
|
tc-end: subtract, 1324318952.000123, failed, Calculated an unexpected value
|
|
tp-end: calculator, 1324318952.002301
|
|
tp-start: files, 1, 1324318952.502348
|
|
tc-start: copy, 1324318952.508291
|
|
tc-se: could not find the cp(1) utility
|
|
tc-end: copy, 1324318953.203145, skipped
|
|
tp-end: files, 1324318953.203800
|
|
.Ed
|
|
.Sh SEE ALSO
|
|
.Xr atf 7
|