Remove unnecessary "../library" prefix from test suite include. This
aligns the test suite with the development branch
Signed-off-by: David Horstmann <david.horstmann@arm.com>
When calling `add_xxx_ciphersuites`, we have to set MODE properly.
This commit adjusts order to address this issue in list_test_case
which matches what we do in a normal execution.
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
This commit includes:
- use subprocess.check_output to report error and capture return
value
- add comment as a reminder for option --list-test-case
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
uniform_title is used to print identical format of $TITLE between
--list-test-case and run_client. In such way, no matter how $TITLE
is developed, --list-test-case will in the same format of test case
description as stored in OUTCOME.CSV.
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
Test case description is printed by different block of code. This
causes code maintenance harder since we need to maintain two parts
of code with same functionality. print_test_title is used to
control test case description in compat.sh
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
Test case description in compat.sh is in format of
[ogm]->[ogm] TLSmode, VERIFY CIPHERSUITE_NAME
This program calls compat.sh to list all potential test case
descriptions then checks test case duplication.
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
The option --list-test-case lists all potential test cases without
executing them. The test case description is identical with $TITLE
during test case execution.
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
It isn't used anywhere.
Keep the command line options of all.sh to avoid breaking any wrapper
scripts that people might have.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
None of the tests actually need GNUTLS_LEGACY (3.3.8): GNUTLS (3.4.10)
works.
Only single-DES actually needs OPENSSL_LEGACY (1.0.1j). For the rest,
OPENSSL (1.0.2g) works.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Every now and then, I see of these programs failing with a super-long
usage message that gives no clue as to what went wrong. (Recently it
happened with a test case in ssl-opt.sh with a fairly long command line
that was entirely correct, except some options were not valid in this
config - the test should have been skipped but wasn't due to some other
bug. It took me longer to figure out than it should have, and could have
if the program had simply reported which param was not recognized.)
Also, have an explicit "help" command, separate "help_ciphersuites", and
have default usage message that's not multiple screens long.
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
All options have reasonable default so the programs don't need arguments
to do something useful.
It is widely accepted for programs that can work without arguments need
not insist on the user passing arguments, see 'ls', 'wc', 'sort', 'more'
and any number of POSIX utilities that all work without arguments.
It is also the historical behaviour of those programs, and something
relied one by at least a few team members.
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
Explain how PSA_CRYPTO_DRIVER_TEST_ALL works and why we have it. Note that
it is incompatible with MBEDTLS_PSA_CRYPTO_CONFIG.
MBEDTLS_PSA_CRYPTO_BUILTIN_KEYS is in the full config, so there's no need to
add it explicitly.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
* Turn the warnings produced when finding non-executed tests that
are not in the allow list into errors.
Signed-off-by: Tomás González <tomasagustin.gonzalezorlando@arm.com>
Introduce the --require-full-coverage in analyze_outcomes.py so that
when analyze_outcomes.py --require-full-coverage is called, those
tests that are not executed and are not in the allowed list issue an
error instead of a warning.
Note that it is useful to run analyze_outcomes.py on incomplete test
results, so this error mode needs to remain optional in the long
term.
Signed-off-by: Tomás González <tomasagustin.gonzalezorlando@arm.com>
The allow list explicits which test cases are allowed to not be
executed when testing. This may be, for example, because a feature
is yet to be developed but the test for that feature is already in
our code base.
Signed-off-by: Tomás González <tomasagustin.gonzalezorlando@arm.com>
When MBEDTLS_USE_PSA_CRYPTO is disabled on a base of full, mention it.
Don't explicitly mention MBEDTLS_PSA_CRYPTO_CONFIG when also mentioning
PSA_WANT_xxx, since PSA_WANT symbols are only meaningful when
MBEDTLS_PSA_CRYPTO_CONFIG is enabled.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Having a wrapper made it harder to use: incompatible with setting
MBEDTLS_CONFIG_FILE, harder to combine with other settings. It was also
surprising since it was the only test config that was structured in that
way.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
