Ada and software testing

[Paul Rubin <no.email@nospam.invalid>]

I wonder if there is good guidance around for software testing in the
Ada world, or if it depends too closely on the application area.  I'm
aware of DO-178B and DO-178C in the aviation world, though I haven't
studied either of them.  Sqlite's document about its testing procedure
is also interesting and maybe a cautionary tale.  Sqlite is a really
nice SQL database whose main misfortune from the Ada perspective is that
it is written in C.  Its testing doc is here:

   https://sqlite.org/testing.html

and a little more info can be found in this interview with the author:

  https://corecursive.com/066-sqlite-with-richard-hipp/#testing-and-aviation-standards


Overview:

1. Sqlite originally had only ad hoc testing.  Then the author
(Dr. Richard Hipp) did some work with Rockwell, heard about DO-178B
there, and embarked on a large effort to strenghten Sqlite's testing in
accordance with DO-178B.  Particularly, the Sqlite team created an
enormous suite of unit tests aiming to get 100% MC/DC test coverage.
That is, for any "if" statement, there must be tests that exercise both
branches of the "if".  This seemingly got Sqlite to be very reliable.

2. Later on, fuzz testing came into vogue, so they started fuzzing
Sqlite.  This in fact found a bunch of crashes and vulnerabilities that
were duly fixed, and nonstop fuzzing was added to the test setup.  But
the testing document (section 4.1.6) notes a tension between MC/DC and
fuzzing: MC/DC requires deep parts of the code to be reachable by test
inputs, while fuzz protection tends to use defensive programming against
"impossible" inputs, resulting in seemingly unreachable code.  Fuzz
testing has been effective enough at finding bugs in C programs that it
has now displaced a lot of static analysis in the C world.

3. Sqlite uses a little bit of static analysis (section 11) but the
document says it has not helped much.  Ada on the other hand uses static
analysis extensively, both in its fine grained type system (compared
with C's) and using tools like SPARK.

4. Bugs found by fuzz testing C programs are typically the standard C
hazards like buffer overflows, undefined behaviour (UB) from bad
arithmetic operands, etc.  I'm of the impression that Ada is less
susceptible to these bugs because of mandatory range checking and less
UB in the language.

Well, that went on for longer than I expected.  My questions are
basically:

    Q1. Are there good recommendations for Ada testing strategies?  Do
    the tests resemble the stuff in the Sqlite doc?

    Q2. Is fuzz testing an important part of Ada testing, and does it
    tend to find many bugs?

Thanks!