From mboxrd@z Thu Jan  1 00:00:00 1970
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	ip-172-31-74-118.ec2.internal
X-Spam-Level: 
X-Spam-Status: No, score=0.6 required=3.0 tests=BAYES_40,NICE_REPLY_A,
	REPLYTO_WITHOUT_TO_CC autolearn=no autolearn_force=no version=3.4.6
Path: eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail
From: "G.B." <bauhaus@notmyhomepage.invalid>
Newsgroups: comp.lang.ada
Subject: Re: Ada and software testing
Date: Fri, 16 Jul 2021 12:01:47 +0200
Organization: A noiseless patient Spider
Message-ID: <scrlec$h26$1@dont-email.me>
References: <871r84cq4r.fsf@nightsong.com>
Reply-To: nonlegitur@notmyhomepage.de
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Fri, 16 Jul 2021 10:01:48 -0000 (UTC)
Injection-Info: reader02.eternal-september.org; posting-host="16ede9abdaa38f7352e6561422ea7240";
	logging-data="17478"; mail-complaints-to="abuse@eternal-september.org";	posting-account="U2FsdGVkX1/A3AVrkVZC91bpDXVWFl+vD3E7ZGpu2CI="
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0)
 Gecko/20100101 Thunderbird/78.11.0
Cancel-Lock: sha1:Ndyo7YBacVZG7NQt5bhZOqCufSY=
In-Reply-To: <871r84cq4r.fsf@nightsong.com>
Content-Language: en-US
Xref: reader02.eternal-september.org comp.lang.ada:62397
List-Id: <comp.lang.ada>

On 12.07.21 02:49, Paul Rubin wrote:

> Well, that went on for longer than I expected.  My questions are
> basically:
> 
>      Q1. Are there good recommendations for Ada testing strategies?  Do
>      the tests resemble the stuff in the Sqlite doc?
> 
>      Q2. Is fuzz testing an important part of Ada testing, and does it
>      tend to find many bugs?

I'd like to add, if I may, a third question, perhaps a a follow-up question,
after having been bitten by a bug that was hidden behind assumptions.

Is there a way of systematically looking for hiding places of bugs
specifically in places external to the program text? And, then, what kind
of mock-ups could establish typical testing patterns? I/O is mentioned
in in the sqlite examples, but what if you do not assume that there
is going to be X .equiv. I/O?

Example: Some external library, of very closed source nature,
exposes an unforeseen behavior. It turns out that a library function
uses a lock, and while waiting for it, the function call times out,
the client program reports failure and terminates normally - with
side effects...

After the fact, after some reading and then some testing, in an adjusted
setup, it all seems plausible. "But, I didn't think of that!".
Educated guesses about what the library might do need to be based
on a vast set of documents, plus the seller of the library
also sells expensive training. Programs need a quick fix, though.

So, what is a proper testing strategy once the programmers have found
that the transitive closure of some call might sometimes incur externally
caused behavior? Such as timeout, or ordering effect due to concurrency?