From: "Björn Lundin" <b.f.lundin@gmail.com>
Subject: Re: Fixed vs float and precision and conversions
Date: Thu, 9 Jul 2020 09:23:30 +0200 [thread overview]
Message-ID: <re6glj$bs1$1@dont-email.me> (raw)
In-Reply-To: <re6cvd$1udg$1@gioia.aioe.org>
Den 2020-07-09 kl. 08:20, skrev Dmitry A. Kazakov:
> On 08/07/2020 21:41, Björn Lundin wrote:
>
>> As I replied to Niklas:
>> It is at least not certain that the number expressed in fixed_type is
>> one of them in the array below.
>
> The array has varying step 0.01 to 10.0, your test has step 0.001. What
> do you expect? Of course it must fail at a very high rate from 9/1 to
> 999/1, practically always.
>
welll, in practice not always.
It only fails when - like my example -
Fixed_Type(float(5.10)) becomes 5.099
and its been running with that construct since 2014,
and I only noticed now. So - not always, actually very seldom.
I did expect floating comparison error to go away using fixed type,
but I forgot about 'garbage in -> garbage out'
I don't have that kind of rounding errors once the values are converted
to fixed_type but I did not realize that I had them in the conversion.
--
Björn
next prev parent reply other threads:[~2020-07-09 7:23 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-07-07 21:10 Fixed vs float and precision and conversions Björn Lundin
2020-07-07 21:30 ` Niklas Holsti
2020-07-08 8:17 ` Björn Lundin
2020-07-07 21:58 ` Shark8
2020-07-08 8:15 ` Björn Lundin
2020-07-08 16:16 ` Shark8
2020-07-08 18:08 ` Björn Lundin
2020-07-08 18:10 ` Björn Lundin
2020-07-08 18:21 ` Niklas Holsti
2020-07-08 19:39 ` Björn Lundin
2020-07-08 20:34 ` Niklas Holsti
2020-07-08 21:24 ` Björn Lundin
2020-07-09 7:11 ` Niklas Holsti
2020-07-08 18:36 ` Dmitry A. Kazakov
2020-07-08 19:41 ` Björn Lundin
2020-07-09 6:20 ` Dmitry A. Kazakov
2020-07-09 7:23 ` Björn Lundin [this message]
2020-07-09 7:49 ` Dmitry A. Kazakov
2020-07-08 21:16 ` Shark8
2020-07-08 21:47 ` Björn Lundin
replies disabled
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox