RARE user and assistance email list

[mc36 <>]

hi,

On 3/7/21 6:39 PM, mc36 wrote:
> hi,
>
> On 3/7/21 5:34 PM, Simon Leinen wrote:
> > Prometheus decided to only use IEEE 754 double-precision floats, which allow 
> > for 53 bits of integer precision
> so well, it explains everything, thanks for the research!


so for the less techie addicted readers i prepared a small demo in attachment.
it does nothing but converts the input to double and prints back to you the 
result:

mc36@noti:~$ ./zzz.sh 123412341234
so 123412341234 stored as 423cbbf2e1f20000 read as 123412341234.000000

the issue is that these floating point numbers store integers in a*(2^b) 
format.
there are (at least) 2 issues with this:

1) when b != 0, you loose precision, you cannot spot small changes:
in the below example, you cannot spot 80k of traffic

mc36@noti:~$ ./zzz.sh 111111111111111111111111111

so 111111111111111111111111111 stored as 4556fa2c59c5caac read as 
111111111111111104952008704.000000
mc36@noti:~$ ./zzz.sh 111111111111111111111199999
so 111111111111111111111199999 stored as 4556fa2c59c5caac read as 
111111111111111104952008704.000000
mc36@noti:~$



2) when b increases between 2 reads, you'll notice a much bigger bump....
in the below example, a single byte causes 4k increase...

mc36@noti:~$ ./zzz.sh 36893488147419101183
so 36893488147419101183 stored as 43ffffffffffffff read as 
36893488147419099136.000000
mc36@noti:~$ ./zzz.sh 36893488147419101184
so 36893488147419101184 stored as 4400000000000000 read as 
36893488147419103232.000000
mc36@noti:~$



regards,
cs

Attachment: zzz.tar
Description: Unix tar archive