MIDI-OX User Forum - Print Page


    
      
        MIDI-OX User Forum
        (http://www.midiox.com/cgi-bin/yabb/YaBB.pl)
      

        HP16C Emulator >> HP16C Questions and Discussion >> Wrong results in FLOAT on 2.1.0.036
        
(Message started by: Herman111552 on Jan 27^th, 2010, 2:15pm)

Title: Wrong results in FLOAT on 2.1.0.036
Post by Herman111552 on Jan 27^th, 2010, 2:15pm

When I'm in Float and subtract 7.25 from 12.5 I receive the answer of 5.3
Process:
12.5 <Enter>
7.25 -
5.3 is displayed.

Title: Re: Wrong results in FLOAT on 2.1.0.036
Post by Jamie OConnell on Jan 30^th, 2010, 3:41pm

You get that answer when the float precision is set to one (1), because of rounding:

[f][FLOAT][1]

If you instead do:

[f][FLOAT][2] (or any precision greater than one)

you will get the result: 5.25

This is identical to the actual HP-16C calculator.

Title: Re: Wrong results in FLOAT on 2.1.0.036
Post by Herman111552 on Feb 1^st, 2010, 8:45am

Thank you. I know I had missed it in the doc. Thanks again.