MIDI-OX User Forum (http://www.midiox.com/cgi-bin/yabb/YaBB.pl)
HP16C Emulator >> HP16C Questions and Discussion >> Wrong results in FLOAT on 2.1.0.036
(Message started by: Herman111552 on Jan 27th, 2010, 2:15pm)

Title: Wrong results in FLOAT on 2.1.0.036
Post by Herman111552 on Jan 27th, 2010, 2:15pm
When I'm in Float and subtract 7.25 from 12.5 I receive the answer of 5.3
Process:
12.5 <Enter>
7.25 -
5.3 is displayed.

Title: Re: Wrong results in FLOAT on 2.1.0.036
Post by Jamie OConnell on Jan 30th, 2010, 3:41pm
You get that answer when the float precision is set to one (1), because of rounding:

[f][FLOAT][1]

If you instead do:

[f][FLOAT][2] (or any precision greater than one)

you will get the result: 5.25

This is identical to the actual HP-16C calculator.

Title: Re: Wrong results in FLOAT on 2.1.0.036
Post by Herman111552 on Feb 1st, 2010, 8:45am
Thank you.  I know I had missed it in the doc.  Thanks again.



MIDI-OX User Forum » Powered by YaBB 1 Gold - SP 1.3.1!
YaBB 2000-2003. All Rights Reserved.