Author |
Topic: Wrong results in FLOAT on 2.1.0.036 (Read 2757 times) |
|
Herman111552
New Member

 MIDI-OX Rules!
Posts: 2
|
 |
Wrong results in FLOAT on 2.1.0.036
« on: Jan 27th, 2010, 2:15pm » |
Quote Modify
|
When I'm in Float and subtract 7.25 from 12.5 I receive the answer of 5.3 Process: 12.5 <Enter> 7.25 - 5.3 is displayed.
|
|
IP Logged |
|
|
|
Jamie OConnell
Administrator
    


Gender: 
Posts: 2027
|
 |
Re: Wrong results in FLOAT on 2.1.0.036
« Reply #1 on: Jan 30th, 2010, 3:41pm » |
Quote Modify
|
You get that answer when the float precision is set to one (1), because of rounding: [f][FLOAT][1] If you instead do: [f][FLOAT][2] (or any precision greater than one) you will get the result: 5.25 This is identical to the actual HP-16C calculator.
|
« Last Edit: Jan 30th, 2010, 4:20pm by Jamie OConnell » |
IP Logged |
--Jamie Music is its own reward.
|
|
|
Herman111552
New Member

 MIDI-OX Rules!
Posts: 2
|
 |
Re: Wrong results in FLOAT on 2.1.0.036
« Reply #2 on: Feb 1st, 2010, 8:45am » |
Quote Modify
|
Thank you. I know I had missed it in the doc. Thanks again.
|
|
IP Logged |
|
|
|
|