User Forum :: Powered by YaBB
« MIDI-OX User Forum - Wrong results in FLOAT on 2.1.0.036 »
Welcome, Guest. Please Login or Register.
Jul 17th, 2018, 3:37am


Home Home Help Help Search Search Members Members Login Login Register Register


   MIDI-OX User Forum
   HP16C Emulator
   HP16C Questions and Discussion
(Moderator: Jamie OConnell)
   Wrong results in FLOAT on 2.1.0.036
« Previous topic | Next topic »
Pages: 1  Reply Reply Notify of replies Notify of replies Send Topic Send Topic Print Print
   Author  Topic: Wrong results in FLOAT on 2.1.0.036  (Read 1669 times)
Herman111552
New Member
*



MIDI-OX Rules!

   


Posts: 2
Wrong results in FLOAT on 2.1.0.036
« on: Jan 27th, 2010, 2:15pm »
Quote Quote Modify Modify

When I'm in Float and subtract 7.25 from 12.5 I receive the answer of 5.3
Process:
12.5 <Enter>
7.25 -
5.3 is displayed.
IP Logged
Jamie OConnell
Administrator
*****






   
WWW Email

Gender: male
Posts: 1982
Re: Wrong results in FLOAT on 2.1.0.036
« Reply #1 on: Jan 30th, 2010, 3:41pm »
Quote Quote Modify Modify

You get that answer when the float precision is set to one (1), because of rounding:
 
[f][FLOAT][1]
 
If you instead do:
 
[f][FLOAT][2] (or any precision greater than one)
 
you will get the result: 5.25
 
This is identical to the actual HP-16C calculator.
« Last Edit: Jan 30th, 2010, 4:20pm by Jamie OConnell » IP Logged

--Jamie
Music is its own reward.

Herman111552
New Member
*



MIDI-OX Rules!

   


Posts: 2
Re: Wrong results in FLOAT on 2.1.0.036
« Reply #2 on: Feb 1st, 2010, 8:45am »
Quote Quote Modify Modify

Thank you.  I know I had missed it in the doc.  Thanks again.
IP Logged
Pages: 1  Reply Reply Notify of replies Notify of replies Send Topic Send Topic Print Print

« Previous topic | Next topic »


MIDI-OX User Forum » Powered by YaBB 1 Gold - SP 1.3.1!
YaBB 2000-2003. All Rights Reserved.