Showing results for 
Search instead for 
Did you mean: 
New member
2 1 0 0
Message 1 of 4
Flag Post


Calculating with decimals Error

HP Recommended
HP Prime



I entered:


13.6+6.4-20 and press the enter key in CAS mode


and it shows -1.13686837722E-13


However, I entered same in Home mode shows 0.


I tried several decimals making 0 in Home mode, always answer is 0.


But in the CAS mode shows smoetimes non-zero answers.


In the CAS mode, 55.8+22.1-77.9 is 0


but 55.8+22.2-78 is -4.54747350886E-13


Is this just me or is this a bug in the CAS Mode??


This is my HP Prime info:

Software Version : 2018 02 12 (13441)

Hardware Version : C

CAS Version 1.4.9

Operating System : V0.050.639


Thank you.

0 Kudos
Tags (1)
Level 8
622 609 117 207
Message 2 of 4
Flag Post
HP Recommended

> Is this just me or is this a bug in the CAS Mode??


Neither. It's due to the fact that CAS mode uses 48-bit binary floating point numbers, unlike Home mode which uses 12-digit BCD floating point numbers.


Therefore, when you type 22.2 into Home, you get exactly 22.2, but in CAS it *displays* as 22.2 but internally is the closest that can be obtained with 48 bits over a power of 2 (in this case, the closest possible value is 195273265093017/2^43, which is exactly equal to 22.199999999999846522769075818359851837158203125, which as you can see is just a tiny bit less than 22.2).  Since numbers are displayed rounded to 12 digits (or less, according to the Number Format you set in Home Settings), it is displayed as 22.2 but internally it's really the strange-looking value above.  This is not a bug; all binary floating-point systems (including almost all computer math software) act that way.


You may ask, Why on earth does CAS use such a strange system, when Home is exact?  Answer: Home's BCD system  isn't infallible either. For example 3*(1/3) --> not 1!  This is not a bug. All BCD systems act that way.


My favorite instantly-goofy result in CAS is: 1-.9-.1 --> not zero!


To avoid needing to learn all of the above, just remember to do all your floating-point real-number number-crunching approximate math in Home, and only use CAS for either EXACT math (when the ENTIRE input contains no decimal points anywhere) and/or SYMBOLIC math (e.g. algebraic expression manipulations, equation solving, and anything else that uses variables which do not have a value already stored in them).


I hope I haven't just confused you further. If I have, then just remember to use Home for anything with a decimal point in it, and CAS for everything else.  :Wink:


Disclaimer: I don't work for HP. I'm just another happy HP Prime user.

Tags (1)
Level 6
Level 6
252 251 42 77
Message 3 of 4
Flag Post
HP Recommended



As Joe explained, all non exact computation systems (which is any system that will yeild decimals) will suffer of this type of issues.


Depending on the specifics of the system implementation, the artefacts will be different. Joe pointed the (1/3)*3 in decimal. You pointed the ones in the CAS (48 bit binary floating point reals). Because we are used to the decimal system's artefacts, but not to the binary ones, we find the 2nd strange and the first ones normal, but this is only due to the fact that we are used to the first ones and not the 2nd ones. (by the way, try 1/3 shift enter *3 enter in cas and you get 1!)


Excel (and all PC systems) will suffer from similar ones. Try entering =100-99.9-0.1 in excel and you will get: -5.6..E-15


Some systems try to hide this from you by rounding things, It looks good at first sight, but causes issues later on when you have a result that is trully 'small' and that the system rounds it down to 0 by mistake!



Wellcome to the wonderfull world of "applied mathematics"!



I am an HP Employee
New member
2 1 0 0
Message 4 of 4
Flag Post
HP Recommended

Thank you so much.


Now I can understand What the difference between CAS and Home mode is.


Again I really appreciate your answer and explain.


0 Kudos
† The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation