Quick Code Tip: Fix Precision with Floating Points

Let’s say you have a floating point value of 200.21, and you convert it to an int, like so:

int amount = (int)(famount * 100);

Depending on processor, libraries, compiler and tons of other stuff, your value may APPEAR to be 200.21, but when you do your conversion the value comes out to 20020. What happened to that extra 1?!

Well, though you may see 200.21, the computer is actually storing 200.209999999… (because, you know, 0.999… == 1).

This doesn’t happen all the time, but when it does, it’s a pain. So how do we fix this?

Just add 0.5. Seriously.

int amount = (int)((famount * 100) + 0.5f));

Bam. Let’s break this down, PEMDAS style.

(famount * 100)
Ok, so we know that famount is not 200.21, but 200.2099999… This now brings us to 20020.99999…

((20020.999…) + 0.5f)
The 0.5 now brings us to 20021.49999… Like magic.

int amount = (int)(20021.4999…)
Saving all the pennies.

Good job. You just saved the world from mathematical inaccuracy.

Until next time, take care. (After that, do whatever the heck you want.)

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s