+ Reply to Thread
Results 1 to 12 of 12

Rounding differences in formula vs. hand-entered

Hybrid View

  1. #1
    Forum Contributor
    Join Date
    01-11-2011
    Location
    Frederick, MD, USA
    MS-Off Ver
    Excel 2007
    Posts
    125

    Re: Rounding differences in formula vs. hand-entered

    Quote Originally Posted by shg View Post
    Your computer, and therefore Excel, use a binary floating point format to store numbers. That format cannot store most decimals exactly, in the same sense that a finite base-10 decimal can't be exactly equal to 1/3. When you add or subtract inexact numbers, you frequently get even less exact numbers.
    These numbers are pretty far from being inexact; they're all rational numbers. There's no "1/3" being forced to decimal, for example.

    Quote Originally Posted by shg View Post
    The manually-entered 99.1 value in F9 is stored as 4058C66666666666 (the hex representation of the 64-bit double-precision number). The SUM formula in F3 returns the result 4058C66666666667; it differs in the least significant bit.
    If I understand you correctly, you're saying that a result of 0.9 is stored differently if it's the result of a calculation versus a hand-entered number?
    Last edited by Gunther Maplethorpe; 12-01-2015 at 03:12 PM. Reason: My typing skillz are sub-par

+ Reply to Thread

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Replies: 0
    Last Post: 04-13-2015, 01:01 PM
  2. [SOLVED] Rounding data entered as Millions, how to please
    By Mike in forum Excel Formulas & Functions
    Replies: 3
    Last Post: 08-04-2006, 06:25 PM
  3. [SOLVED] Rounding data entered as Millions, how to please
    By Mike in forum Excel Formulas & Functions
    Replies: 0
    Last Post: 08-04-2006, 05:40 PM
  4. [SOLVED] Rounding data entered as Millions, how to please
    By Mike in forum Excel Formulas & Functions
    Replies: 0
    Last Post: 08-04-2006, 05:35 PM
  5. [SOLVED] Hand entered dates view wrong
    By Chris in Nebraska in forum Excel General
    Replies: 6
    Last Post: 08-03-2005, 07:05 PM
  6. Replies: 1
    Last Post: 03-13-2005, 01:06 PM
  7. y-axis moves from the left hand side to the right hand side!
    By JP in forum Excel Charting & Pivots
    Replies: 2
    Last Post: 03-10-2005, 09:06 PM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Search Engine Friendly URLs by vBSEO 3.6.0 RC 1