+ Reply to Thread
Results 1 to 5 of 5

An Excel math question

Hybrid View

  1. #1
    Registered User
    Join Date
    03-05-2010
    Location
    ulverston, england
    MS-Off Ver
    Excel 2003
    Posts
    2

    An Excel math question

    Hi, 1st time on this forum in desperation for help.

    I have a spreadsheet which does a number of very simple calculations, but for a particular set of numbers, seems to lose the plot. I have a simplification of the problem here if you could look at it.

    In A1 place number 7.02
    In B1 place number 7.07
    In C!, calculate the difference between A1 & B1 ( =A1-B1)
    In D1, place number 0.05 ( this will be a limit that C1 becomes reportable)
    In E1, do calculation. =IF(ABS(C1)>D1," out of calibration","")

    My problem is that the calculation works for all values placed into A1 & B1 except for the above example. In the example shown, calculation in C1 = D1, but fails. For any other values of A1 & B1 which result in C1=D1, calculation works.
    Am I going mad?
    Thanks.....Peter
    Last edited by nohotash; 03-05-2010 at 05:41 PM.

  2. #2
    Forum Expert JBeaucaire's Avatar
    Join Date
    03-21-2004
    Location
    Bakersfield, CA
    MS-Off Ver
    2010, 2016, Office 365
    Posts
    33,492

    Re: A simple math question

    No, you're not going mad. You've fallen prey to Excel precision issues.

    If you click on the cell with your problem formula and click on the Evaluate Formula icon (Formula Auditing toolbar), step through the formula...you will see that Excel considers the difference between the two values to be -0.050000000007. So, if you ABS() that value, it is slightly greater than 0.05.

    Round your math to specific decimals to eliminate the precision oddities.

    =IF(ROUND(ABS(C1),2)>D1,"out of calibration")

    ===========
    NOTE to mods: I would consider this thread properly titled...
    _________________
    Microsoft MVP 2010 - Excel
    Visit: Jerry Beaucaire's Excel Files & Macros

    If you've been given good help, use the icon below to give reputation feedback, it is appreciated.
    Always put your code between code tags. [CODE] your code here [/CODE]

    ?None of us is as good as all of us? - Ray Kroc
    ?Actually, I *am* a rocket scientist.? - JB (little ones count!)

  3. #3
    Registered User
    Join Date
    03-05-2010
    Location
    ulverston, england
    MS-Off Ver
    Excel 2003
    Posts
    2

    Re: A simple math question

    Many many thanks. I can go home now and have a social drink instead of drinking to forget........
    Last edited by Paul; 03-05-2010 at 03:01 PM.

  4. #4
    Forum Expert JBeaucaire's Avatar
    Join Date
    03-21-2004
    Location
    Bakersfield, CA
    MS-Off Ver
    2010, 2016, Office 365
    Posts
    33,492

    Re: A simple math question

    If that takes care of your need, please click EDIT in your original post, click GO ADVANCED and set the PREFIX box to SOLVED.

    (Also, use the blue "scales" icon in our posts to leave Reputation Feedback, it is appreciated. It is found across from the "time" in each of our posts.)

  5. #5
    Forum Expert royUK's Avatar
    Join Date
    11-18-2003
    Location
    Derbyshire,UK
    MS-Off Ver
    Xp; 2007; 2010
    Posts
    26,200

    Re: A simple math question

    Your post does not comply with Rule 1 of our Forum RULES. Your post title should accurately and concisely describe your problem, not your anticipated solution. Use terms appropriate to a Google search. Poor thread titles, like Please Help, Urgent, Need Help, Formula Problem, Code Problem, and Need Advice will be addressed according to the OP's experience in the forum: If you have less than 10 posts, expect (and respond to) a request to change your thread title. If you have 10 or more posts, expect your post to be locked, so you can start a new thread with an appropriate title.
    To change a Title on your post, click EDIT then Go Advanced and change your title, if 2 days have passed ask a moderator to do it for you.
    Hope that helps.

    RoyUK
    --------
    For Excel Tips & Solutions, free examples and tutorials why not check out my web site

    Free DataBaseForm example

+ Reply to Thread

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Search Engine Friendly URLs by vBSEO 3.6.0 RC 1