+ Reply to Thread
Results 1 to 3 of 3

How to stop unwanted rounding

  1. #1
    Registered User
    Join Date
    07-18-2022
    Location
    Manchester England
    MS-Off Ver
    MS365
    Posts
    91

    How to stop unwanted rounding

    I am using the following code to load a range into a variant, do a simple calculation and then reload the range..
    Please Login or Register  to view this content.
    The normal values of a and b are around .05 and .02. When I first tried this code the range had not been multiplied. I used the locals window and found that a and b were being stored as 1 not 1.04.
    when I increased the value of a and b to 1.55 I found that they were stored as 2. The expected calculation then showed and all the elements were multiplied by 2 not 1.55.
    How can I ensure that the numbers are stored in the correct way and then perform properly in the code?
    John

  2. #2
    Forum Guru
    Join Date
    04-13-2005
    Location
    North America
    MS-Off Ver
    2002/XP, 2007, 2024
    Posts
    16,460

    Re: How to stop unwanted rounding

    When I first tried this code the range had not been multiplied. I used the locals window and found that a and b were being stored as 1 not 1.04.
    Do you mean Inc and Inc2 here, instead of a and b (which you say typically have values of 0.05 and 0.02)? I note that Inc and Inc2 are both dim'ed as Long. When VBA adds 1+0.02, it will get 1.02. When VBA then tries to store 1.02 in a Long, it will round to the nearest integer. It appears to me that the results you are getting are consistent with the data types you are using.

    My guess is that you need to type Inc and Inc2 as Doubles rather than Longs, but that ignores any reasons you have for typing them as Longs. One way or another (whether by changing the type of Inc and Inc2 or introducing new double variables in place of Inc and Inc2), it appears to me that you need to store the 1+a and 1+b results in double variables.
    Quote Originally Posted by shg
    Mathematics is the native language of the natural world. Just trying to become literate.

  3. #3
    Forum Expert romperstomper's Avatar
    Join Date
    08-13-2008
    Location
    England
    MS-Off Ver
    365, varying versions/builds
    Posts
    22,015

    Re: How to stop unwanted rounding

    Have you declared a and b somewhere as either Integer or Long? If so, declare them as Double.
    Everyone who confuses correlation and causation ends up dead.

+ Reply to Thread

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Unwanted Rounding
    By scottintexas in forum Excel Programming / VBA / Macros
    Replies: 5
    Last Post: 02-07-2014, 05:41 PM
  2. [SOLVED] Unwanted rounding
    By darkone1965 in forum Excel Programming / VBA / Macros
    Replies: 2
    Last Post: 05-10-2013, 12:00 AM
  3. Unwanted Rounding
    By IntegraCraig in forum Excel General
    Replies: 12
    Last Post: 10-30-2010, 03:35 PM
  4. [SOLVED] Unwanted rounding of large number
    By Candyman in forum Excel Formulas & Functions
    Replies: 0
    Last Post: 09-06-2005, 12:05 PM
  5. Unwanted rounding of large number
    By Duke Carey in forum Excel Formulas & Functions
    Replies: 15
    Last Post: 09-06-2005, 09:05 AM
  6. Unwanted rounding of large number
    By Duke Carey in forum Excel Formulas & Functions
    Replies: 10
    Last Post: 09-06-2005, 06:05 AM
  7. [SOLVED] Unwanted rounding of large number
    By Candyman in forum Excel Formulas & Functions
    Replies: 0
    Last Post: 09-06-2005, 01:05 AM
  8. Unwanted rounding of large number
    By Candyman in forum Excel Formulas & Functions
    Replies: 0
    Last Post: 09-06-2005, 12:05 AM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Search Engine Friendly URLs by vBSEO 3.6.0 RC 1