+ Reply to Thread
Results 1 to 17 of 17

Compact function to use most recent positive value in column.

Hybrid View

JeremyWyatt6 Compact function to use most... 08-14-2017, 09:33 AM
oeldere Re: Compact function to use... 08-14-2017, 10:55 AM
JeremyWyatt6 Re: Compact function to use... 08-14-2017, 11:18 AM
CK76 Re: Compact function to use... 08-14-2017, 11:03 AM
oeldere Re: Compact function to use... 08-14-2017, 11:11 AM
CK76 Re: Compact function to use... 08-14-2017, 11:25 AM
JeremyWyatt6 Re: Compact function to use... 08-14-2017, 11:35 AM
oeldere Re: Compact function to use... 08-14-2017, 11:32 AM
CK76 Re: Compact function to use... 08-14-2017, 12:03 PM
JeremyWyatt6 Re: Compact function to use... 08-14-2017, 12:17 PM
CK76 Re: Compact function to use... 08-14-2017, 01:17 PM
JeremyWyatt6 Re: Compact function to use... 08-14-2017, 01:35 PM
JohnTopley Re: Compact function to use... 08-14-2017, 02:26 PM
JeremyWyatt6 Re: Compact function to use... 08-14-2017, 02:31 PM
CK76 Re: Compact function to use... 08-15-2017, 12:07 PM
JeremyWyatt6 Re: Compact function to use... 08-15-2017, 12:21 PM
CK76 Re: Compact function to use... 08-15-2017, 12:58 PM
  1. #1
    Registered User
    Join Date
    08-14-2017
    Location
    Nova Scotia, Canada
    MS-Off Ver
    2016
    Posts
    9

    Question Compact function to use most recent positive value in column.

    I am currently working with a very large spreadsheet (650000 rows) and need to use a formula to search a column for the most recently occurring positive number.

    To give some context, the spreadsheet contains flow data from every minute from the passed 16 months. If the net flow is positive it means the storage tank was filling (inflow), if the net flow is negative it means the pumps were activated and the tank was draining. However, while the tank is draining there is still technically an inflow occurring and my goal is to split these up. My estimate is based on the assumption that whichever inflow was occuring just prior to the tank from draining is constantly occurring throughout the outflow time.

    I currently have a column returning me the value I want, but it is with a lookup function which excel cannot handle at this size:

    =IF(I16="No Change", 0, IF(I16="In", 0, IF(I16="Out", ABS(F16+(LOOKUP(2,1/($F$4:F15>0),$F$4:F15))), "Error")))

    The I column contains if statements that check if the net flow is positive or negative, and the F column contains said net flow values.

    Is there a more compact way of performing this task? I have attempted some Index:Match functions with no luck.

    Thanks,

    Jeremy
    Attached Files Attached Files
    Last edited by JeremyWyatt6; 08-14-2017 at 12:17 PM.

  2. #2
    Forum Expert
    Join Date
    05-30-2012
    Location
    The Netherlands
    MS-Off Ver
    Office 365
    Posts
    14,987

    Re: Compact function to use most recent positive value in column.

    a column for the most recently occurring positive number.
    Then probably you can delimed the range for the formula from e.g. row 600.000 to the end.
    Notice my main language is not English.

    I appreciate it, if you reply on my solution.

    If you are satisfied with the solution, please mark the question solved.

    You can add reputation by clicking on the star * add reputation.

  3. #3
    Registered User
    Join Date
    08-14-2017
    Location
    Nova Scotia, Canada
    MS-Off Ver
    2016
    Posts
    9

    Re: Compact function to use most recent positive value in column.

    Sorry I'm not quite sure what you meant by "delimed".

  4. #4
    Forum Expert CK76's Avatar
    Join Date
    06-16-2015
    Location
    ONT, Canada
    MS-Off Ver
    MS365 Apps for enterprise
    Posts
    5,973

    Re: Compact function to use most recent positive value in column.

    I'd recommend using PowerQuery (Get & Transform) since you have Excel 2016.

    It may take some getting used to, but will definitely be more efficient when working with large data set.
    "Progress isn't made by early risers. It's made by lazy men trying to find easier ways to do something."
    ― Robert A. Heinlein

  5. #5
    Forum Expert
    Join Date
    05-30-2012
    Location
    The Netherlands
    MS-Off Ver
    Office 365
    Posts
    14,987

    Re: Compact function to use most recent positive value in column.

    This is a duplicated post.

    Please mark the older question solved.

    https://www.excelforum.com/excel-gen...om-to-top.html

    It is not allowed to duplicate the question according to the forumrules.

  6. #6
    Forum Expert CK76's Avatar
    Join Date
    06-16-2015
    Location
    ONT, Canada
    MS-Off Ver
    MS365 Apps for enterprise
    Posts
    5,973

    Re: Compact function to use most recent positive value in column.

    I believe what he means is to limit the range used in calculation.

    Assuming that latest data is added to bottom of the existing data.
    If you limit the calculation range to Row 600,000 + then you eliminate a lot of overhead in calculation.

  7. #7
    Registered User
    Join Date
    08-14-2017
    Location
    Nova Scotia, Canada
    MS-Off Ver
    2016
    Posts
    9

    Re: Compact function to use most recent positive value in column.

    As of right now I have it set up so the array is locked in cell $F$4:0F"current row", so it keeps getting larger and larger. I recognize that this is definitely causing major strain on excel, however the times between outflows are not constant and I can't really shrink the arrays to a constant size because it may miss an inflow.

    If there wasn't as much data I could find the largest time gap between inflow and outflow and just make all the arrays that size to be safe, but since there are 650000 rows of data this would be quite difficult.

    To add on to this, the lookup function I'm using was found on another forum post but to be honest I don't really understand how it is working.. Not sure what the 2 represents. To my understanding this would mean the lookup is looking for the value 2 in my column but it is somehow correctly finding the last positive number.

  8. #8
    Forum Expert
    Join Date
    05-30-2012
    Location
    The Netherlands
    MS-Off Ver
    Office 365
    Posts
    14,987

    Re: Compact function to use most recent positive value in column.

    @CK76

    I believe what he means is to limit the range used in calculation.
    Yes that is what I meant. Thanks for the help.

  9. #9
    Forum Expert CK76's Avatar
    Join Date
    06-16-2015
    Location
    ONT, Canada
    MS-Off Ver
    MS365 Apps for enterprise
    Posts
    5,973

    Re: Compact function to use most recent positive value in column.

    Upload small sample (say about 50 rows) of data that represent your data (sanitize it of sensitive data).

    Also, your expected output as well. It would help in giving you appropriate solution.

  10. #10
    Registered User
    Join Date
    08-14-2017
    Location
    Nova Scotia, Canada
    MS-Off Ver
    2016
    Posts
    9

    Re: Compact function to use most recent positive value in column.

    Ok I have updated to include sample data. It shows four instances of the outflow being calculated.

  11. #11
    Forum Expert CK76's Avatar
    Join Date
    06-16-2015
    Location
    ONT, Canada
    MS-Off Ver
    MS365 Apps for enterprise
    Posts
    5,973

    Re: Compact function to use most recent positive value in column.

    Ok, yes, this would definitely cause workbook to slowdown with that much data.

    Let me think on logic. I'm going into meeting now and won't be able to look at it in detail until tomorrow.

  12. #12
    Registered User
    Join Date
    08-14-2017
    Location
    Nova Scotia, Canada
    MS-Off Ver
    2016
    Posts
    9

    Re: Compact function to use most recent positive value in column.

    Great, thank you.

    Another thing I noticed is that if there is zero inflow prior to the outflow beginning, instead of simply using the zero value excel continues to search until it hits a number. I'd like it to count a zero as well.

  13. #13
    Forum Expert
    Join Date
    05-05-2015
    Location
    UK
    MS-Off Ver
    Microsoft Excel for Microsoft 365 MSO (Version 2402 Build 16.0.17328.20068) 64-bit
    Posts
    30,769

    Re: Compact function to use most recent positive value in column.

    LOOKUP expects data to be in ascending sequence which is not true of column F so the results are not correct.

  14. #14
    Registered User
    Join Date
    08-14-2017
    Location
    Nova Scotia, Canada
    MS-Off Ver
    2016
    Posts
    9

    Re: Compact function to use most recent positive value in column.

    Quote Originally Posted by JohnTopley View Post
    LOOKUP expects data to be in ascending sequence which is not true of column F so the results are not correct.
    What is strange though is that it is using the correct inflow values aside from the lengths of time with zero inflow. I think if I fully understood how my lookup function was working I might be able to manipulate it to work...

    LOOKUP(2,1/($F$4:F15>0),$F$4:F15))

    I don't understand what the lookup value or vector are in the function..

  15. #15
    Forum Expert CK76's Avatar
    Join Date
    06-16-2015
    Location
    ONT, Canada
    MS-Off Ver
    MS365 Apps for enterprise
    Posts
    5,973

    Re: Compact function to use most recent positive value in column.

    Here, see attached Sheet2.

    Used PowerQuery to do the same calculation and also adjusted to count 0 as well.

    I'll post exact steps to follow a bit later.
    Attached Files Attached Files

  16. #16
    Registered User
    Join Date
    08-14-2017
    Location
    Nova Scotia, Canada
    MS-Off Ver
    2016
    Posts
    9

    Re: Compact function to use most recent positive value in column.

    Excellent, I am curious to find out how this power query works.

    In the mean time I used my original code but shrunk down the data file size as suggested. Now instead of all 16 months shown in one spreadsheet, I have 16 separate monthly spreadsheets. It was quite tedious though.

    Thanks for the dedication to this CK76.

  17. #17
    Forum Expert CK76's Avatar
    Join Date
    06-16-2015
    Location
    ONT, Canada
    MS-Off Ver
    MS365 Apps for enterprise
    Posts
    5,973

    Re: Compact function to use most recent positive value in column.

    Ok, here's steps taken to set it up.
    NOTE: PowerQuery is case sensitive, always use correct syntax and case for "M" or PowerQuery language. If I used lower case in function then it should be kept as is.

    NOTE2: For this to work, it is important that table is sorted oldest to newest (by date & time). If you may change sort order of original table, I'd recommend adding additional step to sort table before Index column is added.

    1. Added Sheet1 M3 & M4 as named range, "vDiam" and "vArea" respectively. To reference it where needed in the Query Editor. (I found later that I only needed vArea)
    2. Copied SCADA portion of data to Sheet2, converted it to Excel Table to load to PowerQuery (PQ)
    3. Load the table to PQ. Then add 0 based Index column [Index]. To be used later in calculations
    4. Open advanced editor in Query Editor (found under "View"). Added line just after "let" to define "mArea" variable to be used in calculation
    let
        mArea = Excel.CurrentWorkbook(){[Name="vArea"]}[Content]{0}[Column1],
    5. Added custom column to return previous row's value for [6_well_level] and named it [PrevWell] using formula below
    =if [Index] > 0 then #"Added Index"{[Index]-1}[6_well_level] else 0
    6. Added custom column for net flow calculation using "mArea" variable defined in step 4. [Net Flow]
    if [Index] > 0 then (([6_well_level]-[PrevWell])*mArea)*60 else 0
    7. Since I'm going to reference table itself to do custom calculation later, at this point, created copy of the table and named it "Result"
    8. In Query Editor, create new blank query (right click on right pane), copy and paste below custom function in the advanced editor. This is used to obtain last positive flow from previous rows.
    (mytable as table, myIndex)=>
    let
        Source = Table.Max(Table.SelectRows(mytable,each [Index] < myIndex and [Net Flow] >= 0),"Index")[Net Flow]
    in
        Source
    9. In Result table, add custom column [Out Flow] with calculation below. This is the reason copy of Table1 was created (to avoid circular reference to the table)
    = if [Net Flow] < 0 then Number.Abs([Net Flow] + GetPositiveFlow(Table1, [Index])) else 0
    10. Add [Net Flow In/Out] with formula below
    = if [Net Flow] = 0 then "No Change" else if [Net Flow] > 0 then "In" else "Out"
    11. Add [Active Pump] column with formula below
    = if [6_p1_amps] > 0 and [6_p2_amps] > 0 then "Both" else if [6_p1_amps] > 0 then "Pump 1" else if [6_p2_amps] > 0 then "Pump 2" else "None"
    12. Remove all columns before [Net Flow] since it's present in original data or just intermediate steps.
    13. Change column data type to appropriate type (Deciman Number or Text)
    14. Load back to the sheet
    Last edited by CK76; 08-15-2017 at 01:03 PM. Reason: See NOTE:

+ Reply to Thread

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. [SOLVED] Function to search column for most recent positive value (Bottom to top).
    By JeremyWyatt6 in forum Excel General
    Replies: 0
    Last Post: 08-14-2017, 08:22 AM
  2. Find most recent date in Column Headings in a Range, insert new column and heading
    By mikey3580 in forum Excel Programming / VBA / Macros
    Replies: 4
    Last Post: 04-07-2014, 08:28 AM
  3. [SOLVED] Find the most recent date in a column based on a name in a different column
    By Tamarissa in forum Excel Formulas & Functions
    Replies: 4
    Last Post: 02-22-2014, 10:10 AM
  4. Replies: 6
    Last Post: 11-13-2012, 07:33 AM
  5. Replies: 0
    Last Post: 06-25-2012, 03:28 PM
  6. [SOLVED] Returning most recent daily data in a column if blanks in column
    By InnesMcc in forum Excel General
    Replies: 6
    Last Post: 11-04-2011, 09:41 AM
  7. [SOLVED] Returning most recent daily data in a column if blanks in column
    By InnesMcc in forum Excel General
    Replies: 2
    Last Post: 11-04-2011, 06:28 AM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Search Engine Friendly URLs by vBSEO 3.6.0 RC 1