Results 1 to 23 of 23

Aggregating data based on a search key, across a very large dataset possible?

Threaded View

  1. #1
    Registered User
    Join Date
    10-09-2014
    Location
    Sydney, Australia
    MS-Off Ver
    2016
    Posts
    14

    Aggregating data based on a search key, across a very large dataset possible?

    Hey friends,

    I have a database full of receipt IDs and need to filter out particular records (i.e. receipts belonging to buyers/sellers that no longer exist). I have spent almost 3-4 hours after work, every day, over the last 5 weeks, and have only finished around ~2200 records, one by one. I underestimated how long it would take, as there are around 330,000 records. So I’m 0.67% of the way there...

    I’m thinking there must be a faster way to do this in Excel?

    So the process is I’m performing:
    1) A look-up of a specific string (in the ‘sub_seller’ column) that can appear in one to many (1:M) records in another table’s (‘Search Table’) column called ‘id’, and
    2) Once found, across all the rows found, aggregates all the values in the column next to it (‘owner_id’), including the search/lookup string itself, and dumping this result set into the adjacent cell in the original table
    3) I’d then like to repeat this process for the other column (obj_buyer) in that same original table.
    4) Once the process (lookup, aggregation, data dump) has been performed for both tables, I’d like to join only the common values across the two data dumps only the common values (i.e. INTERSECT operation) into a separate column (‘INTERSECT’).
    5) I then have a filter/exclusion table, where if any of the values (id’s) in ‘UNION’ (sorry I just realised I meant to use 'INTERSECT') are found in said table, they are to be excluded from the results (which will help to also exclude all the receipts associated with those excluded id’s


    I’ve attached a clean file containing a complete example of this process above. I’ve also put screenshots below to show what each step above looks like / is doing:
    (Also just a note, I realised '1636' (lookup value) should also be in the 'aggregate_seller_owners' column, apologies!

    Step 1)


    Step 2)

    Step 3)

    Step 4)

    Step 5)


    Any ideas? Any assistance would be greatly appreciated.

    Thanks so much guys!
    Attached Images Attached Images
    Last edited by Restrictedz; 08-15-2019 at 07:28 AM. Reason: I meant 'intersection', not 'union'

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Transposing grouped data (large dataset)
    By Bengan1 in forum Excel General
    Replies: 2
    Last Post: 02-20-2018, 08:53 AM
  2. Replies: 3
    Last Post: 10-20-2017, 12:40 AM
  3. Best method of extracting data from large dataset
    By Manchesterpolstudent in forum Excel General
    Replies: 2
    Last Post: 08-07-2017, 03:01 PM
  4. how to perform t test on a large dataset based on dynamic change in values
    By amalchacko in forum Excel Programming / VBA / Macros
    Replies: 0
    Last Post: 07-06-2015, 06:05 PM
  5. Replies: 3
    Last Post: 01-23-2014, 10:13 AM
  6. Create New Worksheets based on one column in a Large Dataset
    By shaolinsnake in forum Excel Programming / VBA / Macros
    Replies: 1
    Last Post: 07-20-2012, 06:29 PM
  7. Filtering on large dataset based on rule
    By not4google in forum Excel General
    Replies: 2
    Last Post: 11-11-2009, 06:15 AM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Search Engine Friendly URLs by vBSEO 3.6.0 RC 1