[coldbox-5.3.0+837] large data-import running slowly

Communities ColdBox HMVC
1

I’m tasked with importing large amounts of data from a remote API. My first attempt is very “ColdBoxy” where I’m using CBORM to handle:

  • see if the row is already in the DB
  • if so, update it, as needed; otherwise insert it
  • etc

The code is beautiful and I see how it’ll be easy to maintain/debug/reuse/etc.

The problem is that, with this much data, it’s really slow.

Should I spend time working on heapsize, garbage-collection, etc, etc?

Or revert to queryExecute()?

Or something else?

TIA.

2

What specifically is slow about it? Some times you can get much better performance on a boring sort of import that processes a lot of records by paralleling it. If you’re on Lusee one of the easiest things to use if the parallel flag on arrayEach() and structEach(). For a mostly DB bound process, tossing 20 threads at it can make it 20 times faster. I also just did this last week in a CommandBox task runner that would process tens of thousands of records from an API by using cbstreams in parallel mode.

Thanks!

~Brad

3

@Brad: “CommandBox task runner that would process tens of thousands of records from an API by using cbstreams in parallel mode” - that’s worthy of a webcast! :slight_smile:

4

ha, it was for a client. I’ll have to see if I can get an example using another API and DB. It was pretty cool and used qb and hyper too.

5

Parallel flag seems promising. Of course, as soon as I turned it on, I ran into what I believe is a race condition. I’m doing this in a handler and I have:

component extends=“coldbox.system.EventHandler” {

property name=“ORMService” inject=“entityService:MyModel”;

function largedataimport( event, rc, prc ) {

MyArray.each(function(element, index) {

// do the work

ORMService.save( ThingToBeSaved );

}, true);

}

}

So I think that my problem is that I’m setting up ORMService outside of the function inside of my each() where, instead, I need a function-scoped version of that thing?

If that’s correct, I’m having trouble figuring out the syntax for assigning a function-scoped entityService:MyModel inside of that function. Can you advise?

6

What’s the exception?

7

Type: org.hibernate.NonUniqueObjectException

Messages: a different object with the same identifier value was already associated with the session: [MyModel#49FDBDA0-6B2E-4DFC-9FD5-9CAED25F7011]

Being thrown by my ORMService.save( ThingToBeSaved ) line.

8

I would have to see the entire source code to give you an answer of why they are overlapping.

– -- You received this message because you are subscribed to the Google Groups “ColdBox Platform” group. For News, visit For Documentation, visit For Bug Reports, visit — You received this message because you are subscribed to the Google Groups “ColdBox Platform” group. To unsubscribe from this group and stop receiving emails from it, send an email to . To post to this group, send email to . To view this discussion on the web visit . For more options, visit .

9

I cannot share “in the clear” but I’ll send to you personally, OK?

For the group, I’ll share solution here if/when we have one.

10

UPDATE: Luis has not reviewed code yet. However, I did speed things up by switching from CBORM to queryExecute(), among other things.

I guess that CBORM is just too much overhead for what I’m attempting to accomplish here?