renderData json type bombs when query results are larger than 35K records [CB4.1.0]

COLDBOX 4.1.0
windows 2008 server
CF 11

event.renderData( data=results, type=“json” statusCode=status );

A query result with more than approximately 35,000 records(could be 40K or 45K - depends on available memory) will kill my response. I don’t get an error, nothing, just an empty response.

However, If I output the response using following code the json response is OK 200, and the resultset can be 300K rows:
jsonData = serializeJSON(results,“struct”);
writeOutput(jsonData);
event.noRender()

Has anyone had limitations on event.renderData() in terms of the number of records returned?

Why are you passing in “struct” as the second parameter to serializeJSON()? It should be a boolean.

https://helpx.adobe.com/coldfusion/cfml-reference/coldfusion-functions/functions-s/serializejson.html

Here is what ColdBox does to render JSON on line 58 of DataMarshaller.cfc:

results = serializeJSON( arguments.data, ( arguments.jsonQueryFormat eq “array”) ? false : true );

Does it take your request a long time to respond? If so, what does Fusion Reactor show that the request is doing? There may be something else downstream that’s handling the variable that contains the JSON that’s getting hung up.

Thanks!

~Brad

ColdBox Platform Evangelist
Ortus Solutions, Corp

E-mail: brad@coldbox.org
ColdBox Platform: http://www.coldbox.org
Blog: http://www.codersrevolution.com

First, the “struct” value as a 2nd parameter is not documented but it works.

false renders:
{“COLUMNS”:[“CITY”,“STATE”],“DATA”:[[“Newton”,“MA”],[“San Jose”,“CA”]]}

true renders:



{“ROWCOUNT”:2, “COLUMNS”:[“CITY”,“STATE”],“DATA”:{“City”:[“Newton”,“San Jose”],“State”:[“MA”,“CA”]}}

“struct” renders:
[{“CITY”:“Newton”,“STATE”:“MA”},{“CITY”:“San Jose”,“STATE”:“CA”}]

the latter of which is the prefer format. Woohoo. It replaces custom scripts like the QueryToArray() or QueryToStruct() that is out there.

Second, the response is slow. 50K → 3 secs query time + 6 secs rendering time, 260K → 17 secs query time + 55 secs rendering time.

Shortly I’ll provide debug info.

This is what I get:

Mar 02, 2016 4:41:59 PM org.apache.catalina.core.StandardHostValve custom
SEVERE: Exception Processing ErrorPage[exceptionType=java.lang.Exception, location=/CFIDE/administrator/templates/errors.jsp]
org.apache.jasper.JasperException: javax.servlet.ServletException: java.lang.IllegalStateException: Cannot create a session after the response has been committed
at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:549)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:455)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:748)
at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:604)
at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:543)
at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:461)
at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:412)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:201)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:422)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:199)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)
Caused by: javax.servlet.ServletException: java.lang.IllegalStateException: Cannot create a session after the response has been committed
at org.apache.jsp.CFIDE.administrator.templates.errors_jsp._jspService(errors_jsp.java:153)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
… 21 more
Caused by: java.lang.IllegalStateException: Cannot create a session after the response has been committed
at org.apache.catalina.connector.Request.doGetSession(Request.java:2935)
at org.apache.catalina.connector.Request.getSession(Request.java:2311)
at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:897)
at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:229)
at org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:543)
at org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:514)
at org.apache.jasper.runtime.PageContextImpl._initialize(PageContextImpl.java:147)
at org.apache.jasper.runtime.PageContextImpl.initialize(PageContextImpl.java:126)
at org.apache.jasper.runtime.JspFactoryImpl.internalGetPageContext(JspFactoryImpl.java:112)
at org.apache.jasper.runtime.JspFactoryImpl.getPageContext(JspFactoryImpl.java:65)
at org.apache.jsp.CFIDE.administrator.templates.errors_jsp._jspService(errors_jsp.java:53)

Hmm, that’s interesting. I originally just meant for you to capture a stack trace while the thread was running, but knowing that there is an actual exception is helpful. It looks like ColdFusion is trying to run a jsp error page for some reason, which is then erroring out when it tries to create a session. The error below seems to be a secondary error. Is there anything else in the logs around the same time. I’d guess perhaps you’re hitting a Tomcat limit on the length of a response? What’s odd is the error page is a jsp page that is in the CFIDE folder.

Thanks!

~Brad

ColdBox Platform Evangelist
Ortus Solutions, Corp

E-mail: brad@coldbox.org
ColdBox Platform: http://www.coldbox.org
Blog: http://www.codersrevolution.com

Brad, here are two stacktraces. First stacktrace says Java Heap Space. Second says ‘exceeded the allowable time limit tag: cfoutput’

When I use serializeJSON(prc.results) with 49,000 query rows:

Mar 03, 2016 10:43:43 AM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [CfmServlet] in context with path [/] threw exception [ROOT CAUSE:
coldfusion.runtime.EventHandlerException: Event handler exception.
at coldfusion.runtime.AppEventInvoker.onRequestStart(AppEventInvoker.java:287)
at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:455)
at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:42)
at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
at coldfusion.filter.PathFilter.invoke(PathFilter.java:142)
at coldfusion.filter.LicenseFilter.invoke(LicenseFilter.java:30)
at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:94)
at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:78)
at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:58)
at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
at coldfusion.CfmServlet.service(CfmServlet.java:219)
at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at coldfusion.inspect.weinre.MobileDeviceDomInspectionFilter.doFilter(MobileDeviceDomInspectionFilter.java:121)
at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:422)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:199)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
] with root cause
javax.servlet.ServletException: ROOT CAUSE:
coldfusion.runtime.EventHandlerException: Event handler exception.
at coldfusion.runtime.AppEventInvoker.onRequestStart(AppEventInvoker.java:287)
at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:455)
at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:42)
at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
at coldfusion.filter.PathFilter.invoke(PathFilter.java:142)
at coldfusion.filter.LicenseFilter.invoke(LicenseFilter.java:30)
at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:94)
at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:78)
at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:58)
at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
at coldfusion.CfmServlet.service(CfmServlet.java:219)
at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at coldfusion.inspect.weinre.MobileDeviceDomInspectionFilter.doFilter(MobileDeviceDomInspectionFilter.java:121)
at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:422)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:199)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space

at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:70)
at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at coldfusion.inspect.weinre.MobileDeviceDomInspectionFilter.doFilter(MobileDeviceDomInspectionFilter.java:121)
at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:422)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:199)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

When I use writeDump(prc.results):

Well yes, I’d never EVER try to to dump a query object with 50k records. cfdump in general is slow and outputs very large amounts of HTML.

Thanks!

~Brad

ColdBox Platform Evangelist
Ortus Solutions, Corp

E-mail: brad@coldbox.org
ColdBox Platform: http://www.coldbox.org
Blog: http://www.codersrevolution.com

I’m learning. Thanks Brad. I have changed how the table grid works now and I’m limiting results and/or using pagination.