ArrayIndexOutOfBoundsException in SES.cfc

Hello all - we have a medium sized coldbox application deployed to production where we are received a strange error on a regular basis that we cannot make heads or tails of. We cannot seem to reproduce it in any environment so far. Here is all of the information I have gathered so far. I will then give you my best hypothesis and what i’m really looking for. I have redacted some specific information, if something is redacted that you think might be relevant to the problem at hand please let me know. This is quite a bit of information, I apologize in advance for the long email.

Error Information
Type java.lang.ArrayIndexOutOfBoundsException
Message Array index out of range: 35
Tag Context C:\wwwroot[redacted1][redacted2]_frameworks\coldbox\system\interceptors\SES.cfc(1024)
C:\wwwroot[redacted1][redacted2]_frameworks\coldbox\system\interceptors\SES.cfc(118)
C:\wwwroot[redacted1][redacted2]_frameworks\coldbox\system\web\context\InterceptorState.cfc(363)
C:\wwwroot[redacted1][redacted2]_frameworks\coldbox\system\web\context\InterceptorState.cfc(254)
C:\wwwroot[redacted1][redacted2]_frameworks\coldbox\system\web\context\InterceptorState.cfc(121)
C:\wwwroot[redacted1][redacted2]_frameworks\coldbox\system\web\services\InterceptorService.cfc(140)
C:\wwwroot[redacted1][redacted2]_frameworks\coldbox\system\web\services\RequestService.cfc(104)
C:\wwwroot[redacted1][redacted2]_frameworks\coldbox\system\Coldbox.cfc(181)
C:\wwwroot[redacted1][redacted2]_frameworks\coldbox\system\Coldbox.cfc(382)
C:\wwwroot[redacted1][redacted2]\Application.cfc(83)

Stack Trace Routes.cfm · GitHub

We started getting the error often enough that we wanted to get some more info and try and figure out what the routes had in it when the error occurred, so we sent an email with this code from a main.cfc handler:

savecontent variable=“local.routesDump” {
writeDump(getInterceptor(“SES”).getRoutes());
}

This gives us a dump that looks like this: coldbox-error - Album on Imgur

As you can see, there are 36 routes in the array - but that is using the getRoutes() method, not dumping the instance.routes directly.

This line is the culprit that throws the error:

1024:

My best guess is that the copy that occurs from this gets a size of 36, but for some reason as it is iterating over to copy the array one of the elements are not accessible, thus throwing the error - but it is something I cannot reproduce at all.

Our production environment is running ACF 10 on Windows 2012. We are running coldbox SEEK 3.7.0 - but we had this same issue when running 3.6.0 - we hoped it would be fixed with the upgrade but it did not. The application gets a sizable amount of traffic - but the occurrences of the error do not seem to be related to traffic (although we cannot confirm that), sometimes we go days without an issue, most often when we do have the error it is maybe one a day, but we have had days where we had four in the same day. But hundreds of users go through the exact same set of events and process that throws the error successfully each day. Also, the users who have received the error have successfully hit other pages in the application before they receive this error. Most users who receive this error are on an ipad, but we have also had users receive this error on a desktop using chrome and firefox.

Here are the contents of my routes.cfm config file:

addRoute(pattern=":handler/:action?/:id?/:id2?/:id3?/:id4?/:id5?/:id6?");
addRoute(pattern=":handler/:action?/:id?/:id2?/:id3?/:id4?/:id5?");
addRoute(pattern=":handler/:action?/:id?/:id2?/:id3?/:id4?");
addRoute(pattern=":handler/:action?/:id?/:id2?/:id3?");
addRoute(pattern=":handler/:action?/:id?/:id2?");
addRoute(pattern=":handler/:action?/:id?");
addRoute(pattern=":handler/:action?");

So here are some questions and what I am trying to figure out:

1). Can anyone figure out how to reproduce this, in our application or in a separate use case - where copying an array receives this error. As you can see, the code does not explicitly refer to an index in the array.

2). According to http://help.adobe.com/en_US/ColdFusion/9.0/Developing/WSc3ff6d0ea77859461172e0811cbec09f0b-7ff1.html, you should use duplicate() around arrays when the arrays contain anything more than simple values. My testing with and without duplicate confirm what this page says, but I can’t seem to find a way to reproduce this issue. I believe that the line that is throwing the error should use duplicate(), but I am not certain that it has anything to do with my issue. Is it something worth trying? (it takes an evening to do the update and then potentially a few days before we would see an error, so in the interest of figuring this out and only changing one thing at a time, I don’t mind to try this but I want to prioritize the attempts at fixing it).

3). Given the routes that are in my routes.cfm config file, is the 36 routes that you see in the dump I link to in the image correct? Here are the patterns in order. As you can see, there are certainly duplicates. I only list the pattern, but all of the other parts of the structure seem the same as well.

MessageBox/
:handler/:action/:id/:id2/:id3/:id4/:id5/:id6/
:handler/:action/:id/:id2/:id3/:id4/:id5/
:handler/:action/:id/:id2/:id3/:id4/
:handler/:action/:id/:id2/:id3/
:handler/:action/:id/:id2/
:handler/:action/:id/
:handler/:action/
:handler/
:handler/:action/:id/:id2/:id3/:id4/:id5/
:handler/:action/:id/:id2/:id3/:id4/
:handler/:action/:id/:id2/:id3/
:handler/:action/:id/:id2/
:handler/:action/:id/
:handler/:action/
:handler/
:handler/:action/:id/:id2/:id3/:id4/
:handler/:action/:id/:id2/:id3/
:handler/:action/:id/:id2/
:handler/:action/:id/
:handler/:action/
:handler/
:handler/:action/:id/:id2/:id3/
:handler/:action/:id/:id2/
:handler/:action/:id/
:handler/:action/
:handler/
:handler/:action/:id/:id2/
:handler/:action/:id/
:handler/:action/
:handler/
:handler/:action/:id/
:handler/:action/
:handler/
:handler/:action/
:handler/

Any help will be greatly appreciated. I have some ability to add things to help debug, but as I said above, the turnaround time wouldn’t be very fast, so I would like to prioritize any ideas for fixes. Let me know if there is anything else I can answer or any other information I can provide.

is auto reload set to false in your routes.cfm?

setAutoReload(false);

It doesn’t look like we are specifying - is the default true?

The default is false. Line 50 of SES.cfc.

Thanks!

~Brad

ColdBox Platform Evangelist
Ortus Solutions, Corp

E-mail: brad@coldbox.org
ColdBox Platform: http://www.coldbox.org
Blog: http://www.codersrevolution.com

looks like the default is false. i bring this up because someone recently had problems with ses/routes and setting that fixed the issue. to me, it does sound like a re-initing due to development environment settings.

In our production environment, we are using these settings in coldbox.cfc:

coldbox.debugMode = false;
coldbox.handlerCaching = true;
coldbox.eventCaching = true;
coldbox.env = “production”;

coldbox.modules.autoReload = false;

I have added my entire routes.cfm file to this gist: Routes.cfm · GitHub
But we aren’t specifying setAutoReload() at all.

perhaps for good measure, add it.

heres what i have in my routes.cfm:

switch(getSetting(“environment”)){
case “DEVELOPMENT”:{
setAutoReload(true);

break;
}

case “STAGING”:{
setAutoReload(true);

break;
}

case “PRODUCTION”:{
setAutoReload(false);

break;
}
}

What update level of CF10? I’m not sure how that code could possibly error. It isn’t ColdBox that’s even trying to loop over the array-- it’s just ColdFusion internally copying the array so it can set a variable. Even if there was another request trying to modify the array at the same time, ColdFusion should handle that with internal locking.

Perhaps put some debugging in every time configure() is called. That will prove whether the interceptor is being reloaded. And some more logging any time the instance.routes array is modified. Then you can at least cross reference whether another thread had recently touched that variable.

Thanks!

~Brad

ColdBox Platform Evangelist
Ortus Solutions, Corp

E-mail: brad@coldbox.org
ColdBox Platform: http://www.coldbox.org
Blog: http://www.codersrevolution.com

Any modules?

signature0.jpg

Luis F. Majano
CEO
Ortus Solutions, Corp
www.ortussolutions.com

ColdBox Platform: http://www.coldbox.org
Linked In: http://www.linkedin.com/pub/3/731/483
Blog: http://www.luismajano.com
IECFUG Manager: http://www.iecfug.com

Social: twitter.com/lmajano facebook.com/lmajano

are the routes reloaded when you use ?fwreinit ?

If so, I may add it as false for everything and then just fwreinit if we were to make a change - it happens so rarely.

Be careful with auto reloading routes. Modules do not like it when this happens and can cause weird side effects, apart that modules do not reload its routes for performance.
The only way routes with auto reload work with modules is to have modules auto reload also turned on.

signature0.jpg

Luis F. Majano
CEO
Ortus Solutions, Corp
www.ortussolutions.com

ColdBox Platform: http://www.coldbox.org
Linked In: http://www.linkedin.com/pub/3/731/483
Blog: http://www.luismajano.com
IECFUG Manager: http://www.iecfug.com

Social: twitter.com/lmajano facebook.com/lmajano

Luis, note that he’s on an older version of ColdBox, so it’s actually on line 1017 of the current dev branch. I at first thought it might be related to modules when I saw that line 1024 was _routes = getModuleRoutes( arguments.module ); but then I realized the actual line throwing the error (per the OP) is

Thanks!

~Brad

ColdBox Platform Evangelist
Ortus Solutions, Corp

E-mail: brad@coldbox.org
ColdBox Platform: http://www.coldbox.org
Blog: http://www.codersrevolution.com

MessageBox - we have forgebox in the modules folder it seems, but I don’t think we are pulling it in - this is the modules directives in our coldbox.cfc:

// Module Directives
modules = {
//Turn to false in production
autoReload = true,
// An array of modules names to load, empty means all of them
include = [“messageBox”],
// An array of modules names to NOT load, empty means none
exclude = []
};

Note that the autoReload is being overridden to false in our environment for production though.

Like Brad mentioned, what version of ColdBox is this?

signature0.jpg

Luis F. Majano
CEO
Ortus Solutions, Corp
www.ortussolutions.com

ColdBox Platform: http://www.coldbox.org
Linked In: http://www.linkedin.com/pub/3/731/483
Blog: http://www.luismajano.com
IECFUG Manager: http://www.iecfug.com

Social: twitter.com/lmajano facebook.com/lmajano

I apologize, I did mention this in my wall-of-text in the original post, but forgot to put it as part of the subject line.

We are running coldbox SEEK 3.7.0 - but we had this same issue when running 3.6.0

Re: coldfusion version:

Version 10,0,10,284825
Tomcat Version 7.0.23.0
Edition Enterprise
Operating System Windows Server 2008 R2 (I mistakenly said 2012 in the original post)
OS Version 6.1

I agree that this is likely a CF bug, but I am having trouble reproducing it in any way.