TechNote (Archived)

ColdFusion MX: Returning large character data record sets causes server instability and Out of Memory errors

When returning large character data queries from a database, the memory print for Macromedia ColdFusion MX is twice as large as that of ColdFusion 5 and can eventually cause the server to hang with java.lang.OutOfMemoryError errors.

This issue is caused by a difference in the size of characters between Java (ColdFusion MX) and C++ (ColdFusion 5). Characters in Java are 16 bits, while characters in C++ are only 8 bits. This is a result of Java translating data into Unicode, which temporarily uses three times the amount of memory that C++ would. The error observed is due to the JVM using more physical memory than is available at that point.

For example, internal testing on a query 41MB in size shows the following behavior:

The raw data is read from the socket by the driver, equaling 41MB.
The data is then translated to Unicode, equaling 82MB. Added to the existing 41MB, it then totals 123MB.
The memory from step one is reclaimed a few seconds after thecfquery returns.
The memory from step two is reclaimed a few seconds after the template ends or the query variable is deleted.

In testing the above example, an attempt was made to use different garbage collection algorithms but the memory requirements did not change nor was memory reclaimed quicker.

WorkAround:

Since the behavior is a result of Java's handling of character data records, avoid the issue by using standard tuning practices to reduce the amount of records returned in exceptionally large queries. For example, write the query to only return those records that will be used rather than returning all the records.