Wednesday, August 15, 2012

Pentaho: MySQL Bulk Load

In MySQL, bulk load operations i.e. "LOAD DATA" has much faster performance than INSERT. In Pentaho's Data Integration module, it is possible to directly perform a bulk load operation into MySQL, using the "MySQL Bulk Loader" Step (in the "Bulk loading" folder).

However, one "gotcha" with it (version 4.3) is that it will crash on an empty stream. You'll either have to find a way to handle it, or use it only in situation where that either doesn't matter or never happens, or hope they fix this in future versions.

No comments:

Post a Comment