The file was several megabytes in size, but every so often the content of one element would not extract correctly. I was convinced the problem must of been with expat, the C parser behind XML::Parser.
It turns out I should of read the manual a little better.
I was resetting a variable in my Char handler each time the handler was called. I was assuming this would only be called once per element. However, it can be called multple times per element and this is what was happening.
For my small test file, the original code was fine. However, resetting the variable in the Start handler and making sure that the Char handler now appends data each call has fixed the problem for all files, including very large ones.