More fixes, improvements
Made some more fixes to the log viewer, and some performance improvements.
Warning: Tech Details Follow!
Basically my method for determining the position each line was in the log file was messed up because the StreamReader class in .NET doesn’t give any feedback on your current position within the file. As a result I had to make my own file reader class (in LogReader.cs) which does maintain the position within the file and reports it back to the application when asked. I also skipped the “Decoding” step as I can make the assumption with the EQ log files that the file format can be directly converted from byte to char.
Once I’d built the new reader, things were going a bit slow so I threw it into the Visual Studio 2005 profiler to see where I was losing my time. Turned out the time was being lost in a number of places:
- Using regular expressions to match the date/time at the start was very expensive, especially given I already knew the exact characters within the line I wanted to pull out. Just extracting the specific characters was an order of magnitude faster.
- Accessing the stream length every line to show the progress indicator was taking a lot of system time. I guess it was checking the length of the log on the file system every single time I asked. Storing this in a local variable before the loop sped things up.
- Constructors of string() and StringBuilder() inside the LogReader class were taking a lot of time. I turned the stringreader into a preallocated buffer that I just set to zero length instead of reallocating every time and things also got much faster.
- Calculating the time as a DateTime every line was taking a fair amount of time. Converting Start and Finish to Unix times and comparing integers runs much faster.
Most of those improvements help the parser run quickly through “dead” times (outside the start/finish boundary) but I could now run through a 180M test log (thanks Keldoth) in under a minute. Made me feel all happy!