Append Operations on Large Files

Friday, January 25, 2008
My team and I were discussing the time it takes for append operations to perform on large files, such as log files. Some suggested that the operation will take longer the larger the file becomes. Others thought the size of the file has no bearing.

I decided to create an experiment. I created a 3.5GB log file using a script and renamed the log file so as to remove any possibility of caching. I ran another script that appended a single line to the logfile. It appended the new line in less than 1 second.

Next, I copied the 3.5GB logfile to a remote server (took 3 minutes to copy), renamed it and tested again. Again, it took less than one second to append a new line. I had another user do the same test from his workstation with the same result.

Conclusion: File size has no bearing on the length of append operations.