Posted 9 years ago
·
Author
adryan87 wrote:damn 20 gb? ) are you ripping all the database ? )
I found a text editor that is meant to work with large files. Using this program I cut out all names that were disabled and pasted them into a new file. Leaving only the retired named in the original file. This new file with only disabled names was over 10 million lines long and came out to 450mb.
After scanning this file, I have found that it is made up of mostly duplicate data. I am in the process of removing all of the duplicate now. So if the original file is anything like this new one, it's mostly duplicates.
This means most of that 20gb is wasted space. Looking back at my code, I can understand how this happened. I rushed to get the retired name finder finished since it was not meant to be anything I released or even used for anything important. But in doing so I wrote code that was super inefficient and duplicated the data every time it went to save it.
Don't worry though, I have re wrote the code and have the name finder scanning again as we speak. I am also cleaning up the old files. So hopefully I'll have something to post real soon.