c++ - Quickly loading large data structures from a file -

c++ - Quickly loading large data structures from a file -

i have big dictionary of english language words (around 70k of them) load memory @ origin of program. loaded radix trie info structure, , each trie node has many links 1 node many others (for illustration word antonyms, "dead" -> "alive", "well"). each node have std::vector<metadata> in contains various miscellaneous metadata program.

now, problem loading time of file. reading file disk, deserialization , allocating info construction thing in general takes lot of time (4-5 seconds).

currently, i'm working on making load asynchronously (or bit bit, fraction of them per frame), due nature of application (it's mobile keyboard), there's plenty of times has loaded quickly.

what can done speed loading? memory pool everything? benchmarking different parts see can optimized, looks like, far, it's little things add together up.

if trie static (i.e. doesn't alter when program's running), build optimized version in array using array indexes in place of pointers. can save info file. startup amounts loading block of info memory.

doing way makes things less convenient (you'll have utilize arrays rather std::vector, example), , might have bit of casting, little thought end compact , fast info construction doesn't suffer allocation overhead associated creating object each node. instead, it's array of varying length structures.

i did application used directed acyclic word graph (dawg). rather rebuild dawg every time programme loaded (a time consuming process), had utility programme created dawg , shipped info file in place of word list.

c++ algorithm serialization data-structures trie