Wednesday, 15 April 2015

Processing large JSON files in PHP -



Processing large JSON files in PHP -

i trying process big (possibly 200m) json files. construction of file array of objects.

so along lines of:

[ {"property":"value", "property2":"value2"}, {"prop":"val"}, ... {"foo":"bar"} ]

each object has arbitrary properties , not necessary share them other objects in array (as in, having same).

i want apply processing on each object in array , file potentially huge, cannot slurp whole file content in memory, decoding json , iterating on php array.

so ideally read file, fetch plenty info each object , process it. sax-type approach ok if there similar library available json.

any suggestion on how deal problem best?

i decided on working on event based parser. it's not quite done yet , edit question link work when roll out satisfying version.

edit:

i worked out version of parser satisfied with. it's available on github:

https://github.com/kuma-giyomu/jsonparser

there's room improvement , welcoming feedback.

php json large-files

No comments:

Post a Comment