memory - Hadoop Namenode Metadata - fsimage and edit logs -

memory - Hadoop Namenode Metadata - fsimage and edit logs -

i understand fsimage loaded memory on startup , farther transactions added edit log rather fsimage performance reasons.

the fsimage in memory gets refreshed when namenode restarted. efficiency, secondary name node periodically checkpoint update fsimage namenode recovery faster. these fine.

but 1 point fail understand this, lets file exists , info file in fsimage in memory. move file different location, updated in edit log. when seek list old file path, complains thats not exists or whatever.

does mean namenode looks @ edit log contradictory purpose of fsimage in memory? or how know file location has changed?

answer looking @ info in edit logs. if info not available in edit logs question stands true use-case when write new file hdfs. while namenode running if remove fsimage file , seek read hdfs file able read.

removing fsimage file running namenode not cause issue read / write operations. when restart namenode, there errors stating image file not found.

let me seek give more explanation help out.

only on start hadoop looks fsimage file, in case if not there, namenode not come , log formatting namenode.

hadoop format -namenode command creates fsimage file (if edit logs present). after namenode startup file metadata fetched edit logs (and if not found info in edit logs searched thru fsimage file). fsimage works checkpoint inforamtion saved lastly time. 1 of reason secondary node keeps on sync (after 1 hr / 1 milliion transactions) edit logs on start lastly checkpoint not much needs synced.

if turn safemode ( command : hdfs dfsadmin -safemode enter) on , utilize savenamespace (command : hdfs dfsadmin -savenamespace), show below mentioned log message.

2014-07-05 15:03:13,195 info org.apache.hadoop.hdfs.server.namenode.fsimage: saving image file /data/hadoop-namenode-data-temp/current/fsimage.ckpt_0000000000000000169 using no compression 2014-07-05 15:03:13,205 info org.apache.hadoop.hdfs.server.namenode.fsimage: image file /data/hadoop-namenode-data-temp/current/fsimage.ckpt_0000000000000000169 of size 288 bytes saved in 0 seconds. 2014-07-05 15:03:13,213 info org.apache.hadoop.hdfs.server.namenode.nnstorageretentionmanager: going retain 2 images txid >= 0 2014-07-05 15:03:13,237 info org.apache.hadoop.hdfs.server.namenode.fseditlog: starting log segment @ 170

memory hadoop metadata