问题现象
程序崩溃,提示MongoDB写入失败,无法再连起。
分析原因
1.首先想到分析mongoDB日志记录
通过 cat /etc/mongod.conf 找到日志所在目录 /var/log/mongodb/mongod.log
2018-11-07T16:50:44.165+0800 W FTDC [ftdc] Uncaught exception in 'FileStreamFailed: Failed to write to interim file buffer for full-time diagnostic data capture: /var/lib/mongo/diagnostic.data/metrics.interim.temp' in full-time diagnostic data capture subsystem. Shutting down the full-time diagnostic data capture subsystem.
2018-11-07T16:51:30.913+0800 E STORAGE [WTCheckpointThread] WiredTiger error (28):handle-write: pwrite: failed to write 4096 bytes at offset 1486848: No space left on device
2018-11-07T16:51:30.914+0800 E STORAGE [WTCheckpointThread] WiredTiger error (28): fatal checkpoint failure: No space left on device
2018-11-07T16:51:30.914+0800 E STORAGE [WTCheckpointThread] WiredTiger error (-31804) WT_SESSION.checkpoint: the process must exit and restart: WT
日志反馈的信息很明确,“磁盘已被写满啦!”, 但是很奇怪,写入量并不大,且只有唯一任务在执行,写满是不可能的。
可能想到的问题是蠕虫病毒,或是由程序递归,死循环等造成的错误数据写入。