Neo4j Embedded database hangs after abnormal application termination

Question

I'm going to make my first application based on Neo4j Embedded database public and ran into a serious blocking issue -

This my previous question with database deleting where I'm faced this issue Effectively clear Neo4j database

I use:

OS: Windows 8.1
jdk1.8.0_51 
neo4j version 2.2.3
lucene version 3.6.2
spring-data-neo4j version 3.4.0.M1

I noticed that everything works properly if I use graphDatabaseService.shutdown(); method before terminating of my application.. Otherwise database is destroyed(not only my application but even Neo4j server also hangs on this corrupted db when I try to load my embedded db with Neo4j Web interface).

Is any way to make Neo4j Embedded database more fault tolerant ? I will lose all my data after the first error (for example blackout event) in the production environment..

I spent a few days on this issue but still have no ideas how to proceed with it. Please help.

UPDATED

info from message.log

2015-07-27 08:51:59.232+0000 INFO  [o.n.k.i.DiagnosticsManager]:   index:
2015-07-27 08:51:59.233+0000 INFO  [o.n.k.i.DiagnosticsManager]:     lucene:
2015-07-27 08:51:59.233+0000 INFO  [o.n.k.i.DiagnosticsManager]:       node:
2015-07-27 08:51:59.233+0000 INFO  [o.n.k.i.DiagnosticsManager]:         node_auto_index:
2015-07-27 08:51:59.234+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _2p.fdt: 2015-07-27T11:50:33+0300 - 44,78 MB
2015-07-27 08:51:59.234+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _2p.fdx: 2015-07-27T11:50:33+0300 - 1,78 MB
2015-07-27 08:51:59.234+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _2p.fnm: 2015-07-27T11:50:33+0300 - 54,00 B
2015-07-27 08:51:59.235+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _2p.frq: 2015-07-27T11:50:34+0300 - 3,99 MB
2015-07-27 08:51:59.235+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _2p.nrm: 2015-07-27T11:50:34+0300 - 684,50 kB
2015-07-27 08:51:59.235+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _2p.prx: 2015-07-27T11:50:34+0300 - 4,12 MB
2015-07-27 08:51:59.236+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _2p.tii: 2015-07-27T11:50:34+0300 - 212,98 kB
2015-07-27 08:51:59.236+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _2p.tis: 2015-07-27T11:50:34+0300 - 21,83 MB
2015-07-27 08:51:59.237+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _2s.cfs: 2015-07-27T11:50:37+0300 - 1,48 MB
2015-07-27 08:51:59.237+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _2t.cfs: 2015-07-27T11:50:38+0300 - 1,55 MB
2015-07-27 08:51:59.237+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _2u.cfs: 2015-07-27T11:50:39+0300 - 1,57 MB
2015-07-27 08:51:59.238+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _2z.cfs: 2015-07-27T11:50:43+0300 - 1,46 MB
2015-07-27 08:51:59.238+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _30.cfs: 2015-07-27T11:50:44+0300 - 12,49 MB
2015-07-27 08:51:59.238+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _31.cfs: 2015-07-27T11:50:45+0300 - 1,22 MB
2015-07-27 08:51:59.238+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _32.cfs: 2015-07-27T11:51:56+0300 - 3,36 MB
2015-07-27 08:51:59.239+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _x.fdt: 2015-07-27T11:49:35+0300 - 19,12 MB
2015-07-27 08:51:59.239+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _x.fdx: 2015-07-27T11:49:35+0300 - 773,77 kB
2015-07-27 08:51:59.239+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _x.fnm: 2015-07-27T11:49:35+0300 - 54,00 B
2015-07-27 08:51:59.240+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _x.frq: 2015-07-27T11:49:35+0300 - 1,72 MB
2015-07-27 08:51:59.240+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _x.nrm: 2015-07-27T11:49:35+0300 - 290,17 kB
2015-07-27 08:51:59.240+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _x.prx: 2015-07-27T11:49:35+0300 - 1,77 MB
2015-07-27 08:51:59.240+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _x.tii: 2015-07-27T11:49:35+0300 - 72,69 kB
2015-07-27 08:51:59.241+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _x.tis: 2015-07-27T11:49:35+0300 - 9,29 MB
2015-07-27 08:51:59.241+0000 INFO  [o.n.k.i.DiagnosticsManager]:           segments.gen: 2015-07-27T11:51:57+0300 - 20,00 B
2015-07-27 08:51:59.241+0000 INFO  [o.n.k.i.DiagnosticsManager]:           segments_2: 2015-07-27T11:51:57+0300 - 2,00 kB
2015-07-27 08:51:59.242+0000 INFO  [o.n.k.i.DiagnosticsManager]:           write.lock: 2015-07-27T11:51:55+0300 - 0,00 B
2015-07-27 08:51:59.242+0000 INFO  [o.n.k.i.DiagnosticsManager]:         - Total: 2015-07-27T11:51:57+0300 - 133,52 MB
2015-07-27 08:51:59.242+0000 INFO  [o.n.k.i.DiagnosticsManager]:       - Total: 2015-07-27T11:48:44+0300 - 133,52 MB
2015-07-27 08:51:59.242+0000 INFO  [o.n.k.i.DiagnosticsManager]:       relationship:
2015-07-27 08:51:59.242+0000 INFO  [o.n.k.i.DiagnosticsManager]:         __rel_types__:
2015-07-27 08:51:59.243+0000 INFO  [o.n.k.i.DiagnosticsManager]:           segments.gen: 2015-07-27T11:50:46+0300 - 20,00 B
2015-07-27 08:51:59.243+0000 INFO  [o.n.k.i.DiagnosticsManager]:           segments_1: 2015-07-27T11:50:46+0300 - 32,00 B
2015-07-27 08:51:59.243+0000 INFO  [o.n.k.i.DiagnosticsManager]:         - Total: 2015-07-27T11:51:53+0300 - 52,00 B
2015-07-27 08:51:59.244+0000 INFO  [o.n.k.i.DiagnosticsManager]:       - Total: 2015-07-27T11:48:44+0300 - 52,00 B
2015-07-27 08:51:59.244+0000 INFO  [o.n.k.i.DiagnosticsManager]:     - Total: 2015-07-27T11:48:44+0300 - 133,52 MB
2015-07-27 08:51:59.244+0000 INFO  [o.n.k.i.DiagnosticsManager]:   - Total: 2015-07-27T11:48:44+0300 - 133,52 MB
2015-07-27 08:51:59.244+0000 INFO  [o.n.k.i.DiagnosticsManager]:   index.db: 2015-07-27T11:48:33+0300 - 411,00 B
2015-07-27 08:51:59.245+0000 INFO  [o.n.k.i.DiagnosticsManager]:   lock: 2015-07-27T11:48:16+0300 - 0,00 B
2015-07-27 08:51:59.249+0000 INFO  [o.n.k.i.DiagnosticsManager]:   messages.log: 2015-07-27T11:51:54+0300 - 47,16 kB
2015-07-27 08:51:59.250+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore: 2015-07-27T11:51:57+0300 - 8,00 kB
2015-07-27 08:51:59.251+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.counts.db.a: 2015-07-27T11:51:57+0300 - 2,80 kB
2015-07-27 08:51:59.251+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.counts.db.b: 2015-07-27T11:50:46+0300 - 2,80 kB
2015-07-27 08:51:59.252+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.id: 2015-07-27T11:51:58+0300 - 9,00 B
2015-07-27 08:51:59.253+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.labeltokenstore.db: 2015-07-27T11:50:46+0300 - 8,00 kB
2015-07-27 08:51:59.254+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.labeltokenstore.db.id: 2015-07-27T11:51:57+0300 - 9,00 B
2015-07-27 08:51:59.255+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.labeltokenstore.db.names: 2015-07-27T11:50:46+0300 - 7,98 kB
2015-07-27 08:51:59.256+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.labeltokenstore.db.names.id: 2015-07-27T11:51:57+0300 - 9,00 B
2015-07-27 08:51:59.259+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.nodestore.db: 2015-07-27T11:51:57+0300 - 5,69 MB
2015-07-27 08:51:59.260+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.nodestore.db.id: 2015-07-27T11:51:58+0300 - 9,00 B
2015-07-27 08:51:59.260+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.nodestore.db.labels: 2015-07-27T11:48:22+0300 - 68,00 B
2015-07-27 08:51:59.261+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.nodestore.db.labels.id: 2015-07-27T11:51:58+0300 - 9,00 B
2015-07-27 08:51:59.265+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.propertystore.db: 2015-07-27T11:51:57+0300 - 31,17 MB
2015-07-27 08:51:59.265+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.propertystore.db.arrays: 2015-07-27T11:48:22+0300 - 128,00 B
2015-07-27 08:51:59.266+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.propertystore.db.arrays.id: 2015-07-27T11:51:57+0300 - 9,00 B
2015-07-27 08:51:59.267+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.propertystore.db.id: 2015-07-27T11:51:57+0300 - 9,00 B
2015-07-27 08:51:59.268+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.propertystore.db.index: 2015-07-27T11:50:46+0300 - 8,00 kB
2015-07-27 08:51:59.269+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.propertystore.db.index.id: 2015-07-27T11:51:57+0300 - 9,00 B
2015-07-27 08:51:59.269+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.propertystore.db.index.keys: 2015-07-27T11:50:46+0300 - 7,98 kB
2015-07-27 08:51:59.270+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.propertystore.db.index.keys.id: 2015-07-27T11:51:57+0300 - 9,00 B
2015-07-27 08:51:59.274+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.propertystore.db.strings: 2015-07-27T11:51:57+0300 - 41,28 MB
2015-07-27 08:51:59.274+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.propertystore.db.strings.id: 2015-07-27T11:51:57+0300 - 9,00 B
2015-07-27 08:51:59.278+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.relationshipgroupstore.db: 2015-07-27T11:51:57+0300 - 151,68 kB
2015-07-27 08:51:59.280+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.relationshipgroupstore.db.id: 2015-07-27T11:51:58+0300 - 9,00 B
2015-07-27 08:51:59.283+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.relationshipstore.db: 2015-07-27T11:51:57+0300 - 37,01 MB
2015-07-27 08:51:59.284+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.relationshipstore.db.id: 2015-07-27T11:51:58+0300 - 9,00 B
2015-07-27 08:51:59.285+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.relationshiptypestore.db: 2015-07-27T11:50:46+0300 - 8,00 kB
2015-07-27 08:51:59.286+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.relationshiptypestore.db.id: 2015-07-27T11:51:57+0300 - 9,00 B
2015-07-27 08:51:59.286+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.relationshiptypestore.db.names: 2015-07-27T11:50:46+0300 - 7,98 kB
2015-07-27 08:51:59.287+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.relationshiptypestore.db.names.id: 2015-07-27T11:51:57+0300 - 9,00 B
2015-07-27 08:51:59.287+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.schemastore.db: 2015-07-27T11:50:46+0300 - 8,00 kB
2015-07-27 08:51:59.288+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.schemastore.db.id: 2015-07-27T11:51:58+0300 - 9,00 B
2015-07-27 08:51:59.288+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.transaction.db.0: 2015-07-27T11:50:43+0300 - 251,72 MB
2015-07-27 08:51:59.289+0000 INFO  [o.n.k.i.DiagnosticsManager]:   neostore.transaction.db.1: 2015-07-27T11:50:48+0300 - 6,12 MB
2015-07-27 08:51:59.289+0000 INFO  [o.n.k.i.DiagnosticsManager]:   schema:
2015-07-27 08:51:59.289+0000 INFO  [o.n.k.i.DiagnosticsManager]:     index:
2015-07-27 08:51:59.289+0000 INFO  [o.n.k.i.DiagnosticsManager]:       lucene:
2015-07-27 08:51:59.289+0000 INFO  [o.n.k.i.DiagnosticsManager]:         1:
2015-07-27 08:51:59.290+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _4v.cfs: 2015-07-27T11:50:35+0300 - 395,64 kB
2015-07-27 08:51:59.290+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _5c.cfs: 2015-07-27T11:50:44+0300 - 49,47 kB
2015-07-27 08:51:59.290+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _5d.fdt: 2015-07-27T11:51:58+0300 - 0,00 B
2015-07-27 08:51:59.291+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _5d.fdx: 2015-07-27T11:51:58+0300 - 0,00 B
2015-07-27 08:51:59.291+0000 INFO  [o.n.k.i.DiagnosticsManager]:           failure-message: 2015-07-27T11:48:27+0300 - 16,00 kB
2015-07-27 08:51:59.291+0000 INFO  [o.n.k.i.DiagnosticsManager]:           segments.gen: 2015-07-27T11:50:44+0300 - 20,00 B
2015-07-27 08:51:59.292+0000 INFO  [o.n.k.i.DiagnosticsManager]:           segments_2: 2015-07-27T11:50:44+0300 - 542,00 B
2015-07-27 08:51:59.292+0000 INFO  [o.n.k.i.DiagnosticsManager]:           write.lock: 2015-07-27T11:48:28+0300 - 0,00 B
2015-07-27 08:51:59.292+0000 INFO  [o.n.k.i.DiagnosticsManager]:         - Total: 2015-07-27T11:51:58+0300 - 461,66 kB
2015-07-27 08:51:59.292+0000 INFO  [o.n.k.i.DiagnosticsManager]:         2:
2015-07-27 08:51:59.293+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _36.cfs: 2015-07-27T11:50:03+0300 - 103,38 kB
2015-07-27 08:51:59.293+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _5b.cfs: 2015-07-27T11:50:44+0300 - 73,52 kB
2015-07-27 08:51:59.293+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _5c.fdt: 2015-07-27T11:51:58+0300 - 0,00 B
2015-07-27 08:51:59.294+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _5c.fdx: 2015-07-27T11:51:58+0300 - 0,00 B
2015-07-27 08:51:59.294+0000 INFO  [o.n.k.i.DiagnosticsManager]:           failure-message: 2015-07-27T11:48:28+0300 - 16,00 kB
2015-07-27 08:51:59.294+0000 INFO  [o.n.k.i.DiagnosticsManager]:           segments.gen: 2015-07-27T11:50:44+0300 - 20,00 B
2015-07-27 08:51:59.295+0000 INFO  [o.n.k.i.DiagnosticsManager]:           segments_2: 2015-07-27T11:50:44+0300 - 542,00 B
2015-07-27 08:51:59.295+0000 INFO  [o.n.k.i.DiagnosticsManager]:           write.lock: 2015-07-27T11:48:28+0300 - 0,00 B
2015-07-27 08:51:59.295+0000 INFO  [o.n.k.i.DiagnosticsManager]:         - Total: 2015-07-27T11:51:58+0300 - 193,45 kB
2015-07-27 08:51:59.295+0000 INFO  [o.n.k.i.DiagnosticsManager]:         3:
2015-07-27 08:51:59.296+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _37.cfs: 2015-07-27T11:50:04+0300 - 789,69 kB
2015-07-27 08:51:59.296+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _4q.cfs: 2015-07-27T11:50:33+0300 - 427,07 kB
2015-07-27 08:51:59.296+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _55.cfs: 2015-07-27T11:50:41+0300 - 104,29 kB
2015-07-27 08:51:59.297+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _5a.cfs: 2015-07-27T11:50:44+0300 - 42,41 kB
2015-07-27 08:51:59.297+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _5b.fdt: 2015-07-27T11:51:58+0300 - 0,00 B
2015-07-27 08:51:59.297+0000 INFO  [o.n.k.i.DiagnosticsManager]:           _5b.fdx: 2015-07-27T11:51:58+0300 - 0,00 B
2015-07-27 08:51:59.297+0000 INFO  [o.n.k.i.DiagnosticsManager]:           failure-message: 2015-07-27T11:48:28+0300 - 16,00 kB
2015-07-27 08:51:59.298+0000 INFO  [o.n.k.i.DiagnosticsManager]:           segments.gen: 2015-07-27T11:50:44+0300 - 20,00 B
2015-07-27 08:51:59.298+0000 INFO  [o.n.k.i.DiagnosticsManager]:           segments_2: 2015-07-27T11:50:44+0300 - 1,01 kB
2015-07-27 08:51:59.298+0000 INFO  [o.n.k.i.DiagnosticsManager]:           write.lock: 2015-07-27T11:48:28+0300 - 0,00 B
2015-07-27 08:51:59.299+0000 INFO  [o.n.k.i.DiagnosticsManager]:         - Total: 2015-07-27T11:51:58+0300 - 1,35 MB
2015-07-27 08:51:59.300+0000 INFO  [o.n.k.i.DiagnosticsManager]:       - Total: 2015-07-27T11:48:28+0300 - 1,99 MB
2015-07-27 08:51:59.300+0000 INFO  [o.n.k.i.DiagnosticsManager]:     - Total: 2015-07-27T11:48:27+0300 - 1,99 MB
2015-07-27 08:51:59.300+0000 INFO  [o.n.k.i.DiagnosticsManager]:     label:
2015-07-27 08:51:59.301+0000 INFO  [o.n.k.i.DiagnosticsManager]:       lucene:
2015-07-27 08:51:59.301+0000 INFO  [o.n.k.i.DiagnosticsManager]:         _3d.cfs: 2015-07-27T11:50:04+0300 - 427,97 kB
2015-07-27 08:51:59.302+0000 INFO  [o.n.k.i.DiagnosticsManager]:         _3d_1.del: 2015-07-27T11:50:44+0300 - 37,00 B
2015-07-27 08:51:59.302+0000 INFO  [o.n.k.i.DiagnosticsManager]:         _4q.cfs: 2015-07-27T11:50:31+0300 - 209,44 kB
2015-07-27 08:51:59.302+0000 INFO  [o.n.k.i.DiagnosticsManager]:         _4q_1.del: 2015-07-27T11:50:44+0300 - 37,00 B
2015-07-27 08:51:59.302+0000 INFO  [o.n.k.i.DiagnosticsManager]:         _5f.cfs: 2015-07-27T11:50:44+0300 - 90,51 kB
2015-07-27 08:51:59.303+0000 INFO  [o.n.k.i.DiagnosticsManager]:         _5g.cfs: 2015-07-27T11:51:56+0300 - 18,07 kB
2015-07-27 08:51:59.303+0000 INFO  [o.n.k.i.DiagnosticsManager]:         _5h.cfs: 2015-07-27T11:51:56+0300 - 108,35 kB
2015-07-27 08:51:59.303+0000 INFO  [o.n.k.i.DiagnosticsManager]:         segments.gen: 2015-07-27T11:51:56+0300 - 20,00 B
2015-07-27 08:51:59.303+0000 INFO  [o.n.k.i.DiagnosticsManager]:         segments_2: 2015-07-27T11:51:56+0300 - 776,00 B
2015-07-27 08:51:59.304+0000 INFO  [o.n.k.i.DiagnosticsManager]:         write.lock: 2015-07-27T11:48:22+0300 - 0,00 B
2015-07-27 08:51:59.304+0000 INFO  [o.n.k.i.DiagnosticsManager]:       - Total: 2015-07-27T11:51:56+0300 - 855,19 kB
2015-07-27 08:51:59.304+0000 INFO  [o.n.k.i.DiagnosticsManager]:     - Total: 2015-07-27T11:48:22+0300 - 855,19 kB
2015-07-27 08:51:59.304+0000 INFO  [o.n.k.i.DiagnosticsManager]:   - Total: 2015-07-27T11:48:27+0300 - 2,82 MB
2015-07-27 08:51:59.305+0000 INFO  [o.n.k.i.DiagnosticsManager]:   store_lock: 2015-07-27T11:48:16+0300 - 0,00 B
2015-07-27 08:51:59.305+0000 INFO  [o.n.k.i.DiagnosticsManager]: --- STARTED diagnostics for KernelDiagnostics:StoreFiles END ---
2015-07-27 08:51:59.378+0000 INFO  [o.n.k.EmbeddedGraphDatabase]: Database is now ready
2015-07-27 08:52:02.592+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 399ms.
2015-07-27 08:52:08.474+0000 INFO  [o.n.k.i.a.i.s.OnlineIndexSamplingJob]: Sampled index :Criterion(name) with 18159 unique values in sample of avg size 18159 taken from index containing 18159 entries
2015-07-27 08:52:11.514+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 354ms.
2015-07-27 08:52:13.692+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 336ms.
2015-07-27 08:52:18.760+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 3728ms.
2015-07-27 08:52:18.764+0000 INFO  [o.n.k.i.a.i.s.OnlineIndexSamplingJob]: Sampled index :CriterionGroup(name) with 1691 unique values in sample of avg size 1691 taken from index containing 1691 entries
2015-07-27 08:52:19.951+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 490ms.
2015-07-27 08:52:25.130+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 4476ms.
2015-07-27 08:52:26.492+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 761ms.
2015-07-27 08:52:33.603+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 5408ms.
2015-07-27 08:52:35.734+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 809ms.
2015-07-27 08:52:43.564+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 5601ms.

During delete operation application hangs on tx.close(); line:

public class Neo4jUtils {

    final static Logger logger = LoggerFactory.getLogger(Neo4jUtils.class);

    private static final int BATCH_SIZE = 10;

    public static void cleanDb(Neo4jTemplate template) {
        logger.info("Cleaning database");
        long deletedNodesCount = 0;
        do {
            GraphDatabaseService graphDatabaseService = template.getGraphDatabaseService();
            Transaction tx = graphDatabaseService.beginTx();
            try {
                Result<Map<String, Object>> result = template.query("MATCH (n) WITH n LIMIT " + BATCH_SIZE + " OPTIONAL MATCH (n)-[r]-() DELETE n, r RETURN count(n) as count", null);
                deletedNodesCount = (long) result.single().get("count");
                tx.success();
                logger.info("Deleted " + deletedNodesCount + " nodes...");
            } catch (Throwable th) {
                logger.error("Error while deleting database", th);
                throw th;
            } finally {
                tx.close();
            }
        } while (deletedNodesCount > 0);
    }

}

UPDATED

I found the general source of this issue but have no idea right now how to fix it.

This is my WebSecurityConfig:

@Configuration
@EnableWebSecurity
@EnableWebMvcSecurity
public class WebSecurityConfig extends WebSecurityConfigurerAdapter {


    @Autowired
    private SocialAuthenticationSuccessHandler socialAuthenticationSuccessHandler;

    @Autowired
    private TokenAuthenticationService tokenAuthenticationService;

    @Autowired
    private TokenAuthenticationFilter tokenAuthenticationFilter;

    @Autowired
    private DBUserDetailsService userDetailsService;

.....

}

I autowired 4 fields that internally use Neo4j(DAO, repositories and so on). Neo4j hangs on the restart after unclean shutdown when these fields are present inside of my WebSecurityConfig. When I comment these fields the issue disappears.

What can be wrong ?

Also your data has to be committed to be recovered, data **in progress** will not be recovered. Can you explain more about your operations, that you execute and your graph model / indexes etc. To see why recovery could take that long? Can you run `bin/neo4j console` to see any relevant output? And please check `data/graph.db/messages.log` for any errors. — Michael Hunger, Jul 27 '15 at 08:24
I have added last lines from my messages.log. Also, I don't know how to run "bin/neo4j" console under Windows OS. I don't have neo4j command in my "bin" folder there. — alexanoid, Jul 27 '15 at 08:58
During delete operation application hangs on "tx.commit()" line. I have added this code into the question. — alexanoid, Jul 27 '15 at 09:06
are you assembling the neo4j jars and your code into one big jar (also called uber jar)? If so, make sure the files in `META-INF/services` get merged and not just stupidly copied one over the other. Also make sure to have `neo4j-lucene-index-.jar` on the classpath — Stefan Armbruster, Jul 28 '15 at 20:26
I'm assembling single war file with Maven `maven-war-plugin`, all libs are placed into `WEB_INF/libs` folder. I have `neo4j-lucene-index-2.2.3.jar` there but I don't have `services` folder under `META-INF` here.. — alexanoid, Jul 28 '15 at 20:35
I have updated my question, I think, I found the general source of this issue but have no idea right now how to fix it. Could you please look into this ? — alexanoid, Jul 29 '15 at 11:55

score 2 · Answer 1 · answered Jul 26 '15 at 19:33

2

Neo4j Enterprise allows you to run multiple Neo4j instances in a cluster providing you high availability and scale out for reads. To survive n failures at the same time you need 2n+1 instances.

In case of a unclean shutdown, the expected behaviour is an automatic recovering operation upon next startup (which might take some time). In the last few years I've never experienced Neo4j in a non-recoverable state after a unclean shutdown.

For embedded use cases be sure to call gdb.shutdown() in a JVM shutdown hook.

answered Jul 26 '15 at 19:33

Stefan Armbruster

39,465
6
87
97

Thanks for your answer. How about power outage ? In this case JVM shutdown hook will never triggered. How to deal in this case with Neo4j embedded ? – alexanoid Jul 27 '15 at 04:28
I'm really worried about our data integrity and right now looks like that I can't rely in this case on Neo4j embedded database.. I'll lose all of my data right after the first abnormal termination of the application. I'd be really happy to use Neo4j Enterprise edition, but AFAIK, it's not free. Right now we are team from only two persons and developing our pet project(small startup), some kind of recommendation system. Is any way to get Neo4j Enterprise edition for free ? – alexanoid Jul 27 '15 at 06:29
No, you don't loose your data in case of an abnormal termination. As mentioned there is an automatic recovery upon next startup, both in embedded and server mode. Which underlying filesystem are you using? – Stefan Armbruster Jul 27 '15 at 06:48
For startups up to 20 employees Neo4j Enterprise is free to use, see http://neo4j.com/startup-program/ – Stefan Armbruster Jul 27 '15 at 06:50
I waited for 3 hours - delete database operation was not completed after restart.. Also, Neo4j server unable to start on this database and it's also hangs.Everything works perfectly if I'm using shutdown() method otherwise hangs each time after application restart. On the working database delete operations is performed for ~1min – alexanoid Jul 27 '15 at 06:56
Thanks for pointing me to Neo4j Enterprise for startups. I have sent my email to Neo4j. – alexanoid Jul 27 '15 at 07:10

score 0 · Accepted Answer · answered Aug 06 '15 at 07:48

I can confirm that this issue related to the following fix - https://github.com/neo4j/neo4j/commit/c9596af0d990d7dde9a9053799b7295985ae3c05

I have applied the changes from this commit and the issue is gone.

So, right now I'm waiting for Neo4j 2.2.4 release.

Do you know the planned release date for Neo4j 2.2.4 ?

Neo4j Embedded database hangs after abnormal application termination

2 Answers2

Linked