“main” java.lang.OutOfMemoryError: Java heap space Error in Stanford Custom Entity Recognition Model training

July 09, 2019, at 03:50 AM

I'm trying to train a custom NER model to recognize 41 entities(the training set has around 6000 lines)

When I try to run the training command provided in the nlp site :

java -cp stanford-ner.jar edu.stanford.nlp.ie.crf.CRFClassifier -prop austen.prop 

This is the error I'm facing :

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at edu.stanford.nlp.optimization.AbstractCachingDiffFunction.ensure(AbstractCachingDiffFunction.java:136)
        at edu.stanford.nlp.optimization.AbstractCachingDiffFunction.derivativeAt(AbstractCachingDiffFunction.java:151)
        at edu.stanford.nlp.optimization.QNMinimizer.evaluateFunction(QNMinimizer.java:1150)
        at edu.stanford.nlp.optimization.QNMinimizer.minimize(QNMinimizer.java:898)
        at edu.stanford.nlp.optimization.QNMinimizer.minimize(QNMinimizer.java:856)
        at edu.stanford.nlp.optimization.QNMinimizer.minimize(QNMinimizer.java:850)
        at edu.stanford.nlp.optimization.QNMinimizer.minimize(QNMinimizer.java:93)
        at edu.stanford.nlp.ie.crf.CRFClassifier.trainWeights(CRFClassifier.java:1935)
        at edu.stanford.nlp.ie.crf.CRFClassifier.train(CRFClassifier.java:1742)
        at edu.stanford.nlp.ie.AbstractSequenceClassifier.train(AbstractSequenceClassifier.java:785)
        at edu.stanford.nlp.ie.AbstractSequenceClassifier.train(AbstractSequenceClassifier.java:756)
        at edu.stanford.nlp.ie.crf.CRFClassifier.main(CRFClassifier.java:3011)

I tried adding -Xmx4096m to my java command to specify the max heap space as 4GB( that is the maximum available space in my machine) but still no luck.

I also tried adding -Xms1024m to specify minimum heap space and yet no different result.

This same command worked flawless without any heap space errors when I tried it to train a model for 20 entities(1500 lines)

Is this heap space related to RAM or the available space?

Should I try training in a machine with more Ram or storage?

Answer 1

If you think you end up with memory-availability issue, here is the guidelines from stanford (Refer back if you can).

  1. Ultimately, if you have tons of features and lots of classes, you need to have lots of memory to train a CRFClassifier. We frequently train models that require several gigabytes of RAM and are used to typing java -mx4g.
  2. You can decrease the memory of the limited-memory quasi-Newton optimizer (L-BFGS). The optimizer maintains a number of past guesses which are used to approximate the Hessian. Having more guesses makes the estimate more accurate, and optimization is faster, but the memory used by the system during optimization is linear in the number of guesses. This is specified by the parameter qnSize. The default is 25. Using 10 is perfectly adequate. If you're short of memory, things will still work with much smaller values, even just a value of 2.
  3. Decrease the order of the CRF. We usually use just first order CRFs (maxLeft=1 and no features that refer to the answer class more than one away - it's okay to refer to word features any distance away). While the code supports arbitrary order CRFs, building second, third, or fourth order CRFs will greatly increase memory usage and normally isn't necessary. Remember: maxLeft refers to the size of the class contexts that your features use (that is, it is one smaller than the clique size). A first order CRF can still look arbitrarily far to the left or right to get information about the observed data context.
  4. Decrease the number of features generated. To see all the features generated, you can set the property printFeatures to true. CRFClassifier will then write (potentially huge) files in the current directory listing the features generated for each token position. Options that generate huge numbers of features include useWordPairs and useNGrams when maxNGramLeng is a large number.
  5. Decrease the number of classes in your model. This may or may not be possible, depending on what your modeling requirements are. But time complexity is proportional to the number of classes raised to the clique size.
  6. Use the flag useObservedSequencesOnly=true. This makes it so that you can only label adjacent words with label sequences that were seen next to each other in the training data. For some kinds of data this actually gives better accuracy, for other kinds it is worse. But unless the label sequence patterns are dense, it will reduce your memory usage.
  7. Of course, shrinking the amount of training data will also reduce the memory needed, but isn't very desirable if you're trying to train the best classifier. You might consider throwing out sentences with no entities in them, though.
  8. If you're concerned about runtime memory usage, some of the above items still apply (number of features and classes, useObservedSequencesOnly, and order of the CRF), but in addition, you can use the flag featureDiffThresh, for example featureDiffThresh=0.05. In training, CRFClassifier will train one model, drop all the features with weight (absolute value) beneath the given threshold, and then train a second model. Training thus takes longer, but the resulting model is smaller and faster at runtime, and usually has very similar performance for a reasonable threshold such as 0.05.
How to get the modified content from google sheets?

How to get the modified content from google sheets?

I am trying to get the modified content after the given time from google sheetsNowhere I can found the api to get the data

How to communicate two services in Android?

How to communicate two services in Android?

I am building app to collect sensoric data received from bluetooth leAdditionally, I need to collect smartphone location from installed GPS sensor

how to get the last modified date for old MS-office files, e.g. .xls and .doc files

how to get the last modified date for old MS-office files, e.g. .xls and .doc files

Is there any APIs in the POI to read out the last modified date for old office files?

How to fix Insert button in Android Studio Application

How to fix Insert button in Android Studio Application

I am trying to build an application using SQLite using the Model-View-Controller architectureMy app has a Container class, a Score class, and a MainActivity class