Memory leaks
Take care of memory leaks and OOM errors before starting the training process
Context: Memory management related exceptions constitute a major portion of fault triggers in deep learning. Memory leaks and OOM (out of memory) errors may during the training, loading of data, writing/saving of data and also during the inferencing
Problem: When a program terminates due to OOM errors during the process of training the network, in absence of a checkpoint, the knowledge gained during the training process is lost. Due to this, the computational energy spent on the training is wasted.
Solution: Factor in the memory availability constraints and possible OOM exceptions while designing the program to train the network and take appropriate steps to avoid them. This may reduce the chances of OOM errors and prevent the corresponding waste of energy.
Example: An example of this pattern can be seen in this stack overflow post .The user runs into out of memory error while training a Keras sequential model due to the inefficient management of the memory in a program that uses a for loop caus- ing a waste of energy spent on the training. This could be avoided by defining the network outside the for loop and reusing the same instance every iteration as suggested in the answer.
- https://stackoverflow.com/questions/63178757/
- https://stackoverflow.com/questions/43695085/
- https://stackoverflow.com/questions/61421983/
- https://stackoverflow.com/questions/47115946/
- https://stackoverflow.com/questions/41255031/
- https://stackoverflow.com/questions/62402825/
- https://stackoverflow.com/questions/60783153/
- https://stackoverflow.com/questions/52700621/
- https://stackoverflow.com/questions/49942110/
- https://stackoverflow.com/questions/47482135/
- https://stackoverflow.com/questions/50690891/
- https://stackoverflow.com/questions/47743936/
- https://stackoverflow.com/questions/54477157/
- https://stackoverflow.com/questions/43299286/
- https://stackoverflow.com/questions/66880262/
- https://stackoverflow.com/questions/51175837/
- https://stackoverflow.com/questions/52443056/
- https://stackoverflow.com/questions/61715631/
- https://stackoverflow.com/questions/46241806/
- https://stackoverflow.com/questions/35695183/
- https://stackoverflow.com/questions/44082297/
- https://stackoverflow.com/questions/46539404/
- https://stackoverflow.com/questions/50331201/
- https://stackoverflow.com/questions/49735217/
- https://stackoverflow.com/questions/48212123/
- https://stackoverflow.com/questions/44327803/
- https://stackoverflow.com/questions/47393991/
- https://stackoverflow.com/questions/49081129/
- https://stackoverflow.com/questions/47286441/
- https://stackoverflow.com/questions/44035651/
- https://stackoverflow.com/questions/39878628/
- https://stackoverflow.com/questions/62115443/
- https://stackoverflow.com/questions/62931082/
- https://stackoverflow.com/questions/63192550/
- https://stackoverflow.com/questions/67056438/
- https://stackoverflow.com/questions/63588069/
- https://stackoverflow.com/questions/59265818/
- https://stackoverflow.com/questions/65107933/
- https://stackoverflow.com/questions/62402825/
- https://stackoverflow.com/questions/54477157/
- https://stackoverflow.com/questions/61715631/
- https://stackoverflow.com/questions/50333532/
- https://stackoverflow.com/questions/50331201/
- https://stackoverflow.com/questions/44327803/
- https://stackoverflow.com/questions/49081129/
- https://stackoverflow.com/questions/47286441/
- https://stackoverflow.com/questions/58636603/
- https://stackoverflow.com/questions/66268742/
- https://stackoverflow.com/questions/64519523/
- https://stackoverflow.com/questions/58898253/
- https://stackoverflow.com/questions/54449159/
- https://stackoverflow.com/questions/64139658/
- https://stackoverflow.com/questions/56910950/
- https://stackoverflow.com/questions/42886049/
- https://stackoverflow.com/questions/43505001/
- https://stackoverflow.com/questions/48257322/
- https://stackoverflow.com/questions/61470732/
- https://stackoverflow.com/questions/48048297/
- https://stackoverflow.com/questions/45252051/
- https://stackoverflow.com/questions/46230501/
- https://stackoverflow.com/questions/50895128/
-
https://stackoverflow.com/questions/62402825/
-
https://stackoverflow.com/questions/62402825/
-
https://stackoverflow.com/questions/62402825/
-
https://stackoverflow.com/questions/62402825/