Towards a catalog of energy patterns for deep learning development

This is an initial open catalog of energy patterns in deep learning development. Our goal is to share the knowledge accross the world and make the deep learning development more energy efficent.

Pre-trained networks

Apply transfer learning with pre-trained networks whenever feasible.
Checkpoint

Use checkpointing at regular intervals for networks with long training times.

Pruning

Prune large networks to reduce the impact of over-parameterization.

Quantization

Use network quantization in applications with memory constraints where a minor loss in performance is acceptable

Distillation

If pre-trained models are too large for a given task, apply knowledge distillation or use distilled versions of pre-trained models.

Efficient read-write

Minimize the memory footprint while performing read/write operations.

Memory Leaks

Take care of memory leaks and OOM errors before starting the training process.

Tensor Operations

Look for built in library functions for tensor operations before writing custom implementations.