GPU Runs Out of Memory

jchp

New member
Hardware: RTX 2070; CPU Ryzen 7 3700x; 32G RAM

Installed Version:

$ pip3 list | grep tensorflow
tensorflow 2.4.1
tensorflow-estimator 2.4.0
$ pip3 --version
pip 21.0.1 from /home/joe/py/tf/lib/python3.6/site-packages/pip (python 3.6)
$ python3 --version
Python 3.6.9
$ modinfo nvidia
version: 450.51.06
$ cat /usr/local/cuda/version.txt
CUDA Version 11.0.228
$ cat /usr/include/cudnn_version.h
#define CUDNN_MAJOR 8
#define CUDNN_MINOR 0
#define CUDNN_PATCHLEVEL 4

Run error:

>>> from ai_benchmark import AIBenchmark
>>> results = AIBenchmark().run()
1/19. MobileNet-V2
1.1 - inference | batch=50, size=224x224: 49.0 ± 2.2 ms
1.2 - training | batch=50, size=224x224: 221.8 ± 0.5 ms
2/19. Inception-V3
2.1 - inference | batch=20, size=346x346: 71.0 ± 0.0 ms
2.2 - training | batch=20, size=346x346: 236 ± 2 ms
3/19. Inception-V4
failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
attempting to perform BLAS operation using StreamExecutor without BLAS support Traceback
0 successful operations.
0 derived errors ignored.

Found the following solution from the web:

# Run the codes below before ai_benchmark.
import tensorflow as tf
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
try:
# Currently, memory growth needs to be the same across GPUs
for gpu in gpus:
tf.config.experimental.set_memory_growth(gpu, True)
logical_gpus = tf.config.experimental.list_logical_devices('GPU')
print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
except RuntimeError as e:
# Memory growth must be set before GPUs have been initialized
print(e)

Results of successful ai_benchmark run:

Device Inference Score: 10742
Device Training Score: 11271
Device AI Score: 22013
 
Top