AUCROC Metric

[vikashg]

Hello all,
While running the validation loop after an epoch I get the following error

Traceback (most recent call last):
  File "/data/Vikash/Tools_A100/Tools/BreastDensityMonai/CAII/app_classification/train_pytorch.py", line 159, in <module>
    main()
  File "/data/Vikash/Tools_A100/Tools/BreastDensityMonai/CAII/app_classification/train_pytorch.py", line 143, in main
    result = auc_metric.aggregate()
  File "/data/Vikash/Tools_A100/Tools/BreastDensityMonai/MONAI/monai/metrics/rocauc.py", line 62, in aggregate
    return compute_roc_auc(y_pred=y_pred, y=y, average=self.average)
  File "/data/Vikash/Tools_A100/Tools/BreastDensityMonai/MONAI/monai/metrics/rocauc.py", line 151, in compute_roc_auc
    auc_values = [_calculate(y_pred_, y_) for y_pred_, y_ in zip(y_pred, y)]
  File "/data/Vikash/Tools_A100/Tools/BreastDensityMonai/MONAI/monai/metrics/rocauc.py", line 151, in <listcomp>
    auc_values = [_calculate(y_pred_, y_) for y_pred_, y_ in zip(y_pred, y)]
  File "/data/Vikash/Tools_A100/Tools/BreastDensityMonai/MONAI/monai/metrics/rocauc.py", line 69, in _calculate
    raise AssertionError("y values must be 0 or 1, can not be all 0 or all 1.")
AssertionError: y values must be 0 or 1, can not be all 0 or all 1.

In order to debug I printed the y_prediction and y_actual. which are as follows

tensor([0.4658, 0.5342], device='cuda:0') tensor([0., 1.], device='cuda:0', dtype=torch.float64) <class 'torch.Tensor'> <class 'torch.Tensor'> torch.Size([2]) torch.Size([2])
tensor([0.5123, 0.4877], device='cuda:0') tensor([0., 1.], device='cuda:0', dtype=torch.float64) <class 'torch.Tensor'> <class 'torch.Tensor'> torch.Size([2]) torch.Size([2])
tensor([0.4366, 0.5634], device='cuda:0') tensor([0., 1.], device='cuda:0', dtype=torch.float64) <class 'torch.Tensor'> <class 'torch.Tensor'> torch.Size([2]) torch.Size([2])

As you can see that not all values are 0 or 1. I know that the AUCmetric has worked with floating point values, when I tried the 2D classification example in the tutorial section

tensor([0.2537, 0.2443, 0.2856, 0.2164], device='cuda:0') tensor([1., 0., 0., 0.]) <class 'torch.Tensor'> <class 'torch.Tensor'> torch.Size([4]) torch.Size([4])
tensor([0.2585, 0.2451, 0.2838, 0.2125], device='cuda:0') tensor([0., 1., 0., 0.]) <class 'torch.Tensor'> <class 'torch.Tensor'> torch.Size([4]) torch.Size([4])
tensor([0.2591, 0.2452, 0.2785, 0.2172], device='cuda:0') tensor([1., 0., 0., 0.]) <class 'torch.Tensor'> <class 'torch.Tensor'> torch.Size([4]) torch.Size([4])
tensor([0.2586, 0.2446, 0.2792, 0.2176], device='cuda:0') tensor([0., 1., 0., 0.]) <class 'torch.Tensor'> <class 'torch.Tensor'> torch.Size([4]) torch.Size([4])
tensor([0.2587, 0.2439, 0.2854, 0.2121], device='cuda:0') tensor([1., 0., 0., 0.]) <class 'torch.Tensor'> <class 'torch.Tensor'> torch.Size([4]) torch.Size([4])
tensor([0.2537, 0.2471, 0.2821, 0.2171], device='cuda:0') tensor([0., 1., 0., 0.]) <class 'torch.Tensor'> <class 'torch.Tensor'> torch.Size([4]) torch.Size([4])
----------
epoch 4/4

Here is the evaluation section of the code

		if (epoch + 1) % val_interval == 0: 
			model.eval()
			with torch.no_grad():
				y_pred = torch.tensor([], dtype=torch.float32, device=device)
				y = torch.tensor([], dtype = torch.float32, device=device)
				for val_data in validation_loader:
					val_images, val_labels = (val_data[0].to(device), val_data[1].to(device))
					y_pred = torch.cat([y_pred, model(val_images)], dim =0)
					y = torch.cat([y, val_labels], dim =0 )
				y_onehot = [y_trans(i) for i in decollate_batch(y)]
				y_pred_act = [y_pred_trans(i) for i in decollate_batch(y_pred)]
			print(len(y_onehot))
			for _y, _z in zip(y_pred_act, y_onehot):
				print(_y, _z, type(_y), type(_z), _y.shape, _z.shape)

			auc_metric(y_pred_act, y_onehot)
			result = auc_metric.aggregate()
			auc_metric.reset()
			del y_pred_act, y_onehot

			metric_values.append(result)
			acc_value = torch.eq(y_pred.argmax(dim=1), y.argmax(dim=1))
			acc_metric = acc_value.sum().item()/len(acc_value)

			if result > best_metric:
				best_metric = result
				best_metric_epoch = epoch + 1 
				torch.save(model.state_dict(), os.path.join(args.base_data_dir, 'best_metric_model.pth'))
				print('saved new best METRIC MODEL')

Any thoughts why the same kind of input is not working.

[Nic-Ma]

Hi @vikashg ,

That because in theory AUC metric can compute with all 0 or all 1 in any class of your labels, please confirm your label values.

Thanks.

[vikashg]

My label values are [1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0] and [0, 0, 0, 1].
I noticed that this error arises when in a batch all the samples are taken from the same class. While we are at it, just confirming that label matrix should be of the format (batch_size, num_classes) and not its transpose

[vikashg]

Hi @Nic-Ma, Here is the code if it helps

def main():
	args = get_arguments()
	img_width = int(args.img_width)
	img_height = int (args.img_height)
	max_epochs = int(args.max_epochs)
	num_classes = int(args.num_classes)

	fid = open(args.data_file, 'r')
	dictionary = json.load(fid)
	image_files = dictionary['images']
	_labels = dictionary['labels']
	y_pred_trans = Compose([EnsureType(), Activations(softmax = True)])
	y_trans = Compose([EnsureType()])


	labels = []
	for _lab in _labels:
		if _lab == 'A':
			new_labels = np.asarray([1, 0, 0, 0],  dtype=float)
		if _lab == 'B':
			new_labels = np.asarray([0, 1, 0, 0], dtype=float)
		if _lab == 'C':
			new_labels = np.asarray([0, 0, 1, 0], dtype=float)
		if _lab == 'D':
			new_labels = np.asarray([0, 0, 0, 1], dtype=float)

		labels.append(new_labels)

	# temp = list(zip(image_files, labels))
	# random.shuffle(temp)
	# image_files, labels = zip(*temp)

	model_dir = os.path.join(args.base_data_dir, 'Trained_Models', args.model_name)
	if os.path.exists(model_dir)==0:
		os.makedirs(model_dir)
	weights_file = os.path.join(model_dir, 'Best_model.pth')

	num_train_images = int(len(image_files)*0.6)
	num_valid_images = int(len(image_files)*0.2)
	num_test_images = len(image_files) - num_train_images - num_valid_images

	num_train_images = 20
	num_valid_images = 10
	num_test_images = len(image_files) - num_train_images - num_valid_images
	
	train_images = image_files[0:num_train_images]
	train_labels = labels[0:num_train_images]

	validation_images = image_files[num_train_images:num_train_images + num_valid_images]
	validation_labels = labels[num_train_images: num_train_images + num_valid_images]

	device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
	
	train_loader, validation_loader = get_data_loaders(train_images, train_labels, validation_images, validation_labels)
	
	# model = inception_v3(num_classes=int(args.num_classes), device=device)
	model = DenseNet121(spatial_dims = 2, in_channels = 3, out_channels = 4).to(device)
	loss_function = torch.nn.CrossEntropyLoss()

	optimizer = torch.optim.Adam(model.parameters(), 1e-5)
	val_interval = 1 
	auc_metric = ROCAUCMetric()

	best_metric = -1
	best_metric_epoch = -1 
	epoch_loss_values = []
	metric_values = []

	print(model)

	step = 0 
	for epoch in range(int(args.max_epochs)):
		model.train()
		epoch_loss = 0 
		print('-' *10)
		print(f"epoch {epoch + 1}/{max_epochs}")
		for batch_data in train_loader:
			step += 1
			inputs, labels = batch_data[0].to(device), batch_data[1].to(device)
			optimizer.zero_grad()
			print(inputs.shape, labels.shape)
			outputs = model(inputs)
			loss = loss_function(outputs, labels)
			loss.backward()
			optimizer.step()
			epoch_loss += loss.item()
			print(f"{step}/{num_train_images // train_loader.batch_size}, "f"train_loss: {loss.item():.4f}")
			epoch_len = num_train_images // train_loader.batch_size
		epoch_loss /= step
		epoch_loss_values.append(epoch_loss)
		print(f"epoch {epoch + 1} average loss: {epoch_loss: .4f}")


		if (epoch + 1) % val_interval == 0: 
			model.eval()
			with torch.no_grad():
				y_pred = torch.tensor([], dtype=torch.float32, device=device)
				y = torch.tensor([], dtype = torch.float32, device=device)
				for val_data in validation_loader:
					val_images, val_labels = (val_data[0].to(device), val_data[1].to(device))
					y_pred = torch.cat([y_pred, model(val_images)], dim =0)
					y = torch.cat([y, val_labels], dim =0 )
				y_onehot = [y_trans(i) for i in decollate_batch(y)]
				y_pred_act = [y_pred_trans(i) for i in decollate_batch(y_pred)]
				print(len(y_onehot))
				for _y, _z in zip(y_pred_act, y_onehot):
					print(_y, _z, type(_y), type(_z), _y.shape, _z.shape)

			auc_metric(y_pred_act, y_onehot)
			result = auc_metric.aggregate()
			auc_metric.reset()
			del y_pred_act, y_onehot

			metric_values.append(result)
			acc_value = torch.eq(y_pred.argmax(dim=1), y.argmax(dim=1))
			acc_metric = acc_value.sum().item()/len(acc_value)

			if result > best_metric:
				best_metric = result
				best_metric_epoch = epoch + 1 
				torch.save(model.state_dict(), os.path.join(args.base_data_dir, 'best_metric_model.pth'))
				print('saved new best METRIC MODEL')


if __name__ == '__main__':
	main()

[vikashg]

Further, if I break down the code

from monai import metrics
for _y, _z in zip(y_pred_act, y_onehot):
        x = metrics.compute_roc_auc(_y, _z)
	auc_metric(_y, _z)
	result = auc_metric.aggregate()

auc_metric.reset()
del y_pred_act, y_onehot
metric_values.append(result)

The above code works. However, if I use the aggregate method the code breaks down.

auc_metric(y_pred_act, y_onehot)
result = auc_metric.aggregate()

[Nic-Ma]

Hi @vikashg ,

Yes, the data shape is (batch_size, num_classes):
https://github.com/Project-MONAI/MONAI/blob/dev/monai/metrics/rocauc.py#L107
Please call auc_metric(_y, _z) for every batch, then only run auc_metric.aggregate() finally to get the result.
Refer to the MeanDice example:
https://github.com/Project-MONAI/tutorials/blob/master/3d_segmentation/torch/unet_evaluation_dict.py#L99

Thanks.

[vikashg]

OK Thanks.