Add numpy fix for MPS on PyTorch 1.12.1 (16b4509f) · Commits · github_fork / Stable Diffusion Webui

When saving training results with torch.save(), an exception is thrown: "RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead." So for MPS, check if Tensor.requires_grad and detach() if necessary.

modules/devices.py

+9 −0

Original line number	Diff line number	Diff line
		@@ -125,7 +125,16 @@ def layer_norm_fix(args, *kwargs):
		return orig_layer_norm(args, *kwargs)


		# MPS workaround for https://github.com/pytorch/pytorch/issues/90532
		orig_tensor_numpy = torch.Tensor.numpy
		def numpy_fix(self, args, *kwargs):
		if self.requires_grad:
		self = self.detach()
		return orig_tensor_numpy(self, args, *kwargs)


		# PyTorch 1.13 doesn't need these fixes but unfortunately is slower and has regressions that prevent training from working
		if has_mps() and version.parse(torch.__version__) < version.parse("1.13"):
		torch.Tensor.to = tensor_to_fix
		torch.nn.functional.layer_norm = layer_norm_fix
		torch.Tensor.numpy = numpy_fix

Admin message