Yes, you will want to use debug.SetPanicOnFault to convert faults at an unexpected (non-nil) address into panics from which you can recover. From the docs:
SetPanicOnFault controls the runtime's behavior when a program faults at an unexpected (non-nil) address. Such faults are typically caused by bugs such as runtime memory corruption, so the default response is to crash the program. Programs working with memory-mapped files or unsafe manipulation of memory may cause faults at non-nil addresses in less dramatic situations; SetPanicOnFault allows such programs to request that the runtime trigger only a panic, not a crash. SetPanicOnFault applies only to the current goroutine. It returns the previous setting.
For the localization of the impact, note that SetPanicOnFault is set at the goroutine level, so a single goroutine can deal with known unsafe access.