To successfully add a 5 Series camera to an NVR 3 Series will require a firmware update on both the NVR and the 5 Series camera.
Simply update the devices firmware, then add the camera in the usual way.
Please ensure your NVR 5 Series is using firmware v18.104.22.16806 or higher
NVR 5 Series Firmware Downloads
Please ensure your 5 Series camera is using firmware v9.0.001.2262 or higher