I have upgraded from Note 4 to Note 5 and I would say it is a worthy upgrade.
The processor, Ram, Graphics, Internal Memory , Camera are all best. I think from last one year Samsung has really changed the game by making there phones more premium and improving the overall build quality.
I have used almost all the note devices and Note 5 has the same feel but with updated features, I am very pleased with the performance of the device.