Did you remove the track tension. Is it loose enough?
Get the rear in the air. Hold a 2X4 on the end of the threaded shaft side where you took the nut off and tap the board with a 4 pound hammer. It should come right off even if you just pull on the right side wheel. All your collars and adjusters will fall right out.
It should come right off. There are collars in the holding everything tight when you put the nut on.