your right the bearing cant move however when the bearing gets rough or the set screw comes loose the jackshaft will start to slip and chew itself up like that. With it being worn that much, I would for sure suggest replacing the top bearing in the chaincase as well, because the shaft would have been moving and stressing/ruining that bearing as well. replace the seal in the backside on chaincase too.
to get the jackshaft out without drilling out 100 rivets, you can remove the long phillips head screw from inside the hood hold down nipple on bellypan, then it will flex out and down slightly enough to squeek out the jackshaft.