I would smooth it down like you propose. The pad is already displaced slightly towards the hub anyways, might as well make certain it has its full range of motion in order to prevent any hang ups & uneven wear. There would be plenty of material left so you would not be weakening anything.