A model to interpret pulse phase shifts in AMXPs: SAX J1808.4-3658 as a proof of concept