Just imagine the inlays arent there and route out like you normally would for the new inlays over the top. That way the old inlay material will get eaten out (no pun intended! lol)
You need to go deep enough so the inlay is flush with the top of the fingerboard. If the inlay is 3mm deep then do the hole 3mm