I was told by my dealer that I could add the port console anytime. So you should be able to remove it.
You'd need a fill piece & carpet to cover the mounting area though.