This article presents AlterVerse, a framework that provides users with novel experiences through visual illusions of interactively altering physical objects in extended reality environments. In the AlterVerse framework, physical objects are seamlessly virtualized by aligning their 3-D virtual replicas with them. It enables users to manipulate (move or rotate) and reshape (deform the shapes or transform the styles of) the physical objects virtually. Simultaneously, the physical objects are visually removed from a physical environment, creating the perception that the altering occurs directly on the physical objects. As a promising application of the AlterVerse framework, the article demonstrates interactive virtual altering of home furniture in interior design or decoration scenarios without physically altering existing furniture.