Anti-circumvention laws already exist and carry penalties including prison, it only takes one judge to create case law that this kind of reverse engineering is circumvention and then it'll disappear from the commercial Internet.

The safest way to go is to create the software from scratch, using reversed engineered proprietary software to make a different, but very similar, product is a legal minefield and largely unnecessary given that the same AI time that goes into reverse engineering the project could be directed towards making a clean room replacement which could completely avoid those legal issues while still replacing the product with a FOSS product.