The biggest downside with it is that the preference system is just complex enough that people think they understand it, but don't, so FUD is easy to spread.
We had an issue two elections ago where our Greens party (a minor party) performed more poorly than they had done in the previous election, losing seats to our center left major party. This was purely a result of the way preferences flowed, and their primary vote percentage didn't change at all, but because it's non trivial to understand how that works and because the outcome isn't intuitively obvious, the narrative developed that the greens had lost relevance, despite their primary vote count not changing.
Explanation of how that played out
Background - Australia (at the time of the last two elections) had two major parties, a center right/right wing party (the Liberals) and a center left party (the Labor party). In 2022, the Liberals lost the election and the government and Labor took power with a simple majority. In 2025, the Liberals were absolutely decimated in the election and Labor won in a landslide. In 2022 and in 2025, the Greens pulled up 3rd, with 12% of the primary vote, but only got 1 seat in 2025 as opposed to 4 in 2022
Preferential voting 101 - Votes are tallied. If no one has over 50%, the candidate with lowest number of votes is ruled out. The second choice for voters voting for the losing party now come in to play and get spread around the remaining parties. If no one has over 50%, the process repeats each time moving further down peoples ranked choice list redistributing their votes, until a candidate gets more than 50% of the votes.
And now how it all played out.
In 2022, Labor won the election, but didn't destroy the Liberals. The liberals still won plenty of seats, but not enough to form government. In seats where Labor was not in the final two parties for the seat, their ranked choice preferences largely favoured the Greens over the Liberals, meaning that the Greens effectively got all of Labors votes. In a handful of instances, this boost was enough to push the Greens in front of the Liberals and give the seat to the Greens. Hence 4 seats.
In 2025, the Liberals were annihilated in the voting. People who vote for the Liberals tend to preference Labor over the Greens though, and thus, because there was a large swing against Liberals, there were a large number of seats where they weren't in the top 2 parties, and in most of these, the Greens didn't see any of the ranked choice/preferential votes. And without that boost, as a minor party, they only claimed 1 seat.
In both cases, 12% of the population put the Greens as their first choice. In 2022, that was 4 seats. In 2025, that was 1 seat.
Edit - It's worth noting, this is still a simplified summary, as there are more than 3 parties, there is a Senate as well as the House of Reps, and the way ranked choice/preferences work are different between the two. And this is also only federal elections. There are state and local government elections as well, which have their own rules.