Maine’s voting system was thrown into potential turmoil on Tuesday by a court opinion that said a process in which voters rank candidates rather than simply picking one — approved by voters in November — violated the State Constitution.

The ranked-choice system allows for winning candidates to be elected by more than 50 percent of voters. But Maine’s Constitution says winners are to be determined by a plurality, in which the winner receives the most votes whether or not that’s a majority of voters.

The opinion is advisory and not a matter of law, but it is significant because Maine is the only state in the nation to have adopted ranked-choice voting. Other states have been watching closely.

After Maine residents approved the ranking method for voting, Republicans and others unhappy with the change asked the Maine Supreme Judicial Court, the state’s highest, for guidance on whether the new system conflicted with the Constitution. The court said unanimously on Tuesday that it did.

The court said the State Legislature had two options: Repeal the voter-approved measure and go back to the plurality system, or amend the Constitution to provide for winners to be chosen by a majority.

If no action is taken, the ranked-choice method will be put in use next year. But, the court warned, results are likely to be challenged, setting the stage for repeat elections, which could cause chaos and governmental instability.

“The potential for a constitutional challenge to those election results and ensuing upheaval is real,” the court said. It called the situation “serious and unusual.”

Last year, The New York Times examined Maine’s decision to adopt ranked-choice voting. Here is part of that Q. and A.:

How does ranked-choice voting work?

Instead of casting a ballot for a single candidate, the voter ranks all of the candidates by preference. So if there are four choices, the voter is asked to rank them one through four.

If no one wins a majority on the first round, the candidate with the fewest votes is eliminated. For voters, that means if the eliminated candidate was your first choice, then your second-choice vote will be applied in the next round of counting.

If your second choice is eliminated, your vote for third choice will be applied — and so on until someone wins a majority.

What are the advantages?

Proponents say the voting method ensures that whoever is elected has the support of a majority of voters. They say this helps increase civility because candidates need to appeal to a spectrum broader than just their base to win over their opponents’ supporters on subsequent ballots. And theoretically it eliminates the possibility of a “spoiler” candidate winning.

What are the disadvantages?

Opponents argue that because it is more complex, ranked-choice voting depresses turnout and leads to more errors in part because it can be confusing. In a rural state like Maine, where half of the communities count ballots by hand, ranked voting could also lead to more errors by the people doing the counting. Opponents argue further that the voting method can still skew the results toward someone not favored by the majority, and it can be costly.

Why did Maine voters approve ranked-choice voting?

It was close — 52 percent in favor to 48 percent against. But a slim majority seemed to want change, and ranked-choice voting had been successful in two mayoral races in Portland. Last year’s measure was put into motion in 2014, but the campaign coincided with rising voter anger over Maine’s governor, Paul R. LePage, a Republican, who has repeatedly embarrassed the state with contentious comments. Mr. LePage was twice elected in three-way races with less than a majority of the vote, which may have given voters an added incentive to approve this measure. The state’s Democratic Party supported it, though ranked-choice voting does not favor one party or another, and it earned the support of the League of Women Voters of Maine. One major concern was that ranked voting violates the state’s Constitution, which calls for elections of the governor and Legislature by a plurality, not a majority, but a constitutional amendment could resolve that.

How much will it cost?

In Maine, the secretary of state’s office said it would need $761,000 in the 2018 fiscal year and $641,000 in the 2019 fiscal year to print additional ballot pages and update ballot machines. The Department of Public Safety said it would need a general fund appropriation of $76,000 and a highway fund allocation of $73,000 in the 2018 and 2019 fiscal years for overtime and fuel to retrieve, secure and return ballots for central counting in Augusta, the state capital.

How has it worked elsewhere?

Ranked-choice voting is used in Ireland and Australia in national elections, and it is used to pick the Oscar nominees for best picture. But only 11 American cities, all of them liberal, have instituted it, so their experience may be too limited to draw firm conclusions.

Here in the United States, it generally appears to be working. But there is some question about whether it decreases turnout. Some studies say it has; others say that when compared with the current system in most cities, it has not. A study of the 2014 mayoral primary in Minneapolis found that voters who were more affluent and white turned out at a higher rate and completed their ballots more accurately than minorities and those in low-income areas.

Howard Dean, a Democrat and former governor of Vermont, favors ranked voting, saying that without a majority vote, “you can’t hold the powerful accountable.” But Gov. Jerry Brown of California, also a Democrat, vetoed a bill in September that would have expanded it in a number of jurisdictions. He called ranked voting “overly complicated and confusing.”

Ranked voting has certainly led to some unexpected results. In the mayor’s race in Oakland, Calif., in 2010, Jean Quan, who came in second in the initial round, had strategically campaigned to be everyone’s second or third choice. In subsequent rounds, she surged to victory on the strength of those second- and third-place votes after other candidates were eliminated. Supporters of the candidate who had come in first in the initial round accused Ms. Quan of gaming the system.

Why have only cities adopted this method until now?

Ranked-choice voting has been easier in cities because they have uniform voting equipment and are geographically compact, reducing issues involving the central counting of ballots.

States have multiple jurisdictions with different kinds of voting equipment and are geographically spread out, hampering central counts. A few states have considered ranked-choice voting but balked in part because of the cost and the difficulties of putting it in place. “This nerdy inside-baseball issue of election administration is the biggest barrier, after people get interested,” said Rob Richie, the executive director of FairVote, a nonprofit that favors ranked-choice voting.

Could ranked-choice voting be used in presidential elections?

Proponents believe that more states will start adopting ranked-choice voting and will eventually allow it for presidential elections. They say that if it had been in place during last year’s primaries, Donald J. Trump would not have won the Republican nomination; he won 13.3 million votes in the primary races, but 16 million were cast against him.

“There’s no doubt Trump would have been stopped in the primaries” if the early states had used ranked-choice voting, said Richard Winger, the editor and publisher of Ballot Access News, which tracks election issues and supports ranked-choice voting. “And obviously Al Gore would have been president if Florida had used it in 2000.”