Nokia, while the world’s largest handset maker, faces an uphill battle to capture smartphone market share – particularly in the U.S. – thanks largely to the progress of the LiMo (Linux Mobile) Foundation and the Google-led Open Handset Alliance (OHA), according to a report released today by ABI Research.

The report, titled “Smartphone and OS markets,” found that after a recent slew of announcements from LiMo and the OHA, the growing support of the Linux platform – as well as support for Google’s operating system Android – is proof that Linux is not a passing fad. By 2013 Linux will take 23% of the smartphone market and will be the second most prevalent solution behind Symbian, according to ABI Research vice president and research director Stuart Carlaw.

“I think we have been pretty foolish on Linux in general because it does offer some unique benefits to vendors in this market,” he said. “The real surprise was that the market is consolidating as quickly as it is. There are a number of different standards around that are going to be available – Open MoCa, probably there for a niche, gadget guys, MIMO and anything QT developed through Nokia – but we really think as far as the commercial availability, it is going to come down to those two, the Open Handset Alliance, Android, and the LiMo foundation products.”

The two organizations operate as separate entities with similar goals, with no plans of merging as some operators have hoped. This might not be a negative thing, however, as the two fill different niches in the market, Carlaw said. The LiMo foundation will be very useful in penetrating the low-end, mid-tier market and branded products, while Android-based products are aimed at more high-end users.

“The OHA is very Google-dominated,” Carlaw said. “The LiMo foundation is broader in its membership and is more carrier driven with some significant OEM support. There are very different agendas from both parties there. There is room in the market for multiple solutions anyway. It would be almost counterproductive for the two to merge.”

The LiMo foundation recently got Verizon on board, while Android has been busy signing up numerous global application developers to code apps for the platform. At Mobile World Congress this year, the LiMo Foundation announced 15 Linux-powered devices from Motorola, NEC, Panasonic and Samsung set to begin shipping commercially by the end of the year. Meanwhile, Nokia remained steadfast in its commitment to its Symbian OS.

The report found that Nokia’s poor position in the United States resulted in a 2007 share of only 4% for Symbian in the American smartphone market. Carlaw attributed this to the fact that Nokia has traditionally struggled with clamshell handsets, which are popular in the U.S., choosing instead to focus on the candy-bar form factor – not as well accepted in this market. To pose a real threat to Linux, Nokia will have to reexamine its entire product portfolio.

“Everything is about delivering a broader portfolio to get there,” Carlaw said. “If they just stuck with Symbian, they are not going to get in the market as quickly as they can. If they took a look at their competitive books and were a bit more open in their platform architecture and software adoption, they will have more to gain than they do to lose…I would be surprised if they don’t have a handset based on the Linux OS in the next three to four years if not well before then.”

If they don’t make this move, Nokia risks being pushed out of the huge smartphone market in the U.S. According to research firm IDC, sales of smartphones – defined as those that include Web browsers, email and video players – more than doubled in the U.S. last quarter to 6.7 million units from Q1 last year. IDC predicts unit shipments in the country will increase 85% this year. Furthermore, if Nokia does fail in this, it could also lose its leadership position in the European markets as well, the ABI report concluded. There it is also challenged by a combination of resurgent Windows and emerging Linux platforms.

“Two years ago, it was such a fragmented market that it could have gone any way. Now, the jury is most definitely in and the day is going to be here, and it is going to be in general based on those two standards with other players in part of it, but not as wide and dominant as they are,” Carlaw said, adding that there will be opportunities for solutions such as Maemo, facilitated by the burgeoning Mobile Internet Device market.