By Adam Jourdan

Alibaba's dominance of online retail in China faces its biggest ever challenge as the firm founded by Jack Ma in an apartment 15 years ago lines up a U.S. initial public offering that could value the firm at around $140 billion.

In a rare blip, Alibaba Group Holdings lost market share last year while its nearest rivals all grew, according to Euromonitor. The market research firm sees China's internet retail market tripling from 2012 to over $300 billion in 2018 as the country's smartphone-savvy shoppers buy everything from plane tickets to sneakers online.

China's biggest social media company, Tencent Holdings Ltd, is leading the revolt, linking the country's most popular messaging app, WeChat, with the number two e-commerce player, JD.com.

An array of smaller rivals is also clawing away at Alibaba's lead, while household retail names like Nike Inc and Gap Inc are increasingly striking out away from the giant's Tmall electronic platform to set up more distinctive online stores of their own.

"In China shopping is a social activity. You want to tell friends about it, recommend it - it's a smartphone activity, and whoever owns that organisational ability also has a hold over how a person shops," said Frank Lavin, Hong Kong-based chief executive of Export Now. Lavin's company helps global firms set up shop in China through Alibaba's Tmall.

Alibaba's e-commerce prospects at home loom large after Ma's firm said on Sunday it was starting plans for a long-awaited listing in the U.S. - potentially the biggest ever IPO by an Internet company - which could surpass the $16 billion raised by social media giant Facebook Inc in 2012.

Alibaba still held a sturdy 45.1 percent of China's e-commerce market last year, down from 46.1 percent a year earlier, according to Euromonitor, and remains bullish in the face of Tencent, JD.com and others. It is beefing up its mobile services to keep up with China's legions of smartphone users.

Alibaba did not respond to repeated requests for comment for this story, though the firm's executive vice-chairman Joe Tsai was upbeat about the firm's e-commerce prospects in an interview with Reuters in Hong Kong last week.

Peeling away

Alibaba is now battling rivals on multiple fronts. Alongside JD.com, heading for a $1.5 billion IPO of its own in the U.S., are well-funded vehicles like household appliance retailer Suning Commerce Group Co Ltd and Wal-Mart Stores Inc's grocery retailer Yihaodian.

Smaller niche players like cosmetics specialist Vipshop Holdings Ltd are also growing in stature. And with global and local brands peeling away from Tmall, the trend is likely to see Alibaba's market share extend its fall, said Bryan Wang, Beijing-based vice-president for Forrester Research.

"We have definitely seen a lot more customers asking us in the last year about how to get away from Tmall," said Wang.

Popular Internet clothing retailer HSTYLE has partly flown the nest. Competing with brands like H&M and Uniqlo, it has branched out from just having a Tmall outlet and now books half its sales through its own site and on JD.com and Tencent.

"As a mature Internet brand we're looking to provide more individual service to our shoppers," Zhao Yingguang, founder and chairman of HSTYLE, told Reuters in an interview, describing his brand as one of the leading women's apparel retailers on Alibaba's platforms. "We go and sell our products where the consumers are."

Rivals rising

Alibaba's vast resources have helped it see off weaker players so far such as Otto Group, 139shop.com, Mecox Lane, Newegg.com and others, but the remaining contenders are more seasoned in competition with Alibaba - and ambitious.

JD.com still lags some way behind Alibaba in second place with a 14 percent market share last year, up fractionally from a year earlier. But its IPO plans and the deal with Tencent - a less well-known name outside China than Alibaba, but worth almost $150 billion by market value - will give it new financial and operational resources.

Tencent, meanwhile, hopes its JD.com tie-up will help it to extend its presence in "the fast-growing physical goods e-commerce market", Tencent president Martin Lau said in a statement. The deal also arms JD.com with the 225 million monthly active users of the WeChat messaging service in China.

As well as technology leading change, consumers themselves are developing new habits, becoming more picky and looking to get more for their money.

"I am leaning toward specialist stores now, because the service is often better than the giant retailers and the delivery I always find is faster," said Grace Lin, 20, a student in Shanghai. "It's not necessarily that I use Tmall less now, but I do use other stores more."

Lin often shops on cosmetics specialist Vipshop's online site. The firm saw revenue climb 145 percent in 2013, while customer numbers shot up 130 percent over the same period, according to an earnings conference call this month. Vipshop almost doubled its market share to 1.8 percent last year.

Suning, a traditional electronics retailer with a strong bricks and mortar presence, is bulking up its online presence, as is rival Bain Capital-backed Gome Electrical Appliances Holding Ltd. Both raised their market share last year.

Yihaodian leverages majority owner Wal-Mart's global name and product range, while other niche players such as Jumei and Yesmywine.com corner specific markets in cosmetics and alcohol.

Ma v Ma

Alibaba's Jack Ma has locked horns repeatedly of late with his rival at Tencent, co-founder Pony Ma - a namesake but unrelated. Rhetoric from the two internet giants has grown increasingly barbed as competition has raged from gaming and microblogs, to taxi hire app price wars and online payment systems that are now the subject of close scrutiny by China's central bank.

When Tencent launched a "red envelope" feature for WeChat, which let users send money gifts by smartphone over the Chinese New Year, Jack Ma wrote it was a "Pearl Harbour attack" on his company's Alipay payment system, referring to the unexpected bombing of the American port during World War II. Tencent said more than 8 million people used the WeChat feature.

Alibaba-invested Weibo Corp, a popular Twitter-like social media platform, filed for a $500 million U.S. IPO of its own on Friday, as it looks to defend its position against fierce competition from Tencent's WeChat.

WeChat does indeed present a threat to Alibaba. In a research report in December, Standard Chartered said its payment function, known in Chinese as Weixin, paired with its wide popularity, could be a "killer weapon" for Tencent to take market share in mobile e-commerce.

China has some of the most active mobile web users in the world. The country's mobile Internet users hit 500 million in 2013, and is estimated to hit 750 million by 2017, according to data from China-based consultancy iResearch.

"The wind is blowing against Alibaba and the biggest risk is coming from the mobile sector," said Zhang Chenhao, executive director at e-commmerce-focused investment advisory firm Gold Sand Capital.

"Firms are all lining up - the school of Tencent or the school of Alibaba - and the camps are forming," said Zhang.