SEO BIGCATS.RU

Lately I've gotten interested in search engine optimization, partly for work, partly for myself. It's unpleasant to do something that others won't then see or find out about, unless you spam-advertise it. I don't know how to advertise (the bans in dikie_koshki and the Ukrainian Wikipedia are proof of that, even though I followed the rules as always), and I'm not going to pay for advertising either, so all that's left is optimization.

On the sites I make at work there's not much room to spread out, and they don't always require optimization anyway. But BIGCATS.RU, which I help administer, is a different matter. Though things don't always go smoothly.

In December I noticed that Google treated www.bigcats.ru, bigcats.ru and bigcats.spb.ru as different sites, ranking them at different positions for the same query. I merged the domains using .htaccess and redirected all requests to bigcats.ru, but Yandex considered www.bigcats.ru the main mirror and refused to follow the redirects, and since it had no way to access the main mirror, over time it removed the entire site from the index except for the home page. I only noticed this at the end of February. That was quickly fixed by adding exceptions for Yandex in the htaccess, and I also created a robots.txt with a host directive, so that over time (within 4 months or never) the main mirror would change in Yandex too. For now, only Yandex's robots can crawl the site without being redirected to bigcats.ru.

The Canonical URL mods were also installed on the forum to add canonical links to the forum, so that Google and other search engines wouldn't index the same forum content available at different addresses. Yandex doesn't support rel="canonical", so for it you have to suffer with exceptions in robots.txt. The Sitemap FX mod was also installed to generate the xml sitemaps of the forum. In the htaccess I had to add an extra redirect to serve /forum/sitemap.php when /forum/sitemap.xml is requested (RewriteRule ^forum/sitemap.xml$ forum/sitemap.php), so that Google wouldn't complain in robots.txt about a sitemap with a php extension. Just recently, based on the phpBB SEO Optimal titles mod, I changed the way the forum's page titles are output, so that the topic name comes at the beginning and the page numbers of multi-page topics are shown. The pages with the new titles are only now starting to come into the index, it'll be interesting to look at the results.

The site itself is harder to optimize, since the structure is illogical and outdated, with a JavaScript menu that robots can't properly crawl. I couldn't generate a sitemap automatically, I had to compile it by hand, entering almost 400 pages. After adding the sitemaps to Google and Yandex Webmaster Tools, the former loaded them fairly quickly and indexed some of the pages, while Yandex took longer to load the sitemaps and not all the links appeared in the index right away, but after a few weeks all of the over 3500 addresses from the sitemaps turned out to be indexed. And Google likewise keeps a third of the pages in its index. As I understand it, that's normal for it.

After being indexed by these two search engines, a lot more visitors come from Yandex; right now for the query "tiger" it comes out to almost 10% of visits, while for vertical search more comes from Google, especially for the query "humor". True, the bounce rate is high, around 70% overall and 75% for referrals from search engines, well, for a subject and structure like this that's normal. I'd like to draw more people to the forum, since it ends up holding more information.

I rely on the data from the Yandex and Google counters, and I've installed them on all the pages of the site and the forum. Analytics turns out better, but personally I prefer the interface. I like Metrica, it's more convenient, has interesting parameters and updates constantly throughout the day. For both counters I set the goal "Post on the forum" and track the statistics of visitors who leave posts.

I removed the rest of the counters, which were only on the home page, they were no use, they only added to the page weight. There were no referrals, for example, from Rambler's top list, nor from the search engine itself. I also placed link-buttons to the forum on the home page instead of the dead chat, RSS and the site's presence on social networks, hiding the addresses by means of the forum redirect, thereby reducing the number of external links. On the home page there's still room for two banners of other sites, if only I could find someone with whom a productive exchange could be arranged. Few come over from "Tigromania", but they stay for a long time, quality visitors. For the "Phoenix" fund it's just a link without an exchange, as I understand it.

I still haven't finished robots.txt, I just can't manage to close off all the pages from Yandex that it shouldn't index on the forum, since re-indexing makes the rankings drop badly, I made sure of this with one of the sites at work. Right now, it comes out as follows:

User-agent: Yandex
Disallow: /forum/faq.php
Disallow: /forum/mcp.php
Disallow: /forum/posting.php
Disallow: /forum/report.php
Disallow: /forum/search.php
Disallow: /forum/style.php
Disallow: /forum/ucp.php
Disallow: /forum/viewtopic.php?p=*
Disallow: /forum/viewtopic.php?t=*
Disallow: /forum/viewtopic.php?f=*&t=*&start=0$
Disallow: /forum/viewtopic.php?f=*&t=*&start=*&st=0&sk=t&sd=a
Disallow: /forum/viewtopic.php?f=*&t=*&st=0&sk=t&sd=a&start=*
Disallow: /forum/viewforum.php?f=*&start=0$
Disallow: /forum/viewtopic.php?f=*&start=*&st=0&sk=t&sd=a
Disallow: /forum/viewtopic.php?f=*&st=0&sk=t&sd=a&start=*
Disallow: /forum/viewtopic.php?f=*&t=*&view=next
Disallow: /forum/viewtopic.php?f=*&t=*&view=next&st=0&sk=t&sd=a
Disallow: /forum/viewtopic.php?f=*&t=*&view=previous
Disallow: /forum/viewtopic.php?f=*&t=*&view=previous&st=0&sk=t&sd=a
Disallow: /cgi_scripts/search/search.pl?query=*
Host: bigcats.ru
Sitemap: http://bigcats.ru/sitemap.xml
Sitemap: http://bigcats.ru/forum/sitemap.xml
Clean-param: sid /forum/index.php
Clean-param: sid&st&sk&sd /forum/viewforum.php
Clean-param: sid&st&sk&sd /forum/viewtopic.php

I can't figure out where Yandex gets the pages with "&st=0&sk=t&sd=a" in the address from, I can't reach such pages myself. I didn't dig deep into the forum scripts, the parameters are needed for sorting and filtering, as I understand it. But the problem remains, I haven't seen such pages in Google's index, while in Yandex the same pages end up indexed twice. At the same time, adding

Clean-param: sid&st&sk&sd /forum/viewforum.php
Clean-param: sid&st&sk&sd /forum/viewtopic.php

affected pages of the form

/forum/viewtopic.php?p=*&st=0&sk=t&sd=a
/forum/viewtopic.php?t=*&st=0&sk=t&sd=a

They disappeared from the index, but right away a bunch of the following ones got added:

/forum/viewtopic.php?f=*&t=*&start=*&st=0&sk=t&sd=a
/forum/viewtopic.php?f=*&t=*&st=0&sk=t&sd=a&start=*
/forum/viewtopic.php?f=*&start=*&st=0&sk=t&sd=a
/forum/viewtopic.php?f=*&st=0&sk=t&sd=a&start=*

I only just now disallowed them in robots.txt.

And still, what exactly are "&st=0&sk=t&sd=a" needed for, where exactly on a phpBB forum do links with these parameters come from, and what's the most effective way to fight their indexing?