How I Migrated a Portal from Joomla 1 to 3
This article describes how I migrated a big website, with over 10000 articles written in Cyrillic and using special characters, from Joomla 1.x to Joomla 3.x.
Task: To migrate the web site preminportal.com.mk from Joomla 1.x to Joomla 3.x. The site was on Joomla 1.0.13, with a 320 MB database and a root folder of 3.5 GB.
In more detail, the database contained:
- jos_content = 15493 rows;
- jos_menu = 131 rows;
- jos_modules = 457 rows;
- jos_sections = 23 rows;
- jos_categories = 127 rows;
- jos_users = 402 rows.
To avoid a big delay of the live site, my idea was to do the whole migration locally on my computer and then just move it to the site hosting. Folders with images, music and other documents I didn't transfer locally, because I decided not to change their location (ex. from images/stories to images/) and avoid massive changes of paths in all articles.
For local work I used USBWebServer. For making it possible to import the database I had to increase memory_limit, upload_max_filesize and post_max_size statements in php.ini to values bigger then 400MB.
In USBWebServer root I created a folder called 'oldsite', and in it I extracted folders from the old preminportal.com.mk site (only the core Joomla 1.0.x folders). On USBWebServer I created a database named 'oldsite', and of course I set the database collation to utf8-general-ci because the site is on Cyrillic (the database name of 'oldsite' is just an example and in your case you need to use the name of your real case database).
In configuration.php I changed these lines:
- $mosConfig_db = 'oldsite';
- $mosConfig_absolute_path = 'C:/USBWebserver-za-Premin\root' (path needs to be the real location of your USBWebsrever);
- $mosConfig_live_site = 'C:/USBWebserver-za-Premin\root' (path needs to be the real location of your USBWebsrever);
- $mosConfig_error_reporting = '0' (only if value is other than zero, to avoid displaying errors);
- $mosConfig_password = 'usbw' (standard password for USBWebserver, if you changed it then use yours);
- $mosConfig_user = 'root' (standard user name for USBWebserver, if you changed it then use yours);
- $mosConfig_live_site = 'http://localhost:8080' (standard url for USBWebserver root folder, and is needed for all parts of the site to load correctly).
For local work with Joomla 1.0.x, I used USBWebSerever version 8, which comes with PHP 5.3.2. This is an older version, but it was necessary in this case, because it has magic_quotes_gpc and they need to be set to On in the php.ini file so the site can be opened. In newer versions of PHP these statements are not used for security reasons.
Now the site is completely replicated locally (without pictures in the articles, but in the end, when the site goes on the server, they will find their paths) and the migration can begin.
I followed instructions from http://docs.joomla.org/J1.5:Migrating_from_1.0.x_to_1.5_Stable (yes, first the site must be migrated to Joomla 1.5.x, and after that to the newest version). Because the site is on Joomla 1.0.13, I downloaded the recommended patch from http://forum.joomla.org/download/file.php?id=36573.
To save you some time: after migrating I found the database had doubled in size, so I had to do the migration over again. In the forums found that the problem can be from articles that are in Trash, so I opened the administrator in the Joomla 1.0 version and found over 700 articles and also 20 menus in Trash, so I emptied it. Also, before migration, remove all unpublished articles, categories and sections from the old site. After doing that I continued the migration process.
Because Joomla 1.0 doesn't have article ordering by state (published/unpublished), on such a big database it was difficult to find all unpublished items in an easy way. From there came the idea to modify some code.
In file ./administrator/components/com_content/admin_content.php line 145 is the line which points to the display of state, and it says to 'display all equal or biger than zero':
"c.state >= 0"
where 0 is unpublished and 1 is published, so a little change to:
"c.state = 0"
and now I have only unpublished articles displayed, so I can select all of them and put the in trash.
There were 830 unpublished articles, and nearly 100 of them were featured, so maybe that also contributing to the 'duplicate' problem. After cleaning all unnecessary articles, I changed the line in admin_content.php back to the original, and now the Article Manager displays 11296 articles, 3560 of them are featured and 2572 are archived.
I also trashed all unpublished categories and sections, menus and sub-menus. There were also 2 sections without articles and one menu without items, so I trashed them as well. I trashed all unpublished modules (except core ones like login that cannnot be trashed).
All third party mambots, modules and components were also uninstalled or sent to trash, and their related tables in the database. All disabled users were also deleted, and all unused templates.
Then in the Trash manager I emptied the trash for all instances. Then I purged the cache. Next, in PHPMyAdmin I selected all tables which had some overhead and optimized them. After that the overhead was 0 for all tables.
Next step, install component Migrator, which migrates the database from Joomla 1.0 to 1.5. This component creates one sql file and that file needs to be imported into a clean instalation of Joomla 1.5.
Now I put the installation files for Joomla 1.5.26 in a file called 'oldsite15, and create a new database named 'oldsite15' with utf8-general-ci collation.
I tried and tried to import the migration file during the 1.5.26 installation. The default offered encoding ISO-8859-1 would not work. I tried using only utf-8 encoding, but always got the following error:
Notice: iconv() [function.iconv]: Detected an illegal character in input string in ...\installation\installer\helper.php on line 873
Line 873 in that helper.php file is:
$buffer = iconv( $srcEncoding, 'utf-8//TRANSLIT', $buffer );
After a little investigation in the forums about how iconv functions and which parameters this function uses, I decided to try with:
$buffer = iconv( $srcEncoding, 'utf-8//IGNORE', $buffer );
I did this because of the first error displayed in the notifications:
Error at the line 9: INSERT INTO jos_migration_backlinks
Query:ï»¿## Created using Migrator 1.3 for Joomla! 1.0
INSERT INTO jos_migration_backlinks VALUES(1,"Ð�Ð¾Ð¼Ð°","index.php?option=com_frontpage&Itemid=1","component/option,com_frontpage/Itemid,1/","");
where in the first row Joomla read the word 'Дома' normally, but after that it changed to strange characters 'Ð�Ð¾Ð¼Ð°', so I guest maybe transliteration is problem.
Now the migration passe with only two minor notifications, and at the end it showed a message for a successful migration. Finally I coould view the site in Joomla 1.5 with normal Cyrillic and all text readable.
In the 1.5 version I installed Akeeba backup 3.4.6 and made a back-up at this point.
Now, on the newest USBWebServer 8.5 version, I'll try to install a new Joomla 3.3.6 and try to migrate the site from 1.5.26. To make this posiblle you need to have both 1.5.26 and 3.3.6 databases on the same server, so here I'm going to install the 1.5.26 version from the backup.
I create a new folder called 'newsite3' and a database called 'newsite3' with utf8-general-ci collation for Joomla 3.3.6, and a database called 'oldsite15' with utf8-general-ci for Joomla 1.5.26.
In a clean Joomla 3.3.6 insatllation I run the redMigrator component. In their options I select to only migrate the oldsite15 database. After 23% redMigrator stalled, so I tried several times with some changes in the options, but the same thing kept happening. I tried increasing the memory_limit after which redMigrator managed to carry over 3300 of 11296 articles before getting stuck again.
Next I tried a component called MigrateMe, which imports a 1.5 database and converts it to version 3. MigrateMe imported the database file and started analyzing but then got stuck.
I heard that SPupgrade worked well, so I gave it a try next, always starting with a fresh Joomla 3 installation.
After installing SPUpgrade, I left all of the settings as default. I connected it with the 'oldsite15' database, and it ran for a little more than two hours. During this time, on Firefox, the only visible sign it was working was a rotating icon in name bar. From time to time in PHPMyAdmin I checked the state of the database, and there I saw changes in row numbers and that was a good sign. After more than two hours site logged itself out and presented the login page.
I logged in and saw a status report from SPUpgrade for what happened, eventual changes and at the bottom the message: 'Successful migration'. There were all the menus, categories, modules and articles. Note that old Joomla 1.0 and 1.5 had sections and categories, whereas in Joomla 3 sections = categories, and categories = subcategories.
Now in the database, the table ucm_history, which holds article versions, was 316 MB. I emptied it, because articles in Joomla 1.0 don't have versions. Versioning was implemented in Joomla 3.
Articles in Joomla 1.0 don't have friendly URL's so I decide to correct that: all articles will have friendly URL's. Note that in this case all old links from other sites to your old unfriendly URL will be lost or need to be changed.
First, because all article titles were in Macedonian Cyrillic, I needed to install the Macedonian language and activate it for the admin part of site. This gave Joomla the ability to automatically transliterate Macedonian Cyrillic titles into Latin characters. For that, you need to open an article to edit it, delete the old alias and save the article. In the process of saving, Joomla will auto-generate a new good transliterated alias. But in my case there are too many articles so I needed faster solution.
On the internet I found the suggestion to use the Batch button. In the database in table xyz_content, all aliases are replaced by empty/nothing. Then create a new Temporary category and remove the selected articles with the Batch button in this new category. When Joomla transfers these articles, it saves each new article which triggers the auto-generation of the alias. After this just transfer the articles back to their original category again with the Batch button. Of course, to avoid messy articles, this process needs to be done category by category. Note, the described process does not work for menus and categories aliases - they need to be opened and saved one by one.
I made new aliases for menus and categories. Now for the articles. First for all articles, via PHPMyAdmin in table xyz _content I changed their alias:
UPDATE `xyz_content` SET `alias`=''
and followed the above suggestions. The proces works but is very slow. To prevent versions of articles I disabled versioning. In the Temporary category the Batch process is very slow, one article at a time. But returning form the Temporary to the original category is fast. I concluded that this way will take too much time, so I decided to find a solution directly in the database.
I asked on official Joomla forum: http://forum.joomla.org/viewtopic.php?f=710&t=868696&sid=e10730d81ad7cbb61c2ed875d857e126. Manicakm suggested I use:
SELECT `title`,replace(replace(lower (`title`),' ','-'),"'",'') FROM `xyz_content`
and to check to see if that type of alias is OK for my purpoase. Checking showed that all new aliases would be in Cyrillic.
After that with the command:
SET alias=replace(replace(lower (`title`),' ','-'),"'",'')
I had all aliases almost friendly, but in Cyrillic. This was not bad, but could I go a step further and make all aliases transliterated in Latin?
pe7er suggested to check with:
SELECT alias, COUNT(*) c FROM xyz_content GROUP BY alias HAVING c > 1
and this command showed that many of the aliases had unfriendly characters like +,.„“()!?"’è”‘ћђ✣✤ etc.
Now these characters needed to be replaced with '-' or empty/nothing according to their position and usage in the title.
I did this with the command:
SET alias = REPLACE(alias,'+','')
SET alias = REPLACE(alias,'✤','-')
After replacing all problematic characters I used the same method to replace all Cyrillic characters in aliases with their Latin transliteration. Finally I had good Latin transliterated friendly URL's.
Now I copied all images, music. Flash files, pdfs, presentations and other documents from the old site according to the paths in the root folder of new Joomla 3.3.6 site. I installed a new template and customized it, installed the latest Akeeba back-up, restored the site on the live server, and the site is fully functional.