Rawr, phpbb2->phpbb3 conversion SUCKS.

I've been trying to convert a phpBB2 forum to a phpBB3 forum.

One of the biggest problems I keep running into is that posts get lost at a certain point. I'm not entirely sure why, or how.

But the issue I'm going to address here is that upgrading a LARGE forum database (ours is at 3 GB, with around 4 million posts) was really slow, especially at step 16 of 28, filling the phpbb_posts table.

Why? To ensure compatibility with other converters, other databases, different designs, the phpBB developers chose to use a SELECT query with LIMIT x, y. Where x = number of rows at which you want to start querying, and y = number of rows to return.

When x gets really big, the speed at which queries will come back will get really slow, in the tens of seconds range.

So instead, I hacked install_convert.php so that when it starts processing phpbb_posts, it'll disregard the skip_rows parameter (which establishes 'x') and instead set x to 0, while adding to the WHERE part of the query a condition stating that we only want to select posts with post_id greater than the largest post_id we have in our new phpBB3 phpbb_posts table.

Here's a diff.
( Read more... )

Mind, I'm still running this right now, so I don't know if it definitely works*, but estimated runtime for my posts table is down to 4 hours and it's holding a steady 330/s row processing rate (as opposed to starting at 330/s and dropping down to less than 100/s row and taking over 14 hours).

*as in, there may be errors down the road?

[edit] As far as I could tell, this worked perfectly.
So if the issue you're experiencing with upgrading phpBB 2.0 to phpBB 3.0 is due to the fact that the upgrade process is too slow at step 17 of 28 (phpbb_posts table), this patch should be safe to use. It will decrease processing time by a significant amount: I think processing step 17 was cut down from 14-16 hours down to a bit over 4 hours.

Whee, crazy exercise.

Saturday:
Biked around in the mornings to assorted garage sales - picked up a fog machine with fog juice for $10.

Then later in the day, I biked to Goodwill, then Walmart / Target / Daiso, then kept going, passing Mountain View and then reaching a Lucky's supermarket on 715 El Camino at which point I thought I reached Sunnyvale (and somehow missed Hankook - the plan was to see how long it would take to bike to my local asian supermarket (like, it can match Safeway in size).)

Looking at google maps though, I was only about halfway between Mountain View and Sunnyvale. :( And said asian supermarket was a good distance away (Lawrence Caltrain station is the closest)

Total: over 16 miles.

Sunday:
Biked to the local Costco and ate samples for a late lunch. :) Yay for getting in without a membership pass. About 4.4 miles one way, 9 miles total.

Also, prices at Costco are higher than Safeway sales. WTF?

On the way back, I hit Safeway on Middlefield and picked up 88¢/lb grapes (w00t!), one avocado (67¢/each), a lime, and a lemon. (The latter two are supposed to be for the water container I'll eventually acquire.)