Welcome to the new BlahBlahCafe!

Having trouble registering? You can contact us at the "Contact us" link at the bottom of the page.

The current state of the forum


Post Posted Sat Sep 05, 2020 8:40 pm
Dr_Jones


User avatar
Posts: 3724
Location: Leiden, Netherlands
Likes given: 193
Likes received  : 402

The story so far.

There's currently three problems with the forum.
1. It is running on a PHP5.x server with a altered version of the phpbb2-forum software called Przemo
2. Somewhere on august 25, there was a crash that caused the forum to stop working. As a result, two tables are completely gone from the forum database. I'll explain this later.
3. The posts-counter has been reset to 0. I will also explain this later, because it has a correlation with 2.

Ok, how to solve these problems?

1. Running the Przemo software on php7 is not possible. Upgrading from phpbb2 -> phpbb3 seems like the only possible solution, phpbb3 is running on php7.
phpbb3 offers an option to import your phpbb2 forum data to phpbb3. I've tested this and as a first result, it seems to be working okay. Of course, there's the usual UTF-8/unicode problems (diacretes, emoji's, etc), but I think that is something I can overcome. Logging in works with my zoolook.nl account, so user data works, and I can see my private messages too. Jay!

2. The missing data. This is the real problem. All topic data is gone and all post data is gone. However, and that's a bit of good news, all the post text still exists.

What does that mean?
Well, if you look at the board hierarchical, you have the following structure

Forums -> topics -> posts -> posts text

This is how it is modeled in the database too.

The forums are News, Oxygene, Articles, etc. Within these forums there are the topics, with in the topics there are the posts.

You can see the forums just fine now, but all the topics are gone. The topic table contains (amongst other things) the topic title and a link to which forum it belongs to.
The posts table is similar, it has a reference to the topic it belongs to and a reference to the forum it belongs to. A bit of redundancy there. It has also a user-id of the one who posted, a datetime value, and many other things. But the posts table does not contain the posts text, that is stored in a different table.
This means there are a bit more than 200,000 orphaned posts. And judging from the highest forum-ID, it is also missing 7,000 topics.

I've tried to get a backup database from the webhosting and it turns out they have a backup of maximum of 7 days old. That means we're a few days too late. This hurts, of course.

Beatle is going to look if there's a backup somewhere on a local disk, but we don't know if there's one and if there is, how old it is.

I've tried to figure out if data could be restored by using some redundancy values, but there's too much data lost. If the topics table only was gone, you could restore it quite easily.

3. I've reset the auto-increment number to the latest post + 1. That should work, I hope. We'll see when I post this message. :-)

Okay, now for what to do next.

I will focus on restoring the topics and posts. If there's no backup, we will have a problem, but I think all is not lost. Firstly, we always have the posts text, that is not gone. I'm hoping people have this forum mirrored somewhere. So if there's someone who has downloaded everything from this forum, let me know. I might restore data from the HTML fiiles.

In the meantime, I can try to port the data to phpbb3 without the utf-8 problems. My deadline for a 100% correct port is 15 september.

Any suggestions and ideas are welcome.
Post Posted Sat Sep 05, 2020 11:40 pm
Adam


User avatar
Posts: 296
Location: Cornwall,U.K.
Likes given: 9
Likes received  : 6

I sucessfully converted a phpbb2 forum to phpbb3 some years ago using their built in converter.
If there is any way you could send me a copy of the forum's SQL database there are a couple of things I'd like to try.
My own forum runs on phpbb 3.2.7 and it would interesting to compare to my database to see how it inserts posts and topics, I can try and convert it but can't guarantee that it will work, but at the end of the day anything is worth a try
Post Posted Sun Sep 06, 2020 12:09 am
Dr_Jones


User avatar
Posts: 3724
Location: Leiden, Netherlands
Likes given: 193
Likes received  : 402

see your PM.
Post Posted Sun Sep 06, 2020 8:54 pm
Dr_Jones


User avatar
Posts: 3724
Location: Leiden, Netherlands
Likes given: 193
Likes received  : 402

I've done a successful import of an old version of the forum, at least, that's what I think. I counted the users from before the import and after the import and excluding the bots all users were imported, I think. The amount of private messages seems to be the same too, so I guess not much data is gone from the import.

Now for the missing forum and posts tables.. this will be a problem. From the Google cache, I've seen we got 6844 topics in July and a total of 202817 posts. As of now, all these 202817 (and more, because August is not included) are orphaned. We're trying everything to get the data and the connection between forums, topics and posts back. This includes using Google cache, archive.org, the access-logs and other means.

Speaking of which: does anyone of you have a mirror of the forum by any chance? Someone who's wget-ed the forum. This might be most useful.
Post Posted Mon Sep 07, 2020 9:23 pm
Dr_Jones


User avatar
Posts: 3724
Location: Leiden, Netherlands
Likes given: 193
Likes received  : 402

Today's update:
Not much today, it's a regular working day for me.

But I've got some help from Adam and 1906, very much appreciated! So I retrieved all topics from the beginning up to december 2006 from an old backup, that means we've only got 14 years to catch up.

Adam is working on the current database and is trying to get a phpbb2 version up and running for testing purposes. I've gotten some useful tips from him and he did some testing on my version of the board. His suggestion to use some extensions resulted in a working shoutbox on the new version, yay! Adam also made a call on facebook if there are people out there who have a mirror of the website somewhere.

1906 and me were trying to find some ways to get the old database-data back, but to no avail. We've thought about some clever ways to reverse engineer things from the Google cache, let's hope this will work. We've thought of some other plans which I've set in motion, but those will be long and time-consuming tasks, I'm afraid.
Post Posted Tue Sep 08, 2020 10:16 pm
Dr_Jones


User avatar
Posts: 3724
Location: Leiden, Netherlands
Likes given: 193
Likes received  : 402

Update: I did not have much time today again (working day again), but I did a quick analysis on the archive.org data. I could restore some topics so that there's now 2988 topics. There used to be 6480 topics, so that's 46%.

The hardest part will be connecting the posts with the topics. I could manage a handful of them so 39885 posts are connected with topics. That's 19%.

Again: mirrors of the websites would be fantastic, these could help a lot with restoring lost topics and the connections between topics and posts.

I'm going to take a few days off now, I also have a social life :-)
Post Posted Thu Sep 10, 2020 11:42 am
Kanta
English Moderator & Miss News

User avatar
Posts: 11059
Location: United Kingdom
Likes given: 79
Likes received  : 310

Thank you for all your hard work. Much appreciated.
Jarregirl YouTube
Concerts attended:
Théâtre Marigny, Paris - 2007
Symphony Hall, Birmingham - 2008
RAH, London - 2008
Wembley Arena, London - 2009
NIA, Birmingham - 2009
POP Bercy, Paris - 2010
NIA, Birmingham - 2010
O2 Arena, London - 2010
Zénith Aréna, Lille - 2010
Port Hercule, Monaco - 2011
TUI Arena, Hannover - 2011
Festival International de Carthage - 2013
Barclaycard Arena, Birmingham - 2016
Post Posted Thu Sep 10, 2020 6:27 pm
Dr_Jones


User avatar
Posts: 3724
Location: Leiden, Netherlands
Likes given: 193
Likes received  : 402

Thanks people (also from the shoutbox), it's a lot of work, but it is fun. Digital archeology (hence the name Dr_Jones :-D) And Kanta, digging through all the old posts, I really admire all the work you have done already!
Post Posted Sat Sep 12, 2020 11:09 am
Velodynamic


User avatar
Posts: 3464
Location: Sweden, far north
Likes given: 2
Likes received  : 7

Even if you don't find topics and posts from the forum's first ten years or so it's still more then we can ask for I think. You're doing a fantasic job Doc.
Maybe it works more randomly then in chronological order? I dunno...
This board has been like a cat with eleven lives in the past. :smoke:
"The worst thing in any form of art is to be betrayed by your own habits."
- JEAN MICHEL JARRE
Post Posted Sat Sep 12, 2020 11:16 am
Kanta
English Moderator & Miss News

User avatar
Posts: 11059
Location: United Kingdom
Likes given: 79
Likes received  : 310

Thanks people (also from the shoutbox), it's a lot of work, but it is fun. Digital archeology (hence the name Dr_Jones :-D) And Kanta, digging through all the old posts, I really admire all the work you have done already!
Thank you. Much appreciated for your very kind words. :D
Jarregirl YouTube
Concerts attended:
Théâtre Marigny, Paris - 2007
Symphony Hall, Birmingham - 2008
RAH, London - 2008
Wembley Arena, London - 2009
NIA, Birmingham - 2009
POP Bercy, Paris - 2010
NIA, Birmingham - 2010
O2 Arena, London - 2010
Zénith Aréna, Lille - 2010
Port Hercule, Monaco - 2011
TUI Arena, Hannover - 2011
Festival International de Carthage - 2013
Barclaycard Arena, Birmingham - 2016
Post Posted Sat Sep 12, 2020 1:04 pm
Adam


User avatar
Posts: 296
Location: Cornwall,U.K.
Likes given: 9
Likes received  : 6

And I now have a working copy of the forum as it is now (Przemo's phpbb2) running on my PC for testing purposes via Wamp server, which allows you to work on any website offline & I have been uploading some of the restored posts to the database via scripts from Dr Jones.
I will also look into building a JMJ themed template for phpbb3 based on a template I have already built for my own forum, which will take a while but no hurry as the priority at the moment is restoring missing posts.
Post Posted Sun Sep 13, 2020 11:43 pm
Dr_Jones


User avatar
Posts: 3724
Location: Leiden, Netherlands
Likes given: 193
Likes received  : 402

Update:
Not much work on the phpbb3 version of the forum today. I will work on that the coming days.

For now, I've worked on digging through old posts via the Google cache and Bing cache. And what do you know? Microsoft beats Google in user-friendlyness! I can get posts automatically from Bing's cache, while Google has anti-bot measures, which are highly annoying.

Anyway, that way I have managed to get the following results:
Total posts 58512 Total topics 3714

That means I've so far restored 54% of the topics data! I did not expect this result at all! Post data is restored for 28%, which is okay.
Topics will have "holes" in them, the longest running threads are a lot less longer, sadly. These are usually the most interesting ones, like the new albums threads, tour threads, ebay thread, etc. I guess we have to live with that. Besides, there's a possibility that I can restore topics to the original topic via quotes. Remember that all the texts are still in the database, so matching might be possible that way. We're thinking of ways to do that.

Here's an screenshot of my development-system:
Image

It's using the standard subsilver template, as Adam said, customizing a template is a low priority right now, but I'm looking forward into seeing that happen.
Post Posted Mon Sep 14, 2020 8:24 am
Kanta
English Moderator & Miss News

User avatar
Posts: 11059
Location: United Kingdom
Likes given: 79
Likes received  : 310

Splendid achievement by both of you.
Jarregirl YouTube
Concerts attended:
Théâtre Marigny, Paris - 2007
Symphony Hall, Birmingham - 2008
RAH, London - 2008
Wembley Arena, London - 2009
NIA, Birmingham - 2009
POP Bercy, Paris - 2010
NIA, Birmingham - 2010
O2 Arena, London - 2010
Zénith Aréna, Lille - 2010
Port Hercule, Monaco - 2011
TUI Arena, Hannover - 2011
Festival International de Carthage - 2013
Barclaycard Arena, Birmingham - 2016
Post Posted Tue Sep 15, 2020 9:07 am
Dr_Jones


User avatar
Posts: 3724
Location: Leiden, Netherlands
Likes given: 193
Likes received  : 402

Last night I've worked on the phpbb3 version. I did some testing on registering for a new user and that worked fine. I got an activation-email, so it looks all works good!
I also worked on transferring the avatars to the new system. The filenames have been changed in the new version, so I've renamed them all.

Today, I expect to get some matched records from 1906 from the printview pages. These contain a full topic, but lack topic and post-ids. That means you have to match via other means. I'm really curious to learn how much can be saved from that.
Post Posted Thu Sep 17, 2020 8:53 pm
Dr_Jones


User avatar
Posts: 3724
Location: Leiden, Netherlands
Likes given: 193
Likes received  : 402

OK, latest update:

Since the end of the month is approaching, we have to chose what to do now. I have a forum running on PHP7 and PHPBB3. A part of the data is restored and the forum is functional. In fact, it's doing pretty well. I'm going to ask some people to take a look at it (not looking at the layout), to test if they can log in and all that. If these tests are okay, we can move the domain to the new forum and it's up and running.

In the meantime, we're working hard on a new style for the forum and restoring more of the lost posts.







  • 2020 Zoolook.nl
    Powered by phpBB forum software