User:Merriam~enwikinews/availability

Tsunami Help and server problems
This site, en.wikinews.org (but not de.wikinews.org), was down on 30 November from about 08:30 to about 17:08 UTC. This estimate is based on the gap in recent changes:


 * (diff) (hist) . . Tsunami Help/Aid Agencies, Donations & Volunteers; 17:15 . . 131.107.71.224 (Talk)
 * (diff) (hist) . . Template:Tsunami Relief Effort; 17:13 . . Nehavish (Talk)
 * (diff) (hist) . . Tsunami Help/Aid Agencies, Donations & Volunteers; 17:11 . . 131.107.71.224 (Talk)
 * (diff) (hist) . . Tsunami Help/Aid Agencies, Donations & Volunteers; 17:08 . . 131.107.71.224 (Talk) (&rarr;Specific Requests)
 * (diff) (hist) . . Tsunami Help/Links; 08:30 . . 195.109.155.71 (Talk)
 * (diff) (hist) . . m Tsunami Help/Tsunami Blog In The Media; 08:30 . . CBasturea (Talk)
 * (diff) (hist) . . New tsunamis to hit India today; 08:25 . . Phoenix (Talk) (Edited dagbladet.no link (which is the actual source for this story))
 * (diff) (hist) . . m Tsunami Help/Tsunami Blog In The Media; 08:24 . . CBasturea (Talk)

Contributors to Tsunami Help were apparently trying to edit during this time. Judging by this experience and other server problems, Wikimedia servers aren't the most suitable for this sort of time-critical activity.

SEA-EAT: "The Wikinews is down rightnow. We are currently working on to move the Wiki pages to a different server. If you have can offer us hosting services and bandwidth, please contact us or post your information here."

The Wikinews Tsunami Help pages are still being edited, but there is also some activity at [http://www.socialtext.net/tsunamihelp/index.cgi? Socialtext].

There was apparently a more general problem with the Wikimedia servers starting at the same time.

From w:User:Jamesday:

The outage on 30 November was from about 08:30 to 10:10 UTC, based on the web server charts. The problem was with memcached, which was having trouble for all Wikimedia hosted projects. It was apparently fixed by restarting memcached. The problems earlier in the week were with the new search, which is currently removed from service. Technically: the new search was refusing all search requests during Java Virtual Machine (JVM) memory garbage collection. The newer Java search been removed from service until it no longer does that.

The problems are entirely unrelated to server load, which was well within the current server capacity. That is: doubling the number of servers would have made no difference at all. The tsunami items had no visible effect at all on server load, so there's no need to wonder whether they are a problem.

The search indexes for the search we're back to using again have typically been updated for all projects twice a week. On my personal to do list is making that more frequent for Wikinews, because of the fast-changing nature of the information here. At the moment, all of the indexes are more out of date than usual - should be taken care of by tomorrow morning (US time).

Some have wondered whether the older search is a problem which allows attacks on the servers. It isn't. It is the highest load operation the database servers perform and we have both load-related controls, which let us turn off a percentage of search if necessary, and an automatic query killer which works for all types of database queries and terminates any which are apparently involved in a denial of service attack on the sites. Because it's the highest load item it's something we turn off temporarily if general load is high enough to cause database slaves to lag or otherwise be unable to keep up. Usually, we're able to keep ahead of load and get more database servers into service before we have to do that - but that wasn't the case in October and November, because two new servers took longer than expected to arrive. That left us having to turn off a fair percentage of search (about 20%) routinely at peak times. With a traffic doubling time of just 8-12 weeks, it can be a challenge to keep ahead of growth if there are delays in getting new equipment. Doesn't stop us from trying though.:)

At present we've largely been operating without the query killer running as we learn what queries it needs to work with on the new MediaWiki 1.4 version. I updated it earlier today to make some changes for MediaWiki 1.4 and that updating will continue as we learn how MediaWiki 1.4 differs from 1.3 in the ability to submit unacceptably high load operations. Most of those aren't search-related. If you'd like to read more about the database load/attack limiter, see Querybane. Jamesday 23:29, 30 Dec 2004 (UTC)