Thijs/wiki.techinc.nl
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
38341 commits 1 branch 0 tags 744 MiB
Commit graph

70 commits

Author SHA1 Message Date
Ariel Glenn
2ec322f6c2 handle naming checkpoint file with first/last pageID when the file is empty 2011-09-23 06:15:20 +00:00
Ariel Glenn
be36f6e1fe getFilename renamed to getFilenames since it can return a list 2011-09-09 07:28:11 +00:00
Ariel Glenn
e30a238ebf uniform comment style, fix a few space issues, address couple issues from comments on r95272 2011-09-08 21:06:15 +00:00
Chad Horohoe
cbd499b538 Remove wfDie() that Ariel keeps trying to resurrect :) 2011-08-30 20:45:36 +00:00
Ariel Glenn
409725f43e check the checkpoint related options only if we specified checkpoints, duh 2011-08-29 22:48:18 +00:00
Ariel Glenn
12e00d6eb9 move some member vars to parent class since they are needed there now, set lastTime and other vars so checkpoints without prefetch work, update progress reporting in parent class to give rates during interval and from start of run 2011-08-28 19:06:52 +00:00
Ariel Glenn
0dcd589b6b define and use closeAndRename() after last write of xml dump file; convert from popen (child inherits all open descriptors and there is no workaround) to proc_open (CLOEXEC set on all descriptors), needed so close and rename doesn't hang forever if a child (prefetcher) is forked 2011-08-27 18:31:03 +00:00
Ariel Glenn
0f90a6b10e remove extraneous hyphen that crept in, grr 2011-08-24 20:43:09 +00:00
Ariel Glenn
76997163b6 fix timestamp stuff, more fallout from bad merge attempt 2011-08-23 00:36:15 +00:00
Ariel Glenn
b6a6354676 fix a couple bad lines in previous commit from bad merge attempt 2011-08-23 00:04:45 +00:00
Ariel Glenn
bc3caf7666 add support for writing out checkpoint files of xml dump at regular intervals (close and rename file based on filename pattern which includes first and last page id written) 2011-08-22 22:45:22 +00:00
Chad Horohoe
b1bfcaeca4 Revert r85034, r81186, r77638: per CR on r77638: XMLReader and XMLWriter are memory-hungry beasts and this script OOMs constantly. 
This was already reverted in 1.17wmf1 (r82930) by Ariel. Doing the same in trunk now. I'm not opposed to using the new pretty XML* classes, but somebody needs to debug why they eat memory for breakfast, lunch and dinner.
 2011-07-20 23:06:24 +00:00
Sam Reed
26e62d5089 Swap = to == in conditional 2011-03-30 18:56:31 +00:00
Ariel Glenn
82915f097b eta shows revs/pages per sec in current interval as well as overall; useful for investigating problems, now that we have logging 2011-03-27 13:27:05 +00:00
Alexandre Emsenhuber
8eb3b596e0 * (bug 27016) Fix for r77638: dumpTextPass.php now consider the "output" parameter 
Also added "output" parameter to the doc since nothing was mentioning it
 2011-01-29 09:00:38 +00:00
Ariel Glenn
585f1a3cce open a series of files for xml input, if supplied (used for multiple xml prefetch files) 2011-01-28 19:20:49 +00:00
Alexandre Emsenhuber
449a2e61bd When opening the subprocess fetchText.php, pass the complete wiki ID through the --wiki parameter so that it works for all setups and not only for wikimedia 2010-12-06 09:20:07 +00:00
Alexandre Emsenhuber
1098d92d42 Follow-up r67324: removed check for empty text since the length is checked explicitly 2010-12-04 19:30:26 +00:00
Alexandre Emsenhuber
6b190f772e Modifier dumpTextPass.php to use XMLReader and XMLWriter rather than xml_* functions 2010-12-03 10:05:01 +00:00
Sam Reed
c15c1c6abb More reversion of r77297, 2 of 2 commits to keep it readable in CR (hopefully) 2010-11-25 22:24:28 +00:00
Antoine Musso
6fe29c9c30 revert r76464 
* mixed style changes and variable removing
* logic changes
* useful comments removed
 2010-11-25 20:01:34 +00:00
Sam Reed
10ecfeb03b Various unused variables, add some braces 
Explicitally define some variables used in hook type calls
 2010-11-10 19:49:04 +00:00
Alexandre Emsenhuber
f916f9e25c One more remplacement to __METHOD__ 2010-10-09 12:30:20 +00:00
Alexandre Emsenhuber
76bebf904c * Standardised file description headers 
* added @file where needed
* added file description headers where needed
 2010-10-03 09:25:28 +00:00
Alexandre Emsenhuber
6da6645192 And while I'm a it: 
* Fixed E_STRICT about different signature of TextPassDumper::dump() and BackupDumper::dump()
* Add the possibility to show the help message...
 2010-10-02 21:58:04 +00:00
Alexandre Emsenhuber
da131336ac Per Platonides; follow-up r74158: XMLReader is included in PHP since 5.1.0 2010-10-02 21:51:44 +00:00
Alexandre Emsenhuber
7e9424bce9 MediaWiki even require PHP 5.1.0+ ;) 2010-10-02 21:40:43 +00:00
Platonides
50e2405a03 Refactor to avoid defining globals inside switches. 2010-08-12 14:25:07 +00:00
Ariel Glenn
fba252c800 clean up space/tab stuff 2010-07-19 18:36:52 +00:00
Ariel Glenn
9f6103db51 intval ( $id ) once in caller instead of in all the getTextXXX functions 2010-07-15 21:25:31 +00:00
Ariel Glenn
da009ed5fb distinguish failed text retrieval from empty text, consolidate text retrieval retry code, die after maxConsecutiveFailedTextRetrievals (of separate revisions) 2010-06-16 20:12:29 +00:00
Ariel Glenn
36ba33c394 XML snapshots: check length of revision text from prefetched content against length in db before using, partial fix for bugs 23264, 18694 2010-06-04 03:10:34 +00:00
Ariel Glenn
a2b7c4fb85 XML snapshots: when retrieving revision text, send the text id back to requester for doublechecking (avoids revision text out of sync errors), partial fix for bugs 23264, 18694 2010-06-04 02:24:56 +00:00
Sam Reed
659778619c Stylize maintenance folder.. 2010-05-22 16:50:39 +00:00
Tim Starling
ad19c032b0 Fix for bug 9413 and the related Malayalam issue reported on wikitech-l. 
* Added $wgFixArchaicUnicode, which, if enabled, converts some deprecated Unicode sequences in Arabic and Malayalam text to their Unicode 5.1 equivalents.
* Added generateNormalizerData.php to generate the relevant data files. Added the generated data files also. 
* Made most things call the new wrapper method $wgContLang->normalize() instead of UtfNormal::cleanUp(), so that Unicode normalization can be customised on a per-language basis.
* Added some generic support for conversion tables to Language so that subclasses can easily implement these kinds of transformations.
 2010-01-04 08:28:50 +00:00
Benjamin Lees
65a9d3729a Follow-up to http://www.mediawiki.org/wiki/Special:Code/MediaWiki/54561: use the class autoloader. 2009-08-19 05:41:42 +00:00
Benjamin Lees
02e4701b82 (bug 19289) importDump.php can now handle bzip2 and 7zip. 
I split out our 7zip stream wrapper into its own file, 7zip.inc.
 2009-08-07 00:53:21 +00:00
Chad Horohoe
567f244e36 Revert r54244 which was stupid and fix this properly. Require commandLine.inc/Maintenance.php using the full path every time. 2009-08-03 21:56:41 +00:00
Brion Vibber
7901baa10b Fix a silly bug that isn't the mystery dump bug we were looking for, but could result in infinite loop during text pass if it fails just at the right time. 2009-05-06 23:47:31 +00:00
Chad Horohoe
be374a912a War on wfElement() and friends. Call the Xml members directly, rather than using old wrappers. 2008-12-14 19:14:21 +00:00
Tim Starling
1daa0dc318 Don't break vim syntax highlighting 2008-06-04 01:48:25 +00:00
Alexandre Emsenhuber
087a9f70c5 WARNING: HUGE COMMIT 
Doxygen documentation update:
* Changed alls @addtogroup to @ingroup. @addtogroup adds the comment to the group description, but doesn't add the file, class, function, ... to the group like @ingroup does. See for example http://svn.wikimedia.org/doc/group__SpecialPage.html where it's impossible to see related files, classes, ... that should belong to that group.
* Added @file to file description, it seems that it should be explicitely decalred for file descriptions, otherwise doxygen will think that the comment document the first class, variabled, function, ... that is in that file.
* Removed some empty comments
* Removed some ?>

Added following groups:
* ExternalStorage
* JobQueue
* MaintenanceLanguage

One more thing: there are still a lot of warnings when generating the doc.
 2008-05-20 17:13:28 +00:00
Brion Vibber
30420e6ffd tweak paths more 2008-01-08 23:58:16 +00:00
Brion Vibber
01aa30f1a1 fix include path regressions with text prefetch 2008-01-08 23:53:20 +00:00
Brion Vibber
60c13d2853 * (bug 12184) Exceptions now sent to stderr instead of stdout for command-line 
scripts, making for cleaner reporting during batch jobs. PHP errors will also
  be redirected in most cases on PHP 5.2.4 and later, switching 'display_errors'
  to 'stderr' at runtime.
 2007-12-06 21:07:49 +00:00
Brion Vibber
54fa21ca82 Do text normalization in dumpTextPass rather than fetchText, to ensure that the force-loaded ICU plugin will be used in the parent process. 2007-11-20 17:34:42 +00:00
Brion Vibber
48f14d8955 Initial implementation of restartable subprocess for text fetching in dumpTextPass.php 2007-11-20 12:58:34 +00:00
Brion Vibber
24cc59f3b9 Quick fix for dump problem: 
Errors are ignored on a connection once it's died, so additional exceptions weren't getting thrown.
We weren't detecting this case and ended up spitting out the rest of the XML without any more text.
Should now die correctly once the timeouts finish.
 2007-08-02 04:15:30 +00:00
Aryeh Gregor
a15c419b3d Remove ?>'s from files. They're pointless, and just asking for people to mess with the files and add trailing whitespace. (Yes, I looked over every one and reverted those that were bogus. Slash-enter a million times in less worked well enough, although it was a bit mind-numbing.) 2007-06-29 01:19:14 +00:00
Brion Vibber
72a4abe588 * Skip additional setting of include_path in commandLine.inc (for non-Wikimedia mode) 
* Fix some scripts that assumed include_path was set with various additional directories
Stuff now seems to mostly work when not overriding include_path.
Taking that out of LocalSettings is the next step... whee!
 2007-06-06 16:01:14 +00:00