wiki.techinc.nl/tests/phpunit/includes/db/LBFactoryTest.php

944 lines
30 KiB
PHP
Raw Normal View History

<?php
/**
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License along
* with this program; if not, write to the Free Software Foundation, Inc.,
* 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
* http://www.gnu.org/copyleft/gpl.html
*
* @file
* @author Antoine Musso
* @copyright © 2013 Antoine Musso
* @copyright © 2013 Wikimedia Foundation Inc.
*/
use MediaWiki\WikiMap\WikiMap;
use Wikimedia\Rdbms\ChronologyProtector;
use Wikimedia\Rdbms\Database;
use Wikimedia\Rdbms\DatabaseDomain;
use Wikimedia\Rdbms\IDatabase;
use Wikimedia\Rdbms\IMaintainableDatabase;
use Wikimedia\Rdbms\IReadableDatabase;
use Wikimedia\Rdbms\LBFactory;
use Wikimedia\Rdbms\LBFactoryMulti;
use Wikimedia\Rdbms\LBFactorySimple;
use Wikimedia\Rdbms\LoadBalancer;
use Wikimedia\Rdbms\LoadMonitorNull;
use Wikimedia\Rdbms\MySQLPrimaryPos;
Replace LoadBalancer/LBFactory callback iteration with generators Callback style iteration made sense before generators existed, but generators make for simpler code. The "call method" variants made sense before closures existed but defeat static analysis. So, in LBFactory: * Add ILBFactory::getAllLBs() * Deprecate ILBFactory::forEachLB() * Remove LBFactory::forEachLBCallMethod(), was protected. * Add LBFactory::getLBsForOwner(), which is protected and has the internal interface in @return. Adding a new abstract method breaks Wikibase tests despite LBFactory not being stable to extend. * Migrate callers. Generators allow you to return/break from the middle of the loop, which implies a little rearrangement for some callers. In LoadBalancer, connections supposedly of type IDatabase were routinely type-hinted as Database in closure parameters so that methods could be called that were not in the interface. So it's convenient to get rid of public iteration methods entirely in favour of private methods returning Database[]. * Hard-deprecate ILoadBalancerForOwner::forEachOpenConnection() and replace it with a private generator method since nothing called it externally except for core tests. * Hard-deprecate ILoadBalancerForOwner::forEachOpenPrimaryConnection() and replace it with a private generator. DeferredUpdates needed it for iterating over IDatabase::explicitTrxActive(), so add ILoadBalancer::explicitTrxActive() as a replacement. * Replace private method LoadBalancer::forEachOpenReplicaConnection() with a generator. Depends-On: If0b382231e27d6d1197fb7b6aef6ab50335df4e5 Change-Id: I64514e77b9bfe737be5b12e1d3c9c49976bb522f
2022-04-26 03:54:59 +00:00
use Wikimedia\TestingAccessWrapper;
/**
* @group Database
objectcache,resourceloader,rdbms,jobqueue: Widen @covers annotations Follows-up I4c7d826c7ec654b, I1287f3979aba1bf1. We lose useful coverage and spend valuable time keeping these accurate through refactors (or worse, forget to do so). The theoretically "bad" accidental coverage is almost never actually bad. Having said that, I'm not removing them wholesale (yet). I've audited each of these specific files to confirm it is a general test of the specified subject class, and also kept it limited to those specified classes. That's imho more than 100% of the benefit for less than 1% of the cost (more because `@covers` is more valuable than the fragile and corrosive individual private method tracking in tests that inevitably get out of date with no local incentive to keep them up to date). Cases like structure tests keep `@coversNothing` etc and we still don't count coverage of other classes. There may be a handful of large legacy classes where some methods are effectively class-like in complexity and that's why it's good for PHPUnit to offer the precision instrument but that doesn't meant we have to use that by-default for everything. I think best practice is to write good narrow unit tests, that reflect how the code should be used in practice. Not to write bad tests and hide part of its coverage within the same class or even namespace. Fortunately, that's generally what we do already it's just that we also kept these annotations still in many cases. This wastes time to keep methods in sync, time to realize (and fix) when other people inevitably didn't keep them in sync, time to find uncovered code only to realize it is already covered, time for a less experienced engineer to feel obligate to and do write a low quality test to cover the "missing" branch in an unrealistic way, time wasted in on-boarding by using such "bad" tests as example for how to use the code and then having to unlearn it months/years later, loss of telemetry in knowing what code actually isn't propertly tested due to being masked by a bad test, and lost oppertunities to find actually ununused/unreachable code and to think about how to instead structure the code such that maybe that code can be removed. ------ Especially cases like LBFactoryTest.php were getting out of hand, and in GlobalIdGeneratorTest.php we even resorted to reminding people with inline comments to keep tags in sync. Change-Id: I69b5385868cc6b451e5f2ebec9539694968bf58c
2023-04-10 21:25:34 +00:00
* @covers \Wikimedia\Rdbms\ChronologyProtector
* @covers \Wikimedia\Rdbms\DatabaseMysqlBase
* @covers \Wikimedia\Rdbms\DatabasePostgres
* @covers \Wikimedia\Rdbms\DatabaseSqlite
* @covers \Wikimedia\Rdbms\LBFactory
* @covers \Wikimedia\Rdbms\LBFactory
* @covers \Wikimedia\Rdbms\LBFactoryMulti
objectcache,resourceloader,rdbms,jobqueue: Widen @covers annotations Follows-up I4c7d826c7ec654b, I1287f3979aba1bf1. We lose useful coverage and spend valuable time keeping these accurate through refactors (or worse, forget to do so). The theoretically "bad" accidental coverage is almost never actually bad. Having said that, I'm not removing them wholesale (yet). I've audited each of these specific files to confirm it is a general test of the specified subject class, and also kept it limited to those specified classes. That's imho more than 100% of the benefit for less than 1% of the cost (more because `@covers` is more valuable than the fragile and corrosive individual private method tracking in tests that inevitably get out of date with no local incentive to keep them up to date). Cases like structure tests keep `@coversNothing` etc and we still don't count coverage of other classes. There may be a handful of large legacy classes where some methods are effectively class-like in complexity and that's why it's good for PHPUnit to offer the precision instrument but that doesn't meant we have to use that by-default for everything. I think best practice is to write good narrow unit tests, that reflect how the code should be used in practice. Not to write bad tests and hide part of its coverage within the same class or even namespace. Fortunately, that's generally what we do already it's just that we also kept these annotations still in many cases. This wastes time to keep methods in sync, time to realize (and fix) when other people inevitably didn't keep them in sync, time to find uncovered code only to realize it is already covered, time for a less experienced engineer to feel obligate to and do write a low quality test to cover the "missing" branch in an unrealistic way, time wasted in on-boarding by using such "bad" tests as example for how to use the code and then having to unlearn it months/years later, loss of telemetry in knowing what code actually isn't propertly tested due to being masked by a bad test, and lost oppertunities to find actually ununused/unreachable code and to think about how to instead structure the code such that maybe that code can be removed. ------ Especially cases like LBFactoryTest.php were getting out of hand, and in GlobalIdGeneratorTest.php we even resorted to reminding people with inline comments to keep tags in sync. Change-Id: I69b5385868cc6b451e5f2ebec9539694968bf58c
2023-04-10 21:25:34 +00:00
* @covers \Wikimedia\Rdbms\LBFactorySimple
* @covers \Wikimedia\Rdbms\LoadBalancer
*/
class LBFactoryTest extends MediaWikiIntegrationTestCase {
private function getPrimaryServerConfig() {
global $wgDBserver, $wgDBport, $wgDBname, $wgDBuser, $wgDBpassword, $wgDBtype;
global $wgSQLiteDataDir;
return [
'serverName' => 'db1',
'host' => $wgDBserver,
'port' => $wgDBport,
'dbname' => $wgDBname,
'user' => $wgDBuser,
'password' => $wgDBpassword,
'type' => $wgDBtype,
'dbDirectory' => $wgSQLiteDataDir,
'load' => 0,
'flags' => DBO_TRX // REPEATABLE-READ for consistency
];
}
public function testLBFactorySimpleServer() {
$servers = [ $this->getPrimaryServerConfig() ];
$factory = new LBFactorySimple( [ 'servers' => $servers ] );
$lb = $factory->getMainLB();
$dbw = $lb->getConnection( DB_PRIMARY );
$this->assertEquals(
$dbw::ROLE_STREAMING_MASTER, $dbw->getTopologyRole(), 'master shows as master' );
$dbr = $lb->getConnection( DB_REPLICA );
$this->assertEquals(
$dbr::ROLE_STREAMING_MASTER, $dbw->getTopologyRole(), 'replica shows as replica' );
$this->assertSame( 'DEFAULT', $lb->getClusterName() );
$this->assertSame( 'my_test_wiki', $factory->resolveDomainID( 'my_test_wiki' ) );
$this->assertSame( $factory->getLocalDomainID(), $factory->resolveDomainID( false ) );
$factory->shutdown();
}
public function testLBFactorySimpleServers() {
$primaryConfig = $this->getPrimaryServerConfig();
$fakeReplica = [ 'serverName' => 'db2', 'load' => 100 ] + $primaryConfig;
$servers = [
$primaryConfig,
$fakeReplica
];
$factory = new LBFactorySimple( [
'servers' => $servers,
'loadMonitor' => [ 'class' => LoadMonitorNull::class ],
] );
$lb = $factory->getMainLB();
$dbw = $lb->getConnection( DB_PRIMARY );
$dbw->ensureConnection();
$this->assertEquals(
$dbw::ROLE_STREAMING_MASTER, $dbw->getTopologyRole(), 'primary shows as primary' );
$dbr = $lb->getConnection( DB_REPLICA );
$dbr->ensureConnection();
$this->assertEquals(
$dbr::ROLE_STREAMING_REPLICA, $dbr->getTopologyRole(), 'replica shows as replica' );
$factory->shutdown();
}
public function testLBFactoryMultiConns() {
$factory = $this->newLBFactoryMultiLBs();
$this->assertSame( 's3', $factory->getMainLB()->getClusterName() );
$dbw = $factory->getMainLB()->getConnection( DB_PRIMARY );
$this->assertEquals(
$dbw::ROLE_STREAMING_MASTER, $dbw->getTopologyRole(), 'master shows as master' );
$dbr = $factory->getMainLB()->getConnection( DB_REPLICA );
$this->assertEquals(
$dbr::ROLE_STREAMING_REPLICA, $dbr->getTopologyRole(), 'replica shows as replica' );
// Destructor should trigger without round stage errors
unset( $factory );
}
public function testLBFactoryMultiRoundCallbacks() {
$called = 0;
$countLBsFunc = static function ( LBFactoryMulti $factory ) {
$count = 0;
Replace LoadBalancer/LBFactory callback iteration with generators Callback style iteration made sense before generators existed, but generators make for simpler code. The "call method" variants made sense before closures existed but defeat static analysis. So, in LBFactory: * Add ILBFactory::getAllLBs() * Deprecate ILBFactory::forEachLB() * Remove LBFactory::forEachLBCallMethod(), was protected. * Add LBFactory::getLBsForOwner(), which is protected and has the internal interface in @return. Adding a new abstract method breaks Wikibase tests despite LBFactory not being stable to extend. * Migrate callers. Generators allow you to return/break from the middle of the loop, which implies a little rearrangement for some callers. In LoadBalancer, connections supposedly of type IDatabase were routinely type-hinted as Database in closure parameters so that methods could be called that were not in the interface. So it's convenient to get rid of public iteration methods entirely in favour of private methods returning Database[]. * Hard-deprecate ILoadBalancerForOwner::forEachOpenConnection() and replace it with a private generator method since nothing called it externally except for core tests. * Hard-deprecate ILoadBalancerForOwner::forEachOpenPrimaryConnection() and replace it with a private generator. DeferredUpdates needed it for iterating over IDatabase::explicitTrxActive(), so add ILoadBalancer::explicitTrxActive() as a replacement. * Replace private method LoadBalancer::forEachOpenReplicaConnection() with a generator. Depends-On: If0b382231e27d6d1197fb7b6aef6ab50335df4e5 Change-Id: I64514e77b9bfe737be5b12e1d3c9c49976bb522f
2022-04-26 03:54:59 +00:00
foreach ( $factory->getAllLBs() as $lb ) {
++$count;
Replace LoadBalancer/LBFactory callback iteration with generators Callback style iteration made sense before generators existed, but generators make for simpler code. The "call method" variants made sense before closures existed but defeat static analysis. So, in LBFactory: * Add ILBFactory::getAllLBs() * Deprecate ILBFactory::forEachLB() * Remove LBFactory::forEachLBCallMethod(), was protected. * Add LBFactory::getLBsForOwner(), which is protected and has the internal interface in @return. Adding a new abstract method breaks Wikibase tests despite LBFactory not being stable to extend. * Migrate callers. Generators allow you to return/break from the middle of the loop, which implies a little rearrangement for some callers. In LoadBalancer, connections supposedly of type IDatabase were routinely type-hinted as Database in closure parameters so that methods could be called that were not in the interface. So it's convenient to get rid of public iteration methods entirely in favour of private methods returning Database[]. * Hard-deprecate ILoadBalancerForOwner::forEachOpenConnection() and replace it with a private generator method since nothing called it externally except for core tests. * Hard-deprecate ILoadBalancerForOwner::forEachOpenPrimaryConnection() and replace it with a private generator. DeferredUpdates needed it for iterating over IDatabase::explicitTrxActive(), so add ILoadBalancer::explicitTrxActive() as a replacement. * Replace private method LoadBalancer::forEachOpenReplicaConnection() with a generator. Depends-On: If0b382231e27d6d1197fb7b6aef6ab50335df4e5 Change-Id: I64514e77b9bfe737be5b12e1d3c9c49976bb522f
2022-04-26 03:54:59 +00:00
}
return $count;
};
$factory = $this->newLBFactoryMultiLBs();
$this->assertSame( 0, $countLBsFunc( $factory ) );
$dbw = $factory->getMainLB()->getConnection( DB_PRIMARY );
$this->assertSame( 1, $countLBsFunc( $factory ) );
// Test that LoadBalancer instances made during pre-commit callbacks in do not
// throw DBTransactionError due to transaction ROUND_* stages being mismatched.
$factory->beginPrimaryChanges( __METHOD__ );
$dbw->onTransactionPreCommitOrIdle( static function () use ( $factory, &$called ) {
++$called;
// Trigger s1 LoadBalancer instantiation during "finalize" stage.
// There is no s1wiki DB to select so it is not in getConnection(),
// but this fools getMainLB() at least.
$factory->getMainLB( 's1wiki' )->getConnection( DB_PRIMARY );
} );
$factory->commitPrimaryChanges( __METHOD__ );
$this->assertSame( 1, $called );
$this->assertEquals( 2, $countLBsFunc( $factory ) );
$factory->shutdown();
$factory->closeAll( __METHOD__ );
$called = 0;
$factory = $this->newLBFactoryMultiLBs();
$this->assertSame( 0, $countLBsFunc( $factory ) );
$dbw = $factory->getMainLB()->getConnection( DB_PRIMARY );
$this->assertSame( 1, $countLBsFunc( $factory ) );
// Test that LoadBalancer instances made during pre-commit callbacks in do not
// throw DBTransactionError due to transaction ROUND_* stages being mismatched.hrow
// DBTransactionError due to transaction ROUND_* stages being mismatched.
$factory->beginPrimaryChanges( __METHOD__ );
$dbw->query( "SELECT 1 as t", __METHOD__ );
$dbw->onTransactionResolution( static function () use ( $factory, &$called ) {
++$called;
// Trigger s1 LoadBalancer instantiation during "finalize" stage.
// There is no s1wiki DB to select so it is not in getConnection(),
// but this fools getMainLB() at least.
$factory->getMainLB( 's1wiki' )->getConnection( DB_PRIMARY );
} );
$factory->commitPrimaryChanges( __METHOD__ );
$this->assertSame( 1, $called );
$this->assertEquals( 2, $countLBsFunc( $factory ) );
$factory->shutdown();
$factory->closeAll( __METHOD__ );
$factory = $this->newLBFactoryMultiLBs();
$dbw = $factory->getMainLB()->getConnection( DB_PRIMARY );
// DBTransactionError should not be thrown
$ran = 0;
$dbw->onTransactionPreCommitOrIdle( static function () use ( &$ran ) {
++$ran;
} );
$factory->commitPrimaryChanges( __METHOD__ );
$this->assertSame( 1, $ran );
$factory->shutdown();
$factory->closeAll( __METHOD__ );
}
private function newLBFactoryMultiLBs() {
global $wgDBserver, $wgDBport, $wgDBname, $wgDBuser, $wgDBpassword, $wgDBtype;
global $wgSQLiteDataDir;
return new LBFactoryMulti( [
'sectionsByDB' => [
's1wiki' => 's1',
'DEFAULT' => 's3'
],
'sectionLoads' => [
's1' => [
'test-db3' => 0,
'test-db4' => 100,
],
's3' => [
'test-db1' => 0,
'test-db2' => 100,
]
],
'serverTemplate' => [
'port' => $wgDBport,
'dbname' => $wgDBname,
'user' => $wgDBuser,
'password' => $wgDBpassword,
'type' => $wgDBtype,
'dbDirectory' => $wgSQLiteDataDir,
'flags' => DBO_DEFAULT
],
'hostsByName' => [
'test-db1' => $wgDBserver,
'test-db2' => $wgDBserver,
'test-db3' => $wgDBserver,
'test-db4' => $wgDBserver
],
'loadMonitor' => [ 'class' => LoadMonitorNull::class ],
] );
}
/**
* @covers \Wikimedia\Rdbms\ChronologyProtector
*/
public function testChronologyProtector() {
$now = microtime( true );
$hasChangesFunc = static function ( $mockDB ) {
$p = $mockDB->writesOrCallbacksPending();
$last = $mockDB->lastDoneWrites();
return is_float( $last ) || $p;
};
// (a) First HTTP request
$m1Pos = new MySQLPrimaryPos( 'db1034-bin.000976/843431247', $now );
$m2Pos = new MySQLPrimaryPos( 'db1064-bin.002400/794074907', $now );
// Primary DB 1
/** @var IDatabase|\PHPUnit\Framework\MockObject\MockObject $mockDB1 */
$mockDB1 = $this->createMock( IDatabase::class );
$mockDB1->method( 'writesOrCallbacksPending' )->willReturn( true );
$mockDB1->method( 'lastDoneWrites' )->willReturn( $now );
$mockDB1->method( 'getPrimaryPos' )->willReturn( $m1Pos );
// Load balancer for primary DB 1
$lb1 = $this->createMock( LoadBalancer::class );
$lb1->method( 'getConnection' )->willReturn( $mockDB1 );
$lb1->method( 'getServerCount' )->willReturn( 2 );
$lb1->method( 'hasReplicaServers' )->willReturn( true );
$lb1->method( 'hasStreamingReplicaServers' )->willReturn( true );
$lb1->method( 'getAnyOpenConnection' )->willReturn( $mockDB1 );
$lb1->method( 'hasOrMadeRecentPrimaryChanges' )->willReturnCallback(
static function () use ( $mockDB1, $hasChangesFunc ) {
return $hasChangesFunc( $mockDB1 );
}
);
$lb1->method( 'getPrimaryPos' )->willReturn( $m1Pos );
$lb1->method( 'getReplicaResumePos' )->willReturn( $m1Pos );
$lb1->method( 'getServerName' )->with( 0 )->willReturn( 'master1' );
// Primary DB 2
/** @var IDatabase|\PHPUnit\Framework\MockObject\MockObject $mockDB2 */
$mockDB2 = $this->createMock( IDatabase::class );
$mockDB2->method( 'writesOrCallbacksPending' )->willReturn( true );
$mockDB2->method( 'lastDoneWrites' )->willReturn( $now );
$mockDB2->method( 'getPrimaryPos' )->willReturn( $m2Pos );
// Load balancer for primary DB 2
$lb2 = $this->createMock( LoadBalancer::class );
$lb2->method( 'getConnection' )->willReturn( $mockDB2 );
$lb2->method( 'getServerCount' )->willReturn( 2 );
$lb2->method( 'hasReplicaServers' )->willReturn( true );
$lb2->method( 'hasStreamingReplicaServers' )->willReturn( true );
$lb2->method( 'getAnyOpenConnection' )->willReturn( $mockDB2 );
$lb2->method( 'hasOrMadeRecentPrimaryChanges' )->willReturnCallback(
static function () use ( $mockDB2, $hasChangesFunc ) {
return $hasChangesFunc( $mockDB2 );
}
);
$lb2->method( 'getPrimaryPos' )->willReturn( $m2Pos );
$lb2->method( 'getReplicaResumePos' )->willReturn( $m2Pos );
$lb2->method( 'getServerName' )->with( 0 )->willReturn( 'master2' );
$bag = new HashBagOStuff();
$cp = new ChronologyProtector(
$bag,
[
'ip' => '127.0.0.1',
'agent' => "Totally-Not-FireFox"
],
null
);
$mockDB1->expects( $this->once() )->method( 'writesOrCallbacksPending' );
$mockDB1->expects( $this->once() )->method( 'lastDoneWrites' );
$mockDB2->expects( $this->once() )->method( 'writesOrCallbacksPending' );
$mockDB2->expects( $this->once() )->method( 'lastDoneWrites' );
// Nothing to wait for on first HTTP request start
$sPos1 = $cp->yieldSessionPrimaryPos( $lb1 );
$sPos2 = $cp->yieldSessionPrimaryPos( $lb2 );
// Record positions in stash on first HTTP request end
$cp->stageSessionPrimaryPos( $lb1 );
$cp->stageSessionPrimaryPos( $lb2 );
$cpIndex = null;
$cp->persistSessionReplicationPositions( $cpIndex );
$this->assertNull( $sPos1 );
$this->assertNull( $sPos2 );
$this->assertSame( 1, $cpIndex, "CP write index set" );
// (b) Second HTTP request
// Load balancer for primary DB 1
$lb1 = $this->createMock( LoadBalancer::class );
$lb1->method( 'getServerCount' )->willReturn( 2 );
$lb1->method( 'hasReplicaServers' )->willReturn( true );
$lb1->method( 'hasStreamingReplicaServers' )->willReturn( true );
$lb1->method( 'getServerName' )->with( 0 )->willReturn( 'master1' );
// Load balancer for primary DB 2
$lb2 = $this->createMock( LoadBalancer::class );
$lb2->method( 'getServerCount' )->willReturn( 2 );
$lb2->method( 'hasReplicaServers' )->willReturn( true );
$lb2->method( 'hasStreamingReplicaServers' )->willReturn( true );
$lb2->method( 'getServerName' )->with( 0 )->willReturn( 'master2' );
$cp = new ChronologyProtector(
$bag,
[
'ip' => '127.0.0.1',
'agent' => "Totally-Not-FireFox"
],
$cpIndex
);
// Get last positions to be reached on second HTTP request start
$sPos1 = $cp->yieldSessionPrimaryPos( $lb1 );
$sPos2 = $cp->yieldSessionPrimaryPos( $lb2 );
// Shutdown (nothing to record)
$cp->stageSessionPrimaryPos( $lb1 );
$cp->stageSessionPrimaryPos( $lb2 );
$cpIndex = null;
$cp->persistSessionReplicationPositions( $cpIndex );
$this->assertNotNull( $sPos1 );
$this->assertNotNull( $sPos2 );
$this->assertSame( $m1Pos->__toString(), $sPos1->__toString() );
$this->assertSame( $m2Pos->__toString(), $sPos2->__toString() );
$this->assertNull( $cpIndex, "CP write index retained" );
$this->assertEquals( '45e93a9c215c031d38b7c42d8e4700ca', $cp->getClientId() );
}
private function newLBFactoryMulti( array $baseOverride = [], array $serverOverride = [] ) {
global $wgDBserver, $wgDBuser, $wgDBpassword, $wgDBname, $wgDBprefix, $wgDBtype;
global $wgSQLiteDataDir;
return new LBFactoryMulti( $baseOverride + [
'sectionsByDB' => [],
'sectionLoads' => [
'DEFAULT' => [
'test-db1' => 1,
],
],
'serverTemplate' => $serverOverride + [
'dbname' => $wgDBname,
'tablePrefix' => $wgDBprefix,
'user' => $wgDBuser,
'password' => $wgDBpassword,
'type' => $wgDBtype,
'dbDirectory' => $wgSQLiteDataDir,
'flags' => DBO_DEFAULT
],
'hostsByName' => [
'test-db1' => $wgDBserver,
],
'loadMonitor' => [ 'class' => LoadMonitorNull::class ],
'localDomain' => new DatabaseDomain( $wgDBname, null, $wgDBprefix ),
'agent' => 'MW-UNIT-TESTS'
] );
}
public function testNiceDomains() {
global $wgDBname;
if ( wfGetDB( DB_PRIMARY )->databasesAreIndependent() ) {
self::markTestSkipped( "Skipping tests about selecting DBs: not applicable" );
return;
}
$factory = $this->newLBFactoryMulti(
[],
[]
);
$lb = $factory->getMainLB();
$db = $lb->getConnectionRef( DB_PRIMARY );
$this->assertEquals(
WikiMap::getCurrentWikiId(),
$db->getDomainID()
);
unset( $db );
/** @var IMaintainableDatabase $db */
$db = $lb->getConnection( DB_PRIMARY, [], $lb::DOMAIN_ANY );
$this->assertSame(
'',
$db->getDomainID(),
'Null domain ID handle used'
);
$this->assertNull(
$db->getDBname(),
'Null domain ID handle used'
);
$this->assertSame(
'',
$db->tablePrefix(),
'Main domain ID handle used; prefix is empty though'
);
$this->assertEquals(
$this->quoteTable( $db, 'page' ),
$db->tableName( 'page' ),
"Correct full table name"
);
$this->assertEquals(
$this->quoteTable( $db, $wgDBname ) . '.' . $this->quoteTable( $db, 'page' ),
$db->tableName( "$wgDBname.page" ),
"Correct full table name"
);
$this->assertEquals(
$this->quoteTable( $db, 'nice_db' ) . '.' . $this->quoteTable( $db, 'page' ),
$db->tableName( 'nice_db.page' ),
"Correct full table name"
);
unset( $db );
$factory->setLocalDomainPrefix( 'my_' );
$db = $lb->getConnection( DB_PRIMARY ); // local domain connection
$this->assertEquals( $wgDBname, $db->getDBname() );
$this->assertEquals(
"$wgDBname-my_",
$db->getDomainID()
);
$this->assertEquals(
$this->quoteTable( $db, 'my_page' ),
$db->tableName( 'page' ),
"Correct full table name"
);
$this->assertEquals(
$this->quoteTable( $db, 'other_nice_db' ) . '.' . $this->quoteTable( $db, 'page' ),
$db->tableName( 'other_nice_db.page' ),
"Correct full table name"
);
$factory->closeAll( __METHOD__ );
$factory->destroy();
}
public function testTrickyDomain() {
global $wgDBname;
if ( wfGetDB( DB_PRIMARY )->databasesAreIndependent() ) {
self::markTestSkipped( "Skipping tests about selecting DBs: not applicable" );
return;
}
$dbname = 'unittest-domain'; // explodes if DB is selected
$factory = $this->newLBFactoryMulti(
[ 'localDomain' => ( new DatabaseDomain( $dbname, null, '' ) )->getId() ],
[
'dbname' => 'do_not_select_me' // explodes if DB is selected
]
);
$lb = $factory->getMainLB();
/** @var IMaintainableDatabase $db */
$db = $lb->getConnection( DB_PRIMARY, [], $lb::DOMAIN_ANY );
$this->assertSame( '', $db->getDomainID(), "Null domain used" );
$this->assertEquals(
$this->quoteTable( $db, 'page' ),
$db->tableName( 'page' ),
"Correct full table name"
);
$this->assertEquals(
$this->quoteTable( $db, $dbname ) . '.' . $this->quoteTable( $db, 'page' ),
$db->tableName( "$dbname.page" ),
"Correct full table name"
);
$this->assertEquals(
$this->quoteTable( $db, 'nice_db' ) . '.' . $this->quoteTable( $db, 'page' ),
$db->tableName( 'nice_db.page' ),
"Correct full table name"
);
unset( $db );
$factory->setLocalDomainPrefix( 'my_' );
$db = $lb->getConnection( DB_PRIMARY, [], "$wgDBname-my_" );
$this->assertEquals(
$this->quoteTable( $db, 'my_page' ),
$db->tableName( 'page' ),
"Correct full table name"
);
$this->assertEquals(
$this->quoteTable( $db, 'other_nice_db' ) . '.' . $this->quoteTable( $db, 'page' ),
$db->tableName( 'other_nice_db.page' ),
"Correct full table name"
);
$this->assertEquals(
$this->quoteTable( $db, 'garbage-db' ) . '.' . $this->quoteTable( $db, 'page' ),
$db->tableName( 'garbage-db.page' ),
"Correct full table name"
);
$lb->reuseConnection( $db ); // don't care
$factory->closeAll( __METHOD__ );
$factory->destroy();
}
public function testInvalidSelectDB() {
if ( wfGetDB( DB_PRIMARY )->databasesAreIndependent() ) {
$this->markTestSkipped( "Not applicable per databasesAreIndependent()" );
}
$dbname = 'unittest-domain'; // explodes if DB is selected
$factory = $this->newLBFactoryMulti(
[ 'localDomain' => ( new DatabaseDomain( $dbname, null, '' ) )->getId() ],
[
'dbname' => 'do_not_select_me' // explodes if DB is selected
]
);
$lb = $factory->getMainLB();
/** @var IDatabase $db */
$db = $lb->getConnection( DB_PRIMARY, [], $lb::DOMAIN_ANY );
$this->expectException( \Wikimedia\Rdbms\DBUnexpectedError::class );
$db->selectDomain( 'garbagedb' );
}
public function testInvalidSelectDBIndependent() {
$dbname = 'unittest-domain'; // explodes if DB is selected
$factory = $this->newLBFactoryMulti(
[ 'localDomain' => ( new DatabaseDomain( $dbname, null, '' ) )->getId() ],
[
// Explodes with SQLite and Postgres during open/USE
'dbname' => 'bad_dir/do_not_select_me'
]
);
$lb = $factory->getMainLB();
// FIXME: this should probably be lower (T235311)
$this->expectException( \Wikimedia\Rdbms\DBConnectionError::class );
if ( !$factory->getMainLB()->getServerAttributes( 0 )[Database::ATTR_DB_IS_FILE] ) {
$this->markTestSkipped( "Not applicable per ATTR_DB_IS_FILE" );
}
/** @var IDatabase $db */
$this->assertNotNull( $lb->getConnectionInternal( DB_PRIMARY, [], $lb::DOMAIN_ANY ) );
}
public function testInvalidSelectDBIndependent2() {
$dbname = 'unittest-domain'; // explodes if DB is selected
$factory = $this->newLBFactoryMulti(
[ 'localDomain' => ( new DatabaseDomain( $dbname, null, '' ) )->getId() ],
[
// Explodes with SQLite and Postgres during open/USE
'dbname' => 'bad_dir/do_not_select_me'
]
);
$lb = $factory->getMainLB();
// FIXME: this should probably be lower (T235311)
$this->expectException( \Wikimedia\Rdbms\DBExpectedError::class );
if ( !$lb->getConnection( DB_PRIMARY )->databasesAreIndependent() ) {
$this->markTestSkipped( "Not applicable per databasesAreIndependent()" );
}
$db = $lb->getConnectionInternal( DB_PRIMARY );
$db->selectDomain( 'garbage-db' );
}
public function testRedefineLocalDomain() {
global $wgDBname;
if ( wfGetDB( DB_PRIMARY )->databasesAreIndependent() ) {
self::markTestSkipped( "Skipping tests about selecting DBs: not applicable" );
return;
}
$factory = $this->newLBFactoryMulti(
[],
[]
);
$lb = $factory->getMainLB();
$conn1 = $lb->getConnectionRef( DB_PRIMARY );
$this->assertEquals(
WikiMap::getCurrentWikiId(),
$conn1->getDomainID()
);
unset( $conn1 );
$factory->redefineLocalDomain( 'somedb-prefix_' );
$this->assertEquals( 'somedb-prefix_', $factory->getLocalDomainID() );
$domain = new DatabaseDomain( $wgDBname, null, 'pref_' );
$factory->redefineLocalDomain( $domain );
Replace LoadBalancer/LBFactory callback iteration with generators Callback style iteration made sense before generators existed, but generators make for simpler code. The "call method" variants made sense before closures existed but defeat static analysis. So, in LBFactory: * Add ILBFactory::getAllLBs() * Deprecate ILBFactory::forEachLB() * Remove LBFactory::forEachLBCallMethod(), was protected. * Add LBFactory::getLBsForOwner(), which is protected and has the internal interface in @return. Adding a new abstract method breaks Wikibase tests despite LBFactory not being stable to extend. * Migrate callers. Generators allow you to return/break from the middle of the loop, which implies a little rearrangement for some callers. In LoadBalancer, connections supposedly of type IDatabase were routinely type-hinted as Database in closure parameters so that methods could be called that were not in the interface. So it's convenient to get rid of public iteration methods entirely in favour of private methods returning Database[]. * Hard-deprecate ILoadBalancerForOwner::forEachOpenConnection() and replace it with a private generator method since nothing called it externally except for core tests. * Hard-deprecate ILoadBalancerForOwner::forEachOpenPrimaryConnection() and replace it with a private generator. DeferredUpdates needed it for iterating over IDatabase::explicitTrxActive(), so add ILoadBalancer::explicitTrxActive() as a replacement. * Replace private method LoadBalancer::forEachOpenReplicaConnection() with a generator. Depends-On: If0b382231e27d6d1197fb7b6aef6ab50335df4e5 Change-Id: I64514e77b9bfe737be5b12e1d3c9c49976bb522f
2022-04-26 03:54:59 +00:00
/** @var LoadBalancer $lbWrapper */
$lbWrapper = TestingAccessWrapper::newFromObject( $lb );
$n = iterator_count( $lbWrapper->getOpenConnections() );
$this->assertSame( 0, $n, "Connections closed" );
$conn2 = $lb->getConnectionRef( DB_PRIMARY );
$this->assertEquals(
$domain->getId(),
$conn2->getDomainID()
);
unset( $conn2 );
$factory->closeAll( __METHOD__ );
$factory->destroy();
}
private function quoteTable( IReadableDatabase $db, $table ) {
if ( $db->getType() === 'sqlite' ) {
return $table;
} else {
return $db->addIdentifierQuotes( $table );
}
}
public function testCPPosIndexCookieValues() {
$time = 1526522031;
$agentId = md5( 'Ramsey\'s Loyal Presa Canario' );
$this->assertEquals(
'3@542#c47dcfb0566e7d7bc110a6128a45c93a',
LBFactory::makeCookieValueFromCPIndex( 3, 542, $agentId )
);
$lbFactory = $this->newLBFactoryMulti();
$lbFactory->setRequestInfo( [ 'IPAddress' => '10.64.24.52', 'UserAgent' => 'meow' ] );
$this->assertEquals(
'1@542#c47dcfb0566e7d7bc110a6128a45c93a',
LBFactory::makeCookieValueFromCPIndex( 1, 542, $agentId )
);
$this->assertSame(
null,
LBFactory::getCPInfoFromCookieValue( "5#$agentId", $time - 10 )['index'],
'No time set'
);
$this->assertSame(
null,
LBFactory::getCPInfoFromCookieValue( "5@$time", $time - 10 )['index'],
'No agent set'
);
$this->assertSame(
null,
LBFactory::getCPInfoFromCookieValue( "0@$time#$agentId", $time - 10 )['index'],
'Bad index'
);
$this->assertSame(
2,
LBFactory::getCPInfoFromCookieValue( "2@$time#$agentId", $time - 10 )['index'],
'Fresh'
);
$this->assertSame(
2,
LBFactory::getCPInfoFromCookieValue( "2@$time#$agentId", $time + 9 - 10 )['index'],
'Almost stale'
);
$this->assertSame(
null,
LBFactory::getCPInfoFromCookieValue( "0@$time#$agentId", $time + 9 - 10 )['index'],
'Almost stale; bad index'
);
$this->assertSame(
null,
LBFactory::getCPInfoFromCookieValue( "2@$time#$agentId", $time + 11 - 10 )['index'],
'Stale'
);
$this->assertSame(
$agentId,
LBFactory::getCPInfoFromCookieValue( "5@$time#$agentId", $time - 10 )['clientId'],
'Live (client ID)'
);
$this->assertSame(
null,
LBFactory::getCPInfoFromCookieValue( "5@$time#$agentId", $time + 11 - 10 )['clientId'],
'Stale (client ID)'
);
}
public function testSetDomainAliases() {
$lb = $this->newLBFactoryMulti();
$origDomain = $lb->getLocalDomainID();
$this->assertEquals( $origDomain, $lb->resolveDomainID( false ) );
$this->assertEquals( "db-prefix_", $lb->resolveDomainID( "db-prefix_" ) );
$lb->setDomainAliases( [
'alias-db' => 'realdb',
'alias-db-prefix_' => 'realdb-realprefix_'
] );
$this->assertEquals( 'realdb', $lb->resolveDomainID( 'alias-db' ) );
$this->assertEquals( "realdb-realprefix_", $lb->resolveDomainID( "alias-db-prefix_" ) );
}
public function testGetChronologyProtectorTouched() {
$store = new HashBagOStuff;
$lbFactory = $this->newLBFactoryMulti( [
'cpStash' => $store,
'cliMode' => false
] );
$lbFactory->setRequestInfo( [ 'ChronologyClientId' => 'ii' ] );
// 2019-02-05T05:03:20Z
$mockWallClock = 1549343000.0;
$priorTime = $mockWallClock; // reference time
$lbFactory->setMockTime( $mockWallClock );
$lbWrap = TestingAccessWrapper::newFromObject( $lbFactory );
$cpWrap = TestingAccessWrapper::newFromObject( $lbWrap->getChronologyProtector() );
$cpWrap->store->set(
$cpWrap->key,
$cpWrap->mergePositions(
false,
[],
[
$lbFactory::CLUSTER_MAIN_DEFAULT => $priorTime
]
),
3600
);
$mockWallClock += 1.0;
$touched = $lbFactory->getChronologyProtectorTouched();
$this->assertEquals( $priorTime, $touched );
}
public function testReconfigureWithOneReplica() {
$primaryConfig = $this->getPrimaryServerConfig();
$fakeReplica = [ 'load' => 100, 'serverName' => 'replica' ] + $primaryConfig;
$conf = [ 'servers' => [
$primaryConfig,
$fakeReplica
] ];
// Configure an LBFactory with one replica
$factory = new LBFactorySimple( $conf );
$lb = $factory->getMainLB();
$this->assertSame( 2, $lb->getServerCount() );
$con = $lb->getConnectionInternal( DB_REPLICA );
$ref = $lb->getConnection( DB_REPLICA );
// Call reconfigure with the same config, should have no effect
$factory->reconfigure( $conf );
$this->assertSame( 2, $lb->getServerCount() );
$this->assertTrue( $con->isOpen() );
$this->assertTrue( $ref->isOpen() );
// Call reconfigure with empty config, should have no effect
$factory->reconfigure( [] );
$this->assertSame( 2, $lb->getServerCount() );
$this->assertTrue( $con->isOpen() );
$this->assertTrue( $ref->isOpen() );
// Reconfigure the LBFactory to only have a single server.
$conf['servers'] = [ $this->getPrimaryServerConfig() ];
$factory->reconfigure( $conf );
// The LoadBalancer should have been reconfigured automatically.
$this->assertSame( 1, $lb->getServerCount() );
// Reconfiguring should not close connections immediately.
$this->assertTrue( $con->isOpen() );
// Connection refs should detect the config change, close the old connection,
// and get a new connection.
$this->assertTrue( $ref->isOpen() );
$this->assertSame( IDatabase::ROLE_STREAMING_MASTER, $ref->getTopologyRole() );
// The old connection should have been closed by DBConnRef.
$this->assertFalse( $con->isOpen() );
}
public function testReconfigureWithThreeReplicas() {
$primaryConfig = $this->getPrimaryServerConfig();
$replica1Config = [ 'serverName' => 'db2', 'load' => 0 ] + $primaryConfig;
$replica2Config = [ 'serverName' => 'db3', 'load' => 1 ] + $primaryConfig;
$replica3Config = [ 'serverName' => 'db4', 'load' => 1 ] + $primaryConfig;
$conf = [ 'servers' => [
$primaryConfig,
$replica1Config,
$replica2Config,
$replica3Config
] ];
// Configure an LBFactory with two replicas
$factory = new LBFactorySimple( $conf );
$lb = $factory->getMainLB();
$this->assertSame( 4, $lb->getServerCount() );
$this->assertSame( 'db1', $lb->getServerName( 0 ) );
$this->assertSame( 'db2', $lb->getServerName( 1 ) );
$this->assertSame( 'db3', $lb->getServerName( 2 ) );
$this->assertSame( 'db4', $lb->getServerName( 3 ) );
$con = $lb->getConnectionInternal( DB_REPLICA );
$ref = $lb->getConnection( DB_REPLICA );
// Call reconfigure with the same config, should have no effect
$factory->reconfigure( $conf );
$this->assertSame( 4, $lb->getServerCount() );
$this->assertSame( 'db1', $lb->getServerName( 0 ) );
$this->assertSame( 'db2', $lb->getServerName( 1 ) );
$this->assertSame( 'db3', $lb->getServerName( 2 ) );
$this->assertSame( 'db4', $lb->getServerName( 3 ) );
$this->assertTrue( $con->isOpen() );
$this->assertTrue( $ref->isOpen() );
// Call reconfigure with empty config, should have no effect
$factory->reconfigure( [] );
$this->assertSame( 4, $lb->getServerCount() );
$this->assertSame( 'db1', $lb->getServerName( 0 ) );
$this->assertSame( 'db2', $lb->getServerName( 1 ) );
$this->assertSame( 'db3', $lb->getServerName( 2 ) );
$this->assertSame( 'db4', $lb->getServerName( 3 ) );
$this->assertTrue( $con->isOpen() );
$this->assertTrue( $ref->isOpen() );
// Reconfigure the LBFactory to only have a two servers (server indexes shifted).
$conf['servers'] = [ $primaryConfig, $replica2Config, $replica3Config ];
$factory->reconfigure( $conf );
// The LoadBalancer should have been reconfigured automatically.
$this->assertSame( 3, $lb->getServerCount() );
$this->assertSame( 'db1', $lb->getServerName( 0 ) );
$this->assertSame( false, $lb->getServerInfo( 1 ) );
$this->assertSame( 'db3', $lb->getServerName( 2 ) );
$this->assertSame( 'db4', $lb->getServerName( 3 ) );
// Reconfiguring should not close connections immediately.
$this->assertTrue( $con->isOpen() );
// Connection refs should detect the config change, close the old connection,
// and get a new connection.
$this->assertTrue( $ref->isOpen() );
$this->assertSame( IDatabase::ROLE_STREAMING_REPLICA, $ref->getTopologyRole() );
// The old connection should have been closed by DBConnRef.
$this->assertFalse( $con->isOpen() );
}
public function testAutoReconfigure() {
$primaryConfig = $this->getPrimaryServerConfig();
$fakeReplica = [ 'load' => 100, 'serverName' => 'replica1' ] + $primaryConfig;
$conf = [
'servers' => [
$primaryConfig,
$fakeReplica
],
];
// The config callback should return $conf, reflecting changes
// made to the local variable.
$conf['configCallback'] = static function () use ( &$conf ) {
static $calls = 0;
$calls++;
if ( $calls == 1 ) {
return $conf;
} else {
unset( $conf['servers'][1] );
return $conf;
}
};
// Configure an LBFactory with one replica
$factory = new LBFactorySimple( $conf );
$lb = $factory->getMainLB();
$this->assertSame( 2, $lb->getServerCount() );
$con = $lb->getConnectionInternal( DB_REPLICA );
$ref = $lb->getConnection( DB_REPLICA );
// Nothing changed, autoReconfigure() should do nothing.
$factory->autoReconfigure();
$this->assertSame( 2, $lb->getServerCount() );
$this->assertTrue( $con->isOpen() );
$this->assertTrue( $ref->isOpen() );
// Now autoReconfigure() should detect the change and reconfigure all LoadBalancers.
$factory->autoReconfigure();
// The LoadBalancer should have been reconfigured now.
$this->assertSame( 1, $lb->getServerCount() );
// Reconfiguring should not close connections immediately.
$this->assertTrue( $con->isOpen() );
// Connection refs should detect the config change, close the old connection,
// and get a new connection.
$this->assertTrue( $ref->isOpen() );
$this->assertSame( IDatabase::ROLE_STREAMING_MASTER, $ref->getTopologyRole() );
// The old connection should have been called by DBConnRef.
$this->assertFalse( $con->isOpen() );
}
}