wiki.techinc.nl/tests/phpunit/includes/debug/logger/monolog/KafkaHandlerTest.php

228 lines
6.4 KiB
PHP
Raw Normal View History

Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
<?php
/**
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License along
* with this program; if not, write to the Free Software Foundation, Inc.,
* 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
* http://www.gnu.org/copyleft/gpl.html
*
* @file
*/
namespace MediaWiki\Logger\Monolog;
use MediaWikiTestCase;
use Monolog\Logger;
use Wikimedia\TestingAccessWrapper;
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
/**
* @covers \MediaWiki\Logger\Monolog\KafkaHandler
*/
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
class KafkaHandlerTest extends MediaWikiTestCase {
protected function setUp() {
if ( !class_exists( 'Monolog\Handler\AbstractProcessingHandler' )
|| !class_exists( 'Kafka\Produce' )
) {
$this->markTestSkipped( 'Monolog and Kafka are required for the KafkaHandlerTest' );
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
}
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
parent::setUp();
}
public function topicNamingProvider() {
return [
[ [], 'monolog_foo' ],
[ [ 'alias' => [ 'foo' => 'bar' ] ], 'bar' ]
];
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
}
/**
* @dataProvider topicNamingProvider
*/
public function testTopicNaming( $options, $expect ) {
$produce = $this->getMockBuilder( 'Kafka\Produce' )
->disableOriginalConstructor()
->getMock();
$produce->expects( $this->any() )
->method( 'getAvailablePartitions' )
->will( $this->returnValue( [ 'A' ] ) );
$produce->expects( $this->once() )
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
->method( 'setMessages' )
->with( $expect, $this->anything(), $this->anything() );
$produce->expects( $this->any() )
->method( 'send' )
->will( $this->returnValue( true ) );
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
$handler = new KafkaHandler( $produce, $options );
$handler->handle( [
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
'channel' => 'foo',
'level' => Logger::EMERGENCY,
'extra' => [],
'context' => [],
] );
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
}
public function swallowsExceptionsWhenRequested() {
return [
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
// defaults to false
[ [], true ],
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
// also try false explicitly
[ [ 'swallowExceptions' => false ], true ],
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
// turn it on
[ [ 'swallowExceptions' => true ], false ],
];
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
}
/**
* @dataProvider swallowsExceptionsWhenRequested
*/
public function testGetAvailablePartitionsException( $options, $expectException ) {
$produce = $this->getMockBuilder( 'Kafka\Produce' )
->disableOriginalConstructor()
->getMock();
$produce->expects( $this->any() )
->method( 'getAvailablePartitions' )
->will( $this->throwException( new \Kafka\Exception ) );
$produce->expects( $this->any() )
->method( 'send' )
->will( $this->returnValue( true ) );
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
if ( $expectException ) {
$this->setExpectedException( 'Kafka\Exception' );
}
$handler = new KafkaHandler( $produce, $options );
$handler->handle( [
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
'channel' => 'foo',
'level' => Logger::EMERGENCY,
'extra' => [],
'context' => [],
] );
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
if ( !$expectException ) {
$this->assertTrue( true, 'no exception was thrown' );
}
}
/**
* @dataProvider swallowsExceptionsWhenRequested
*/
public function testSendException( $options, $expectException ) {
$produce = $this->getMockBuilder( 'Kafka\Produce' )
->disableOriginalConstructor()
->getMock();
$produce->expects( $this->any() )
->method( 'getAvailablePartitions' )
->will( $this->returnValue( [ 'A' ] ) );
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
$produce->expects( $this->any() )
->method( 'send' )
->will( $this->throwException( new \Kafka\Exception ) );
if ( $expectException ) {
$this->setExpectedException( 'Kafka\Exception' );
}
$handler = new KafkaHandler( $produce, $options );
$handler->handle( [
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
'channel' => 'foo',
'level' => Logger::EMERGENCY,
'extra' => [],
'context' => [],
] );
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
if ( !$expectException ) {
$this->assertTrue( true, 'no exception was thrown' );
}
}
public function testHandlesNullFormatterResult() {
$produce = $this->getMockBuilder( 'Kafka\Produce' )
->disableOriginalConstructor()
->getMock();
$produce->expects( $this->any() )
->method( 'getAvailablePartitions' )
->will( $this->returnValue( [ 'A' ] ) );
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
$mockMethod = $produce->expects( $this->exactly( 2 ) )
->method( 'setMessages' );
$produce->expects( $this->any() )
->method( 'send' )
->will( $this->returnValue( true ) );
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
// evil hax
$matcher = TestingAccessWrapper::newFromObject( $mockMethod )->matcher;
TestingAccessWrapper::newFromObject( $matcher )->parametersMatcher =
new \PHPUnit_Framework_MockObject_Matcher_ConsecutiveParameters( [
[ $this->anything(), $this->anything(), [ 'words' ] ],
[ $this->anything(), $this->anything(), [ 'lines' ] ]
] );
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
$formatter = $this->createMock( \Monolog\Formatter\FormatterInterface::class );
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
$formatter->expects( $this->any() )
->method( 'format' )
->will( $this->onConsecutiveCalls( 'words', null, 'lines' ) );
$handler = new KafkaHandler( $produce, [] );
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
$handler->setFormatter( $formatter );
for ( $i = 0; $i < 3; ++$i ) {
$handler->handle( [
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
'channel' => 'foo',
'level' => Logger::EMERGENCY,
'extra' => [],
'context' => [],
] );
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
}
}
public function testBatchHandlesNullFormatterResult() {
$produce = $this->getMockBuilder( 'Kafka\Produce' )
->disableOriginalConstructor()
->getMock();
$produce->expects( $this->any() )
->method( 'getAvailablePartitions' )
->will( $this->returnValue( [ 'A' ] ) );
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
$produce->expects( $this->once() )
->method( 'setMessages' )
->with( $this->anything(), $this->anything(), [ 'words', 'lines' ] );
$produce->expects( $this->any() )
->method( 'send' )
->will( $this->returnValue( true ) );
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
$formatter = $this->createMock( \Monolog\Formatter\FormatterInterface::class );
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
$formatter->expects( $this->any() )
->method( 'format' )
->will( $this->onConsecutiveCalls( 'words', null, 'lines' ) );
$handler = new KafkaHandler( $produce, [] );
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
$handler->setFormatter( $formatter );
$handler->handleBatch( [
[
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
'channel' => 'foo',
'level' => Logger::EMERGENCY,
'extra' => [],
'context' => [],
],
[
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
'channel' => 'foo',
'level' => Logger::EMERGENCY,
'extra' => [],
'context' => [],
],
[
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
'channel' => 'foo',
'level' => Logger::EMERGENCY,
'extra' => [],
'context' => [],
],
] );
Produce monolog messages through kafka+avro This allows a logging channel to be configured to write directly to kafka. Logs can be serialized either to json blobs or the more compact apache avro format. The Kafka handler for monolog needs a list of one of more kafka servers to query cluster metadata from. This should be able to use any monolog formatter, although some like JsonFormatter require you to disable formatBatch as Kafka protocol would prefer to encode each record independently in the protocol. This requires the nmred/kafka-php library, version >= 1.3.0. Adds a new formatter which serializes to the apache avro format. This is a compact binary format which uses pre- defined schemas. This initial implementation is very simple and takes the plain schemas as a constructor argument. Adds a new option to MonologSpi to wrap handlers in a BufferHandler. This doesn't flush until the request shuts down and prevents any network requests in the logger from adding latency to web requests. Related mediawiki/vendor update: Ibfe4bd2036ae8e998e2973f07bd9a6f057691578 The necessary config is something like: array( 'loggers' => array( 'CirrusSearchRequests' => array( 'handlers' => array( 'kafka' ), ), ), 'handlers' => array( 'kafka' => array( 'factory' => '\\MediaWiki\\Logger\\Monolog\\KafkaHandler::factory', 'args' => array( 'localhost:9092' ), 'formatter' => 'avro', 'buffer' => true, ), ), 'formatters' => array( 'avro' => array( 'class' => '\\MediaWiki\\Logger\\Monolog\\AvroFormatter', 'args' => array( array( 'CirrusSearchRequests' => array( 'type' => 'record', 'name' => 'CirrusSearchRequests' 'fields' => array( ... ) ), ), ), ), ), ) Bug: T106256 Change-Id: I6ee744b3e5306af0bed70811b558a543eed22840
2015-08-04 18:02:47 +00:00
}
}