Only creating account via the web interface does not work on the beta
cluster, because of CAPTCHA.
Bug: T323824
Change-Id: I7e9c6169f1c72450b4b620a8712629af7637f06b
Wikimedia CI now serves MediaWiki over Apache (T286649) which mean we
can now do concurrent requests!
Change-Id: I5513535ac43138df1215a4896267aaa689bb2f56
dotenv is used only in core. It's usage was introduced into
wdio-defaults.conf.js by mistake in 8a4fab5, while moving functionality
from wdio.conf.js.
This patch moves dotenv usage back to wdio.conf.js, where it should be.
Bug: T283613
Change-Id: I6ec96248cd0b829ef44e574b7e6dc98cf363ee3a
So far, each repository had it's own wdio.conf.js file. That lead to a
lot of duplicated code. This patch moves core's wdio.conf.js file to
wdio-mediawiki package. The file can be used by core, extensions and
skins. Each repository can override settings, if needed.
Bug: T281484
Change-Id: Icd098a2adf8f6183aad5fb6b33196c67fd809e15
The jobs are running a subset of tests. (Tests containing string
`@daily` in the name.) To find the tests we are using
`--mochaOpts.grep` WebdriverIO command line option. The filter is
executed after the tests have already started. Since some test files
have no tests with string `@daily`, they are skipped. That results in
JUnit reporter creating a couple of empty XML files. Jenkins thinks
empty files are failures and fails the build.
This commit creates a suite of files that have tests with string
`@daily` and runs only those files.
Bug: T281505
Change-Id: Id844da81feac49f2a2b6820e98ead98d7e932e9f
JUnit reporter configuration changed since WebdriverIO v6. Before this
patch, the reporter was creating files in format
wdio-*-junit-reporter.log while Jenkins expects files in format
WDIO.xunit*.xml.
Bug: T214686
Change-Id: Iba68a48e3b3f512c5004446b21cdfde6558e8b15
When debugging, it's useful to have screenshots from multiple test runs.
At the moment, file name is created only with test suite name and test
name. When a new test (suite) is executed, screenshots from the previous
one are overwritten.
This patch adds date and time in a slightly modified ISO 8601 format.
(Colons and dots are replaced with dashes.)
Example old file name:
Special%3ARecentChanges-shows-page-creation.png
Example new file name:
Special%3ARecentChanges-shows-page-creation-2021-04-27T12-58-01-822Z.png
Bug: T280970
Change-Id: I90fbcdaf5cb22ea455fd55e718e8db3b3c1dfaea
I've removed `test` argument from afterTest hook by mistake in 2320f00.
It's needed by `filePath()`.
Linters didn't catch this and tests didn't fail, so this mistake went
unnoticed.
Bug: T280355
Change-Id: I1bc142d8282b7a7266233c69d0b27f8700c8f473
There is code in wdio.conf.js to only take screenshots of failed tests.
For a while now, `test.passed` is returning `undefined`, so screenshots
are taken for both passing and failing tests. The change probably
happened when we upgraded webdriverio to v5 or v6, but nobody noticed
it.
I actually like the current behavior. We take videos for both passing
and failing tests, so it makes sense to take screenshots too. Both
screenshots and videos are small files.
(The biggest screenshot is 539 KB, the biggest video is 21 KB.)
Bug: T280355
Change-Id: I8a73afcda19d6dc1cad75256835448488aff37d5
A typical output when running Selenium tests contains a lot of
`Screenshot location...` messages. They are not useful and should be
removed.
Bug: T280343
Change-Id: I462a3764f4a26c4231ff48797c4834ba80a4079d
By default, Mocha will abort the test suite if it takes more than 60
seconds. That is usually plenty when you're just running tests, but it's
usually not enough when you're debugging. At the moment, you have to
manually increase `mochaOpts` `timeout` in `wdio.conf.js`.
This patch checks if DEBUG is set, and if so, increases `timeout` from
60 seconds to 60 minutes.
This patch simplifies debugging. Instead of having to manually edit
`wdio.conf.js`, increasing `timeout` is done by:
DEBUG=true npm run selenium-test
Debugging is documented at
https://www.mediawiki.org/wiki/Selenium/How-to/Debug_with_browser.debug()#mochaOpts
Bug: T278827
Change-Id: Ied8e7eab34d9472b841b803ada6541dd12f85976
Since WebdriverIO v5, Puppeteer is available as a Chromedriver
alternative.
Puppeteer and Chromedriver behave almost the same, but there were a few
minor failures that needed to be fixed.
Puppeteer is bundled with WebdriverIO. Chromedriver needs to be installed
(and started/stopped) separately. Getting rid of Chromedriver simplifies
our documentation, among other things.
The commit updates tests/selenium/wdio.conf.js to use Puppeteer.
It also fixes a few minor problems in tests/selenium/specs/page.js
introduced by the change.
Bug: T269566
Change-Id: I143c44304461980a2be98eb60741184fa7671343
Removed default values for MEDIAWIKI_USER, MEDIAWIKI_PASSWORD, MW_SERVER
and MW_SCRIPT_PATH in wdio.conf.js in order to use values from the .env file.
Bug: T247312
Change-Id: I728c705b476f7741982d7e7c3e0ed219f2fcd054
Earlier versions of WebdriverIO lacked support for video recording.
Starting with WebdriverIO v5, a new package wdio-video-reporter
has been introduced which is used to replace ffmpeg and Xvfb.
Limitations:
- Alert messages and pop-ups do not get displayed in the videos.
New Packages:
- wdio-video-reporter
Bug: T255253
Change-Id: Id443779959bfac55eae42e16e10d1f20c25375bb
This reverts commit babbbbc6e5.
Reason for revert: Video recording only works with WebdriverIO v5. It doesn't work with v6.
Bug: T252523
Change-Id: I5dd0bb5ef03bcf5337207c63d26755e983634787
Earlier versions of WebdriverIO lacked support for video recording.
Starting with WebdriverIO v5, a new package wdio-video-reporter
has been introduced which is used to replace ffmpeg and Xvfb.
Limitations:
- Alert messages and pop-ups do not get displayed in the videos.
Bug: T252523
Change-Id: I6c964467066eb30e28a3435ddf7831a4e296b8f5
This package was serving two purposes:
* Install chromedriver automatically.
* Start chromedriver automatically when using the
'npm run selenium' entry point (e.g. for local development).
For the CI entry points (selenium-test, selenium-daily) the
chromedriver is skipped. For those, we expect the CI runner to
have started a chromedriver server already, which allows the
server to continue running between different the test suites
of different extensions etc.
By letting WDIO start the chromedriver for local development,
we had the benefit of 'npm run selenium' directly being an alias
to the 'wdio' command. This made it easy to pass additional
arguments like --spec or --mochaOpts for running specific tests
only and overall makes things easier to reason about from a
developer perspective with fewer (required) layers of
indirection you see and need to understand when debugging.
But, this has also caused problems. Specifically, it means
developers have to use the same version of Chrome locally
as we specify in package.json which isn't very flexible. While
we could try to keep this up-to-date roughly, that too isn't
an option because the secure environments we recommend for local
use (Quibble and Fresh-node) both use a Debian distro with
typically older versions of Chromium than your local auto-updating
Google Chrome.
For now, let's go back to moving the responsibility of installing
chromedriver together with the responsibility to install Chrome
itself.
== Changes ==
For CI, nothing changes (it already installs both).
For developers using Quibble, Vagrant or Fresh nothing changes
(already installs both).
For developers who make their own secure environment, they will
need to make sure chromedriver is installed, just as we required
between 2017 and Aug 2019 before my commit 1955a8aa56.
== How ==
wdio-chromedriver cannot start a foreign install because the
protocol can change between versions, so unfortunately together
with not using chromedriver from npm, we also can't use
the wdio-chromedriver autostart mechanism. Instead, I'm bringing
back Antoine's shell script that did this prior to ebda397c54.
But, with a few tweaks so that it forwards shell arguments,
this way developers continue to enjoy the benefit of 1955a8aa56,
which is that 'npm run selenium -- --spec foo.js' works.
Bug: T234610
Change-Id: I03ca94315d0611d2b6abccc12cf9076d41b2c00e
The one-var rule only applies to 'var' where it matters that we
don't let the engine implicitly hoist variables leading to
confusing code that may be unsafe to refactor in a straight-forward
manner.
This doesn't apply to ES6 const and let. While I updated the
rule in 1955a8aa5 to allow ES6, I did not enforce it. Fixed now.
When assigning a value, each assignment should be its own
statement (disallow awkward comma separated assignments).
When not assigning a value, and the lines are next to each other,
they should be in one statement. This makes sense for this small
directory and might make sense to propose for the wikimedia preset
at some point.
Note that the rule I added was not required to make the build
pass. The changes I made are allowed by default by the wikimedia
preset for es6+. The rule I added disallows the previous way.
Change-Id: I26cda095a8a4589084e36273038877f2dd1fc50b
* Options no longer needed or no longer exist in wdio v5:
- coloredLogs: Now always on. The underlying 'chalk' library can still
be influenced via the FORCE_COLOR environment variable.
- screenshotPath: Removed. Was already disabled in our config.
- deprecationWarnings: Meh.
- 'sync: true' – On by default when `@wdio/sync` is installed.
The wdio v5 config generator doesn't recommend setting manually.
* The selenium.sh script was removed. It existed to start and stop
chromedriver for local use by developers. This is now done by
the wdio-chromedriver-service. In WMF CI, Quibble starts its own
chromedriver (as optimisation, reused across gated repos),
which is why the 'selenium-test' entry points remains and skips
this.
* The wdio-mediawiki package now requires wdio v5 and Node 10.
This doesn't affect extension repos because versions are pinned.
Upgrade may happen at the earliest convenience.
* Several WDIO methods changed names or signature. Full list at:
<https://github.com/webdriverio/webdriverio/blob/v5.13.2/CHANGELOG.md#v500-2018-12-20>
Highlights:
- browser.element() is now browser.findElement() with "$()" as alias.
- browser.localStorage replaced by browser.setLocalStorage.
- browser.deleteCookie() requires `name` param. To delete all at once,
there is a new method browser.deleteAllCookies().
- Commands that return data no longer wrapped in `{ value: … }`.
Values are now returned directly.
- Custom config keys are now under browser.config instead of browser.options.
Renamed our username/password keys to be mw-prefixed, to avoid clashes
and reduce confusion with similar config keys.
- browser.click(selector) and browser.getText(selector) no longer exist.
Use $(selector).click() or .getText() instead.
* Fix "no such alert" warning from specs/page.js by removing the apparently
redundant code.
Bug: T234002
Bug: T213268
Change-Id: I908997569ca8457997af30cb29e98ac41fae3b64
I removed these in e78484109a because they seemed redundant
given that the library used by wdio-sauce-service already reads
these when present. However, our current version of this package
clobbers this unconditionally with config.user/config.key, thus
not letting it reach that default.
Also add sauceConnect:true which means developers can run
localhost tests against sauce labs if they so choose.
Change-Id: I8d31a23374b2b9b6cd6f1405c7a2f1001da08f1a
Also, remove the redundant unprefixed `user` and `key` options.
These would be cleaner to set through `sauceConnectOpts`
as `sauceConnectOpts { username: …, accesskey: … }`, but even
that isn't needed because wdio-sauce-service already reads these
same environment variables directly. No further config needed.
Also remove the duplication of default variables we don't have
any particular preference for yet so that enjoy improvements to
these when upgrading.
Change-Id: I0d4802f83509b0b49f775560b508d9929ae9979e
So that we can use different framework and let developers easily upgrade
webdriver.io, selenium tests for extensions and skins must now be run
directly at the root of the extension or skin:
cd extensions/Foobar
npm install
npm run-script selenium-test
All repositories had the entry point introduced and Quibble 0.0.34
supports that new scheme. Thus Wikimedia CI runs extensions / skins
tests twice: once via mediawiki/core and once via the 'selenium-test'
script.
Remove the glob specs for extensions / skins from wdio.conf.
Bug: T199116
Change-Id: If19db4e45b56989e2c15240a9ed0e8a55a940fb9
This reverts commit c3878c08b5.
Now that we're running on node10, this will fail until we fix
things.
Bug: T213268
Bug: T222406
Change-Id: I5935fc5d5bc23978e50275d3c99ac870b3b82f49
This is so that we can migrate to node 10 in CI. It should only land
for an hour or so, and is to be reverted as soon as we get CI up to
node 10 and working, lest our suites of selenium tests become
subject to bitrot.
Bug: T222406
Change-Id: I9473e0213c070a0e963913fd81062c6bbf9ec2f2
These test must be run from the extension's repo, because they use a custom
script to modify LocalSettings.php.
Bug: T190829
Change-Id: I920c7faa3ce91f1560f383575aa6a6adfcadd9c7
To match the Minerva repo per I3a48ecb768a, currently
requires duplication of logic here because Quibble ignores
per-repo settings (tracked as T199116).
Bug: T222517
Change-Id: I857458c4fd9185dea03566183f40216252c533cd
1) stdout/stderr log improvement
Our wdio test suite records video of Selenium test session using ffmpeg,
its stdout/stderr are relayed via console.log() however the new line was
not stripped which is confusing and in case the buffer contains multiple
lines, only the first line had the 'ffmpeg stderr: ' prefix.
Ensure each ffmpeg line is prefixed by logging them individually.
2) exit code / signal
childProcess 'close' event is being passed the exit code and the signal.
We only handled the exit code, thus when ffmpeg is send a signal we
logged:
ffmpeg exited with code null
Handle the case when the childProcess is send a signal (typically
SIGINT).
Change-Id: I9f509690baa9b7981399a09448582f45c30800d7
When the wdio configuration has screenshotPath, when a test fail a
screenshot is automatically taken. However there is no support to
specify a filename, we thus went with our own handler in afterTest hook
to have the screenshot names after the full test name.
As a result we had two screenshots taken:
A) CHROME-Error-xxx.png
B) Suite-Sometest.png
Set screenshotPath to null which disable the automatic screenshoting.
Adjust rest of code to use logPath as a base path to save screenshots.
As a side effect, it works around a nasty race condition in the
reporting. The report is generated while some test still has an
afterTest hook running. The report fail to generate and cause nodejs to
exit. The exact fix still has to be figured out though. T216818.
Bug: T216818
Change-Id: I4b53c98bd4de0d9ff850bb883ba899dd1dce0274
In addition to two existing modes (headless and visible), this patch
introcues another mode, headless recording.
If DISPLAY environment variable is not set, Chrome will run in it's
built-in headless mode.
If it is set, and the value starts with colon (`:`), Chrome will
run in headless mode using Xvfb and record video of every test using FFmpeg.
If the value does not start with colon, Chrome will be visible.
Bug: T179188
Change-Id: Ic1723c5f2d57a28201caf6ba7056cb73fb74a957
Similar to custom location of VisualEditor tests, Wikibase has tests in repo and client folders.
Bug: T180228
Change-Id: I3debe8bed9a760aea374529dbe007aa5a72421aa
This is new package will be reusable by other repositories for
their browser tests, without having to reference the internal
selenium/pageobjects/ directory from MediaWiki core.
In addition to not requiring direct imports, it will also avoid
problems in the future by allowing the package to be versioned
and iterated upon without forcing an atomic global upgrade
(or broken master builds), everytime we change something.
See wdio-mediawiki/README for details.
Within MediaWiki core itself, the package is used using the
'file' specifier in its package.json, so that we always test
and develop using its working copy, which makes drafting and
testing changes easier.
Also misc changes to make wdio.conf easier to understand.
Bug: T193088
Change-Id: I547a7899e7a97693a93567dd763784e637433d55
Without this, the tests sometimes fail like this:
> Error: ENOENT: no such file or directory, open './log/should-be-creatable.png'
> at screenshot() - saveScreenshot.js:52:17
> at saveScreenshot("./log/should-be-creatable.png")
This seems to race with the junit plugin, which uses mkdirp to
create it if missing, but the screenshot handling is separate
from that. WebdriverIO's own screenshot handling also does this
so it makes sense for ours to do that, too.
I considered trying to re-use WebdriverIO's save mechanism,
directly but it's not publicly exposed and only used for the
crash scenario, so for now we'll have to keep our own.
Also:
* Add to gitignore.
* Update default to use __dirname instead of './' because the
latter will somtimes be mediawiki/ and sometimes be selenium/
depending on whether you run all tests or some tests.
* Remove trailing slash from default logPath, and instead add the
slash in filePath.
Reason:
- The LOG_DIR used by Jenkins doesn't end in a slash either
(currently not failing because we no longer use that job,
and let quibble run the tests instead, which doesn't set
the LOG_DIR).
- The WDIO docs and example also use screenshotPath without
trailing slash.
- Without this, setting LOG_DIR=/tmp/something results in
filenames like /tmp/somethingexample.png.
Bug: T193088
Change-Id: I6550f9315bae89f96a791f7ae8cc2fbec5ee8dd5
This is functionally a no-op, purely refactoring (mostly style).
* Consistently require packages at the top of a file.
(e.g. MWBot in edit.page.js).
* Remove unused .call(this) from mwbot interaction closures,
which didn't use 'this'.
* Use Node.js regular Promise chaining with then(), instead of
complex bluebird.coroutine generator function yields, which
are intended to emulate async-await, but the syntax is quite
error-prone for inexperienced developers and hard to debug.
Once we require Node 7+ for the selenium tests, we can use
async-await here natively, but until then, might as well use
regular then() syntax, which we already use elsewhere in the
tests, and is also what MWBot documentation uses.
* Also applied some minor whitespace changes for consistency
among these files and other MediaWiki JS. E.g. no empty line
before the first statement of a function. Add a new line between
different methods, and between the end of a class and the
export statement.
* Remove 'use strict' from test files. The patterns that would expose
the bad non-strict behaviour are mostly already forbidden by ESLint,
and the run-time optimisation to disable non-strict can't be noticed
in tests (more useful in prod where e.g. the same process would run
a function 1 million times). Main reason here is to keep things
simple for new-comers and reduce boilerplate, given that these tests
will mainly be worked on by browser-JS developers, not Node.js devs,
and we don't currently use strict mode in our front-end code, either.
* Remove unused bluebird dependency.
Bug: T193088
Change-Id: I59f9211299e8e884c28c7733bcee3b7b28542610
It seems relPath() is used for all other paths here. When I was
running a single test from the /tests/selenium, I noticed wdio
was logging to the console that it excluded ./extensions/CirrusSearch,
which doesn't make sense from that directory. The other paths
are fine.
Change-Id: Ic89ae0a91cdb6590b7190ea44969e653e9e2d6ea
I was unable to get the tests passing locally in MediaWiki-Vagrant.
The main slowdown appears to be from MWBot logging-in for each
command anew. Which works fine, but just seems to be quite slow.
Change-Id: I3c0abc79cac48c26d952b0bbe73574aa48b2d70e
* Remove redundant user/pass for Jenkins, which is unused now
that Jenkins jobs set it through ENV instead.
See also T188520 and Ibb388741fee9 (integration/config).
* Use simple default operator (||) instead of elaborate
undefined check.
* Document that the inline user/password defaults relate to
MediaWiki vagrant.
* Set 'logPath' for Jenkins based on LOG_PATH env (from mw-set-env.sh),
instead of assuming where it will be relative to mediawiki-core.
* Put new lines between blocks and functions.
* Document --no-sandbox hack from 4ff5652501.
Change-Id: I4c7856d260ded8c94e991a27863b80c9eea2e33e
