Right now, this was badly needed for the wayland subtest, as it seems
like waiting for a frame to be ready wasn't enough for the screenshot to
show what we want. In practice, even if the machine was idle, it could
sometimes take a few extra milliseconds for the app to first appear on
the display.
This was worse when the machine is under stress, which is often the case
with CI. For example, the command below showed a ~20% failure rate on my
laptop with four cores:
go test -c -o test && stress ./test -test.run EndToEnd/Wayland
Add a generic withRetries helper function, which allows us to keep
trying some action up to a timeout, with sleeps in between that start at
100ms and keep doubling until 2s. The function also logs before each
sleep, in case the user is confused why their test is stuck for
potentially may seconds at once.
Refactor the wantColors function into a separate function that returns
an error, as we can no longer directly report errors via *testing.T. It
still reports all the mismatches at once, which is useful. It can now be
used on to pof withRetries with a thin wrapper.
While at it, make the X11 subtest use withRetries to wait for the X
server to be ready. It was using a simpler method with a fixed number of
static sleeps. It's now more consistent, and a bit better overall.
With the changes above, the 'stress' command from earlier can get past
100 runs on my laptop with no failures at all.
Finally, fix a rogue log.Fatal call I had somehow missed.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
This type contains all the common bits, such as *testing.T, as well as
the channel and method used to wait for blocking until a frame is ready.
It also allows us to initialise this base separately from Start, which
keeps the exported method simpler to understand.
The base type is embedded into the specific driver types, so that the
code remains simple. While at it, start embedding *testing.T too, so
that we can write d.Fatalf instead of d.t.Fatalf. The drivers will only
have a small number of exported methods as per the interface, so it's
easy to keep those from colliding with the method set on T.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
It passes the whole e2e test flow on my real device, a OnePlus 5 running
LineageOS 16.0 (Android 9).
I was also successful at running it against an x86-64 Android 8.0
emulator, but I'm not including any of that just yet. A patch later this
week will include a piece of code to set up and start an emulator, which
CI can then use to run the test.
Also stop requiring the screen dimensions to be enforced when running in
non-headless mode. An Android emulator can run at an arbitrary
resolution, and even in headless mode, but a real Android device will
have its own predefined resolution. Forcing the test user to set the
-headless=false flag to not get annoying "unexpected dimensions" errors
would be annoying.
That check doesn't really mean much, as our test app doesn't care about
the screen resolution. And we were only doing the check sometimes. Drop
it entirely, making the resolution parameters merely a hint so that we
can keep the drivers a bit more consistent.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
I've seen a couple of "timed out waiting for a frame to be ready" errors
on CI in the past week. I think two seconds is a bit low, if the machine
in question is not very powerful or under stress - which is common for
CI environments.
Raise the timeout to 5s.
While at it, add some log lines to each e2e test, and mark waitForFrame
as a helper func, so that its errors show up at the caller's location.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Newer Windows NDKs add the "x86_64" platform suffix like other
OS'es. Remove the special case.
Newer NDKs are also installed in "ndk/x.y.z" versioned directories
instead of in "ndk-bundle". Support that.
Signed-off-by: Elias Naur <mail@eliasnaur.com>
For example, if the test app fails to start on wayland, we'd block
~forever (ten minutes) waiting for it to render its first frame.
We don't have a good solution right now. But at least we can use a
relatively short timeout, to help out the human who rightfully expects
a result within ten seconds.
While at it, remove a sway "get_seats" command, which was a leftover
from my debugging of what input devices are available when running
headless.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
We already wait for the app to render itself via the "frame ready"
stdout prints. Waiting for the canvas to be ready isn't really necessary
anymore. It helped us while we had to rely on sleeps, but that's no
longer the case.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
The js/wasm bug meant that printing to stdout in the e2e app would crash
the runtime, so we couldn't synchronize with the app like in the other
drivers.
Now that the bug is fixed, we can do the same as the other drivers just
fine.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Now that we use tip due to breaking changes in Go's JS APIs, let's
replace our hacky cleanup list code with 1.14's upcoming
testing.T.Cleanup.
Adding more deliberate uses of tip would ususally be best avoided, but
these will only affect developers working on gio's tests, not regular
users of gio.
While at it, remove some debug t.Logf calls I forgot to remove.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
WebAssembly builds require Go 1.14 since the breaking change in Go,
golang.org/cl/203600.
gofmt -w -s . as well.
Signed-off-by: Elias Naur <mail@eliasnaur.com>
This means we can deduplicate some of the logic, and keep it all in one
place.
Start expanding the logic too; the tests are slow, so they should be
skipped on 'go test -short'. The ones we have so far all run in a matter
of seconds on an average laptop today, but future tests will probably
require heavier work like wine or kvm.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
In an earlier commit, we made it possible to run the e2e tests with the
race detector enabled everywhere via GOFLAGS=-race go test.
However, that's not at all standard; most users will simply use 'go test
-race'. Moreover, having 'go test -race' run the test program with the
race detector, but not the e2e gio app, is a bit useless.
Instead, have the tests detect when they run with the race detector, and
enable the race detector in the test app too. As before, the JS test is
skipped whenever -race is used.
This also means we can test with -race in the same way in each of the
modules, which simplifies CI.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Spotted the javac one by chance when reading the code, so I ran the tool
and fixed another small bug while at it.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Right now, we can only run the e2e gio app with -race, not without
-race, because the flag is hard-coded in the tests.
The reason for this change was that 'GOFLAGS=-race go test' would fail
with the JS test, since js/wasm doesn't support the race detector. Fix
that by skipping the JS test when -race is used.
Now, we can run multiple levels of -race:
go test # no -race at all
go test -race # -race for the tests, not the e2e gio app
GOFLAGS=-race go test # -race for everything (best-effort)
To detect the race detector being on, we use a file with a build tag.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
It's not possible to set GOFLAGS=-race because some programs are
built for webassembly where -race is not supported.
Signed-off-by: Elias Naur <mail@eliasnaur.com>
Clicking doesn't quite work yet, but everything else does. We use a
custom sway config to ensure that it's a minimalist setup with no bar or
borders, like the other drivers.
The generic test now adapts to the window's real size when running in
non-headless mode, since tiling window managers resize some drivers like
sway. The default headless mode still expects the exact size that we
specify, as no real windows are at play.
While at it, clean up some now unused code from the x11 file.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
We can instead synchronize with the gio app via stdout. We need three
states, since we need to first invalidate a frame and then print when
the next frame is drawn.
This is not happening on the JS test yet, because stdout printing
crashes in that case. See the comment.
This change should make the X11 test a bit faster on fast machines,
while making it more stable in small or headless machines like CI.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Instead rely on more tiny standalone tools. In this case, scrot lets us
take png screenshots, and works well.
On the plus side, we remove some large X Go deps, and we don't need
nearly as much code.
While at it, skip if any of the tools are missing, and actually defer
the cleanup funcs so that they run when we fail the test early.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
The test app now responds to mouse clicks; clicking on one of the four
sections of the app flips it to red color until clicked again.
Add a Click method to the TestDriver interface, and implement it in both
of the current drivers. Unfortunately, I failed at implementing it in
X11 with the xdg library, after a few wasted hours. Instead, start
relying on more external tools which are simple to use and not heavy to
install.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Elias Naur