Fix a long-standing TODO: instead of each sub-test handling its own
output separately, just make each expose its output via an io.Reader.
Then, the shared driverBase code can tell if any of the lines contain
the magic "gio frame ready" string.
Reduces the amount of code a bit, but most importantly, it keeps the "is
a frame ready?" logic in a single place.
In the future, this also enables us to do more with all the e2e test app
output consistently. For example, we might want to add a -debug flag to
always log output lines as they happen.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Since Wine is heavily tied to X11, we build its end-to-end test driver
on top of X11's. We use the same mechanism to start an X server, take
screenshots, and issue clicks.
Its only quirk is that it was difficult to get the screenshots to line
up with Gio's window. The comments cover what we ended up with. The
display dimensions are now part of driverBase, so that methods other
than Start can also use them - this is necessary for the wine driver to
crop screenshots.
We also use a sleep for now; a comment explains why, and a TODO is left
for future Dan to deal with. What we have now works, and I've spent
enough hours on this patch as it is.
Adding Wine to CI, and ensuring that the test passes there, is left for
a follow-up patch.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
First, move from debian unstable to testing, since sway was promoted to
testing as of earlier this week.
Second, use the --sync option when using xdotool to move an X11 mouse.
This makes the command block until the mouse has finished moving to the
specified location, removing a potential race with the following
'xdotool click' command.
Third, deduplicate some logic into driverBase: tempDir to create a
temporary directory within a test, and needPrograms to skip a test if
the required programs aren't available.
Lastly, split the code that starts the X11 server into a method, so that
the future Wine e2e driver can reuse it. Since Wine is tightly coupled
with X11, we can reuse a good part of the code, including the X11 server
and the xdotool mechanisms.
We also add a TODO to perhaps improve the handling of the app's output
under each of the e2e test cases.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Right now, this was badly needed for the wayland subtest, as it seems
like waiting for a frame to be ready wasn't enough for the screenshot to
show what we want. In practice, even if the machine was idle, it could
sometimes take a few extra milliseconds for the app to first appear on
the display.
This was worse when the machine is under stress, which is often the case
with CI. For example, the command below showed a ~20% failure rate on my
laptop with four cores:
go test -c -o test && stress ./test -test.run EndToEnd/Wayland
Add a generic withRetries helper function, which allows us to keep
trying some action up to a timeout, with sleeps in between that start at
100ms and keep doubling until 2s. The function also logs before each
sleep, in case the user is confused why their test is stuck for
potentially may seconds at once.
Refactor the wantColors function into a separate function that returns
an error, as we can no longer directly report errors via *testing.T. It
still reports all the mismatches at once, which is useful. It can now be
used on to pof withRetries with a thin wrapper.
While at it, make the X11 subtest use withRetries to wait for the X
server to be ready. It was using a simpler method with a fixed number of
static sleeps. It's now more consistent, and a bit better overall.
With the changes above, the 'stress' command from earlier can get past
100 runs on my laptop with no failures at all.
Finally, fix a rogue log.Fatal call I had somehow missed.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
This type contains all the common bits, such as *testing.T, as well as
the channel and method used to wait for blocking until a frame is ready.
It also allows us to initialise this base separately from Start, which
keeps the exported method simpler to understand.
The base type is embedded into the specific driver types, so that the
code remains simple. While at it, start embedding *testing.T too, so
that we can write d.Fatalf instead of d.t.Fatalf. The drivers will only
have a small number of exported methods as per the interface, so it's
easy to keep those from colliding with the method set on T.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
For example, if the test app fails to start on wayland, we'd block
~forever (ten minutes) waiting for it to render its first frame.
We don't have a good solution right now. But at least we can use a
relatively short timeout, to help out the human who rightfully expects
a result within ten seconds.
While at it, remove a sway "get_seats" command, which was a leftover
from my debugging of what input devices are available when running
headless.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Now that we use tip due to breaking changes in Go's JS APIs, let's
replace our hacky cleanup list code with 1.14's upcoming
testing.T.Cleanup.
Adding more deliberate uses of tip would ususally be best avoided, but
these will only affect developers working on gio's tests, not regular
users of gio.
While at it, remove some debug t.Logf calls I forgot to remove.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
This means we can deduplicate some of the logic, and keep it all in one
place.
Start expanding the logic too; the tests are slow, so they should be
skipped on 'go test -short'. The ones we have so far all run in a matter
of seconds on an average laptop today, but future tests will probably
require heavier work like wine or kvm.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
In an earlier commit, we made it possible to run the e2e tests with the
race detector enabled everywhere via GOFLAGS=-race go test.
However, that's not at all standard; most users will simply use 'go test
-race'. Moreover, having 'go test -race' run the test program with the
race detector, but not the e2e gio app, is a bit useless.
Instead, have the tests detect when they run with the race detector, and
enable the race detector in the test app too. As before, the JS test is
skipped whenever -race is used.
This also means we can test with -race in the same way in each of the
modules, which simplifies CI.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Right now, we can only run the e2e gio app with -race, not without
-race, because the flag is hard-coded in the tests.
The reason for this change was that 'GOFLAGS=-race go test' would fail
with the JS test, since js/wasm doesn't support the race detector. Fix
that by skipping the JS test when -race is used.
Now, we can run multiple levels of -race:
go test # no -race at all
go test -race # -race for the tests, not the e2e gio app
GOFLAGS=-race go test # -race for everything (best-effort)
To detect the race detector being on, we use a file with a build tag.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
It's not possible to set GOFLAGS=-race because some programs are
built for webassembly where -race is not supported.
Signed-off-by: Elias Naur <mail@eliasnaur.com>
Clicking doesn't quite work yet, but everything else does. We use a
custom sway config to ensure that it's a minimalist setup with no bar or
borders, like the other drivers.
The generic test now adapts to the window's real size when running in
non-headless mode, since tiling window managers resize some drivers like
sway. The default headless mode still expects the exact size that we
specify, as no real windows are at play.
While at it, clean up some now unused code from the x11 file.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Daniel Martí