Fix a long-standing TODO: instead of each sub-test handling its own
output separately, just make each expose its output via an io.Reader.
Then, the shared driverBase code can tell if any of the lines contain
the magic "gio frame ready" string.
Reduces the amount of code a bit, but most importantly, it keeps the "is
a frame ready?" logic in a single place.
In the future, this also enables us to do more with all the e2e test app
output consistently. For example, we might want to add a -debug flag to
always log output lines as they happen.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Since Wine is heavily tied to X11, we build its end-to-end test driver
on top of X11's. We use the same mechanism to start an X server, take
screenshots, and issue clicks.
Its only quirk is that it was difficult to get the screenshots to line
up with Gio's window. The comments cover what we ended up with. The
display dimensions are now part of driverBase, so that methods other
than Start can also use them - this is necessary for the wine driver to
crop screenshots.
We also use a sleep for now; a comment explains why, and a TODO is left
for future Dan to deal with. What we have now works, and I've spent
enough hours on this patch as it is.
Adding Wine to CI, and ensuring that the test passes there, is left for
a follow-up patch.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
We were using 'go run . <args>' before, which works fine, but does mean
re-linking a new binary and throwing it away at each invocation. Given
that the end-to-end tests don't do all that much work besides building
the tiny red.go app, this amount of extra work was noticeable.
We can obtain statistics for the JS sub-test, which used 'go run', via
the perflock and benchcmd tools:
$ go test -c
$ perflock -governorp% benchcmd EndToEnd/JS ./gogio.test -test.run=EndToEnd/JS
After capturing those numbers before and after the change, we can then
compare them with benchstat. The CPU cost of the subtest is halved:
name old time/op new time/op delta
EndToEnd/JS 1.42s ± 2% 1.07s ± 3% -25.04% (p=0.008 n=5+5)
name old user-time/op new user-time/op delta
EndToEnd/JS 1.46s ± 3% 0.75s ± 5% -48.34% (p=0.008 n=5+5)
name old sys-time/op new sys-time/op delta
EndToEnd/JS 366ms ±13% 224ms ± 7% -38.79% (p=0.008 n=5+5)
An alternative here would have been to refactor main.go to allow being
called directly. However, that would have required a non-trivial
refactor, since flag parsing is done via globals. Given that the
TestMain method is asy and keeps the main function simple, we've
decided to avoid a refactor.
While at it, remove the sleep in the Android driver to wait for the app
to come up on screen. Since we retry screenshots now, we no longer need
a static sleep. On average, we still need one retry for the initial
screenshot, but that's just 100ms versus the old 500ms. The maximum wait
time is also 2s here, which should scale better for slower devices.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
First, move from debian unstable to testing, since sway was promoted to
testing as of earlier this week.
Second, use the --sync option when using xdotool to move an X11 mouse.
This makes the command block until the mouse has finished moving to the
specified location, removing a potential race with the following
'xdotool click' command.
Third, deduplicate some logic into driverBase: tempDir to create a
temporary directory within a test, and needPrograms to skip a test if
the required programs aren't available.
Lastly, split the code that starts the X11 server into a method, so that
the future Wine e2e driver can reuse it. Since Wine is tightly coupled
with X11, we can reuse a good part of the code, including the X11 server
and the xdotool mechanisms.
We also add a TODO to perhaps improve the handling of the app's output
under each of the e2e test cases.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
This type contains all the common bits, such as *testing.T, as well as
the channel and method used to wait for blocking until a frame is ready.
It also allows us to initialise this base separately from Start, which
keeps the exported method simpler to understand.
The base type is embedded into the specific driver types, so that the
code remains simple. While at it, start embedding *testing.T too, so
that we can write d.Fatalf instead of d.t.Fatalf. The drivers will only
have a small number of exported methods as per the interface, so it's
easy to keep those from colliding with the method set on T.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
For example, if the test app fails to start on wayland, we'd block
~forever (ten minutes) waiting for it to render its first frame.
We don't have a good solution right now. But at least we can use a
relatively short timeout, to help out the human who rightfully expects
a result within ten seconds.
While at it, remove a sway "get_seats" command, which was a leftover
from my debugging of what input devices are available when running
headless.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
We already wait for the app to render itself via the "frame ready"
stdout prints. Waiting for the canvas to be ready isn't really necessary
anymore. It helped us while we had to rely on sleeps, but that's no
longer the case.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
The js/wasm bug meant that printing to stdout in the e2e app would crash
the runtime, so we couldn't synchronize with the app like in the other
drivers.
Now that the bug is fixed, we can do the same as the other drivers just
fine.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Now that we use tip due to breaking changes in Go's JS APIs, let's
replace our hacky cleanup list code with 1.14's upcoming
testing.T.Cleanup.
Adding more deliberate uses of tip would ususally be best avoided, but
these will only affect developers working on gio's tests, not regular
users of gio.
While at it, remove some debug t.Logf calls I forgot to remove.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
This means we can deduplicate some of the logic, and keep it all in one
place.
Start expanding the logic too; the tests are slow, so they should be
skipped on 'go test -short'. The ones we have so far all run in a matter
of seconds on an average laptop today, but future tests will probably
require heavier work like wine or kvm.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Right now, we can only run the e2e gio app with -race, not without
-race, because the flag is hard-coded in the tests.
The reason for this change was that 'GOFLAGS=-race go test' would fail
with the JS test, since js/wasm doesn't support the race detector. Fix
that by skipping the JS test when -race is used.
Now, we can run multiple levels of -race:
go test # no -race at all
go test -race # -race for the tests, not the e2e gio app
GOFLAGS=-race go test # -race for everything (best-effort)
To detect the race detector being on, we use a file with a build tag.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
The test app now responds to mouse clicks; clicking on one of the four
sections of the app flips it to red color until clicked again.
Add a Click method to the TestDriver interface, and implement it in both
of the current drivers. Unfortunately, I failed at implementing it in
X11 with the xdg library, after a few wasted hours. Instead, start
relying on more external tools which are simple to use and not heavy to
install.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Now we implement the "red background" end-to-end test exactly once.
While at it, start using a 800x600 window size, which is a bit more
realistic than 600x600, and will catch if we got either dimension wrong.
The interface only has two methods for now, but it will be expanded in
the future to also support input such as clicks.
Keeping state in the test driver, such as a context or a connection, is
a bit awkward but necessary so that we don't have to repeat arguments
over and over. The same applies to testing.T.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
chromedp was defaulting to log.Printf, which is not good for tests.
The xgb and xgbutil logs were suppressed if -v wasn't given, but they
were sent straight to os.Stderr otherwise:
=== RUN TestX11
=== PAUSE TestX11
=== CONT TestX11
XGB: conn.go:47: Could not get authority info: EOF
XGB: conn.go:48: Trying connection without authority info...
--- PASS: TestX11 (0.87s)
Instead, direct their loggers to an io.Writer implementation that sends
its output to t.Logf:
=== RUN TestX11
=== PAUSE TestX11
=== CONT TestX11
TestX11: x11_test.go:187: XGB: conn.go:47: Could not get authority info: EOF
TestX11: x11_test.go:187: XGB: conn.go:48: Trying connection without authority info...
--- PASS: TestX11 (0.86s)
We do end up with duplicate log prefixes, but at least we don't write
straight to stderr, which will be a problem as we add more concurrent
tests.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
On FreeBSD the X11 test seems to succeed except for the alpha
value:
--- FAIL: TestX11 (2.04s)
js_test.go:138: got 0xffff000000000000 at (5,5), want 0xffff00000000ffff
js_test.go:138: got 0xffff000000000000 at (595,595), want 0xffff00000000ffff
FAIL
Ignore alpha values for now.
Signed-off-by: Elias Naur <mail@eliasnaur.com>
Right now it's very similar to the JS test on Chrome. Like it, this one
just runs the "red.go" gio app, takes a screenshot, and expects to see
red.
It also supports the -headless flag; when true, Xvfb is used and it's
entirely headless and hidden. Otherwise, Xephyr is used and once can see
the test in action. If the tool isn't installed, the test is skipped.
We need to add xgb as a dependency, so that we can connect to the X
server and interact with it, like taking screenshots.
Finally, this is an initial version, and a number of TODOs are left for
a later time. They'll get fixed in follow-up patches.
While at it, start making all tests parallel, since the end-to-end tests
take about a second each and neither are very cpu-intensive.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
After some investigation, it turns out that both Alpine and Arch suffer
from the same bug - their packages completely lack SwiftShader.
The current workaround is still the best that we have right now. But at
least we can actually provide a good explanation why, and a TODO to
improve this once the issues filed with the distros are fixed.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
For example, if the browser doesn't have webgl at all, the gio app will
fail to load. This will result in the screenshot being incorrect,
without an apparent reason:
--- FAIL: TestJSOnChrome (0.89s)
js_test.go:122: got 0xffffffffffffffff at (5,5), want 0xffff00000000ffff
js_test.go:122: got 0xffffffffffffffff at (595,595), want 0xffff00000000ffff
The underlying webgl error was accessible if one added a sleep and ran
'go test -headless=false', allowing to open the console and see the
error messages.
Instead, capture them via chromedp and print them to the test's logger:
--- FAIL: TestJSOnChrome (0.89s)
js_test.go:79: console log: "2019/10/29 12:41:07 app: webgl is not supported"
js_test.go:79: console warning: "exit code:", 1
js_test.go:122: got 0xffffffffffffffff at (5,5), want 0xffff00000000ffff
js_test.go:122: got 0xffffffffffffffff at (595,595), want 0xffff00000000ffff
JS Exceptions are a completely different mechanism, so they're not
covered by this patch. We can add them at a later time if needed.
While at it, update to the latest tagged version of chromedp.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
First, add a headless boolean flag that defaults to true. That way, one
can run 'go test -headless=false' to, for example, see how Chrome runs
the webassembly endtoend test.
Second, skip the Chrome test if the browser isn't installed.
While at it, run 'gofmt -s' on the package.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Package ui is now only about units except for the Config.Now method.
Remove Now and rename Config to Converter. Add layout.Config to
replace the old ui.Config.
Signed-off-by: Elias Naur <mail@eliasnaur.com>
This commit adds the first fully end-to-end test. It builds a very
simple Gio app, loads it on Chrome, and checks that it works.
To control Chrome, we use chromedp, a library in pure Go that takes care
of starting the browser and talking to it via the devtools protocol.
We add the test directly in the cmd module, since it mainly interacts
with the gogio tool, and also because the code might turn into some sort
of 'gogio test' command in the future. This does add chromedp and ui as
test dependencies to go.mod, but GOPROXY should allow a 'go get' of
gogio to not download their entire source code archives.
We don't replace ui with ../../ui in the go.mod, to ensure that testing
the cmd module works from anywhere without unintended differences.
The test app being used is inside a testdata directory, to ensure it's
not go-gettable, and that it doesn't otherwise affect the cmd module.
Finally, the test itself is pretty simple. The app just paints a red
background, and the test verifies that, once loaded, the background of
the browser viewport is indeed red.
The test does currently require Chrome or Chromium to be installed,
which is fine for now. It may also require a GPU, though I don't have a
headless machine to check for sure. The test uses Chrome in headless
mode though, so it doesn't open up any visible browser window.
All in all, the test succeeds in just over a second on my laptop with
Chromium 77.0.3865.75.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Elias Naur