First, use wineboot instead of winecfg to set up the WINEPREFIX. It's
the right tool for it.
Second, when initialising WINEPREFIX, use output pipes instead of
CombinedOutput. The latter will wait for *all* output to be copied to a
buffer, including the output from grandchildren processes. Since wine
starts wineserver automatically, and wineserver lingers for three
seconds by default, this is bad. We would waste three seconds waiting
for wineserver doing nothing, and then the next wine call (to start the
app) would need to start wineserver all over.
Instead, with pipes we can get cmd.Run to only wait for the parent
process to finish. wineserver stays running but we don't care. And, when
we start the gio app, we very likely reuse the same wineserver process.
Third, disable wine-gecko and wine-mono. This ensures we don't get stuck
if they're not installed, and speeds up wineboot by avoiding work we
don't need.
The time to set up WINEPREFIX goes down form ~6s to ~1s, and the overall
subtest run-time goes down from ~10s to ~3s.
Finally, copiously document all of the precious data I've gathered above
after hours of debugging.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Fix a long-standing TODO: instead of each sub-test handling its own
output separately, just make each expose its output via an io.Reader.
Then, the shared driverBase code can tell if any of the lines contain
the magic "gio frame ready" string.
Reduces the amount of code a bit, but most importantly, it keeps the "is
a frame ready?" logic in a single place.
In the future, this also enables us to do more with all the e2e test app
output consistently. For example, we might want to add a -debug flag to
always log output lines as they happen.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
This way, if the user has a custom winecfg, it can't possibly affect the
tests. I was encountering this as DXVK does not work on virtual Xorg
servers (which we use), and Gio thus failed to render on such a
combination.
>From the numbers below, it can be seen that setting up a new WINEPREFIX
takes roughly five seconds:
$ rm -rf ~/.cache/gio-e2e-wine
$ go test -run EndToEnd/Windows
PASS
ok gioui.org/cmd/gogio 16.369s
$ go test -run EndToEnd/Windows
PASS
ok gioui.org/cmd/gogio 11.810s
A repeated run still has a slow "wine winecfg /?", for some reason. Add
a TODO since I can see it taking a third of the time on my terminal. I
haven't been able to properly investigate why, unfortunately. As far as
I can tell, winecfg is just faster when run with a terminal instead of
an output buffer. They might use isatty on stdout/stderr.
The overall time to run the wine sub-test is increased from ~5s to ~11s,
but it's worth it to make it run everywhere. It looks like there is
plenty of room as per the TODO above, as winecfg seems to mostly do
nothing. We're also not too worried, as all e2e subtests run in
parallel.
Fixes#106.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
This way, a Gio app's logs can be filtered uniquely, which wasn't
possible before since the tag "gio" would be the same between gio apps.
Since the app ID is supplied at build time, inject it via a variable
with the linker's help. The variable is only used on Android for now,
but that's OK. It might be useful for other platforms or other internal
packages in the future.
Fixes#84.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
While launching an app on Android, the hard-coded theme is used for
the color. That color is by default black, and results in a jarring
transition to the actual app background.
The default Gio color is white, so use that for the theme background
as well.
This change will break for "dark mode" programs and similar. A future
improvement would be to reflect the actual app background in the theme.
Signed-off-by: Elias Naur <mail@eliasnaur.com>
Not strictly necessary, but makes embedding a Gio Android Activity
easier; adding
<activity android:name="org.gioui.GioActivity"
android:theme="@style/Theme.GioApp"
android:configChanges="orientation|keyboardHidden"
android:windowSoftInputMode="adjustResize">
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
</activity>
should be enough.
Signed-off-by: Elias Naur <mail@eliasnaur.com>
Installing it on Debian was enough, with the only wrinkle that
propagating -race won't work when we're cross-compiling, since
cross-compilation disables CGo by default.
For now, just skip the test in that edge case. If we want to use the
race detector on Windows in the future, we need to get a Windows CI
builder somehow.
Tested on my fork; see https://builds.sr.ht/~mvdan/job/164899.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Since Wine is heavily tied to X11, we build its end-to-end test driver
on top of X11's. We use the same mechanism to start an X server, take
screenshots, and issue clicks.
Its only quirk is that it was difficult to get the screenshots to line
up with Gio's window. The comments cover what we ended up with. The
display dimensions are now part of driverBase, so that methods other
than Start can also use them - this is necessary for the wine driver to
crop screenshots.
We also use a sleep for now; a comment explains why, and a TODO is left
for future Dan to deal with. What we have now works, and I've spent
enough hours on this patch as it is.
Adding Wine to CI, and ensuring that the test passes there, is left for
a follow-up patch.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
We were using 'go run . <args>' before, which works fine, but does mean
re-linking a new binary and throwing it away at each invocation. Given
that the end-to-end tests don't do all that much work besides building
the tiny red.go app, this amount of extra work was noticeable.
We can obtain statistics for the JS sub-test, which used 'go run', via
the perflock and benchcmd tools:
$ go test -c
$ perflock -governorp% benchcmd EndToEnd/JS ./gogio.test -test.run=EndToEnd/JS
After capturing those numbers before and after the change, we can then
compare them with benchstat. The CPU cost of the subtest is halved:
name old time/op new time/op delta
EndToEnd/JS 1.42s ± 2% 1.07s ± 3% -25.04% (p=0.008 n=5+5)
name old user-time/op new user-time/op delta
EndToEnd/JS 1.46s ± 3% 0.75s ± 5% -48.34% (p=0.008 n=5+5)
name old sys-time/op new sys-time/op delta
EndToEnd/JS 366ms ±13% 224ms ± 7% -38.79% (p=0.008 n=5+5)
An alternative here would have been to refactor main.go to allow being
called directly. However, that would have required a non-trivial
refactor, since flag parsing is done via globals. Given that the
TestMain method is asy and keeps the main function simple, we've
decided to avoid a refactor.
While at it, remove the sleep in the Android driver to wait for the app
to come up on screen. Since we retry screenshots now, we no longer need
a static sleep. On average, we still need one retry for the initial
screenshot, but that's just 100ms versus the old 500ms. The maximum wait
time is also 2s here, which should scale better for slower devices.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
First, move from debian unstable to testing, since sway was promoted to
testing as of earlier this week.
Second, use the --sync option when using xdotool to move an X11 mouse.
This makes the command block until the mouse has finished moving to the
specified location, removing a potential race with the following
'xdotool click' command.
Third, deduplicate some logic into driverBase: tempDir to create a
temporary directory within a test, and needPrograms to skip a test if
the required programs aren't available.
Lastly, split the code that starts the X11 server into a method, so that
the future Wine e2e driver can reuse it. Since Wine is tightly coupled
with X11, we can reuse a good part of the code, including the X11 server
and the xdotool mechanisms.
We also add a TODO to perhaps improve the handling of the app's output
under each of the e2e test cases.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Right now, this was badly needed for the wayland subtest, as it seems
like waiting for a frame to be ready wasn't enough for the screenshot to
show what we want. In practice, even if the machine was idle, it could
sometimes take a few extra milliseconds for the app to first appear on
the display.
This was worse when the machine is under stress, which is often the case
with CI. For example, the command below showed a ~20% failure rate on my
laptop with four cores:
go test -c -o test && stress ./test -test.run EndToEnd/Wayland
Add a generic withRetries helper function, which allows us to keep
trying some action up to a timeout, with sleeps in between that start at
100ms and keep doubling until 2s. The function also logs before each
sleep, in case the user is confused why their test is stuck for
potentially may seconds at once.
Refactor the wantColors function into a separate function that returns
an error, as we can no longer directly report errors via *testing.T. It
still reports all the mismatches at once, which is useful. It can now be
used on to pof withRetries with a thin wrapper.
While at it, make the X11 subtest use withRetries to wait for the X
server to be ready. It was using a simpler method with a fixed number of
static sleeps. It's now more consistent, and a bit better overall.
With the changes above, the 'stress' command from earlier can get past
100 runs on my laptop with no failures at all.
Finally, fix a rogue log.Fatal call I had somehow missed.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
This type contains all the common bits, such as *testing.T, as well as
the channel and method used to wait for blocking until a frame is ready.
It also allows us to initialise this base separately from Start, which
keeps the exported method simpler to understand.
The base type is embedded into the specific driver types, so that the
code remains simple. While at it, start embedding *testing.T too, so
that we can write d.Fatalf instead of d.t.Fatalf. The drivers will only
have a small number of exported methods as per the interface, so it's
easy to keep those from colliding with the method set on T.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
It passes the whole e2e test flow on my real device, a OnePlus 5 running
LineageOS 16.0 (Android 9).
I was also successful at running it against an x86-64 Android 8.0
emulator, but I'm not including any of that just yet. A patch later this
week will include a piece of code to set up and start an emulator, which
CI can then use to run the test.
Also stop requiring the screen dimensions to be enforced when running in
non-headless mode. An Android emulator can run at an arbitrary
resolution, and even in headless mode, but a real Android device will
have its own predefined resolution. Forcing the test user to set the
-headless=false flag to not get annoying "unexpected dimensions" errors
would be annoying.
That check doesn't really mean much, as our test app doesn't care about
the screen resolution. And we were only doing the check sometimes. Drop
it entirely, making the resolution parameters merely a hint so that we
can keep the drivers a bit more consistent.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
I've seen a couple of "timed out waiting for a frame to be ready" errors
on CI in the past week. I think two seconds is a bit low, if the machine
in question is not very powerful or under stress - which is common for
CI environments.
Raise the timeout to 5s.
While at it, add some log lines to each e2e test, and mark waitForFrame
as a helper func, so that its errors show up at the caller's location.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Newer Windows NDKs add the "x86_64" platform suffix like other
OS'es. Remove the special case.
Newer NDKs are also installed in "ndk/x.y.z" versioned directories
instead of in "ndk-bundle". Support that.
Signed-off-by: Elias Naur <mail@eliasnaur.com>
For example, if the test app fails to start on wayland, we'd block
~forever (ten minutes) waiting for it to render its first frame.
We don't have a good solution right now. But at least we can use a
relatively short timeout, to help out the human who rightfully expects
a result within ten seconds.
While at it, remove a sway "get_seats" command, which was a leftover
from my debugging of what input devices are available when running
headless.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
We already wait for the app to render itself via the "frame ready"
stdout prints. Waiting for the canvas to be ready isn't really necessary
anymore. It helped us while we had to rely on sleeps, but that's no
longer the case.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
The js/wasm bug meant that printing to stdout in the e2e app would crash
the runtime, so we couldn't synchronize with the app like in the other
drivers.
Now that the bug is fixed, we can do the same as the other drivers just
fine.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Now that we use tip due to breaking changes in Go's JS APIs, let's
replace our hacky cleanup list code with 1.14's upcoming
testing.T.Cleanup.
Adding more deliberate uses of tip would ususally be best avoided, but
these will only affect developers working on gio's tests, not regular
users of gio.
While at it, remove some debug t.Logf calls I forgot to remove.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
WebAssembly builds require Go 1.14 since the breaking change in Go,
golang.org/cl/203600.
gofmt -w -s . as well.
Signed-off-by: Elias Naur <mail@eliasnaur.com>
This means we can deduplicate some of the logic, and keep it all in one
place.
Start expanding the logic too; the tests are slow, so they should be
skipped on 'go test -short'. The ones we have so far all run in a matter
of seconds on an average laptop today, but future tests will probably
require heavier work like wine or kvm.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
In an earlier commit, we made it possible to run the e2e tests with the
race detector enabled everywhere via GOFLAGS=-race go test.
However, that's not at all standard; most users will simply use 'go test
-race'. Moreover, having 'go test -race' run the test program with the
race detector, but not the e2e gio app, is a bit useless.
Instead, have the tests detect when they run with the race detector, and
enable the race detector in the test app too. As before, the JS test is
skipped whenever -race is used.
This also means we can test with -race in the same way in each of the
modules, which simplifies CI.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Spotted the javac one by chance when reading the code, so I ran the tool
and fixed another small bug while at it.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Right now, we can only run the e2e gio app with -race, not without
-race, because the flag is hard-coded in the tests.
The reason for this change was that 'GOFLAGS=-race go test' would fail
with the JS test, since js/wasm doesn't support the race detector. Fix
that by skipping the JS test when -race is used.
Now, we can run multiple levels of -race:
go test # no -race at all
go test -race # -race for the tests, not the e2e gio app
GOFLAGS=-race go test # -race for everything (best-effort)
To detect the race detector being on, we use a file with a build tag.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
It's not possible to set GOFLAGS=-race because some programs are
built for webassembly where -race is not supported.
Signed-off-by: Elias Naur <mail@eliasnaur.com>
Clicking doesn't quite work yet, but everything else does. We use a
custom sway config to ensure that it's a minimalist setup with no bar or
borders, like the other drivers.
The generic test now adapts to the window's real size when running in
non-headless mode, since tiling window managers resize some drivers like
sway. The default headless mode still expects the exact size that we
specify, as no real windows are at play.
While at it, clean up some now unused code from the x11 file.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
We can instead synchronize with the gio app via stdout. We need three
states, since we need to first invalidate a frame and then print when
the next frame is drawn.
This is not happening on the JS test yet, because stdout printing
crashes in that case. See the comment.
This change should make the X11 test a bit faster on fast machines,
while making it more stable in small or headless machines like CI.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Instead rely on more tiny standalone tools. In this case, scrot lets us
take png screenshots, and works well.
On the plus side, we remove some large X Go deps, and we don't need
nearly as much code.
While at it, skip if any of the tools are missing, and actually defer
the cleanup funcs so that they run when we fail the test early.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
The test app now responds to mouse clicks; clicking on one of the four
sections of the app flips it to red color until clicked again.
Add a Click method to the TestDriver interface, and implement it in both
of the current drivers. Unfortunately, I failed at implementing it in
X11 with the xdg library, after a few wasted hours. Instead, start
relying on more external tools which are simple to use and not heavy to
install.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Search for imports of the form gioui.org/app/permission/* and add
required permissions to AndroidManifest.xml.
Signed-off-by: Greg Pomerantz <gmp.gio@wow.st>
On Android, in addition to adding jar files found in the source
directory of the program being compiled, cmd/gogio also searches
every dependency for jar files to include in the output APK.
Signed-off-by: Greg Pomerantz <gmp.gio@wow.st>
This vastly simplifies our code, and saves us the ugly math.
While at it, establish that a TestDriver must have a white background,
which is already satisfied by both existing implementations.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
The e2e app now splits the window into four rectangles and paints them
differently.
The first advantage is that we now test that we see the entire Gio app.
Before, with a solid background color, we could be seeing a small part
of the window and we wouldn't tell the difference.
The second advantage is that we test more colors. In particular, the
fourth color includes a different alpha value, which renders the same on
JS and X11.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Now we implement the "red background" end-to-end test exactly once.
While at it, start using a 800x600 window size, which is a bit more
realistic than 600x600, and will catch if we got either dimension wrong.
The interface only has two methods for now, but it will be expanded in
the future to also support input such as clicks.
Keeping state in the test driver, such as a context or a connection, is
a bit awkward but necessary so that we don't have to repeat arguments
over and over. The same applies to testing.T.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
chromedp was defaulting to log.Printf, which is not good for tests.
The xgb and xgbutil logs were suppressed if -v wasn't given, but they
were sent straight to os.Stderr otherwise:
=== RUN TestX11
=== PAUSE TestX11
=== CONT TestX11
XGB: conn.go:47: Could not get authority info: EOF
XGB: conn.go:48: Trying connection without authority info...
--- PASS: TestX11 (0.87s)
Instead, direct their loggers to an io.Writer implementation that sends
its output to t.Logf:
=== RUN TestX11
=== PAUSE TestX11
=== CONT TestX11
TestX11: x11_test.go:187: XGB: conn.go:47: Could not get authority info: EOF
TestX11: x11_test.go:187: XGB: conn.go:48: Trying connection without authority info...
--- PASS: TestX11 (0.86s)
We do end up with duplicate log prefixes, but at least we don't write
straight to stderr, which will be a problem as we add more concurrent
tests.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
On FreeBSD the X11 test seems to succeed except for the alpha
value:
--- FAIL: TestX11 (2.04s)
js_test.go:138: got 0xffff000000000000 at (5,5), want 0xffff00000000ffff
js_test.go:138: got 0xffff000000000000 at (595,595), want 0xffff00000000ffff
FAIL
Ignore alpha values for now.
Signed-off-by: Elias Naur <mail@eliasnaur.com>
First, pick a random display number between 1 and 100,000. The pool is
large enough that we don't need to think about collisions for now.
Second, wait for the X server to expose its socket for up to 1s, instead
of doing a single static sleep of 200ms. The average time we actually
need to sleep on my laptop is around 5ms, so this gives a noticeable
speed-up.
Signed-off-by: Daniel Martí <mvdan@mvdan.cc>
Daniel Martí