Windows 10: Introducing WinAppDriver UI Recorder

Brink · Jun 21, 2018

A new open-sourced tool is now available for the Windows Application Driver (WinAppDriver) community: the WinAppDriver UI Recorder tool. This tool will enable users to easily create automated UI tests.

For those of you not familiar with WinAppDriver, it is a UI automation service for Windows 10 that users can use to test their applications. We recently released our v1.1 preview that you can read more about here.

What is UI Recorder

For many in the WinAppDriver community, Inspect has been the most common tool for users to select UI elements and view their attribute data. Though Inspect serves its intended purpose of viewing accessibility data, it falls behind when it comes to supporting scenarios specifically for UI automation, such as being able to generate XPath queries.

For situations such as these, the WinAppDriver UI Recorder tool hopes to fill in the gaps from Inspect and serve as its alternative.

As for its initial release, the UI Recorder tool will enable the following two key scenarios:

1) Inspecting UI elements and retrieving their XPath expressions

2) Generating C# code for certain actions (mouse click) when “Record” is active

Generated Code can be pasted into the UI Recorder Template folder for WinAppDriver playback

We’re hoping that with this tool, users will have a simpler and more intuitive approach in writing automation scripts for WinAppDriver.

Getting Started

The code for the UI Recorder is open-sourced and available on WinAppDriver’s GitHub repo here. It’s recommended to use Visual Studio 2017 to start building and compiling it. Once compiled, you can immediately start using UI Recorder.

In addition to access to the source, a zipped executable can be found here on our GitHub Releases section.

Using UI Recorder

The UI Recorder tool aims to provide an intuitive, and simplistic, user interface that is divided into two panels, as seen below:

UI Recorder tracks both keyboard and mouse interactions against an application interface—representing a UI action. When Recording is active, both the top and bottom panels are dynamically updated with varying UI element information every time a new UI action takes place. The Top Panel shows the generated XPath query of the currently selected UI element, and the Bottom Panel shows the raw XML information for the same element. You can navigate to the C# Code tab on the bottom panel to view generated C# code of the recorded action which you can use on a WinAppDriver test.

The following animation provides an example of the recording process:

The code recorded can be coped over to the clipboard and pasted onto the WinAppDriver UI Recorder template project to be replayed.

Providing Feedback

With the UI Recorder tool being open-source, we highly encourage the community to submit any PRs with changes or enhancements, and to post suggestions on how they would like to see the UI Recorder grow.
Please use the GitHub Issues Board to provide any feedback on the UI Recorder tool – we look forward to hearing about any suggestions, feature requests, or bug reports!

Staying Informed

To stay up to date with WinAppDriver news follow @mrhassanuz.

Summary

A new tool for WinAppDriver is available now: UI Recorder. Now users have an intuitive way automate UI with WinAppDriver by not only being able to generate XPath expressions on the fly, but also C# code from by recording UI action events through mouse clicks.
Click to expand...

Source: Introducing WinAppDriver UI Recorder - Windows Developer Blog

:)

Try3 · Jun 21, 2018

Steps Recorder there... but not there

Are you trying to use this? - C:\Windows\System32\psr.exe
[Start menu, Windows accessories, Steps recorder - but often referred to as the Problem steps recorder]

It is supposed to launch a small UI dialog and it should also be shown on the Taskbar but, from your description, this does not happen. Does anything happen at all?

Denis

Ryan Fra · Jun 21, 2018

Can not record audio.

Hi,

We suggest that you run troubleshooting for Audio Recording by following these steps:

Type troubleshooting on the search box.

Click View all.

Select Recording Audio.

Follow the steps to troubleshoot Recording Audio.

Let us know if you need further help.

Brink · Jun 21, 2018

Support for Pen with Windows Application Driver v1.1 Preview

Windows Application Driver (WinAppDriver) is continuing Microsoft’s investment in UI test automation tools for Windows 10, and now, we’re excited to announce the next release of WinAppDriver—version 1.1!

A preview is available today— bringing support for Pen automation. The full v1.1 release is also on the horizon and will feature support for Multi-Touch in addition to Pen.

pxv8elFZY7g

What is WinAppDriver?

For those of you who aren’t familiar with WinAppDriver or UI automation, WinAppDriver is an open-standards based UI automation service designed to work with all kinds of Windows 10 applications including WPF, WinForms, legacy Win32, and UWP. By complying with an open-standard, WinAppDriver users will be able to leverage the robust UI automation ecosystem already provided by Appium and Selenium.

What’s new in the v1.1 Preview

In v1.1, we’re aligning with the W3C WebDriver standard, and as a result, implementing the Actions API to bring in advanced input device support.

The Preview release of v1.1 includes the following:

WinAppDriver updated for Pen, including support for advanced Pen functionality:

Pressure

Tilt X & Tilt Y

Twist

Barrel button

Eraser support

Appium-Dotnet-Driver NuGet Package

This is a preview Nuget package with updated bindings to enable Pen automation on WinAppDriver.

Samples & Documentation on GitHub

Note that the full release will also include support for Multi-Touch – more details on that below.

Getting Started with Pen

You can download the preview version of WinAppDriver on our GitHub page here: Releases · Microsoft/WinAppDriver · GitHub.

To get started on using Pen, we highly recommend checking out our new Sticky Notes sample here.

Let’s sketch out a quick smile

To demonstrate something a little more complex than a few strokes on a sticky note, we tried drawing out the following smiley face through the following steps–

Step 1 – Using Pen to draw a basic Circle

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25 //Initiate a Pen object using the custom Dotnet Driver Bindings.
PointerInputDevice penDevice = new PointerInputDevice(PointerKind.Pen);
ActionSequence drawSequence = new ActionSequence(penDevice, 0);

//Set starting position of circle by its center point.
var centerX = canvasCoordinate.X + canvasSize.Width / 5 + 285;
var centerY = canvasCoordinate.Y + canvasSize.Height / 5 + 270;
//Radius of circle.
var radius = 200;
/* This value will dictate the granularity of the number of stokes to complete the strokes. The more steps, the less blocky the circle will be. Note: Sticky Notes will interpolate between the steps anyway, so it will not appear blocky if Pen is not lifted. */
var steps = 50;
// These two variables will calculate either the X or Y coordinate around the center of the circle for a given step.
int xValue = (int)(centerX + radius * Math.Cos(2 * Math.PI * 0 / steps));
int yValue = (int)(centerY + radius * Math.Sin(2 * Math.PI * 0 / steps));
drawSequence.AddAction(penDevice.CreatePointerMove(CoordinateOrigin.Viewport, xValue, yValue, TimeSpan.Zero));
drawSequence.AddAction(penDevice.CreatePointerUp(PointerButton.PenFrontTip));
/* Function to draw circle by calculating coordinates around center point and brushing through them. */
for (var i = 0; i <= steps; i++) {
xValue = (int)(centerX + radius * Math.Cos(2 * Math.PI * i / steps));
yValue = (int)(centerY + radius * Math.Sin(2 * Math.PI * i / steps)); drawSequence.AddAction(penDevice.CreatePointerDown(PointerButton.PenContact));
drawSequence.AddAction(penDevice.CreatePointerMove(CoordinateOrigin.Viewport, xValue, yValue, TimeSpan.Zero));
}
//Lifting pen up once Circle is drawn. drawSequence.AddAction(penDevice.CreatePointerUp(PointerButton.PenContact));
//Final step would be to execute the sequence.
newStickyNoteSession.PerformActions(new List<ActionSequence> { drawSequence });
Step 2 – Adding in the smile

We’ll have to get a little clever with this part, and modify the original for-loop from step 1 and add supplementary code following it. This will continue the same sequence.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27 /* Loop modified to iterate past opening step and closing step of circle. This will create an opening for the smiley ìmouthî. */
for (var i = 0; i < steps; i++) {
xValue = (int)(centerX + radius * Math.Cos(2 * Math.PI * i / steps));
yValue = (int)(centerY + radius * Math.Sin(2 * Math.PI * i / steps));
if (i > 1){
drawSequence.AddAction(penDevice.CreatePointerDown(PointerButton.PenContact));

} drawSequence.AddAction(penDevice.CreatePointerDown(PointerButton.PenContact));
drawSequence.AddAction(penDevice.CreatePointerMove(CoordinateOrigin.Viewport, xValue, yValue, TimeSpan.Zero));
}
drawSequence.AddAction(penDevice.CreatePointerUp(PointerButton.PenContact));
/* The following vars will calculate X & Y coordinates for start and end point of the smile. */
var xSmile = (int)(centerX + radius * Math.Cos(2 * Math.PI * 1 / steps));
var ySmile = (int)(centerY + radius * Math.Sin(2 * Math.PI * 1 / steps));
var xSmile2 = (int)(centerX + radius * Math.Cos(2 * Math.PI * (steps - 1) / steps));
var ySmile2 = (int)(centerY + radius * Math.Sin(2 * Math.PI * (steps - 1) / steps));

/* Continue previous sequence and execute the Pen actions. */
drawSequence.AddAction(penDevice.CreatePointerMove(CoordinateOrigin.Viewport, xSmile, ySmile, TimeSpan.Zero));
drawSequence.AddAction(penDevice.CreatePointerDown(PointerButton.PenContact));
drawSequence.AddAction(penDevice.CreatePointerMove(CoordinateOrigin.Viewport, centerX, centerY, TimeSpan.FromMilliseconds(400), new PenInfo { Pressure = .600f }));
drawSequence.AddAction(penDevice.CreatePointerUp(PointerButton.PenContact));
drawSequence.AddAction(penDevice.CreatePointerDown(PointerButton.PenContact));
drawSequence.AddAction(penDevice.CreatePointerMove(CoordinateOrigin.Viewport, xSmile2, ySmile2, TimeSpan.FromMilliseconds(400)));

// Execute sequence.
newStickyNoteSession.PerformActions(new List<ActionSequence> { drawSequence });
The additional pressure applied to the “smile strokes”—this is to add depth to the smile. The sketch should appear as the following:

Step 3 – Eyes

If we wanted to finish up and add eyes, it would look something like the following:

There’s a lot of clever ways you can go on about adding the eyes, so we’ll let you decide which way is best. The full code for our design will be included as part of the Sticky Notes sample on GitHub!

Details on full 1.1

The full release of v1.1 will come with the following additions:

WinAppDriver

Pen—support for Pen will be carried over to full 1.1 release from the Preview.

Multi-Touch—support for Multi-Touch will be added in as well through the Actions API. The following touch modifiers will be supported:

Pressure

Twist

New Samples & Bindings

Samples from the Preview will be further expanded to demonstrate Pen and Multi-Touch functionality. The samples will incorporate a private Appium-Dotnet-Driver Nuget feed that will enable Actions implementation via the new bindings. We’re looking into having these changes be merged into the official Appium .Net Driver, and eventually be rolled-up to the Selenium Namespace in the future.

Release Date

We’re targeting this June for the full release of v1.1—stay tuned to our GitHub board for more info!

Moving Forward

The WinAppDriver team will continue to work on adding new features, resolving bugs, and improving performance. We’ve been looking into popular community requests, and as such, have a couple of cool things in the pipeline for 1.2 and beyond—one in particular being to improve performance with XPath handling.

We’ll also be releasing a new tool for the community – more details on this to follow in the coming weeks. Stay tuned!

How do I provide feedback?

Please provide feedback on our Github issue board – we look forward to hearing about any suggestions, feature requests, or bug reports!

Microsoft/WinAppDriver

If you have any cool sketches done through v1.1 that you’d like to share—do so on the GitHub board! It may even be featured in a future blog post!

Stay Informed

To stay up to date with WinAppDriver news follow @mrhassanuz.

Summary

The v1.1 Preview is available now—enabling users to automate Pen scenarios. Full release for v1.1 to follow, and with it will bring support for Multi-Touch as well. Head over to our releases page on GitHub to download the preview, and get a jump-start on Pen automation by reviewing our updated samples.
Click to expand...

Source: Announcing support for Pen with Windows Application Driver v1.1 - Preview available now - Windows Developer Blog

Windows 10: Introducing WinAppDriver UI Recorder

Introducing WinAppDriver UI Recorder

Introducing WinAppDriver UI Recorder

Introducing WinAppDriver UI Recorder

Introducing WinAppDriver UI Recorder - Similar Threads - Introducing WinAppDriver Recorder

WinAppDriver version 1.2.99 - Need help to rectify the issue -...

WinAppDriver version 1.2.99 - Need help to rectify the issue -...

WinAppDriver open source

WinAppDriver open source

CodedUI to WinAppDriver/Appium - automated conversion

CodedUI to WinAppDriver/Appium - automated conversion

20H1 continues the proud tradition of introducing UI bugs.

Introducing .NET Multi-platform App UI

Introducing Skype call recording

Users found this page by searching for:

winappdriver ui recorder

WinAppDriver UI Recorder usage

win app ui recorder