Windows 10: Support for Pen with Windows Application Driver v1.1 Preview

Brink · May 17, 2018

Windows Application Driver (WinAppDriver) is continuing Microsoft’s investment in UI test automation tools for Windows 10, and now, we’re excited to announce the next release of WinAppDriver—version 1.1!

A preview is available today— bringing support for Pen automation. The full v1.1 release is also on the horizon and will feature support for Multi-Touch in addition to Pen.

[youtube]pxv8elFZY7g[/youtube]

What is WinAppDriver?

For those of you who aren’t familiar with WinAppDriver or UI automation, WinAppDriver is an open-standards based UI automation service designed to work with all kinds of Windows 10 applications including WPF, WinForms, legacy Win32, and UWP. By complying with an open-standard, WinAppDriver users will be able to leverage the robust UI automation ecosystem already provided by Appium and Selenium.

What’s new in the v1.1 Preview

In v1.1, we’re aligning with the W3C WebDriver standard, and as a result, implementing the Actions API to bring in advanced input device support.

The Preview release of v1.1 includes the following:

WinAppDriver updated for Pen, including support for advanced Pen functionality:

Pressure

Tilt X & Tilt Y

Twist

Barrel button

Eraser support

Appium-Dotnet-Driver NuGet Package

This is a preview Nuget package with updated bindings to enable Pen automation on WinAppDriver.

Samples & Documentation on GitHub

Note that the full release will also include support for Multi-Touch – more details on that below.

Getting Started with Pen

You can download the preview version of WinAppDriver on our GitHub page here: Releases · Microsoft/WinAppDriver · GitHub.

To get started on using Pen, we highly recommend checking out our new Sticky Notes sample here.

Let’s sketch out a quick smile *Smile

To demonstrate something a little more complex than a few strokes on a sticky note, we tried drawing out the following smiley face through the following steps–

Step 1 – Using Pen to draw a basic Circle

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25 //Initiate a Pen object using the custom Dotnet Driver Bindings.
PointerInputDevice penDevice = new PointerInputDevice(PointerKind.Pen);
ActionSequence drawSequence = new ActionSequence(penDevice, 0);

//Set starting position of circle by its center point.
var centerX = canvasCoordinate.X + canvasSize.Width / 5 + 285;
var centerY = canvasCoordinate.Y + canvasSize.Height / 5 + 270;
//Radius of circle.
var radius = 200;
/* This value will dictate the granularity of the number of stokes to complete the strokes. The more steps, the less blocky the circle will be. Note: Sticky Notes will interpolate between the steps anyway, so it will not appear blocky if Pen is not lifted. */
var steps = 50;
// These two variables will calculate either the X or Y coordinate around the center of the circle for a given step.
int xValue = (int)(centerX + radius * Math.Cos(2 * Math.PI * 0 / steps));
int yValue = (int)(centerY + radius * Math.Sin(2 * Math.PI * 0 / steps));
drawSequence.AddAction(penDevice.CreatePointerMove(CoordinateOrigin.Viewport, xValue, yValue, TimeSpan.Zero));
drawSequence.AddAction(penDevice.CreatePointerUp(PointerButton.PenFrontTip));
/* Function to draw circle by calculating coordinates around center point and brushing through them. */
for (var i = 0; i <= steps; i++) {
xValue = (int)(centerX + radius * Math.Cos(2 * Math.PI * i / steps));
yValue = (int)(centerY + radius * Math.Sin(2 * Math.PI * i / steps)); drawSequence.AddAction(penDevice.CreatePointerDown(PointerButton.PenContact));
drawSequence.AddAction(penDevice.CreatePointerMove(CoordinateOrigin.Viewport, xValue, yValue, TimeSpan.Zero));
}
//Lifting pen up once Circle is drawn. drawSequence.AddAction(penDevice.CreatePointerUp(PointerButton.PenContact));
//Final step would be to execute the sequence.
newStickyNoteSession.PerformActions(new List<ActionSequence> { drawSequence });
Step 2 – Adding in the smile

We’ll have to get a little clever with this part, and modify the original for-loop from step 1 and add supplementary code following it. This will continue the same sequence.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27 /* Loop modified to iterate past opening step and closing step of circle. This will create an opening for the smiley ìmouthî. */
for (var i = 0; i < steps; i++) {
xValue = (int)(centerX + radius * Math.Cos(2 * Math.PI * i / steps));
yValue = (int)(centerY + radius * Math.Sin(2 * Math.PI * i / steps));
if (i > 1){
drawSequence.AddAction(penDevice.CreatePointerDown(PointerButton.PenContact));

} drawSequence.AddAction(penDevice.CreatePointerDown(PointerButton.PenContact));
drawSequence.AddAction(penDevice.CreatePointerMove(CoordinateOrigin.Viewport, xValue, yValue, TimeSpan.Zero));
}
drawSequence.AddAction(penDevice.CreatePointerUp(PointerButton.PenContact));
/* The following vars will calculate X & Y coordinates for start and end point of the smile. */
var xSmile = (int)(centerX + radius * Math.Cos(2 * Math.PI * 1 / steps));
var ySmile = (int)(centerY + radius * Math.Sin(2 * Math.PI * 1 / steps));
var xSmile2 = (int)(centerX + radius * Math.Cos(2 * Math.PI * (steps - 1) / steps));
var ySmile2 = (int)(centerY + radius * Math.Sin(2 * Math.PI * (steps - 1) / steps));

/* Continue previous sequence and execute the Pen actions. */
drawSequence.AddAction(penDevice.CreatePointerMove(CoordinateOrigin.Viewport, xSmile, ySmile, TimeSpan.Zero));
drawSequence.AddAction(penDevice.CreatePointerDown(PointerButton.PenContact));
drawSequence.AddAction(penDevice.CreatePointerMove(CoordinateOrigin.Viewport, centerX, centerY, TimeSpan.FromMilliseconds(400), new PenInfo { Pressure = .600f }));
drawSequence.AddAction(penDevice.CreatePointerUp(PointerButton.PenContact));
drawSequence.AddAction(penDevice.CreatePointerDown(PointerButton.PenContact));
drawSequence.AddAction(penDevice.CreatePointerMove(CoordinateOrigin.Viewport, xSmile2, ySmile2, TimeSpan.FromMilliseconds(400)));

// Execute sequence.
newStickyNoteSession.PerformActions(new List<ActionSequence> { drawSequence });
The additional pressure applied to the “smile strokes”—this is to add depth to the smile. The sketch should appear as the following:

Step 3 – Eyes

If we wanted to finish up and add eyes, it would look something like the following:

There’s a lot of clever ways you can go on about adding the eyes, so we’ll let you decide which way is best. The full code for our design will be included as part of the Sticky Notes sample on GitHub!

Details on full 1.1

The full release of v1.1 will come with the following additions:

WinAppDriver

Pen—support for Pen will be carried over to full 1.1 release from the Preview.

Multi-Touch—support for Multi-Touch will be added in as well through the Actions API. The following touch modifiers will be supported:

Pressure

Twist

New Samples & Bindings

Samples from the Preview will be further expanded to demonstrate Pen and Multi-Touch functionality. The samples will incorporate a private Appium-Dotnet-Driver Nuget feed that will enable Actions implementation via the new bindings. We’re looking into having these changes be merged into the official Appium .Net Driver, and eventually be rolled-up to the Selenium Namespace in the future.

Release Date

We’re targeting this June for the full release of v1.1—stay tuned to our GitHub board for more info!

Moving Forward

The WinAppDriver team will continue to work on adding new features, resolving bugs, and improving performance. We’ve been looking into popular community requests, and as such, have a couple of cool things in the pipeline for 1.2 and beyond—one in particular being to improve performance with XPath handling.

We’ll also be releasing a new tool for the community – more details on this to follow in the coming weeks. Stay tuned!

How do I provide feedback?

Please provide feedback on our Github issue board – we look forward to hearing about any suggestions, feature requests, or bug reports!

https://github.com/Microsoft/WinAppDriver/issues

If you have any cool sketches done through v1.1 that you’d like to share—do so on the GitHub board! It may even be featured in a future blog post!

Stay Informed

To stay up to date with WinAppDriver news follow @mrhassanuz.

Summary

The v1.1 Preview is available now—enabling users to automate Pen scenarios. Full release for v1.1 to follow, and with it will bring support for Multi-Touch as well. Head over to our releases page on GitHub to download the preview, and get a jump-start on Pen automation by reviewing our updated samples.
Click to expand...

Source: Announcing support for Pen with Windows Application Driver v1.1 - Preview available now - Windows Developer Blog

:)

Entegy · May 17, 2018

Pen for 950XL???

There was nothing announced about supporting a pen nor is any kind of pen tech listed in the tech specs of the phone in the Microsoft store.

ArkEngel · May 17, 2018

No windows 10 preview for Lumia 720??

Hi Anjaiah,

here's the link to supported phones for the technical preview.

Register, install and engage | Windows Insider Program

Notes

Some phones, regions, and mobile operators might not support Technical Preview. If your phone isn't listed, and no preview builds
appear under Enroll in the Windows Insider app, then your phone isn't supported and it may or may not be supported in the future.

To see which Windows Phone operating system (OS) version is installed on your phone, go to Settings > About > More info

Windows 10: Support for Pen with Windows Application Driver v1.1 Preview

Support for Pen with Windows Application Driver v1.1 Preview

Support for Pen with Windows Application Driver v1.1 Preview

Support for Pen with Windows Application Driver v1.1 Preview - Similar Threads - Support Pen Application

Digital Pen Support

Digital Pen Support

Pen support

Pen support

Want to uninstall Driver Support 1

Want to uninstall Driver Support 1

Pen and Touch Support

Application preview

Windows Ink 1: Introduction to Ink and Pen