.NET - UI automation with the WPF UIAutomation framework

A few weeks ago I published an article about White, an open-source framework to automate Windows applications. Now I found out that the Windows Presentation Foundation (WPF) also provides a new Accessibility API called Microsoft UI Automation.

• http://msdn.microsoft.com/en-us/library/ms753388.aspx
• http://msdn.microsoft.com/en-us/library/system.windows.automation.aspx

AutomationElements and control patterns

The UI Automation framework enables you to automate Win32, Windows Forms and WPF applications. Every part of a UI (window, button, menu, …) is represented as an AutomationElement. An AutomationElement is not really a control; it is an object that represents the accessible properties of a control. Elements are contained in a tree structure, with the desktop as the root element.

By passing conditions to the FindFirst() method you can access all controls in another Windows application. You still need tools like UI Spy or WinSpector to find the AutomationId of controls.

AutomationElements also expose control patterns which provide properties for specific control types (window, button, menu, …). These patterns can also be used to call methods. There are several interesting patterns like InvokePattern (click), ExpandCollapsePattern, WindowPattern (close), TextPattern (manipulate text), … When using UI Spy you will see that all available patterns will be listed in the Properties window.

Automate Windows Calculator

To compare the features and performance of WPF UIAutomation to White, I have implemented the same example as in my previous article. So the demo application will start the Windows Calculator, calculate a sum and display the decimal and binary result in a WPF window. Because this example will demonstrate several techniques, not everything is optimal. There will probably be better and shorter ways to accomplish the same.

I have created a small WPF application but the UIAutomation libraries are not restricted to WPF. You could also use them in a WinForms or Console application.

First you need to add a reference to the UIAutomationClient.dll, UIAutomationTypes.dll and UIAutomationProviders.dll assemblies.

<Window x:Class="ScipBe.Demo.UIAutomation.WindowUIAutomation"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    Title="UI Automation" Height="400" Width="400">
    <DockPanel LastChildFill="True">
        <Button DockPanel.Dock="Top" Click="ButtonAutomate_Click">Automate Calculator</Button>
        <Button DockPanel.Dock="Top" Click="ButtonAutomateHelper_Click">Automate Calculator (helper class)</Button>
        <TextBox DockPanel.Dock="Top" Name="textBoxResult1"></TextBox>
        <TextBox DockPanel.Dock="Top" Name="textBoxResult2"></TextBox>
        <Button DockPanel.Dock="Top" Click="ButtonEvents_Click">Subscribe to events</Button>
        <ListBox DockPanel.Dock="Top" Name="listBoxEventLog"></ListBox>
    </DockPanel>
</Window>

using System.Windows;
using System.Windows.Automation;
using System.Windows.Forms;
using System.Diagnostics;
using System.Threading;
using System.Windows.Threading;
 
namespace ScipBe.Demo.UIAutomation
{
  public partial class WindowUIAutomation : Window
  {
    public WindowUIAutomation()
    {
      InitializeComponent();
    }
 
    private void ButtonAutomate_Click(object sender, RoutedEventArgs e)
    {
      // Launch Windows Calculator (calc.exe) and find the main window
      Process.Start("calc");
      Thread.Sleep(500);
      PropertyCondition findWindow = new PropertyCondition(AutomationElement.ClassNameProperty, "SciCalc");
      AutomationElement window = AutomationElement.RootElement.FindFirst(TreeScope.Children, findWindow);
 
      // Get a reference to the Edit TextBox (AutomationId = "403")
      PropertyCondition findTextBox = new PropertyCondition(AutomationElement.AutomationIdProperty, "403");
      AutomationElement textBox = window.FindFirst(TreeScope.Descendants, findTextBox);
 
      // Get a reference to the RadioButton "Dec" (AutomationId = "307")
      PropertyCondition findRadioButtonDecimal = new PropertyCondition(AutomationElement.AutomationIdProperty, "307");
      AutomationElement radioButtonDecimal = window.FindFirst(TreeScope.Descendants, findRadioButtonDecimal);
      // If this RadioButton can not be found, the calculator uses the Standard view
      // Call the Scientfic/Wetenschappelijk menu to make more options visible
      if (radioButtonDecimal == null)
      {  
        AutomationElement menuItemView = window.FindFirst(TreeScope.Descendants, 
          new PropertyCondition(AutomationElement.AutomationIdProperty, "Item 2"));
        if (menuItemView != null)
        {
          ExpandCollapsePattern pattern = menuItemView.GetCurrentPattern(ExpandCollapsePattern.Pattern) as ExpandCollapsePattern;
          pattern.Expand();
        }
        AutomationElement menuItemScientific = window.FindFirst(TreeScope.Descendants,
          new PropertyCondition(AutomationElement.AutomationIdProperty, "Item 304"));
        if (menuItemScientific != null)
        {
          InvokePattern pattern = menuItemScientific.GetCurrentPattern(InvokePattern.Pattern) as InvokePattern;
          pattern.Invoke();
        }
      }
 
      // Press F6 to make sure we are using the decimal numerical system
      SendKeys.SendWait("{F6}");
 
      // Focus the Edit TextBox and enter the text 54 
      textBox.SetFocus();
      Thread.Sleep(100);
      SendKeys.SendWait("45");
 
      // Press button "3" (AutomationId = "127") so the value will be 543
      AutomationElement button3 = window.FindFirst(TreeScope.Descendants, 
        new PropertyCondition(AutomationElement.AutomationIdProperty, "127"));
      if (button3 != null)
      {
        InvokePattern pattern = button3.GetCurrentPattern(InvokePattern.Pattern) as InvokePattern;
        pattern.Invoke();
      }
 
      // Press the "+" button (AutomationId = "92")
      AutomationElement buttonPlus = window.FindFirst(TreeScope.Descendants, 
        new PropertyCondition(AutomationElement.AutomationIdProperty, "92"));
      if (buttonPlus != null)
      {
        InvokePattern pattern = buttonPlus.GetCurrentPattern(InvokePattern.Pattern) as InvokePattern;
        pattern.Invoke();
      }
 
      // Enter the second value. Press button "6" (AutomationId = "130")
      AutomationElement button6 = window.FindFirst(TreeScope.Descendants, 
        new PropertyCondition(AutomationElement.AutomationIdProperty, "130"));
      (button6.GetCurrentPattern(InvokePattern.Pattern) as InvokePattern).Invoke();
 
      // Enter the second value. Press button "7" (AutomationId = "131")
      AutomationElement button7 = window.FindFirst(TreeScope.Descendants,
        new PropertyCondition(AutomationElement.AutomationIdProperty, "131"));
      (button7.GetCurrentPattern(InvokePattern.Pattern) as InvokePattern).Invoke();
 
      // Press the "=" button to calculate the result (AutomationId = "112")
      AutomationElement buttonEquals = window.FindFirst(TreeScope.Descendants,
        new PropertyCondition(AutomationElement.NameProperty, "="));
      (buttonEquals.GetCurrentPattern(InvokePattern.Pattern) as InvokePattern).Invoke();
 
      // Focus the Edit TextBox again and copy the result to the clipboard
      // by sending the keyboard combination CTRL+C
      textBox.SetFocus();
      Thread.Sleep(100);
      SendKeys.SendWait("^C");
 
      // Press F8 to change the numerical system from decimal to binary
      SendKeys.SendWait("{F8}");
 
      // Show the decimal result via the text on the clipboard
      // and the corresponding binary value by accessing the text of the Edit TextBox
      textBoxResult1.Text = "543 + 67 = " + System.Windows.Forms.Clipboard.GetText() + " (dec)";
      textBoxResult2.Text = "543 + 67 = " + (textBox.GetCurrentPattern(TextPattern.Pattern) as TextPattern).DocumentRange.GetText(-1) + " (bin)";
 
      // Close the Calculator application
      (window.GetCurrentPattern(WindowPattern.Pattern) as WindowPattern).Close();
    }
  }
}

The UI Automation framework does not support sending keystrokes, but it still can be done by calling the Send() or SendWait() methods of the SendKeys class which is a part of the System.Windows.Forms namespace.

UIAutomationHelper

As you have noticed, a lot of code is repeated when accessing controls. I’m not a big fan of helper classes but in this case it looks interesting to create a helper class with static methods to isolate some code.

public static class UIAutomationHelper
{
  public static AutomationElement LaunchApplication(string fileName)
  {
    return LaunchApplication(fileName, "");
  }
 
  public static AutomationElement LaunchApplication(string fileName, string arguments)
  {
    Process process = new Process();
    process.StartInfo.FileName = fileName;
    process.StartInfo.Arguments = arguments;
    process.Start();
    process.WaitForInputIdle(1000);
    return (AutomationElement.FromHandle(process.MainWindowHandle));
  }
 
  public static AutomationElement GetElementByName(AutomationElement window, string name)
  {
    if (window == null)
      return null;
 
    return window.FindFirst(TreeScope.Descendants,
      new PropertyCondition(AutomationElement.NameProperty, name));
  }
 
  public static AutomationElement GetElementById(AutomationElement window, string id)
  {
    if (window == null)
      return null;
 
    return window.FindFirst(TreeScope.Descendants,
      new PropertyCondition(AutomationElement.AutomationIdProperty, id));
  }
 
  public static bool InvokeElementByName(AutomationElement window, string name)
  {
    return InvokeElement(window, new PropertyCondition(AutomationElement.NameProperty, name)); 
  }
 
  public static bool InvokeElementById(AutomationElement window, string id)
  {
    return InvokeElement(window, new PropertyCondition(AutomationElement.AutomationIdProperty, id)); 
  }
 
  public static bool InvokeElement(AutomationElement window, System.Windows.Automation.Condition condition)
  {
    if (window == null)
      return false;
 
    AutomationElement element = window.FindFirst(TreeScope.Descendants, condition);
    if (element != null)
    {
      InvokePattern pattern = element.GetCurrentPattern(InvokePattern.Pattern) as InvokePattern;
      pattern.Invoke();
      return true;
    }
 
    return false;
  }
}

The next example shows the source code when using my UIAutomationHelper class. The code is more compact and more readable. Of course the final result is still the same.

private void ButtonAutomateHelper_Click(object sender, RoutedEventArgs e)
{
  // Launch Windows Calculator (calc.exe) and find the main window
  AutomationElement window = UIAutomationHelper.LaunchApplication("calc");
 
  // Get a reference to the Edit TextBox (AutomationId = "403")
  AutomationElement textBox = UIAutomationHelper.GetElementById(window, "403");
 
  // Get a reference to the RadioButton "Dec" (AutomationId = "307")
  AutomationElement radioButtonDecimal = UIAutomationHelper.GetElementById(window, "307");
  // If this RadioButton can not be found, the calculator uses the Standard view
  // Call the Scientfic/Wetenschappelijk menu to make more options visible
  if (radioButtonDecimal == null)
  {
    AutomationElement menuItemView = UIAutomationHelper.GetElementById(window, "Item 2");
    if (menuItemView != null)
    {
      ExpandCollapsePattern pattern = menuItemView.GetCurrentPattern(ExpandCollapsePattern.Pattern) as ExpandCollapsePattern;
      pattern.Expand();
    }
    UIAutomationHelper.InvokeElementById(window, "Item 304");
  }
 
  // Press F6 to make sure we are using the decimal numerical system
  SendKeys.SendWait("{F6}");
 
  // Focus the Edit TextBox and enter the text 54 
  textBox.SetFocus();
  Thread.Sleep(100);
  SendKeys.SendWait("45");
 
  // Press button "3" (AutomationId = "127") so the value will be 543
  UIAutomationHelper.InvokeElementById(window, "127");
 
  // Press the "+" button (AutomationId = "92")
  UIAutomationHelper.InvokeElementById(window, "92");
 
  // Enter the second value. Press button "6" (AutomationId = "130")
  UIAutomationHelper.InvokeElementById(window, "130");
 
  // Enter the second value. Press button "7" (AutomationId = "131")
  UIAutomationHelper.InvokeElementById(window, "131");
 
  // Press the "=" button to calculate the result (AutomationId = "112")
  UIAutomationHelper.InvokeElementByName(window, "=");
 
  // Focus the Edit TextBox again and copy the result to the clipboard
  // by sending the keyboard combination CTRL+C
  textBox.SetFocus();
  Thread.Sleep(100);
  SendKeys.SendWait("^C");
 
  // Press F8 to change the numerical system from decimal to binary
  SendKeys.SendWait("{F8}");
 
  // Show the decimal result via the text on the clipboard
  // and the corresponding binary value by accessing the text of the Edit TextBox
  textBoxResult1.Text = "543 + 67 = " + System.Windows.Forms.Clipboard.GetText() + " (dec)";
  textBoxResult2.Text = "543 + 67 = " + (textBox.GetCurrentPattern(TextPattern.Pattern) as TextPattern).DocumentRange.GetText(-1) + " (bin)";
 
  // Close the Calculator application
  (window.GetCurrentPattern(WindowPattern.Pattern) as WindowPattern).Close();
}

Compared to the White example, this application is a lot faster. The UI Automation framework is more complex but also more advanced. Furthermore it supports some very nice features like subscribing to events.

Subscribing to events

The AddAutomationEventHandler() and RemoveAutomationEventHandler() methods of the static Automation class can be used to subscribe or unsubscribe to events like InvokePattern.InvokedEvent, AutomationElement.MenuOpenedEvent, TextPattern.TextChangedEvent, WindowPattern.WindowOpenedEvent, WindowPattern.WindowClosedEvent, …

The next example will demonstrate how to subscribe to events of the Calculator. The log in our WPF Window will be updated when clicking the operator buttons (+, -, /, *, =) or when closing the Calculator application.

private void ButtonEvents_Click(object sender, RoutedEventArgs e)
{
  listBoxEventLog.Items.Clear();
 
  AutomationElement window = UIAutomationHelper.LaunchApplication("calc");
 
  // Subscribe to WindowClosedEvent. OnApplicationClose will be triggered
  Automation.AddAutomationEventHandler(WindowPattern.WindowClosedEvent, 
    window, TreeScope.Element, OnApplicationClose);
 
  // Subscribe to InvokedEvent of '=' button. OnEqualsClick will be triggered
  Automation.AddAutomationEventHandler(InvokePattern.InvokedEvent, 
    UIAutomationHelper.GetElementByName(window, "="), TreeScope.Element, OnEqualsClick);
 
  // Subscribe to InvokedEvent of operator buttons. OnUIAutomationEvent will be triggered
  Automation.AddAutomationEventHandler(InvokePattern.InvokedEvent, 
    UIAutomationHelper.GetElementByName(window, "+"), TreeScope.Element, OnUIAutomationEvent);
  Automation.AddAutomationEventHandler(InvokePattern.InvokedEvent, 
    UIAutomationHelper.GetElementByName(window, "-"), TreeScope.Element, OnUIAutomationEvent);
  Automation.AddAutomationEventHandler(InvokePattern.InvokedEvent, 
    UIAutomationHelper.GetElementByName(window, "*"), TreeScope.Element, OnUIAutomationEvent);
  Automation.AddAutomationEventHandler(InvokePattern.InvokedEvent, 
    UIAutomationHelper.GetElementByName(window, "/"), TreeScope.Element, OnUIAutomationEvent);
}
 
private void OnApplicationClose(object sender, AutomationEventArgs e)
{
  // Use the Dispatcher Invoke method to access the UI thread.
  // If you don't use this, a System.InvalidOperationException 
  // (The calling thread cannot access this object because a different thread owns it)
  // will be raised
  this.Dispatcher.Invoke(DispatcherPriority.Normal, (MethodInvoker)(() =>
  {
    listBoxEventLog.Items.Add("Close window");
  }));
}
 
private void OnEqualsClick(object sender, AutomationEventArgs e)
{ 
  this.Dispatcher.Invoke(DispatcherPriority.Normal, (MethodInvoker)(() =>
  {
    listBoxEventLog.Items.Add("Click button '=' ");
  }));
}
 
private void OnUIAutomationEvent(object sender, AutomationEventArgs e)
{
  // Get source element
  AutomationElement sourceElement = sender as AutomationElement;
 
  if (e.EventId == InvokePattern.InvokedEvent)
  {
    this.Dispatcher.Invoke(DispatcherPriority.Normal, (MethodInvoker)(() =>
    {
      // Log some info like ControlType, AutomationId, Name and Event
      listBoxEventLog.Items.Add(string.Format("ControlType={0} - AutomationId={1} - Name='{2}' - Event={3}",
        sourceElement.Current.ControlType.LocalizedControlType,
        sourceElement.Current.AutomationId,
        sourceElement.Current.Name,
        e.EventId.ProgrammaticName));
    }));
  }
}

I hope you like these examples and that this will encourage you to take a closer look at WPF UIAutomation. The UI Automation framework can be very useful when integrating your application with other applications which do no have a COM interface.

It is also used a lot in automated test frameworks. If you’re interested in UI testing, then you have to take a look at the open-source UI Automation Verify (UIA Verify) Test Automation Framework

Remarks or ideas, just let me know.