[Hosts] Converting manual release-check-list tests to UI-Test Automation and Adding more UI-Test-cases (#37657)

* Add more UI-Test, refactor UITestAutomation

* Convert manual test-case to automation

UI-Tests:

Validating Empty-view is shown if no entries in the list.
Validating Empty-view is NOT shown if 1 or more entries in the list.
Validating Add-an-entry HyperlinkButton in Empty-view works correctly.
Validating Adding-entry Button works correctly.
Validating the Add button should be Disabled if more than 9 hosts in one entry.
Validating the Add button should be Enabled if less or equal 9 hosts in one entry.
Validating error message should be shown if not run as admin.
Validating Warning-Dialog will be shown if 'Show a warning at startup' toggle is On.
Validating Warning-Dialog will NOT be shown if 'Show a warning at startup' toggle is Off.
Validating click 'Quit' button in Warning-Dialog, the Hosts File Editor window would be closed.
Validating click 'Accept' button in Warning-Dialog, the Hosts File Editor window would NOT be closed.

---------

Co-authored-by: Jerry Xu <nxu@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

src/modules/Hosts/Hosts.UITests/HostModuleTests.cs

@@ -2,7 +2,7 @@
 // The Microsoft Corporation licenses this file to you under the MIT license.
 // See the LICENSE file in the project root for more information.
 
-using System;
+using System.Collections.Generic;
 using System.Threading.Tasks;
 using Microsoft.PowerToys.UITest;
 using Microsoft.VisualStudio.TestTools.UnitTesting;
@@ -18,8 +18,18 @@ namespace Hosts.UITests
         }
 
         /// <summary>
-        /// Test if Empty-view is shown when no entries are present.
-        /// And 'Add an entry' button from Empty-view is functional.
+        /// Test Empty-view in the Hosts-File-Editor
+        /// <list type="bullet">
+        /// <item>
+        /// <description>Validating Empty-view is shown if no entries in the list.</description>
+        /// </item>
+        /// <item>
+        /// <description>Validating Empty-view is NOT shown if 1 or more entries in the list.</description>
+        /// </item>
+        /// <item>
+        /// <description>Validating Add-an-entry HyperlinkButton in Empty-view works correctly.</description>
+        /// </item>
+        /// </list>
         /// </summary>
         [TestMethod]
         public void TestEmptyView()
@@ -28,20 +38,25 @@ namespace Hosts.UITests
             this.RemoveAllEntries();
 
             // 'Add an entry' button (only show-up when list is empty) should be visible
-            Assert.IsTrue(this.FindAll<Button>(By.Name("Add an entry")).Count == 1, "'Add an entry' button should be visible in the empty view");
+            Assert.IsTrue(this.FindAll<HyperlinkButton>("Add an entry").Count == 1, "'Add an entry' button should be visible in the empty view");
 
             // Click 'Add an entry' from empty-view for adding Host override rule
-            this.Find<Button>(By.Name("Add an entry")).Click();
+            this.Find<HyperlinkButton>("Add an entry").Click();
 
             this.AddEntry("192.168.0.1", "localhost", false, false);
 
             // Should have one row now and not more empty view
-            Assert.IsTrue(this.FindAll<Button>(By.Name("Delete")).Count == 1, "Should have one row now");
-            Assert.IsTrue(this.FindAll<Button>(By.Name("Add an entry")).Count == 0, "'Add an entry' button should be invisible if not empty view");
+            Assert.IsTrue(this.FindAll<Button>("Delete").Count == 1, "Should have one row now");
+            Assert.IsTrue(this.FindAll<HyperlinkButton>("Add an entry").Count == 0, "'Add an entry' button should be invisible if not empty view");
         }
 
         /// <summary>
-        /// Test if 'New entry' button is functional
+        /// Test Adding-entry Button in the Hosts-File-Editor
+        /// <list type="bullet">
+        /// <item>
+        /// <description>Validating Adding-entry Button works correctly.</description>
+        /// </item>
+        /// </list>
         /// </summary>
         [TestMethod]
         public void TestAddingEntry()
@@ -49,11 +64,155 @@ namespace Hosts.UITests
             this.CloseWarningDialog();
             this.RemoveAllEntries();
 
-            Assert.IsTrue(this.FindAll<Button>(By.Name("Delete")).Count == 0, "Should have no row after removing all");
+            Assert.IsTrue(this.FindAll<Button>("Delete").Count == 0, "Should have no row after removing all");
 
             this.AddEntry("192.168.0.1", "localhost", true);
 
-            Assert.IsTrue(this.FindAll<Button>(By.Name("Delete")).Count == 1, "Should have one row now");
+            Assert.IsTrue(this.FindAll<Button>("Delete").Count == 1, "Should have one row now");
+        }
+
+        /// <summary>
+        /// Test Multiple-hosts validation logic
+        /// <list type="bullet">
+        /// <item>
+        /// <description>Validating the Add button should be Disabled if more than 9 hosts in one entry.</description>
+        /// </item>
+        /// <item>
+        /// <description>Validating the Add button should be Enabled if less or equal 9 hosts in one entry.</description>
+        /// </item>
+        /// </list>
+        /// </summary>
+        [TestMethod]
+        public void TestTooManyHosts()
+        {
+            this.CloseWarningDialog();
+
+            // only at most 9 hosts allowed in one entry
+            string validHosts = string.Join(" ", "host_1", "host_2", "host_3", "host_4", "host_5", "host_6", "host_7", "host_8", "host_9");
+
+            // should not allow to add more than 9 hosts in one entry, hosts are separated by space
+            string inValidHosts = validHosts + " more_host";
+
+            this.Find<Button>("New entry").Click();
+
+            Assert.IsFalse(this.Find<Button>("Add").Enabled, "Add button should be Disabled by default");
+
+            this.Find<TextBox>("Address").SetText("127.0.0.1");
+
+            this.Find<TextBox>("Hosts").SetText(validHosts);
+            Assert.IsTrue(this.Find<Button>("Add").Enabled, "Add button should be Enabled with validHosts");
+
+            this.Find<TextBox>("Hosts").SetText(inValidHosts);
+            Assert.IsFalse(this.Find<Button>("Add").Enabled, "Add button should be Disabled with inValidHosts");
+
+            this.Find<Button>("Cancel").Click();
+        }
+
+        /// <summary>
+        /// Test Error-message in the Hosts-File-Editor
+        /// <list type="bullet">
+        /// <item>
+        /// <description>Validating error message should be shown if not run as admin.</description>
+        /// </item>
+        /// </list>
+        /// </summary>
+        [TestMethod]
+        public void TestErrorMessageWithNonAdminPermission()
+        {
+            this.CloseWarningDialog();
+            this.RemoveAllEntries();
+
+            // Add new URL override and a warning tip should be shown
+            this.AddEntry("192.168.0.1", "localhost", true);
+
+            Assert.IsTrue(
+                this.FindAll<TextBlock>("The hosts file cannot be saved because the program isn't running as administrator.").Count == 1,
+                "Should display host-file saving error if not run as administrator");
+        }
+
+        /// <summary>
+        /// Test Filter-panel function in the Hosts-File-Editor
+        /// <list type="bullet">
+        /// <item>
+        /// <description>Validating Address filter matching pattern: contains, endsWith, startsWith, exactly-match.</description>
+        /// </item>
+        /// <item>
+        /// <description>Validating Hosts filter matching pattern: contains, endsWith, startsWith, exactly-match.</description>
+        /// </item>
+        /// <item>
+        /// <description>Validating click Filters Button to open filter-panel, and click Filter Button again to close filter-panel.</description>
+        /// </item>
+        /// </list>
+        /// </summary>
+        [TestMethod]
+        public void TestFilterControl()
+        {
+            this.CloseWarningDialog();
+            this.RemoveAllEntries();
+
+            for (int i = 0; i < 10; i++)
+            {
+                this.AddEntry("192.168.0." + i, "localhost_" + i, true);
+            }
+
+            // Open-filter-panel
+            this.Find<Button>("Filters").Click();
+            Assert.IsTrue(this.FindAll<Button>("Clear filters").Count == 1, "Filter panel should be opened afer click Filter Button");
+
+            var addressFilterCases = new KeyValuePair<string, int>[]
+            {
+                // contains text, expected matched more rows
+                new("168.0", 10),
+
+                // ends with text, expected matched 1 row
+                new("168.0.1", 1),
+
+                // starts with text, expected matched more rows
+                new("192.168.", 10),
+
+                // full text, expected matched 1 row
+                new("192.168.0.1", 1),
+
+                // no-matching text, expected matched no row
+                new("127.0.0", 0),
+
+                // empty filter, should display all rows
+                new(string.Empty, 10),
+            };
+
+            foreach (var (addressFilter, expectedCount) in addressFilterCases)
+            {
+                this.Find<TextBox>("Address").SetText(addressFilter);
+                Assert.IsTrue(this.Find("Entries").FindAll<Button>("Delete").Count == expectedCount);
+            }
+
+            var hostFilterCases = new KeyValuePair<string, int>[]
+            {
+                // contains text, expected matched more rows
+                new("host_", 10),
+
+                // ends with text, expected matched 1 row
+                new("host_4", 1),
+
+                // starts with text, expected matched more rows
+                new("localhost", 10),
+
+                // full text, expected matched 1 row
+                new("localhost_5", 1),
+
+                // empty filter, should display all rows
+                new(string.Empty, 10),
+            };
+
+            foreach (var (hostFilterCase, expectedCount) in hostFilterCases)
+            {
+                this.Find<TextBox>("Hosts").SetText(hostFilterCase);
+                Assert.IsTrue(this.Find("Entries").FindAll<Button>("Delete").Count == expectedCount);
+            }
+
+            // Close-filter-panel
+            this.Find<Button>("Filters").Click();
+            Assert.IsFalse(this.FindAll<Button>("Clear filters").Count == 1, "Filter panel should be closed afer click Filter Button");
         }
 
         private void AddEntry(string ip, string host, bool active = true, bool clickAddEntryButton = true)
@@ -61,21 +220,21 @@ namespace Hosts.UITests
             if (clickAddEntryButton)
             {
                 // Click 'Add an entry' for adding Host override rule
-                this.Find<Button>(By.Name("New entry")).Click();
+                this.Find<Button>("New entry").Click();
             }
 
             // Adding a new host override localhost -> 192.168.0.1
-            Assert.IsFalse(this.Find<Button>(By.Name("Add")).Enabled, "Add button should be Disabled by default");
+            Assert.IsFalse(this.Find<Button>("Add").Enabled, "Add button should be Disabled by default");
 
-            Assert.IsTrue(this.Find<TextBox>(By.Name("Address")).SetText(ip, false).Text == ip);
-            Assert.IsTrue(this.Find<TextBox>(By.Name("Hosts")).SetText(host, false).Text == host);
+            Assert.IsTrue(this.Find<TextBox>("Address").SetText(ip).Text == ip);
+            Assert.IsTrue(this.Find<TextBox>("Hosts").SetText(host).Text == host);
 
-            this.Find<ToggleSwitch>(By.Name("Active")).Toggle(active);
+            this.Find<ToggleSwitch>("Active").Toggle(active);
 
-            Assert.IsTrue(this.Find<Button>(By.Name("Add")).Enabled, "Add button should be Enabled after providing valid inputs");
+            Assert.IsTrue(this.Find<Button>("Add").Enabled, "Add button should be Enabled after providing valid inputs");
 
             // Add the entry
-            this.Find<Button>(By.Name("Add")).Click();
+            this.Find<Button>("Add").Click();
 
             // 0.5 second delay after adding an entry
             Task.Delay(500).Wait();
@@ -84,25 +243,25 @@ namespace Hosts.UITests
         private void CloseWarningDialog()
         {
             // Find 'Accept' button which come in 'Warning' dialog
-            if (this.FindAll<Window>(By.Name("Warning")).Count > 0 &&
-                this.FindAll<Button>(By.Name("Accept")).Count > 0)
+            if (this.FindAll("Warning").Count > 0 &&
+                this.FindAll<Button>("Accept").Count > 0)
             {
                 // Hide Warning dialog if any
-                this.Find<Button>(By.Name("Accept")).Click();
+                this.Find<Button>("Accept").Click();
             }
         }
 
         private void RemoveAllEntries()
         {
             // Delete all existing host-override rules
-            foreach (var deleteBtn in this.FindAll<Button>(By.Name("Delete")))
+            foreach (var deleteBtn in this.FindAll<Button>("Delete"))
             {
                 deleteBtn.Click();
-                this.Find<Button>(By.Name("Yes")).Click();
+                this.Find<Button>("Yes").Click();
             }
 
             // Should have no row left, and no more delete button
-            Assert.IsTrue(this.FindAll<Button>(By.Name("Delete")).Count == 0);
+            Assert.IsTrue(this.FindAll<Button>("Delete").Count == 0);
         }
     }
 }

src/modules/Hosts/Hosts.UITests/HostsSettingTests.cs

@@ -0,0 +1,129 @@
+// Copyright (c) Microsoft Corporation
+// The Microsoft Corporation licenses this file to you under the MIT license.
+// See the LICENSE file in the project root for more information.
+
+using System;
+using System.Threading.Tasks;
+using Microsoft.PowerToys.UITest;
+using Microsoft.VisualStudio.TestTools.UnitTesting;
+
+namespace Hosts.UITests
+{
+    [TestClass]
+    public class HostsSettingTests : UITestBase
+    {
+        /// <summary>
+        /// Test Warning Dialog at startup
+        /// <list type="bullet">
+        /// <item>
+        /// <description>Validating Warning-Dialog will be shown if 'Show a warning at startup' toggle is On.</description>
+        /// </item>
+        /// <item>
+        /// <description>Validating Warning-Dialog will NOT be shown if 'Show a warning at startup' toggle is Off.</description>
+        /// </item>
+        /// <item>
+        /// <description>Validating click 'Quit' button in Warning-Dialog, the Hosts File Editor window would be closed.</description>
+        /// </item>
+        /// <item>
+        /// <description>Validating click 'Accept' button in Warning-Dialog, the Hosts File Editor window would NOT be closed.</description>
+        /// </item>
+        /// </list>
+        /// </summary>
+        [TestMethod]
+        public void TestWarningDialog()
+        {
+            this.LaunchFromSetting(showWarning: true);
+
+            // Validating Warning-Dialog will be shown if 'Show a warning at startup' toggle is on
+            Assert.IsTrue(this.FindAll("Warning").Count > 0, "Should show warning dialog");
+
+            // Quit Hosts File Editor
+            this.Find<Button>("Quit").Click();
+
+            // Wait for 500 ms to make sure Hosts File Editor is closed
+            Task.Delay(500).Wait();
+
+            // Validating click 'Quit' button in Warning-Dialog, the Hosts File Editor window would be closed
+            Assert.IsTrue(this.IsHostsFileEditorClosed(), "Hosts File Editor should be closed after click Quit button in Warning Dialog");
+
+            // Re-attaching to Setting Windows
+            this.Session.Attach(PowerToysModule.PowerToysSettings);
+
+            this.Find<Button>("Launch Hosts File Editor").Click();
+
+            // wait for 500 ms to make sure Hosts File Editor is launched
+            Task.Delay(500).Wait();
+
+            this.Session.Attach(PowerToysModule.Hosts);
+
+            // Should show warning dialog
+            Assert.IsTrue(this.FindAll("Warning").Count > 0, "Should show warning dialog");
+
+            // Quit Hosts File Editor
+            this.Find<Button>("Accept").Click();
+
+            Task.Delay(500).Wait();
+
+            // Validating click 'Accept' button in Warning-Dialog, the Hosts File Editor window would NOT be closed
+            Assert.IsFalse(this.IsHostsFileEditorClosed(), "Hosts File Editor should NOT be closed after click Accept button in Warning Dialog");
+
+            // Close Hosts File Editor window
+            this.Session.Find<Window>("Hosts File Editor").Close();
+
+            // Restore back to PowerToysSettings Session
+            this.Session.Attach(PowerToysModule.PowerToysSettings);
+
+            this.LaunchFromSetting(showWarning: false);
+
+            // Should NOT show warning dialog
+            Assert.IsTrue(this.FindAll("Warning").Count == 0, "Should NOT show warning dialog");
+
+            // Host Editor Window should not be closed
+            Assert.IsFalse(this.IsHostsFileEditorClosed(), "Hosts File Editor should NOT be closed");
+
+            // Close Hosts File Editor window
+            this.Session.Find<Window>("Hosts File Editor").Close();
+
+            // Restore back to PowerToysSettings Session
+            this.Session.Attach(PowerToysModule.PowerToysSettings);
+        }
+
+        private bool IsHostsFileEditorClosed()
+        {
+            try
+            {
+                this.Session.FindAll<Window>("Hosts File Editor");
+            }
+            catch (Exception ex)
+            {
+                // Validate if editor window closed by checking exception.Message
+                return ex.Message.Contains("Currently selected window has been closed");
+            }
+
+            return false;
+        }
+
+        private void LaunchFromSetting(bool showWarning = false, bool launchAsAdmin = false)
+        {
+            // Goto Hosts File Editor setting page
+            if (this.FindAll<NavigationViewItem>("Hosts File Editor").Count == 0)
+            {
+                // Expand Advanced list-group if needed
+                this.Find<NavigationViewItem>("Advanced").Click();
+            }
+
+            this.Find<NavigationViewItem>("Hosts File Editor").Click();
+
+            this.Find<ToggleSwitch>("Enable Hosts File Editor").Toggle(true);
+            this.Find<ToggleSwitch>("Launch as administrator").Toggle(launchAsAdmin);
+            this.Find<ToggleSwitch>("Show a warning at startup").Toggle(showWarning);
+
+            // launch Hosts File Editor
+            this.Find<Button>("Launch Hosts File Editor").Click();
+
+            Task.Delay(500).Wait();
+
+            this.Session.Attach(PowerToysModule.Hosts);
+        }
+    }
+}

src/modules/Hosts/Hosts.UITests/Release-Test-Checklist-Migration-Progress.md

@@ -0,0 +1,26 @@
+## This is for tracking UI-Tests migration progress for Hosts File Editor Module
+Refer to [release check list] (https://github.com/microsoft/PowerToys/blob/releaseChecklist/doc/releases/tests-checklist-template.md#hosts-file-editor) for all manual tests.
+
+### Existing Manual Test-cases run by previous PowerToys owner
+For existing manual test-cases, we will convert them to UI-Tests and run them in CI and Release pipeline
+
+ * Launch Host File Editor:
+   - [x] Verify the application exits if "Quit" is clicked on the initial warning. (**HostsSettingTests.TestWarningDialog**)
+   - [x] Launch Host File Editor again and click "Accept". The module should not close. (**HostModuleTests.TestEmptyView**)
+   - [ ] Launch Host File Editor again and click "Accept". The module should not close. Open the hosts file (`%WinDir%\System32\Drivers\Etc`) in a text editor that auto-refreshes so you can see the changes applied by the editor in real time. (VSCode is an editor like this, for example)
+   - [ ] Enable and disable lines and verify they are applied to the file.
+   - [ ] Add a new entry and verify it's applied.
+   - [ ] Add manually an entry with more than 9 hosts in hosts file (Windows limitation) and verify it is split correctly on loading and the info bar is shown.
+   - [x] Try to filter for lines and verify you can find them. (**HostModuleTests.TestFilterControl**)
+   - [ ] Click the "Open hosts file" button and verify it opens in your default editor. (likely Notepad)
+ * Test the different settings and verify they are applied:
+   - [ ] Launch as Administrator.
+   - [x] Show a warning at startup. (**HostsSettingTests.TestWarningDialog**)
+   - [ ] Additional lines position.
+
+### Additional UI-Tests cases
+  - [x] Add manually an entry with more than 9 hosts and Add button should be disabled. (**HostModuleTests.TestTooManyHosts**)
+  - [x] Add manually an entry with less or equal 9 hosts and Add button should be enabled. (**HostModuleTests.TestTooManyHosts**)
+  - [x] Should show empty view if no entries. (**HostModuleTests.TestEmptyView**)
+  - [x] Add a new entry with valid or invalid input (**HostModuleTests.TestAddHost**)
+  - [x] Show save host file error if not run as Administrator. (**HostModuleTests.TestErrorMessageWithNonAdminPermission**)