Skip to content

Commit 6264320

Browse files
freakboy3742miss-islington
authored andcommitted
gh-137973: Add a non-parallel test plan to the iOS testbed project (GH-138018)
Modifies the iOS testbed project to add a test plan. This simplifies the iOS test runner, as we can now use the built-in log streaming to see test results. It also allows for some other affordances, like providing a default LLDB config, and using a standardized mechanism for specifying test arguments. (cherry picked from commit 2ba2287b85eea3cc3a71d77c6bcf9eb5670ca05d) Co-authored-by: Russell Keith-Magee <russell@keith-magee.com>
1 parent 132b866 commit 6264320

File tree

11 files changed

+271
-346
lines changed

11 files changed

+271
-346
lines changed

.gitignore

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,6 @@ iOS/testbed/Python.xcframework/ios-*/lib
8080
iOS/testbed/Python.xcframework/ios-*/Python.framework
8181
iOS/testbed/iOSTestbed.xcodeproj/project.xcworkspace
8282
iOS/testbed/iOSTestbed.xcodeproj/xcuserdata
83-
iOS/testbed/iOSTestbed.xcodeproj/xcshareddata
8483
Mac/Makefile
8584
Mac/PythonLauncher/Info.plist
8685
Mac/PythonLauncher/Makefile

Doc/using/ios.rst

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -374,6 +374,17 @@ You can also open the testbed project in Xcode by running:
374374
375375
This will allow you to use the full Xcode suite of tools for debugging.
376376

377+
The arguments used to run the test suite are defined as part of the test plan.
378+
To modify the test plan, select the test plan node of the project tree (it
379+
should be the first child of the root node), and select the "Configurations"
380+
tab. Modify the "Arguments Passed On Launch" value to change the testing
381+
arguments.
382+
383+
The test plan also disables parallel testing, and specifies the use of the
384+
``iOSTestbed.lldbinit`` file for providing configuration of the debugger. The
385+
default debugger configuration disables automatic breakpoints on the
386+
``SIGINT``, ``SIGUSR1``, ``SIGUSR2``, and ``SIGXFSZ`` signals.
387+
377388
App Store Compliance
378389
====================
379390

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
The iOS test runner has been simplified, resolving some issues that have
2+
been observed using the runner in GitHub Actions and Azure Pipelines test
3+
environments.

iOS/README.rst

Lines changed: 15 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -293,7 +293,7 @@ project, and then boot and prepare the iOS simulator.
293293
Debugging test failures
294294
-----------------------
295295

296-
Running ``make test`` generates a standalone version of the ``iOS/testbed``
296+
Running ``make testios`` generates a standalone version of the ``iOS/testbed``
297297
project, and runs the full test suite. It does this using ``iOS/testbed``
298298
itself - the folder is an executable module that can be used to create and run
299299
a clone of the testbed project.
@@ -316,12 +316,26 @@ This is the equivalent of running ``python -m test -W test_os`` on a desktop
316316
Python build. Any arguments after the ``--`` will be passed to testbed as if
317317
they were arguments to ``python -m`` on a desktop machine.
318318

319+
Testing in Xcode
320+
^^^^^^^^^^^^^^^^
321+
319322
You can also open the testbed project in Xcode by running::
320323

321324
$ open my-testbed/iOSTestbed.xcodeproj
322325

323326
This will allow you to use the full Xcode suite of tools for debugging.
324327

328+
The arguments used to run the test suite are defined as part of the test plan.
329+
To modify the test plan, select the test plan node of the project tree (it
330+
should be the first child of the root node), and select the "Configurations"
331+
tab. Modify the "Arguments Passed On Launch" value to change the testing
332+
arguments.
333+
334+
The test plan also disables parallel testing, and specifies the use of the
335+
``iOSTestbed.lldbinit`` file for providing configuration of the debugger. The
336+
default debugger configuration disables automatic breakpoints on the
337+
``SIGINT``, ``SIGUSR1``, ``SIGUSR2``, and ``SIGXFSZ`` signals.
338+
325339
Testing on an iOS device
326340
^^^^^^^^^^^^^^^^^^^^^^^^
327341

@@ -336,40 +350,3 @@ select the root node of the project tree (labeled "iOSTestbed"), then the
336350
(this will likely be your own name), and plug in a physical device to your
337351
macOS machine with a USB cable. You should then be able to select your physical
338352
device from the list of targets in the pulldown in the Xcode titlebar.
339-
340-
Running specific tests
341-
^^^^^^^^^^^^^^^^^^^^^^
342-
343-
As the test suite is being executed on an iOS simulator, it is not possible to
344-
pass in command line arguments to configure test suite operation. To work
345-
around this limitation, the arguments that would normally be passed as command
346-
line arguments are configured as part of the ``iOSTestbed-Info.plist`` file
347-
that is used to configure the iOS testbed app. In this file, the ``TestArgs``
348-
key is an array containing the arguments that would be passed to ``python -m``
349-
on the command line (including ``test`` in position 0, the name of the test
350-
module to be executed).
351-
352-
Disabling automated breakpoints
353-
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
354-
355-
By default, Xcode will inserts an automatic breakpoint whenever a signal is
356-
raised. The Python test suite raises many of these signals as part of normal
357-
operation; unless you are trying to diagnose an issue with signals, the
358-
automatic breakpoints can be inconvenient. However, they can be disabled by
359-
creating a symbolic breakpoint that is triggered at the start of the test run.
360-
361-
Select "Debug > Breakpoints > Create Symbolic Breakpoint" from the Xcode menu, and
362-
populate the new brewpoint with the following details:
363-
364-
* **Name**: IgnoreSignals
365-
* **Symbol**: UIApplicationMain
366-
* **Action**: Add debugger commands for:
367-
- ``process handle SIGINT -n true -p true -s false``
368-
- ``process handle SIGUSR1 -n true -p true -s false``
369-
- ``process handle SIGUSR2 -n true -p true -s false``
370-
- ``process handle SIGXFSZ -n true -p true -s false``
371-
* Check the "Automatically continue after evaluating" box.
372-
373-
All other details can be left blank. When the process executes the
374-
``UIApplicationMain`` entry point, the breakpoint will trigger, run the debugger
375-
commands to disable the automatic breakpoints, and automatically resume.

0 commit comments

Comments
 (0)