Skip to content

Transform PointerEvents to the local coordinate system of the event receiver #32192

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
May 31, 2019

Conversation

goderbauer
Copy link
Member

@goderbauer goderbauer commented May 7, 2019

Note: This PR is a follow-up to #31894.

Description

Before this change, PointerEvents would always be expressed in the global coordinate system of the screen. The change adds localPosition and localDelta to PointerEvents, which are expressed in the local coordinate system of the event receiver. This means - among other things - that scrolling now sticks to your finger even if the Scrollable has been scaled up or down. Furthermore, a rotated scrollable can now be scrolled by moving the finger in the (rotated) direction of the scrollview.

Furthermore, this change also fixes hit testing for perspective transforms. For example, if a view is rotated along its x axis, buttons now continue to react to touch events.

Related Issues

Fixes #29916
Fixes #18408
Fixes #31958

Tests

I added the following tests:

  • wrapping HitTestResult and its subclasses
  • transforming PointerEvents and related helper methods
  • PointerRouter transforms events before delivering them to the route
  • PointerSignalResolver delivers the transformed event
  • Tests for changed GestureRecognizers to ensure they work in transformed subtrees
  • Tests for new MatrixUtil functions
  • Tests for position transforming methods on BoxHitTestResult and SliverHitTestResult
  • Listener receives transformed events for touch/signal/mouse
  • Scrolling now works as expected (sticks to the finger) in transformed subtrees (incl. perspective transform).

Checklist

Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

  • I read the Contributor Guide and followed the process outlined there for submitting PRs.
  • I signed the CLA.
  • I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
  • I updated/added relevant documentation (doc comments with ///).
  • All existing and new tests are passing.
  • The analyzer (flutter analyze --flutter-repo) does not report any problems on my PR.
  • I am willing to follow-up on review comments in a timely manner.

Breaking Change

Does your PR require Flutter developers to manually update their apps to accommodate your change?

@goderbauer goderbauer added the framework flutter/packages/flutter repository. See also f: labels. label May 7, 2019
@ooyemyintaung007
Copy link

Thank

@gspencergoog
Copy link
Contributor

gspencergoog commented May 8, 2019

From the test failures, looks like you need to check for a null onHover callback before trying to call it.

Also, I've been working on trying to make a Iterable<S> findAll<S>(Offset regionOffset) on the Layer that will return all of the found annotations at that offset, instead of just the first one.

Otherwise, we can't nest Listeners (i.e. we can't put a button that handles hover detection inside of another Listener that also handles mouse enter/exit).

How will this change with this PR? WIll I need to apply a transform at each child layer? (And maybe I should wait until this lands?)

@gspencergoog
Copy link
Contributor

Actually, in looking at your code some more, it looks like my change was orthogonal to yours, so I committed it: #32350

@goderbauer
Copy link
Member Author

@gspencergoog You're right about the null check. Sorry for being confused. Thanks for looking at it!

@goderbauer goderbauer marked this pull request as ready for review May 9, 2019 08:46
@goderbauer goderbauer requested review from gspencergoog and Hixie May 9, 2019 08:46
@VerTiGoEtrex
Copy link

Eager to see this merged -- wish I had seen this before I started trying to learn how to "unproject" perspective matrices for hit-testing. Thanks for your work -- it works perfectly!

thanks

/// Returns the transformation of `position` into the coordinate system
/// described by `transform`.
///
/// The z-value of `position` is assumed to be 0.0. If `transform` is null,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not just assumed to be zero, it's set to zero (or, really, ignored and zero used instead). And why is that? Wouldn't you want Z to also be preserved if set?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this function is used to convert global to local coordinates, it appears to have the some problem with out of plane rotations as the Transform widget which I tried to describe here #18408 (comment).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gspencergoog Offsets only have an x and y value - and the z value is always assumed to be zero. The method is just explicitly documenting that.

@spkersten The secret is in the transform matrix that's passed into this. We take the paint transform, set the third row and third column to (0,0,1,0) and then invert it to get a transform matrix that ignores z.

/// Usually, the [global] [Offset] is in the coordinate space of the screen
/// after conversion to logical pixels and the [local] offset is the same
/// [Offset], but transformed to a local coordinate space.
class CombinedOffset {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a big fan of this name (it's not an Offset subclass)...

Maybe something based on the word Position? I don't have a good idea...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there some way we can avoid making a public class at all for this? It's not clear to me that it does anything outside of this use case we need for this class.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed to OffsetPair.

This class may be useful for people implementing their own gesture detectors, so I think it is ok to be part of the public API.

@Hixie
Copy link
Contributor

Hixie commented May 10, 2019

LGTM modulo comments (especially we need a better name than CombinedOffset)

Copy link
Contributor

@spkersten spkersten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've made some comments on the transformations, based on what I found for the Transform widget's hit testing. Please consider my comments as doubts that could be looked into and not as facts, since I've spend only a limited amount of time with this PR so far.

expect(PointerEvent.transformPosition(Matrix4.identity(), position), position);
final Matrix4 transform = Matrix4.translationValues(10, 20, 0);
expect(PointerEvent.transformPosition(transform, position), const Offset(20.0 + 10.0, 30.0 + 20.0));
});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having this test with a transformation containing an out of plane rotation (rotateX or rotateY) would be useful as that is a less obvious case that has a bug in the Transform widget.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's the complex "Perspective transform on scrollable" test at the very bottom of the PR that rotates around X and verifies that scrolling still works.

Copy link
Contributor

@spkersten spkersten May 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hope it's okay, but created the test that I proposed to show that it fails: https://github.com/goderbauer/flutter/compare/s2...spkersten:goderbauer-s2-rotation-test?expand=1

Edit: If I replace:
final Matrix4 inverseTransform = Matrix4.inverted(transform);
with
final Matrix4 inverseTransform = Matrix4.inverted(PointerEvent.removePerspectiveTransform(transform));
the test passes.

So it probably comes down to me misunderstanding the code. Played around with some hit testing an transformations and it all seems to work fine. I'm sorry for the confusion.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've looked at the "Perspective transform on scrollable" test and it made me really confused. It also fails when I removed the perspective (setEntry(3,2, 0.001)) and only keep the rotation. So I add the widget tree in an app to see what happens:

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) => MaterialApp(home: Foo());
}

class Foo extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return Transform(
      transform: Matrix4.identity()
        ..setEntry(3, 2, 0.001)
        ..rotateX(math.pi / 4)
      ,
      child: Container(
        padding: EdgeInsets.all(16),
        color: Colors.yellow,
        child: Center(
          child: Container(
            width: 200,
            child: ListView.builder(
//            controller: controller,
              cacheExtent: 0.0,
              itemBuilder: (BuildContext context, int index) {
                return Container(
                  height: 100.0,
                  color: index % 2 == 0 ? Colors.blue : Colors.red,
                  child: Text('Tile $index'),
                );
              },
            ),
          ),
        ),
      ),
    );
  }
}

I've added the DecoratedBox to see more clearly what is going on. Here is a screenshot:
Screenshot 2019-05-24 at 19 42 30
So it looks like the children aren't transformed at all!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/goderbauer/flutter/compare/s2...spkersten:goderbauer-s2-rotation-test?expand=1

Yes, applying MatrixUtils.transformPoint in both directions will not work as that method completely drops the z value on the floor as you already found out. If you use that method to transform a point P(x, y, 0) on the xy-plane of a x-rotated render plane you will end up with a point P'(x', y', z') and z' will most likely not be 0. If you now take the x' and y' of that point and run it through transformPoint again (which just sets z to 0), you will not end up with x and y again.

In combination with PointerEvent.removePerspectiveTransform it works because that method modifies the transform matrix in a way to map point P(x, y, 0) from the xy-plane of the render plane to the point P'(x', y', 0) on the xy-plane of the screen (that's the point on the screen at which P will be drawn). Because z is zero for both points, we can now use MatrixUtils.transformPoint to correctly transform between the two.

Hope this makes sense? Looking at the other example you posted now...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

About your second comment: That looks like an iOS-specific rendering bug. On Android your example renders fine. Let me do a little investigation and file a separate bug about it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, it looks like it was a bug with the old engine version that this PR was synced to. I just pulled in master and everything renders correctly. (And hit testing also works as expected.)

/// Returns the transformation of `position` into the coordinate system
/// described by `transform`.
///
/// The z-value of `position` is assumed to be 0.0. If `transform` is null,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this function is used to convert global to local coordinates, it appears to have the some problem with out of plane rotations as the Transform widget which I tried to describe here #18408 (comment).

/// The resulting matrix can be used to determine the point in the local
/// coordinate system of the transformed component that is exactly under the
/// user's finger when the user is touching the screen.
static Matrix4 paintTransformToPointerEventTransform(Matrix4 paintTransform) {
Copy link
Contributor

@spkersten spkersten May 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why this would work without inverting the matrix. Let's say the paint transformation is to scale both x and y by a factor 2: T = diag(2, 2, 1, 1). That means pixels of the widget are visually presented twice as far from the origin than in the local coordinate system. Then when a user taps the screen on one such pixel, that tap position will be twice as far from the origin than in the local coordinate system as well. So to do hit testing in the local coordinate system, the point would have to be transformed by diag(0.5, 0.5, 1, 1). That should generalise to the inverse of T.

I see now that the RenderObject is still responsible for the inversion, so maybe just the name of this function is confusing to me. And the reason why the perspective would have to be removed is unclear to me.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I renamed this function to make it clearer what it actually does.

Copy link
Contributor

@gspencergoog gspencergoog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

32384589-a60f0e74-c078-11e7-9bc1-e5b5287aea9d

@goderbauer
Copy link
Member Author

I took out the event transformation for mouse-specific events (enter/hover/exit) since supporting those correctly will require some re-structuring of how the MouseTracker works. I filed #33675 to fix that. Please upvote the issue if transformed mouse events are important for you.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
framework flutter/packages/flutter repository. See also f: labels.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Scaling a widget also scales its drag gestures Scrollables do not obey MediaQuery devicePixelRatio Transforms break UI hit testing
8 participants