863676 - Sporadic display issue with soft keyboard

Status: RESOLVED FIXED

(Core Graveyard :: Widget: WinRT, defect)

Description

[Jim Mathies [:jimm]]

There is a random display issue with the skb, which appears to have something to do with UIA or widget or maybe it's just some weird timing issue. Here's a UIA trace after a tap on an in-content text input:

mozilla::widget::winrt::UIABridge::GetPropertyValue
mozilla::widget::winrt::UIABridge::Navigate
NavigateDirection_Parent
mozilla::widget::winrt::UIABridge::Navigate
NavigateDirection_Parent
mozilla::widget::winrt::UIABridge::GetFocus
Focus element flags: 1 1 0
mozilla::widget::winrt::UIATextElement::SetFocusInternal
mozilla::widget::winrt::UIATextElement::GetPropertyValue
mozilla::widget::winrt::UIATextElement::GetPropertyValue
mozilla::widget::winrt::UIATextElement::Navigate
mozilla::widget::winrt::UIATextElement::GetPropertyValue
UIATextElement::GetPropertyValue: idProp=30016
mozilla::widget::winrt::UIATextElement::GetPropertyValue
mozilla::widget::winrt::UIATextElement::Navigate
mozilla::widget::winrt::UIABridge::GetPropertyValue
mozilla::widget::winrt::UIABridge::GetPropertyValue
mozilla::widget::winrt::UIABridge::Navigate
mozilla::widget::winrt::UIABridge::Navigate
mozilla::widget::winrt::UIABridge::GetPropertyValue
mozilla::widget::winrt::UIATextElement::get_BoundingRectangle
mozilla::widget::winrt::UIATextElement::GetPropertyValue
UIATextElement::GetPropertyValue: idProp=30003
mozilla::widget::winrt::UIATextElement::GetPropertyValue
UIATextElement::GetPropertyValue: idProp=49228
UIATextElement: Unhandled property
mozilla::widget::winrt::UIATextElement::GetPatternProvider
UIATextElement::GetPatternProvider=10014
** TextPattern requested from element.
mozilla::widget::winrt::UIATextElement::GetPatternProvider
UIATextElement::GetPatternProvider=10029
mozilla::widget::winrt::UIATextElement::GetPropertyValue
UIATextElement::GetPropertyValue: idProp=30019
mozilla::widget::winrt::UIATextElement::GetPatternProvider
UIATextElement::GetPatternProvider=10002
** ValuePattern requested from element.
mozilla::widget::winrt::UIATextElement::GetPatternProvider
UIATextElement::GetPatternProvider=10002
** ValuePattern requested from element.
mozilla::widget::winrt::UIATextElement::get_IsReadOnly
mozilla::widget::winrt::UIABridge::FocusChangeEvent
Focus element flags: 1 1 0
mozilla::widget::winrt::UIATextElement::AnnounceFocus
mozilla::widget::winrt::UIATextElement::GetPropertyValue
mozilla::widget::winrt::UIATextElement::GetPropertyValue
mozilla::widget::winrt::UIATextElement::Navigate
mozilla::widget::winrt::FrameworkView::OnSoftkeyboardShown
************* metro_softkeyboard_shown 
mozilla::widget::winrt::FrameworkView::OnSoftkeyboardHidden
************* metro_softkeyboard_hidden 


I have not been able to track down the cause of this. The keyboard hidden event fires for a reason that isn't apparent in widget logging. I've managed to trap the callback event in the debugger but unfortunately the stack isn't of much use since it's sent to metrofx async over com, so whatever triggers it is off the stack.

I'm at a loss as to the cause.

I've also tested with TSF disabled and can still reproduce, so I'm pretty sure that module isn't involved.

This may be due to the nature of UIABridge which isn't a real UIA implementation. It's just a light weight UIA wrapper that holds a single IRawElementProviderSimple who's bounds change when the user moves around the document.

Comment 1

[Jim Mathies [:jimm]]

What we might consider is to go ahead and implement bug 762817. That would take some work, but it would allow us to latch into our existing accessibility layer fully and remove the temporary bridge code.

Comment 2

[Jim Mathies [:jimm]]

Attachment: text input test case

Comment 8

[Marco Mucci [:MarcoM]]

Hey Jim, no points required.  This is a dependent work item part of Story Bug 855297.

Comment 9

[Jim Mathies [:jimm]]

Not sure why this blocked bug 861945, they two are unrelated.

Comment 10

[Jim Mathies [:jimm]]

Should we put time into trying to nail down our remaining keyboard display problem for 26?

Comment 11

[Asa Dotzler [:asa]]

(In reply to Jim Mathies [:jimm] from comment #10)
> Should we put time into trying to nail down our remaining keyboard display
> problem for 26?

Yes, please. Let's add this to the Preview list. Thanks, Jim.

Comment 12

[Marco Mucci [:MarcoM]]

Hey Jim, are you taking the full story (Bug 855297) or just this work item for Iteration #12?  If it's the whole story, I'll add it to the iteration and the MP list.  If it's just the work item, no need to do anything else.  Thanks.

Comment 13

[Jim Mathies [:jimm]]

(In reply to Marco Mucci [:MarcoM] from comment #12)
> Hey Jim, are you taking the full story (Bug 855297) or just this work item
> for Iteration #12?  If it's the whole story, I'll add it to the iteration
> and the MP list.  If it's just the work item, no need to do anything else. 
> Thanks.

Those others should probably be triaged independently. I'm only planning on working on this bug.

Comment 14

[Jim Mathies [:jimm]]

Attachment: debug patch

Comment 15

[Jim Mathies [:jimm]]

Attachment: wip

Comment 16

[Jim Mathies [:jimm]]

Attachment: wip

Comment 17

[Jim Mathies [:jimm]]

Attachment: textboxes.html

Comment 18

[Jim Mathies [:jimm]]

Attachment: twotextareas.html

Comment 19

[Jim Mathies [:jimm]]

Attachment: forms.html

Comment 20

[Jim Mathies [:jimm]]

Hey Juan, I think I may have a good fix for this. I've been testing on a surface pro with this try build - 

https://tbpl.mozilla.org/?tree=Try&showall=0&rev=12e7404f9314

without issue. Curious if QA might have the time to take this build for a spin on some other hardware to see if the problem persists?

Comment 21

[juan becerra [:juanb]]

I'll take a look now with the three devices I have (Surface Pro, Dell XPS, Iconia W3).

Comment 22

[Jim Mathies [:jimm]]

(In reply to juan becerra [:juanb] from comment #21)
> I'll take a look now with the three devices I have (Surface Pro, Dell XPS,
> Iconia W3).

Thanks!

Comment 23

[juan becerra [:juanb]]

I have been testing this on the three machines listed in comment #21 with some of the examples here, as well just playing around with only the skb, and I haven't seen any issues with the software keyboard itself.

Comment 24

[Jim Mathies [:jimm]]

Attachment: patch v.1

A lot of this is logging and some refining of the UIABridge code. The key change here is the added WM_GETOBJECT handler in MetroWidget.

Comment 25

[Brian R. Bondy [:bbondy]]

Comment on attachment 785138 [details] [diff] [review]
patch v.1

Review of attachment 785138 [details] [diff] [review]:
-----------------------------------------------------------------

::: widget/windows/winrt/FrameworkView.cpp
@@ +367,5 @@
>      return S_OK;
>    sKeyboardIsVisible = false;
>    memset(&sKeyboardRect, 0, sizeof(Rect));
>    MetroUtils::FireObserver("metro_softkeyboard_hidden");
> +  aArgs->put_EnsuredFocusedElementInView(true);

Can aArg ever be passed in a nullptr value?

Comment 26

[Jim Mathies [:jimm]]

(In reply to Brian R. Bondy [:bbondy] from comment #25)
> Comment on attachment 785138 [details] [diff] [review]
> patch v.1
> 
> Review of attachment 785138 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: widget/windows/winrt/FrameworkView.cpp
> @@ +367,5 @@
> >      return S_OK;
> >    sKeyboardIsVisible = false;
> >    memset(&sKeyboardRect, 0, sizeof(Rect));
> >    MetroUtils::FireObserver("metro_softkeyboard_hidden");
> > +  aArgs->put_EnsuredFocusedElementInView(true);
> 
> Can aArg ever be passed in a nullptr value?

Don't think winrt would send us a null args param.

Comment 27

[Jim Mathies [:jimm]]

https://hg.mozilla.org/integration/fx-team/rev/859a06fa227c

Comment 28

[Tim Taubert [:ttaubert] (inactive)]

https://hg.mozilla.org/mozilla-central/rev/859a06fa227c