Dynamic Screenshare Source (Share-this-surface-instead)

[eladalon1983]

Problem statement

User who share one display surface often wish to make a quick transition to sharing another.

Example 1

• The user prepares to deliver a presentation over VC.
• Before the call starts, the user sets up N tabs, each pre-navigated to a specific location, relevant application already set up. Note that some setup might be involved here. Maybe the user has set up some of these applications to a very specific state.
• The user begins call over a VC application.
• The user interacts with VC application, then with browser's dialog, to share the first tab.
• The user interacts with the first shared tab.
• The user wishes to switch to second tab.
• What now? The user has to reactivate the VC tab, re-engage screensharing, etc. Very suboptimal!

Example 2

Same thing, but now that user is recording a lecture using something like Loom.

Chrome's current solution

Chrome currently offers an extension API that presents a Share this tab instead button.

The user just hops to the next tab, presses the button, and continues the presentation.
0 beats skipped.
This solution is implementable by user agents without any API changes. In theory.

But...

Not all applications are able to smoothly deal with a change of source. What about applications that rely on tight-coupling with the captured tab? A change of source breaks that connection. The capturing application can detect that using Capture Handle, but what can it do then? Inform the user why some features suddenly disappear?!

The user is not interested in excuses.
The user is not interested in the distinction between application and browser.
The user protests: "If I was not supposed to press that button - why did you ever show it to me?"

Solution

The browser should not guess whether a share-this-tab-instead button should be presented to the user. (Or any other UA-specific mechanism for switching sources.)

So how about:

enum SourceChangePreferenceEnum {
  "include",
  "exclude"
};

dictionary DisplayMediaStreamConstraints {
  SourceChangePreferenceEnum sourceChange;
};

Is this safe?

The application does NOT gain control over changing the source. It only opts into the user agent exposing to the user the option to change the source. The user remains in full control over whether the source is changed, what it is changed to, and when it is changed.

[jan-ivar]

Chrome currently offers an extension API that presents a Share this tab instead button.

Nice feature! Why isn't Chrome offering it in getDisplayMedia? I thought that's what #170 was for.

What about applications that rely on tight-coupling with the captured tab?

I'd say such applications are mistaken. There's no API I know of that allows tight coupling between an app and a tab.

CaptureHandle allows tight coupling with a captured application, not its tab. Yes, the user agent is obligated to fire the capturehandlechange event when that apps' tab is navigated to replace it, but that's fairly abstract: you switch from interacting with one captured app to the next (one that supports interacting).

A change of source breaks that connection.

How is that different from the user navigating the captured tab?

The capturing application can detect that using Capture Handle, but what can it do then?

The same thing it does when the user navigates the captured tab today?

Inform the user why some features suddenly disappear?!

Features tied to a captured application disappearing when the captured application disappears, seems like it works as intended. Can you rephrase as a problem?

It seems to me user agents can implement a "Share this tab instead" in-browser button today, by firing the capturehandlechange event when capture transitions from documentA in tabA to documentB in tabB, just as if the user navigated tabA from documentA to documentB.

Features backed by CaptureHandle should then "just work" if documentB supports them, and disappear otherwise, just like today. Is anything undesirable about that?

I see no reason for new API surface.

[eladalon1983]

Nice feature! Why isn't Chrome offering it in getDisplayMedia? I thought that's what #170 was for.

A colleague of mine is working on it.

I'd say such applications are mistaken. There's no API I know of that allows tight coupling between an app and a tab.

For self-capture, the two are one and the same. You might be aware that it's now possible to bring your Meet calls into Docs. If you also choose to capture the current tab, it gets cropped to remove the talking heads and share remotely only the doc. Multiple parts of this complex experience (which I name as an example) assume that the current tab is captured. What should they do if the user switches to capturing another tab?

• Keep transmitting the new video remotely but inform the user it's a different tab?
• Break of the capture and ask the user politely to refrain from clicking the share-instead-button again?

Multiple challenges arise that could be avoided if the browser did not put up the button.

How is that different from the user navigating the captured tab?

1. We're not going to prevent the user from navigating tabs; I hope the reasons are self-evident. But should we also allow share-this-tab-instead for any capturer/capturee pair? Are the two things, navigation and share-this-instead, equally fundemental user rights?
2. When navigation happens inside of a website, e.g. by clicking a link, it's very likely that the next document will again set a capture-handle, and communication can resume. You might be able to keep the same communication channel going.
3. When self-capturing, navigation of the top-level document breaks off the capture.

Features tied to a captured application disappearing when the captured application disappears, seems like it works as intended. Can you rephrase as a problem?
...
Is anything undesirable about that?

Glad to rephrase. Look at it from the point of view of the capturing application and its developer. The browser puts this big, shiny, tempting button in front of the user. The user clicks it, and then the app "stops working" (i.e. can still stream the captured content, but not remote-control it anymore). The user doesn't know why. The user blames the app. If the app's developer is upset with the browser, is that unreasonable? Who put up that big shiny button? Was it not... the browser?!

I see no reason for new API surface.

We have just started this conversation. Allow me the opportunity to convince you to the contrary.

[jan-ivar]

You might be aware that it's now possible to bring your Meet calls into Docs.

Thanks for this context! I've tried it in Chrome now.

What should they do if the user switches to capturing another tab?

The same thing as if the user had chosen a different tab in the first place? — e.g. when I first click the "Present now" button, I'm presented with these choices: (it lets me select "Other Tab", even though it defaults to "This Tab")

But when I choose the second tab, I get this error:

Is a "Share this tab instead" button only a problem for apps that want to limit users to self-capture?

If so, I'm not supportive of apps using getDisplayMedia in this way, and opposed to new APIs to aid such use.

It seems like Docs should be using getViewportMedia here. Happy to discuss in w3c/mediacapture-viewport#13 instead.

[eladalon1983]

The same thing as if the user had chosen a different tab in the first place?

Yes, these are similar issues. (But not identical - failure to take off is not the same as crashing mid-flight.) Does it matter though? Should our inability to fix one issue prevent us from addressing the other? We had good reasons to force a full selection on the user even when the application does not want it. Do we have a good reason to mandate that source-changing would either be supported always or never? Why not allow it to fit the app's needs?

Is a "Share this tab instead" button only a problem for apps that want to limit users to self-capture?

I don't think so, but it is the clearer case, and the one that most concerns me. So let's keep things simple and stick with self-capture initially.

If so, I'm not supportive of apps using getDisplayMedia in this way, and opposed to new APIs to aid such use.

Could you elaborate why? I am cognizant of our previous discussions on adjacent topics. For example, I don't think they apply here. This does not push the user towards self-capture to begin with, and does not prevent the user from stopping the capture. So those probably aren't your concerns. What are your concerns, then?

[eladalon1983]

During the 2022-05-17 WebRTC WG interim meeting, recorded here, the consensus was to proceed with this as a hint. I'll send a PR.

[jan-ivar]

The minutes of that meeting are here.