🐛 [cloud_firestore] experimentalForceLongPolling option

[jbaptisteroesch]

Users on certain corporate networks are experiencing the issue, Could not reach Cloud Firestore backend. Backend didn't respond within 10 seconds likely due to proxies, ad blockers, etc. This issue is known and the team is currently addressing it in the Firebase JS SDK. See: firebase/firebase-js-sdk#1674.

However, it appears no similar action has been taken on the Flutter SDK. An issue was raised a few months ago, but was quickly closed without a definitive resolution. See: #10534.

The options experimentalForceLongPolling or experimentalAutoDetectLongPolling do not appear to be available in the cloud_firestore plugin (v4.8.1).
Did I overlook something, is it a planned feature, or does it not exist at all? Is there a workaround if these settings are unavailable?

The issue is difficult to reproduce and highly situational. I was previously unaware of the problem and am finding it challenging to consider moving the project to another backend service.

Thank you in advance for any assistance.

[darshankawar]

@jbaptisteroesch
Please check this issue and underlying comments related to the said option and team's response on it and see if it helps in your case or not.

[jbaptisteroesch]

@darshankawar when trying the given answer in the issue, I'm facing multiple errors (see screen below).

index.html (pastebin):
js code added on lines 99/100/105/106

[darshankawar]

Although https://www.reddit.com/r/Firebase/comments/uw6qa8/cloud_functions_unexpected_token_export/ isn't related to cloud_firestore, but can you take a look at the comments and see if it helps pertaining to the issue you mentioned above ?

[jbaptisteroesch]

The Reddit post wasn't particularly helpful, but I tried another solution that didn't return any errors on my end.

Here's the code:

<script src="https://www.gstatic.com/firebasejs/9.23.0/firebase-app-compat.js"></script>
  <script src="https://www.gstatic.com/firebasejs/9.23.0/firebase-firestore-compat.js"></script>
  <script>
    // Your web app's Firebase configuration
    var firebaseConfig = {
      apiKey: "",
      authDomain: "",
      databaseURL: "",
      projectId: "",
      storageBucket: "",
      messagingSenderId: "",
      appId: "",
      measurementId: ""
    };
    // Initialize Firebase
    firebase.initializeApp(firebaseConfig);
    firebase.firestore().settings({ experimentalForceLongPolling: true })

However, I have a minor concern about it. I couldn't find any other place than the index.html file to store the firebaseConfigelement and all the keys are visible. According to this post, this isn't a security issue because the protection of data should be handled through database rules. I'm curious to know if this is accurate. I appreciate your guidance.

[darshankawar]

Thanks for the update @jbaptisteroesch
You can check out below links and see if they help to answer your question:

https://firebase.google.com/docs/functions/config-env?gen=2nd
https://firebase.google.com/docs/functions/config-env?gen=2nd#secret-manager
https://cloud.google.com/secret-manager/docs

[willcalderbank]

Sorry @darshankawar I think your missing the issue here.

We are having the same underlying issue which was fixed by setting experimentalForceLongPolling on the js SDK. This used to be done as mentioned on this issue #6635 (as @darshankawar stated) however since then cloud_firestore is no longer initialized in the index and instead is initialized by Firebase.initializeApp in dart there is no way of pushing settings into the underlying JS sdk.

The issue we have is that experimentalForceLongPolling setting isn't exposed via the dart library.

Other settings such as 'enablePersistence' and 'cacheSizeBytes' are exposed, 'experimentalForceLongPolling' is not.

Thanks

Edit: apologues I've just dug through the source enablePersistence is not able to be set via settings but a different call, however cacheSizeBytes, host, ssl, and ignoreUndefinedProperties can be

[willcalderbank]

I've been digging through the docs and source code, amazingly 4 days ago version 4.8.2 was released which updated the underlying sdk which now has experimentalAutoDetectLongPolling set to true by default. Its fairly hidden in the doc's and in no way obvious that this changed.

@jbaptisteroesch that may have fixed your problem. However I still think its a good idea to have these settings exposed in the dart lib.

[jbaptisteroesch]

@darshankawar thanks to this post found on stackoverlow I was able to fix the exposed key issue. However, now that this seems to work, I have a message in the console:

firebase/firestore: Firestore (9.23.0): You are overriding the original host. If you did not intend to override your settings, use {merge: true}.

Does this means that i don't need to initialize Firebase in my main.dart? You can find below my code in the two files.

index.html

<head>
<script src="https://www.gstatic.com/firebasejs/9.23.0/firebase-app-compat.js"></script>
<script src="https://www.gstatic.com/firebasejs/9.23.0/firebase-firestore-compat.js"></script>
</head>
<body>
<script>
  document.addEventListener("firebase-api-key-loaded", function () {
    var firebaseConfig = {
      apiKey: window.FIREBASE_API_KEY,
      authDomain: window.FIREBASE_AUTH_DOMAIN,
      databaseURL: window.FIREBASE_DATABASE_URL,
      projectId: window.FIREBASE_PROJECT_ID,
      storageBucket: window.FIREBASE_STORAGE_BUCKET,
      messagingSenderId: window.FIREBASE_MESSAGING_SENDER_ID,
      appId: window.FIREBASE_APP_ID,
      measurementId: window.FIREBASE_MEASUREMENT_ID
    };
    firebase.initializeApp(firebaseConfig);
    firebase.firestore().settings({experimentalForceLongPolling: true})
  });
</script>
</body>

main.dart

void main() async {
await Firebase.initializeApp(
      options: DefaultFirebaseOptions.currentPlatform,
    );
}

@willcalderbank I was enable to find a part of code related to this issue in the 4.8.2 version in this PR.

[jcwasher]

@jbaptisteroesch That console message has always been there for us. I think the issue is that you do need to initialize your app in main, which will override the settings, but maybe you can get away with not doing so if you don't also support mobile apps. Haven't tried that because our app is cross-platform.

But we've run into the same issue. A lot of our users work for companies with strict network security, so experimentalForceLongPolling has been very important to us. We were stuck on older versions, but it has gotten to the point where it needed to be addressed. I have forked the flutterfire repo and added both long polling options to settings. Just got it caught up with cloud_firestore 4.8.2.

I'm going to leave the experimentalAutoDetectLongPolling default value as false in my fork even though it sounds like they finally flipped that to true in the JS SDK. Reason being that both long polling options cannot be true at the same time. So we will overwrite that value if flutterfire is currently wrapping that version of the JS SDK. I will plan on maintaining this fork until they remove experimentalForceLongPolling. Here are the notes in the JS docs on that:

Forces the SDK’s underlying network transport (WebChannel) to use long-polling. Each response from the backend will be closed immediately after the backend sends data (by default responses are kept open in case the backend has more data to send). This avoids incompatibility issues with certain proxies, antivirus software, etc. that incorrectly buffer traffic indefinitely. Use of this option will cause some performance degradation though. This setting cannot be used with experimentalAutoDetectLongPolling and may be removed in a future release. If you find yourself using it to work around a specific network reliability issue, please tell us about it in firebase/firebase-js-sdk#1674 setting cannot be used in a Node.js environment.

Btw we haven't rolled this into production yet, but I have confirmed that the network requests are following the long polling pattern. We plan to roll out a release including this in the next couple of weeks.

Here's what you need to add to your pubspec.yaml if you want to give the fork a try:

dependency_overrides:
  # Override to allow for experimentalForceLongPolling
  cloud_firestore:
    git:
      url: https://github.com/jcwasher/flutterfire.git
      path: packages/cloud_firestore/cloud_firestore
      ref: fcd904c983309213b90bf2d66826899c89ef5f73
  cloud_firestore_platform_interface:
    git:
      url: https://github.com/jcwasher/flutterfire.git
      path: packages/cloud_firestore/cloud_firestore_platform_interface
      ref: fcd904c983309213b90bf2d66826899c89ef5f73
  cloud_firestore_web:
    git:
      url: https://github.com/jcwasher/flutterfire.git
      path: packages/cloud_firestore/cloud_firestore_web
      ref: fcd904c983309213b90bf2d66826899c89ef5f73

And then in your main.dart:

  await Firebase.initializeApp(options: DefaultFirebaseOptions.currentPlatform);
  FirebaseFirestore.instance.settings = Settings(
    experimentalForceLongPolling: true,
  );

[darshankawar]

Thanks for the updates. Based on the reports and feedback, keeping this issue open for team's input.

/cc @russellwheatley

[vietstone-ng]

@jbaptisteroesch That console message has always been there for us. I think the issue is that you do need to initialize your app in main, which will override the settings, but maybe you can get away with not doing so if you don't also support mobile apps. Haven't tried that because our app is cross-platform.

But we've run into the same issue. A lot of our users work for companies with strict network security, so experimentalForceLongPolling has been very important to us. We were stuck on older versions, but it has gotten to the point where it needed to be addressed. I have forked the flutterfire repo and added both long polling options to settings. Just got it caught up with cloud_firestore 4.8.2.

I'm going to leave the experimentalAutoDetectLongPolling default value as false in my fork even though it sounds like they finally flipped that to true in the JS SDK. Reason being that both long polling options cannot be true at the same time. So we will overwrite that value if flutterfire is currently wrapping that version of the JS SDK. I will plan on maintaining this fork until they remove experimentalForceLongPolling. Here are the notes in the JS docs on that:

Forces the SDK’s underlying network transport (WebChannel) to use long-polling. Each response from the backend will be closed immediately after the backend sends data (by default responses are kept open in case the backend has more data to send). This avoids incompatibility issues with certain proxies, antivirus software, etc. that incorrectly buffer traffic indefinitely. Use of this option will cause some performance degradation though. This setting cannot be used with experimentalAutoDetectLongPolling and may be removed in a future release. If you find yourself using it to work around a specific network reliability issue, please tell us about it in firebase/firebase-js-sdk#1674 setting cannot be used in a Node.js environment.

Btw we haven't rolled this into production yet, but I have confirmed that the network requests are following the long polling pattern. We plan to roll out a release including this in the next couple of weeks.

Here's what you need to add to your pubspec.yaml if you want to give the fork a try:

dependency_overrides:
  # Override to allow for experimentalForceLongPolling
  cloud_firestore:
    git:
      url: https://github.com/jcwasher/flutterfire.git
      path: packages/cloud_firestore/cloud_firestore
      ref: fcd904c983309213b90bf2d66826899c89ef5f73
  cloud_firestore_platform_interface:
    git:
      url: https://github.com/jcwasher/flutterfire.git
      path: packages/cloud_firestore/cloud_firestore_platform_interface
      ref: fcd904c983309213b90bf2d66826899c89ef5f73
  cloud_firestore_web:
    git:
      url: https://github.com/jcwasher/flutterfire.git
      path: packages/cloud_firestore/cloud_firestore_web
      ref: fcd904c983309213b90bf2d66826899c89ef5f73

And then in your main.dart:

  await Firebase.initializeApp(options: DefaultFirebaseOptions.currentPlatform);
  FirebaseFirestore.instance.settings = Settings(
    experimentalForceLongPolling: true,
  );

Oh, it seems this solution does not work anymore, at least in my situation. This is my updated pubspec

dependency_overrides:
  # Override to allow for experimentalForceLongPolling
  cloud_firestore:
    git:
      url: https://github.com/jcwasher/flutterfire.git
      path: packages/cloud_firestore/cloud_firestore
      ref: dc2f319478b246f41619c818103ae3f86dd8d36f
  cloud_firestore_platform_interface:
    git:
      url: https://github.com/jcwasher/flutterfire.git
      path: packages/cloud_firestore/cloud_firestore_platform_interface
      ref: dc2f319478b246f41619c818103ae3f86dd8d36f
  cloud_firestore_web:
    git:
      url: https://github.com/jcwasher/flutterfire.git
      path: packages/cloud_firestore/cloud_firestore_web
      ref: dc2f319478b246f41619c818103ae3f86dd8d36f

I confirm that when I write a JS app with firebase-js-sdk and turn on this option, it works. But this does not for my Flutter app. Am I doing something wrong?

[jcwasher]

Hi @vietstone-ng, it looks like you got this working on a custom fork? Let me know if you are still having troubles and I can share my new SHAs with you. I recently updated my fork.

[vietstone-ng]

@jcwasher I tried to replicate your work on cloud_firestore, cloud_firestore_platform_interface and cloud_firestore_web into my custom fork, but I'm still having the problem. Could you share your new SHAs?

When I tried to create a JS app and turn on this option, it works as described in firebase/firebase-js-sdk#1674 (comment). But no luck with Flutter.

[jcwasher]

@vietstone-ng Sure, here are my latest.

dependency_overrides:
  cloud_firestore:
    git:
      url: https://github.com/jcwasher/flutterfire.git
      path: packages/cloud_firestore/cloud_firestore
      ref: dc2f319478b246f41619c818103ae3f86dd8d36f
  cloud_firestore_platform_interface:
    git:
      url: https://github.com/jcwasher/flutterfire.git
      path: packages/cloud_firestore/cloud_firestore_platform_interface
      ref: dc2f319478b246f41619c818103ae3f86dd8d36f
  cloud_firestore_web:
    git:
      url: https://github.com/jcwasher/flutterfire.git
      path: packages/cloud_firestore/cloud_firestore_web
      ref: dc2f319478b246f41619c818103ae3f86dd8d36f

[vietstone-ng]

@jcwasher Thank you very much. But it does not work in my situation.

I opened Chrome's dev tool and saw that the settings are passed correctly to JS layer, but the error still happens.

Below is my pubspec:

# pubspec.yaml
...
dependencies:
  # Firebase
  firebase_core: ^2.24.2
  firebase_auth: ^4.16.0
  firebase_messaging: ^14.7.10
  cloud_firestore: ^4.14.0
  firebase_crashlytics: ^3.4.9
  cloud_functions: ^4.6.0
  firebase_storage: ^11.6.7
  firebase_remote_config: ^4.3.15

  # Firebase web
  firebase_core_web: 2.12.0
  firebase_auth_web: ^5.8.13
  firebase_messaging_web: ^3.5.18
  cloud_firestore_web: ^3.9.0
  cloud_functions_web: ^4.6.11

dependency_overrides:
  cloud_firestore:
    git:
      url: https://github.com/jcwasher/flutterfire.git
      path: packages/cloud_firestore/cloud_firestore
      ref: dc2f319478b246f41619c818103ae3f86dd8d36f
  cloud_firestore_platform_interface:
    git:
      url: https://github.com/jcwasher/flutterfire.git
      path: packages/cloud_firestore/cloud_firestore_platform_interface
      ref: dc2f319478b246f41619c818103ae3f86dd8d36f
  cloud_firestore_web:
    git:
      url: https://github.com/jcwasher/flutterfire.git
      path: packages/cloud_firestore/cloud_firestore_web
      ref: dc2f319478b246f41619c818103ae3f86dd8d36f
...

The attached image is my devtool where JS's initializeFirestore is called with experimentalAutoDetectLongPolling = true

and the log:

Launching lib/main.dart on Chrome in debug mode...
This app is linked to the debug service: ws://127.0.0.1:63126/JM2sWtLQWk0=/ws
Debug service listening on ws://127.0.0.1:63126/JM2sWtLQWk0=/ws
Connecting to VM Service at ws://127.0.0.1:63126/JM2sWtLQWk0=/ws
WARNING: found an existing <meta name="viewport"> tag. Flutter Web uses its own viewport configuration for better compatibility with Flutter. This tag will be replaced.
[2024-04-11T06:50:33.053Z]  @firebase/firestore:
Bad state: No element
[cloud_firestore/unavailable] Failed to get document because the client is offline.
Error: TypeError: null: type 'Null' is not a subtype of type 'Map<dynamic, dynamic>'
dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/errors.dart 297:3  throw_
dart-sdk/lib/_internal/js_shared/lib/rti.dart 1385:3                         _failedAsCheck
dart-sdk/lib/_internal/js_shared/lib/rti.dart 1363:3                         _generalAsCheckImplementation
dart-sdk/lib/_internal/js_dev_runtime/patch/async_patch.dart 45:50           <fn>
dart-sdk/lib/async/zone.dart 1661:54                                         runUnary
dart-sdk/lib/async/future_impl.dart 162:18                                   handleValue
dart-sdk/lib/async/future_impl.dart 838:44                                   handleValueCallback
dart-sdk/lib/async/future_impl.dart 867:13                                   _propagateToListeners
dart-sdk/lib/async/future_impl.dart 643:5                                    [_completeWithValue]
dart-sdk/lib/async/future_impl.dart 713:7                                    callback
dart-sdk/lib/async/schedule_microtask.dart 40:11                             _microtaskLoop
dart-sdk/lib/async/schedule_microtask.dart 49:5                              _startMicrotaskLoop
dart-sdk/lib/_internal/js_dev_runtime/patch/async_patch.dart 181:7           <fn>
[2024-04-11T06:50:41.978Z]  @firebase/firestore:

I also noticed that the JS's DEFAULT_AUTO_DETECT_LONG_POLLING flag is enabled by default, it means experimentalAutoDetectLongPolling is enabled if I do not set any value.

But it does not work in my situation. Can someone help me to check what problem is happening?

[jcwasher]

@vietstone-ng A few questions...

1. Are you using the flutterfire cli to set up your Firebase project?
2. Why are you specifying versions for Firebase sub dependencies? Did you have conflicts with other packages? It could be that something is not being reconciled correctly between versions.
3. Have you tried setting experimentalForceLongPolling to true?
4. What does your main look like?

Here are the versions I'm specifying for my other Firebase packages. Fwiw I'm not using some of the others that you are:

cloud_firestore: 4.15.9
firebase_auth: 4.17.9
firebase_core: 2.27.1

Note it may be helpful to not use the ^ before the version number, as that may be upgrading packages in such a way that causes conflicts with the fork.

[vietstone-ng]

@jcwasher Thank you very much for your time.

I used flutterfire cli the set up Firebase, and tried both experimentalAutoDetectLongPolling / experimentalForceLongPolling
I tried js code and saw that there're 3 js ways to work with Firestore: compat/namspaced API, normal modular API, and lite modular API.
The normal modular API does not work in my situation, and this is the way FlutterFire used to work on the web.
I reported it here: firebase/firebase-js-sdk#8176

So the problem seems the JS code does not work in my situation, not Dart code

[Justus-M]

My customers are getting the same error from your last log "Failed to get document because the client is offline." on certain corporate networks.

In my case it is on Flutter Windows