Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feat: Threaded MutationsBatcher #722

Merged
merged 24 commits into from
May 10, 2023
Merged

Feat: Threaded MutationsBatcher #722

Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
3010cd5
Feat: Threaded MutationsBatcher
Mariatta Jan 12, 2023
dbf80c3
Make _MutationsBatchQueue private
Mariatta Jan 13, 2023
9ef3ec5
Fix the tests.
Mariatta Jan 16, 2023
148d1c0
Unit test adjustments
Mariatta Jan 17, 2023
f312659
Add flush_interval.
Mariatta Jan 18, 2023
e657321
Add test for the batcher time interval
Mariatta Jan 18, 2023
00e03e5
Adjust parameters
Mariatta Jan 31, 2023
649cd9b
Raise errors from mutations.
Mariatta Feb 1, 2023
1a5ff05
test coverage
Mariatta Feb 2, 2023
70f1cc8
Use `from_grpc_status`
Mariatta Feb 10, 2023
fd06c2f
Update MutationsBatchError
Mariatta Feb 16, 2023
b912792
Add flow control to control inflight requests
Mariatta Mar 9, 2023
cc334ec
Improve test coverage
Mariatta Mar 10, 2023
177414f
Submit the batch when queue's empty
Mariatta Mar 10, 2023
d907180
Update google/cloud/bigtable/batcher.py
Mariatta Mar 23, 2023
d0a6ba3
Changes based on PR review
Mariatta Mar 29, 2023
06bc2c1
Update google/cloud/bigtable/batcher.py
Mariatta Apr 5, 2023
c0584b5
Update google/cloud/bigtable/batcher.py
Mariatta Apr 5, 2023
9899e45
Changes based on PR review
Mariatta Apr 5, 2023
51b063b
Docstring updates
Mariatta Apr 5, 2023
1e42c1b
Remove unused variable, is_open
Mariatta Apr 6, 2023
c1c50ea
Adjust the logic for when the batch is larger than available resources.
Mariatta May 5, 2023
414fe87
Make `_flush_async` private
Mariatta May 9, 2023
c2943ad
Update test case for flush async
Mariatta May 9, 2023
File filter

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update google/cloud/bigtable/batcher.py 
Co-authored-by: Mattie Fu <mattiefu@google.com>
  • Loading branch information
@Mariatta @mutianf
Mariatta and mutianf committed Apr 5, 2023
commit d907180063a45fc9739771bbacc07b5d45192c9f
2 changes: 1 addition & 1 deletion google/cloud/bigtable/batcher.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@

MAX_MUTATIONS_SIZE = 100 * 1024 * 1024 # 100MB # max inflight byte size.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe rename this variable to indicate it's for flow control.

Suggested change
MAX_MUTATIONS_SIZE = 100 * 1024 * 1024 # 100MB # max inflight byte size.
MAX_OUTSTANDING_BYTES = 100 * 1024 * 1024 # 100MB # max inflight byte size.

MAX_OUTSTANDING_ELEMENTS = 100000 # max inflight requests.
MAX_OUTSTANDING_ELEMENTS = 100000 # max inflight mutations.


class MutationsBatchError(Exception):
Expand Down