Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: improve row merging #619

Merged
merged 4 commits into from
Aug 4, 2022
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
fix: improve row merging perf by 10x
The underlying GAPIC client uses protoplus for all requests and responses. However the underlying protos for ReadRowsResponse are never exposed to end users directly: the underlying chunks get merged into logic rows. The readability benefits provided by protoplus for ReadRows do not justify the costs. This change unwraps the protoplus messages and uses the raw protobuff message as input for row merging. This improves row merging performance by 10x. For 10k rows, each with 100 cells where each cell is 100 bytes and in groups of 100 rows per ReadRowsResponse, cProfile showed a 10x improvement:

old:          124266037 function calls in 68.208 seconds
new:          13042837 function calls in 7.787 seconds
  • Loading branch information
igorbernstein2 committed Aug 4, 2022
commit d5969da8032682f0c571247c5ea0229b6cfdd420
19 changes: 12 additions & 7 deletions google/cloud/bigtable/row_data.py
Original file line number Diff line number Diff line change
Expand Up @@ -474,7 +474,11 @@ def _read_next(self):

def _read_next_response(self):
"""Helper for :meth:`__iter__`."""
return self.retry(self._read_next, on_error=self._on_error)()
resp_protoplus = self.retry(self._read_next, on_error=self._on_error)()
# unwrap the underlying protobuf, there is a significant amount of
# overhead that protoplus imposes for very little gain. The protos
# are not user visible, so we just use the raw protos for merging.
return data_messages_v2_pb2.ReadRowsResponse.pb(resp_protoplus)

def __iter__(self):
"""Consume the ``ReadRowsResponse`` s from the stream.
Expand Down Expand Up @@ -543,11 +547,12 @@ def _process_chunk(self, chunk):
def _update_cell(self, chunk):
if self._cell is None:
qualifier = None
if "qualifier" in chunk:
qualifier = chunk.qualifier
if chunk.HasField("qualifier"):
qualifier = chunk.qualifier.value

family = None
if "family_name" in chunk:
family = chunk.family_name
if chunk.HasField("family_name"):
family = chunk.family_name.value

self._cell = PartialCellData(
chunk.row_key,
Expand Down Expand Up @@ -577,8 +582,8 @@ def _validate_chunk_reset_row(self, chunk):

# No reset with other keys
_raise_if(chunk.row_key)
_raise_if("family_name" in chunk)
_raise_if("qualifier" in chunk)
_raise_if(chunk.HasField("family_name"))
_raise_if(chunk.HasField("qualifier"))
_raise_if(chunk.timestamp_micros)
_raise_if(chunk.labels)
_raise_if(chunk.value_size)
Expand Down