Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

perf: improve row merging #619

Merged
merged 4 commits into from
Aug 4, 2022
Merged

perf: improve row merging #619

merged 4 commits into from
Aug 4, 2022

Commits on Aug 4, 2022

  1. fix: improve row merging perf by 10x 

    The underlying GAPIC client uses protoplus for all requests and responses. However the underlying protos for ReadRowsResponse are never exposed to end users directly: the underlying chunks get merged into logic rows. The readability benefits provided by protoplus for ReadRows do not justify the costs. This change unwraps the protoplus messages and uses the raw protobuff message as input for row merging. This improves row merging performance by 10x. For 10k rows, each with 100 cells where each cell is 100 bytes and in groups of 100 rows per ReadRowsResponse, cProfile showed a 10x improvement:

old:          124266037 function calls in 68.208 seconds
new:          13042837 function calls in 7.787 seconds
    @igorbernstein2
    igorbernstein2 committed Aug 4, 2022
    Configuration menu
    Copy the full SHA
    d5969da View commit details
    Browse the repository at this point in the history

  2. fix tests 

    The previous approach of duck typing the protobuf messages to plain python objects no longer works as we need to shuck protoplus now
    @igorbernstein2
    igorbernstein2 committed Aug 4, 2022
    Configuration menu
    Copy the full SHA
    609cddd View commit details
    Browse the repository at this point in the history

  3. lint

    @igorbernstein2
    igorbernstein2 committed Aug 4, 2022
    Configuration menu
    Copy the full SHA
    f310b6b View commit details
    Browse the repository at this point in the history

  4. lint again

    @igorbernstein2
    igorbernstein2 committed Aug 4, 2022
    Configuration menu
    Copy the full SHA
    a4debf4 View commit details
    Browse the repository at this point in the history