Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: GoogleCloudPlatform/public-datasets-pipelines Loading
base: v4.0.0
Choose a base ref
...
head repository: GoogleCloudPlatform/public-datasets-pipelines Loading
compare: v4.1.0
Choose a head ref
  • 15 commits
  • 127 files changed
  • 5 contributors

Commits on May 26, 2022

  1. chore: Update README.md (#365)

    * Update README.md
    
    * feat: Update README.md
    adlersantos committed May 26, 2022
    Configuration menu
    Copy the full SHA
    673104c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    bfb9f23 View commit details
    Browse the repository at this point in the history

Commits on May 27, 2022

  1. Configuration menu
    Copy the full SHA
    e7a726a View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    4c4bbc8 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    e4f5274 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    ddd3dac View commit details
    Browse the repository at this point in the history

Commits on May 31, 2022

  1. Configuration menu
    Copy the full SHA
    dac5fb8 View commit details
    Browse the repository at this point in the history

Commits on Jun 1, 2022

  1. Feat: Onboard EPA Historical Air Quality (#370)

    * fix: Resolving issues as yet unresolved.
    
    * fix: black issue
    
    * fix: resolved datatype date not known
    nlarge-google committed Jun 1, 2022
    Configuration menu
    Copy the full SHA
    ba473c6 View commit details
    Browse the repository at this point in the history

Commits on Jun 2, 2022

  1. Feat: Onboard Cloud Storage Geo Index (#367)

    * fix: Added partitioning and clustering. Not production ready.
    
    * fix: Works for landsat_index
    
    * fix: Added sentinel-2 pipeline as well as clustering.
    
    * fix: Clean up.
    
    * fix: attempt to resolve pod not started.
    
    * fix: various fixes to pipeline.yaml which caused 'pod not started' issues.  Added bigquery to requirements.txt
    
    * fix: Resolve resource issues in pipeline.yaml.
    
    * fix: Moved schema JSON files as per code review request.
    
    * fix: Exposed variable values into pipeline.yaml as per code review.
    
    * Update pipeline.yaml
    
    Co-authored-by: Adler Santos <adlersantos@google.com>
    nlarge-google and adlersantos committed Jun 2, 2022
    Configuration menu
    Copy the full SHA
    63cdb2a View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2022

  1. Feat: Onboard EPA Historical Air Quality (#373)

    * fix: Resolving issues as yet unresolved.
    
    * fix: black issue
    
    * fix: resolved datatype date not known
    
    * fix: submit new changes following issues in production after deployment.
    
    * fix: miscellaneous fixes.
    
    * fix: Annual summaries now works.
    
    * fix: co_hourly_summary now successful.
    
    * fix: black hook issues
    
    * fix: data type corrections for some fields.
    
    * fix: Resolved file naming breaking issue in ozone_hourly pipeline. Not production ready.
    
    * fix: Now fixed annual summaries.  Ready for AF test in dev.
    
    * fix: Tested full load all pipelines.
    
    * fix: Replicate full load fixes into incremental load dag.
    
    * fix: Resolved black hooks
    nlarge-google committed Jun 6, 2022
    Configuration menu
    Copy the full SHA
    4f4c87e View commit details
    Browse the repository at this point in the history

Commits on Jun 8, 2022

  1. Feat: Onboard The General Index Dataset (#342)

    * feat: onboarding early un sdg version
    
    * feat: onboarding early un sdg version
    
    * feat: Initial checkin.  Not production ready.
    
    * fix: loads test data into dataframe.  Not production ready.
    
    * fix: issues with datatypes when loading data.
    
    * fix: Tested locally. runs as intended.
    
    * fix: Changes to variables in pipeline.yaml preparing for multiple pipelines.
    
    * fix: misc changes.  not ready for production.
    
    * fix: completed loading one file from end-to-end.
    
    * fix: expanded pipeline.yaml to include all data files for download and processing
    
    * fix: Added deletion of source data if it is being reloaded.
    
    * fix: next update
    
    * fix: Testing in AF.
    
    * fix: Increased number of nodes in cluster in AF to ensure resources are available for running the pipelines.
    
    * fix: Resolved download file reference in code.
    
    * fix: Fixed delete source data function.
    
    * fix: resolved yaml lint issues
    
    * fix: black errors
    
    * fix: black errors
    
    * fix: code fix, production ready
    
    * fix: Fixed the bad charecters from data giving loading issues. Production ready
    gkodukula committed Jun 8, 2022
    Configuration menu
    Copy the full SHA
    67d7216 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d3dba58 View commit details
    Browse the repository at this point in the history

Commits on Jun 10, 2022

  1. Configuration menu
    Copy the full SHA
    c7cd9dd View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    0f1c696 View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2022

  1. chore(main): release 4.1.0 (#366)

    * chore(main): release 4.1.0
    
    * Update CHANGELOG.md
    
    Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
    Co-authored-by: Adler Santos <adlersantos@google.com>
    release-please[bot] and adlersantos committed Jun 14, 2022
    Configuration menu
    Copy the full SHA
    cdbca70 View commit details
    Browse the repository at this point in the history
Loading