Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Save pass@k result & use custom tokenizer #20

Merged
merged 5 commits into from
Jul 9, 2024
Merged

Save pass@k result & use custom tokenizer #20

merged 5 commits into from
Jul 9, 2024

Conversation

marianna13
Copy link
Contributor

Hey there!

Two changes that would be nice to have:

  1. Save pass@k result after evaluation (in a JSON file).
  2. The ability to load custom tokenizer with HF AutoTokenizer (i.e. if tokenizer name is different from the model name).
@terryyz terryyz self-requested a review July 8, 2024 14:38
terryyz
terryyz reviewed Jul 9, 2024
View reviewed changes
bigcodebench/evaluate.py
@@ -277,6 +277,12 @@ def stucking_checker():
if not os.path.isfile(result_path):
with open(result_path, "w") as f:
json.dump(results, f, indent=2)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe adding if not os.path.isfile(pass_at_k_path):?
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I think a better way is to check if at least the Pass@1 scores are the same and decide whether we need to rewrite the result_path and pass_at_k_path. Wdyt?
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, sounds good
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I'll merge the PR after your update.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done
@terryyz
Copy link
Collaborator

terryyz commented Jul 9, 2024

@marianna13 LGTM, only minor updates are needed. Did you test w/ these changes with some models?
@marianna13
Copy link
Contributor Author

I tried with ibm-granite/granite-3b-code-base and openai-community/gpt2 tokenizer. Also I tried apple/OpenELM-1_1B and meta-llama/Llama-2-7b-hf tokenizer ( it didn't work but it's not bc of the tokenizer, there's a problem with OpenELM config)
@terryyz terryyz merged commit bbe93d6 into bigcode-project:main Jul 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
2 participants
@marianna13 @terryyz