I tested berkley leaderboard for qwen2-72b-Instruct and codestral

[bks5881]

I get the following results. if anyone wants to use it.

Rank	Overall Acc	Model	Model Link	Organization	License	AST Summary	Exec Summary	Simple Function AST	Python Simple Function AST	Java Simple Function AST	JavaScript Simple Function AST	Multiple Functions AST	Parallel Functions AST	Parallel Multiple AST	Simple Function Exec	Python Simple Function Exec	REST Simple Function Exec	Multiple Functions Exec	Parallel Functions Exec	Parallel Multiple Exec	Relevance Detection	Cost ($ Per 1k Function Calls)	Latency Mean (s)	Latency Standard Deviation (s)	Latency 95th Percentile (s)
1	84.68%	Qwen2-72B-Instruct	MY_URL	Qwen	lala	45.07%	65.43%	84.80%	91.25%	59.00%	0.00%	95.50%	0.00%	0.00%	85.71%	0.00%	85.71%	90.00%	86.00%	0.00%	73.75%	N/A	N/A	N/A	N/A
2	39.06%	Codestral-22B-v0.1	MY_URL	codestral	lala	33.24%	28.59%	27.45%	34.25%	7.00%	14.00%	39.50%	31.50%	34.50%	32.35%	47.00%	11.43%	28.00%	24.00%	30.00%	87.08%	N/A	N/A	N/A	N/A

[CharlieJCJ]

Hi @bks5881 , thanks for the efforts in applying BFCL on new open source models.

One thing to point out is that through eyeballing, it seems that the results for Qwen2-72B-Instruct in your table don't result to the number in Overall Acc column (84.68%).

[bks5881]

I am not sure what could have caused the difference, I basically built my own handler that calls openai compatible endpoint and just ran the tests.