README.md

metadata

language:
  - en
license: apache-2.0
datasets:
  - SenseLLM/ReflectionSeq-GPT
  - SenseLLM/ReflectionSeq-DS
model-index:
  - name: ReflectionCoder-CL-34B
    results:
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: IFEval (0-Shot)
          type: HuggingFaceH4/ifeval
          args:
            num_few_shot: 0
        metrics:
          - type: inst_level_strict_acc and prompt_level_strict_acc
            value: 40.08
            name: strict accuracy
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=SenseLLM/ReflectionCoder-CL-34B
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: BBH (3-Shot)
          type: BBH
          args:
            num_few_shot: 3
        metrics:
          - type: acc_norm
            value: 14.26
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=SenseLLM/ReflectionCoder-CL-34B
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MATH Lvl 5 (4-Shot)
          type: hendrycks/competition_math
          args:
            num_few_shot: 4
        metrics:
          - type: exact_match
            value: 1.96
            name: exact match
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=SenseLLM/ReflectionCoder-CL-34B
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: GPQA (0-shot)
          type: Idavidrein/gpqa
          args:
            num_few_shot: 0
        metrics:
          - type: acc_norm
            value: 0.11
            name: acc_norm
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=SenseLLM/ReflectionCoder-CL-34B
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MuSR (0-shot)
          type: TAUR-Lab/MuSR
          args:
            num_few_shot: 0
        metrics:
          - type: acc_norm
            value: 10.4
            name: acc_norm
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=SenseLLM/ReflectionCoder-CL-34B
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MMLU-PRO (5-shot)
          type: TIGER-Lab/MMLU-Pro
          config: main
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 4.71
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=SenseLLM/ReflectionCoder-CL-34B
          name: Open LLM Leaderboard

ReflectionCoder: Learning from Reflection Sequence for Enhanced One-off Code Generation

📄 Paper • 🏠 Repo • 🤖 Models • 📚 Datasets

Introduction

ReflectionCoder is a novel approach that effectively leverages reflection sequences constructed by integrating compiler feedback to improve one-off code generation performance. Please refer to our paper and repo for more details!

---

Models

Model	Checkpoint	Size	HumanEval (+)	MBPP (+)	License
ReflectionCoder-CL-7B	🤗 HF Link	7B	75.0 (68.9)	72.2 (61.4)	Llama2
ReflectionCoder-CL-34B	🤗 HF Link	34B	70.7 (66.5)	68.4 (56.6)	Llama2
ReflectionCoder-DS-6.7B	🤗 HF Link	6.7B	80.5 (74.4)	81.5 (69.6)	DeepSeek
ReflectionCoder-DS-33B	🤗 HF Link	33B	82.9 (76.8)	84.1 (72.0)	DeepSeek

Datasets

Dataset	Link	License
ReflectionSeq-GPT	🤗 HF Link	License
ReflectionSeq-DS	🤗 HF Link	License

How to Use

Chat Format

Following chat templates of most models, we use two special tokens to wrap the message of user and assistant, i.e., <|user|>, <|assistant|>, and <|endofmessage|>. Furthermore, we use two special tokens to wrap the content of different blocks, i.e., <|text|> and <|endofblock|>. You can use the following template to prompt our ReflectionCoder.

import torch
from transformers import pipeline

chat = [
    {"role": "user", "content": "<Your code instruction here>"}
]

generator = pipeline(
    model="SenseLLM/ReflectionCoder-CL-34B",
    task="text-generation",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

result = generator(chat, max_length=128, num_return_sequences=1)

print(result)

Please refer to our GitHub Repo for more technical details.

Citation

If you find this repo useful for your research, please kindly cite our paper:

@misc{ren2024reflectioncoder,
    title={ReflectionCoder: Learning from Reflection Sequence for Enhanced One-off Code Generation}, 
    author={Houxing Ren and Mingjie Zhan and Zhongyuan Wu and Aojun Zhou and Junting Pan and Hongsheng Li},
    year={2024},
    eprint={2405.17057},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

Acknowledgments

We thank the following amazing projects that truly inspired us:

• CodeLlama
• DeepSeek-Coder
• WizardCoder
• Evol-CodeAlpaca-v1
• MagiCoder
• EvalPlus
• OpenCoderInterpreter

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	11.92
IFEval (0-Shot)	40.08
BBH (3-Shot)	14.26
MATH Lvl 5 (4-Shot)	1.96
GPQA (0-shot)	0.11
MuSR (0-shot)	10.40
MMLU-PRO (5-shot)	4.71