codify-llama-2-7b

This model is a fine-tuned version of Llama-2-7b-hf on the ALPACA_20k dataset.

Intended uses & limitations

1. Load the model as a Hugging Face Pipeline:

from transformers import pipeline

pipe = pipeline('text-generation', model='mohammedaly22/Codify-LLama-2-7b')

2. Prepare the instruction template

from string import Template

prompt_template_inference = Template("""You are a world class software engineer answering coding questions. Below is an
instruction that describes a coding task, paired with an  optional input that
provides further context. Write a response that accurately completes the task if
the instruction is code-related, else, you should reponse that you don't know the answer
as it is outside the context of coding. Note, you should stop generation after reaching the <EOG> token.

### Instruction:
$instruction

### Input:
$input

### Response:
""")

3. Create an instruction prompt using the above template

instruction = "Write a Python function that creates a simple 2-layer neural network using Keras for performing binary classification"
input = "input shape of the neural network will be a vector of 200 elements"
prompt = prompt_template_inference.substitute({"instruction": instruction, "input": input})

This is the final instruction prompt that will be passed to the pipeline

You are a world class software engineer answering coding questions. Below is an
instruction that describes a coding task, paired with an  optional input that
provides further context. Write a response that accurately completes the task if
the instruction is code-related, else, you should reponse that you don't know the answer
as it is outside the context of coding. Note, you should stop generation after reaching the <EOG> token.

### Instruction:
Write a Python function that creates a simple 2-layer neural network using Keras for performing binary classification

### Input:
input shape of the neural network will be a vector of 200 elements

### Response:

4. Passing the instruction prompt to the pipeline

output = pipe(
    prompt,
    do_sample=True,
    return_full_text=False,
    max_new_tokens=200,
    clean_up_tokenization_spaces=True
)

Here is the generated code of the model:

def build_simple_neural_network(): 
    return Model(
      inputs=Input(shape=(200,)),
      outputs=Dense(2, activation="softmax"),
      name="simple_neural_network"
    )

<EOG>

Training procedure

BitsAndBytes hyperparameters

• use_4bit: True
• bnb_4bit_compute_dtype: "float16"
• bnb_4bit_quant_type: "nf4"
• use_double_nested_quant: False

LoRA configurations

• lora_r: 64
• lora_alpha: 16
• lora_dropout: 0.1

Training hyperparameters

The following hyperparameters were used during training:

• num_train_epochs: 1
• fp16: False
• bf16: False
• per_device_train_batch_size: 4
• per_device_eval_batch_size: 4
• gradient_accumulation_steps: 1
• gradient_checkpointing: True
• max_grad_norm: 0.3
• learning_rate: 2e-4
• weight_decay: 0.001
• optim: "paged_adamw_32bit"
• lr_scheduler_type: "cosine"
• max_steps: -1
• warmup_ratio: 0.03
• group_by_length: True
• save_steps: 0
• logging_steps: 50

Training results

Step	Training Loss
50	1.377900
100	0.368700
150	0.336600
200	0.334800
250	0.332300
300	0.333700
350	0.322100
400	0.317000
450	0.320800
500	0.308400
550	0.321900
600	0.310700
650	0.322100
700	0.327700
750	0.322000
800	0.311300
850	0.321800
900	0.318700
950	0.321600
1000	0.314900
1050	0.321700
1100	0.307600
1150	0.315800
1200	0.316800
1250	0.314200
1300	0.310400
1350	0.308000
1400	0.318600
1450	0.309700
1500	0.307600
1550	0.296800
1600	0.305800
1650	0.307400
1700	0.327400
1750	0.306100
1800	0.309900
1850	0.316300
1900	0.299500
1950	0.315700
2000	0.307600