umarbutler
commited on
Commit
•
fa3ebea
1
Parent(s):
33e3c53
Update README.md
Browse files
README.md
CHANGED
@@ -56,12 +56,12 @@ The training dataset was subsequently fed to [GPT2](https://huggingface.co/gpt2)
|
|
56 |
| Weight decay | 0.01 |
|
57 |
| Warmup ratio | 0.06 |
|
58 |
|
59 |
-
After training for 3 epochs, or 465,441 steps, over a period of ~25 hours on two GeForce RTX 4090s, the model achieved a loss of 0.61.
|
60 |
|
61 |
## Limitations 🚧
|
62 |
Although the model has not been tested for bias, one would expect it to exhibit much of the same, if not all, the biases of [GPT2](https://huggingface.co/gpt2).
|
63 |
|
64 |
-
One might also expect the model to exhibit a bias towards the type of language employed in legislation and regulations (its source materials) as well as towards Commonwealth law (the largest source of legislation in [Open Australian Legal Corpus](https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus) at the time of the model's creation).
|
65 |
|
66 |
Finally, it is worth noting that the model may lack knowledge of Victorian, Northern Territory and Australian Capital Territory law as licensing restrictions had prevented their inclusion in the training data.
|
67 |
|
|
|
56 |
| Weight decay | 0.01 |
|
57 |
| Warmup ratio | 0.06 |
|
58 |
|
59 |
+
After training for 3 epochs, or 465,441 steps, over a period of ~25 hours on two GeForce RTX 4090s, the model achieved a training loss of 0.61.
|
60 |
|
61 |
## Limitations 🚧
|
62 |
Although the model has not been tested for bias, one would expect it to exhibit much of the same, if not all, the biases of [GPT2](https://huggingface.co/gpt2).
|
63 |
|
64 |
+
One might also expect the model to exhibit a bias towards the type of language employed in legislation and regulations (its source materials) as well as towards Commonwealth law (the largest source of legislation in the [Open Australian Legal Corpus](https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus) at the time of the model's creation).
|
65 |
|
66 |
Finally, it is worth noting that the model may lack knowledge of Victorian, Northern Territory and Australian Capital Territory law as licensing restrictions had prevented their inclusion in the training data.
|
67 |
|