optimize pipeline between device fwd and host bwd

README.md

@@ -13,9 +13,13 @@ This model only contains the `GaudiConfig` file for running the [GPT2](https://h
 **This model contains no model weights, only a GaudiConfig.**
 
 This enables to specify:
+- `use_habana_mixed_precision`: whether to use Habana Mixed Precision (HMP)
+    - `hmp_opt_level`: optimization level for HMP, see [here](https://docs.habana.ai/en/latest/PyTorch/PyTorch_Mixed_Precision/PT_Mixed_Precision.html#configuration-options) for a detailed explanation
+    - `hmp_bf16_ops`: list of operators that should run in bf16
+    - `hmp_fp32_ops`: list of operators that should run in fp32
+    - `hmp_is_verbose`: verbosity
 - `use_fused_adam`: whether to use Habana's custom AdamW implementation
 - `use_fused_clip_norm`: whether to use Habana's fused gradient norm clipping operator
-- `use_torch_autocast`: whether to use PyTorch's autocast mixed precision
 
 ## Usage
 

gaudi_config.json

@@ -1,5 +1,5 @@
 {
+  "use_habana_mixed_precision": false,
   "use_fused_adam": true,
-  "use_fused_clip_norm": true,
-  "use_torch_autocast": true
+  "use_fused_clip_norm": true
 }