File size: 4,319 Bytes
483de47 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 |
## LaMa: Resolution-robust Large Mask Inpainting with Fourier Convolutions
:grapes: [[Official Project Page](https://advimman.github.io/lama-project/)] :apple:[[LaMa Cleaner](https://github.com/Sanster/lama-cleaner)]
We use the highly organized code [lama-cleaner](https://github.com/Sanster/lama-cleaner) to simplify the demo code for users.
<div align="center">
</div>
## Abstract
> Modern image inpainting systems, despite the significant progress, often struggle with large missing areas, complex geometric structures, and high-resolution images. We find that one of the main reasons for that is the lack of an ef-fective receptive field in both the inpainting network andthe loss function. To alleviate this issue, we propose anew method called large mask inpainting (LaMa). LaM ais based on: a new inpainting network architecture that uses fast Fourier convolutions, which have the image-widereceptive field
a high receptive field perceptual loss; large training masks, which unlocks the potential ofthe first two components. Our inpainting network improves the state-of-the-art across a range of datasets and achieves excellent performance even in challenging scenarios, e.g.completion of periodic structures. Our model generalizes surprisingly well to resolutions that are higher than thoseseen at train time, and achieves this at lower parameter & compute costs than the competitive baselines.
## Table of Contents
- [Installation](#installation)
- [LaMa Demos](#paint-by-example-demos)
- [Diffuser Demo](#paintbyexample-diffuser-demos)
- [PaintByExample with SAM](#paintbyexample-with-sam)
## TODO
- [x] LaMa Demo with lama-cleaner
- [x] LaMa with SAM
- [ ] LaMa with GroundingDINO
- [ ] LaMa with Grounded-SAM
## Installation
We're using lama-cleaner for this demo, install it as follows:
```bash
pip install lama-cleaner
```
Please refer to [lama-cleaner](https://github.com/Sanster/lama-cleaner) for more details.
Then install Grounded-SAM follows [Grounded-SAM Installation](https://github.com/IDEA-Research/Grounded-Segment-Anything#installation) for some extension demos.
## LaMa Demos
Here we provide the demos for `LaMa`
### LaMa Demo with lama-cleaner
```bash
cd playground/LaMa
python lama_inpaint_demo.py
```
with the highly organized code lama-cleaner, this demo can be done in about 20 lines of code. The result will be saved as `lama_inpaint_demo.jpg`:
<div align="center">
| Input Image | Mask | Inpaint Output |
|:----:|:----:|:----:|
|  |  |  |
</div>
### LaMa with SAM
```bash
cd playground/LaMa
python sam_lama.py
```
**Tips**
To make it better for inpaint, we should **dilate the mask first** to make it a bit larger to cover the whole region (Thanks a lot for [Inpaint-Anything](https://github.com/geekyutao/Inpaint-Anything) and [Tao Yu](https://github.com/geekyutao) for this)
The `original mask` and `dilated mask` are shown as follows:
<div align="center">
| Mask | Dilated Mask |
|:---:|:---:|
|  |  |
</div>
And the inpaint result will be saved as `sam_lama_demo.jpg`:
| Input Image | SAM Output | Dilated Mask | LaMa Inpaint |
|:---:|:---:|:---:|:---:|
|  |  |  |  |
|