Update Model

README.md

@@ -15,6 +15,11 @@ pipeline_tag: image-classification
 
 # SuSy - Synthetic Image Detector
 
+![susy-logo](susy_logo.jpeg)
+
+- **Repository:** https://github.com/HPAI-BSC/SuSy
+- **Dataset:** https://huggingface.co/datasets/HPAI-BSC/SuSy-Dataset
+
 ## Model Details
 
 <!-- Provide a longer summary of what this model is. -->
@@ -33,7 +38,7 @@ The model can be used as a detector by either taking the class with the highest
 
 ### Model Description
 
-- **Developed by:** Pablo Bernabeu, Enrique Lopez and Dario Garcia-Gasulla from [HPAI](https://hpai.bsc.es/)
+- **Developed by:** [Pablo Bernabeu Perez](https://huggingface.co/pabberpe), [Enrique Lopez Cuena](https://huggingface.co/Cuena) and [Dario Garcia Gasulla](https://huggingface.co/dariog) from [HPAI](https://hpai.bsc.es/)
 - **Model type:** Spatial-Based Synthetic Image Detection and Recognition Convolutional Neural Network
 - **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
 
@@ -58,8 +63,12 @@ Out-of-scope uses include the following:
 * Detection of manually edited images using traditional tools.
 * Detection of images automatically downscaled and/or upscaled. These are considered as non-synthetic samples in the model training phase. 
 * Detection of inpainted images.
-* Detection of synthetic vs manually crafted illustrations. The model is trained only on photorealistic samples.
-* Attribution of synthetic images to their generative model if the model was not included in the training data. Although some generalization capabilities are expected, reliability in this case cannot be estimated.
+* Detection of synthetic vs manually crafted illustrations. The model is trained mainly on photorealistic samples.
+* Attribution of synthetic images to their generative model if the model was not included in the training data. AThis model may not be used to train generative models or tools aimed at lthough some generalization capabilities are expected, reliability in this case cannot be estimated.
+
+### Forbidden Uses
+
+This model may not be used to train generative models or tools aimed at purposefully deceiving the model or creating misleading content.
 
 ## Bias, Risks, and Limitations
 
@@ -71,7 +80,7 @@ The model may be biased in the following ways:
 
 The model has the following technical limitations:
 
-* The performance of the model may be influenced by transformations and editions performed on the images. While the model was trained on some alterations (JPEG compression, downscaling, and downscaling+upscaling) there are other alterations applicable to images that could reduce the model accuracy.
+* The performance of the model may be influenced by transformations and editions performed on the images. While the model was trained on some alterations (blur, brightness, compression and gamma) there are other alterations applicable to images that could reduce the model accuracy.
 * The model will not be able to attribute synthetic images to their generative model if the model was not included in the training data.
 * The model is trained on patches with high gray-level contrast. For images composed entirely by low contrast regions, the model may not work as expected.
 
@@ -113,20 +122,22 @@ See `test_image.py` and `test_patch.py` for other examples on how to use the mod
 
 <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
 
-|      Dataset      | Year |  Test | Train |
-|:-----------------:|:----:|------:|------:|
-|        COCO       | 2017 | 1,234 | 4,201 |
-|   dalle-3-images  | 2023 |   330 | 1,317 |
-|    diffusiondb    | 2022 | 1,234 | 4,201 |
-| midjourney-images | 2023 |   246 |   980 |
-|   midjourney-tti  | 2022 |   906 | 3,624 |
-|   realisticSDXL   | 2023 | 1,234 | 4,201 |
+The dataset is available at: https://huggingface.co/datasets/HPAI-BSC/SuSy-Dataset
+
+| Dataset           | Year | Train | Validation | Test  |
+|-------------------|------|-------|------------|-------|
+| COCO              | 2017 | 2,967 | 1,234      | 1,234 |
+| dalle-3-images    | 2023 | 987   | 330        | 330   |
+| diffusiondb       | 2022 | 2,967 | 1,234      | 1,234 |
+| realisticSDXL     | 2023 | 2,967 | 1,234      | 1,234 |
+| midjourney-tti    | 2022 | 2,718 | 906        | 906   |
+| midjourney-images | 2023 | 1,845 | 617        | 617   |
 
 #### Authentic Images
 
 - [COCO](https://cocodataset.org/)
 
-We use a random subset of the COCO dataset, containing 5,435 images, for the authentic images in our training dataset. The partitions are made respecting the original COCO splits, with 4,201 images in the training partition and 1,234 in the test partition.
+We use a random subset of the COCO dataset, containing 5,435 images, for the authentic images in our training dataset. The partitions are made respecting the original COCO splits, with 2,967 images in the training partition and 1,234 in the validation and test partitions.
 
 #### Synthetic Images
 
@@ -136,12 +147,14 @@ We use a random subset of the COCO dataset, containing 5,435 images, for the aut
 - [midjourney-texttoimage](https://www.kaggle.com/datasets/succinctlyai/midjourney-texttoimage)
 - [realistic-SDXL](https://huggingface.co/datasets/DucHaiten/DucHaiten-realistic-SDXL)
 
-For the diffusiondb dataset, we use a random subset of 5,435 images, with 4,201 in the training partition and 1,234 in the test partition. We use only the realistic images from the realisticSDXL dataset, with images in the realistic-2.2 split in our training data and the realistic-1 split for our test partition. The remaining datasets are used in their entirety, with 80% of the images in the training partition and 20% in the test partition.
+For the diffusiondb dataset, we use a random subset of 5,435 images, with 2,967 in the training partition and 1,234 in the validation and test partitions. We use only the realistic images from the realisticSDXL dataset, with images in the realistic-2.2 split in our training data and the realistic-1 split for our test partition. The remaining datasets are used in their entirety, with 60% of the images in the training partition, 20% in the validation partition and 20% in the test partition.
 
 ### Training Procedure
 
 <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
 
+The training code is available at: https://github.com/HPAI-BSC/SuSy
+
 #### Preprocessing
 
 **Patch Extraction**
@@ -150,12 +163,14 @@ To prepare the training data, we extract 240x240 patches from the images, minimi
 
 **Data Augmentation**
 
-| Technique                | Probability | Other Parameters                                                                                        |
-|--------------------------|:-----------:|---------------------------------------------------------------------------------------------------------|
-| HorizontalFlip           |     0.35    | -                                                                                                      |
-| RandomBrightnessContrast |     0.50    | brightness\_limit=0.2 contrast\_limit=0.2                                                               |
-| RandomGamma              |     0.50    | gamma\_limit=(80, 120)                                                                                  |
-| CoarseDropout            |     0.50    | min\_holes=1, max\_holes=3 min\_height=64, max\_height=100, min\_width=64, max\_width=100 fill\_value=0 |
+| Technique                | Probability | Other Parameters                          |
+|--------------------------|:-----------:|-------------------------------------------|
+| HorizontalFlip           |     0.50    | -                                         |
+| RandomBrightnessContrast |     0.20    | brightness\_limit=0.2 contrast\_limit=0.2 |
+| RandomGamma              |     0.20    | gamma\_limit=(80, 120)                    |
+| AdvancedBlur             |     0.20    |                                           |
+| GaussianBlur             |     0.20    |                                           |
+| JPEGCompression          |     0.20    | quality\_lower=75 quality\_upper=100      |
 
 
 #### Training Hyperparameters
@@ -168,13 +183,15 @@ To prepare the training data, we extract 240x240 patches from the images, minimi
 - Factor: 0.1
 - Patience: 4
 - Batch Size: 128
-- Epochs: 50
-- Early Stopping: 8
+- Epochs: 10
+- Early Stopping: 2
 
 ## Evaluation
 
 <!-- This section describes the evaluation protocols and provides the results. -->
 
+The evaluation code is available at: https://github.com/HPAI-BSC/SuSy
+
 ### Testing Data, Factors & Metrics
 
 #### Testing Data
@@ -182,54 +199,64 @@ To prepare the training data, we extract 240x240 patches from the images, minimi
 <!-- This should link to a Dataset Card if possible. -->
 
 - Test Split of our Training Dataset
+- Synthetic Images generated with [Stable Diffusion 3 Medium](https://huggingface.co/stabilityai/stable-diffusion-3-medium) and [FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) using prompts from [Gustavosta/Stable-Diffusion-Prompts](https://huggingface.co/datasets/Gustavosta/Stable-Diffusion-Prompts)
 - Synthetic Images in the Wild: Dataset containing 210 Authentic and Synthetic Images obtained from Social Media Platforms
 - [Flickr 30k Dataset](https://www.kaggle.com/datasets/hsankesara/flickr-image-dataset)
+- [Google Landmarks v2](https://github.com/cvdfoundation/google-landmark)
+- [Synthbuster](https://zenodo.org/records/10066460)
 
 #### Metrics
 
 <!-- These are the evaluation metrics being used, ideally with a description of why. -->
 
-- Accuracy: The proportion of correctly classified images.
-- F1 Score: The harmonic mean of precision and recall.
+- Recall: The proportion of correctly classified positive instances out of all actual positive instances in a dataset.
 
 ### Results
 
 <!-- This section provides the results of the evaluation. -->
 
-#### Test Split
-
-Task | Detection F1 Score | Recognition F1 Score
---- | --- | ---
-Original Images | 0.9867 | 0.9041
-JPEG Compressed Images | 0.9918 | 0.9141
-Downscaled Images | 0.9761 | 0.7470
-Downscaled+Upscaled Images | 0.9868 | 0.8266
-
-#### Synthetic Images in the Wild
-
-79.55% Detection Accuracy
-
-#### Flickr30k Dataset
-
-99.19% Detection Accuracy
+#### Authentic Sources
+
+| Dataset             | Model | Year | Recall |
+|---------------------|-------|------|--------|
+| Flickr30k           | -     | 2014 | 90.53  |
+| Google Landmarks v2 | -     | 2020 | 64.54  |
+| In-the-wild         | -     | 2024 | 33.06  |
+
+#### Synthetic Sources
+
+| Dataset     | Model                     | Year | Recall |
+|-------------|---------------------------|------|--------|
+| Synthbuster | Glide                     | 2021 | 53.50  |
+| Synthbuster | Stable Diffusion 1.3      | 2022 | 87.00  |
+| Synthbuster | Stable Diffusion 1.4      | 2022 | 87.10  |
+| Synthbuster | Stable Diffusion 2        | 2022 | 68.40  |
+| Synthbuster | DALL-E 2                  | 2022 | 20.70  |
+| Synthbuster | MidJourney V5             | 2023 | 73.10  |
+| Synthbuster | Stable Diffusion XL       | 2023 | 79.50  |
+| Synthbuster | Firefly                   | 2023 | 40.90  |
+| Synthbuster | DALL-E 3                  | 2023 | 88.60  |
+| Authors     | Stable Diffusion 3 Medium | 2024 | 93.23  |
+| Authors     | Flux.1-dev                | 2024 | 96.46  |
+| In-the-wild | Mixed/Unknown             | 2024 | 89.90  |
 
 ### Summary
 
-The model obtains performs well in the test split, with high detection and recognition F1 scores. The model shows robustness to the JPEG compressed images for both tasks while the performance in the downscaled and rescaled images suffers in the recognition task, but the detection task remains stable.
+The results for authentic image datasets reveal varying detection performance across different sources. Recall rates range from 33.06% for the In-the-wild dataset to 90.53% for the Flickr30k dataset. The Google Landmarks v2 dataset shows an intermediate recall rate of 64.54%. These results indicate a significant disparity in the detectability of authentic images across different datasets, with the In-the-wild dataset presenting the most challenging case for SuSy.
 
-The model is also evaluated in our Synthetic Images in the Wild dataset, which contains 220 images obtained from social media platforms, with 121 real images and 99 AI-generated images. The difficuly of this dataset lies in the fact that the images are uploaded to social media by a wide range of users, so the images may have different resolutions, lighting conditions and quality, additionally they may have been edited or compressed. Regarding the synthetic images, the generation process is unknown, so the model has to generalize to unseen generative models. The dataset was tested by 10 human evaluators, which achieved an average detection accuracy of 72.22% and a best detection accuracy of 78.73%. The model achieves a detection accuracy of 79.55% in this dataset at its best threshold.
+The results for synthetic image datasets show varying detection performance across different image generation models. Recall rates range from 20.70% for DALL-E 2 (2022) to 96.46% for Flux.1-dev (2024). Stable Diffusion models generally exhibited high detectability, with versions 1.3 and 1.4 (2022) showing recall rates above 87%. More recent models tested by the authors, such as Stable Diffusion 3 Medium (2024) and Flux.1-dev (2024), demonstrate even higher detectability with recall rates above 93%. The in-the-wild mixed/unknown synthetic dataset from 2024 showed a high recall of 89.90%, indicating effective detection across various unknown generation methods. These results suggest an overall trend of improving detection capabilities for synthetic images, with newer generation models generally being more easily detectable.
 
-Finally, the model shows excellent performance in the Flickr30k dataset. This dataset contains authentic images, so it serves the purpose of testing the number of false positives generated by the model. The model achieves a detection accuracy of 99.19% in this dataset at its best threshold.
+It must be noted that these metrics were computed using the center-patch of images, instead of using the patch voting mechanisms described previously. This strategy allows a more fair comparison with other state-of-the-art methods although it hinders the performance of SuSy.
 
 ## Environmental Impact
 
 <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
 
-- **Hardware Type:** 2xH100
-- **Hours used:** 15
+- **Hardware Type:** H100
+- **Hours used:** 16
 - **Hardware Provider:** Barcelona Supercomputing Center (BSC)
 - **Compute Region:** Spain
-- **Carbon Emitted:** 2.11kg
+- **Carbon Emitted:** 0.63kg
 
 ## Citation
 
@@ -238,9 +265,18 @@ Finally, the model shows excellent performance in the Flickr30k dataset. This da
 **BibTeX:**
 
 ```bibtex
-@thesis{bernabeu2024stair,
+@misc{bernabeu2024susy,
+    title={Present and Future Generalization of Synthetic Image Detectors},
+    author={Bernabeu Perez, Pablo and Lopez Cuena, Enrique and Garcia Gasulla, Dario},
+    year={2024},
+    month={09}
+}
+```
+
+```bibtex
+@thesis{bernabeu2024aidetection,
     title={Detecting and Attributing AI-Generated Images with Machine Learning},
-    author={Bernabeu Pérez, Pablo},
+    author={Bernabeu Perez, Pablo},
     school={UPC, Facultat d'Informàtica de Barcelona, Departament de Ciències de la Computació},
     year={2024},
     month={06}
@@ -249,7 +285,7 @@ Finally, the model shows excellent performance in the Flickr30k dataset. This da
 
 ## Model Card Authors
 
-[Pablo Bernabeu](https://huggingface.co/pabberpe) and [Dario Garcia-Gasulla](https://huggingface.co/dariog)
+[Pablo Bernabeu Perez](https://huggingface.co/pabberpe) and [Dario Garcia Gasulla](https://huggingface.co/dariog)
 
 ## Model Card Contact
 

SuSy.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0f2916d1fe0967380340860a0d046f625fd4bb34359157bc318707770a002ce9
-size 50810328
+oid sha256:fa10fae300ee2742c7a373b6c3332c2595b461954b8f5616d2d382ef2751020e
+size 50810392

config.json

@@ -0,0 +1,3 @@
+{
+    "description": "This JSON file does not contain any functional data. Its presence allows Hugging Face to monitor downloads for this repository."
+}