Release Model

README.md

@@ -1,3 +1,256 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+datasets:
+- HuggingFaceM4/COCO
+- ehristoforu/dalle-3-images
+- poloclub/diffusiondb
+- ehristoforu/midjourney-images
+- nateraw/midjourney-texttoimage
+- duchaiten/duchaiten-realistic-sdxl
+tags:
+- vision
+- image-classification
+pipeline_tag: multi-class-image-classification
+---
+
+# SuSy - Synthetic Image Detector
+
+## Model Details
+
+<!-- Provide a longer summary of what this model is. -->
+
+SuSy is a Spatial-Based Synthetic Image Detection and Recognition Model, designed and trained to detect synthetic images and attribute them to a generative model (i.e., two StableDiffusion models, two Midjourney versions and DALL·E 3). The model takes image patches of size 224x224 as input, and outputs the probability of the image being authentic or having been created by each of the aforementioned generative models.
+
+![model-architecture](model_architecture.png)
+
+The model is based on a CNN architecture and is trained using a supervised learning approach. It's design is based on [previous work](https://upcommons.upc.edu/handle/2117/395959), originally intended for video superresolution detection, adapted here for the tasks of synthetic image detection and recognition. The architecture consists of two modules: a feature extractor and a multi-layer perceptron (MLP), as it's quite light weight. SuSy has a total of 12.7M parameters, with the feature extractor accounting for 12.5M parameters and the MLP accounting for the remaining 197K.
+
+The CNN feature extractor consists of five stages following a [ResNet-18](https://arxiv.org/abs/1512.03385) scheme. The output of each of the blocks is used as input for various bottleneck modules that are arranged in a staircase pattern. The bottleneck modules consist of three 2D convolutional layers. Each level of bottlenecks takes input at a later stage than the previous level, and each bottleneck module takes input from the current stage and, except the first bottleneck of each level, from the previous bottleneck module.
+
+The outputs of each level of bottlenecks and stage 4 are passed to a 2D adaptative average pooling layer and then concatenated to form the feature map feeding the MLP. The MLP consists of three fully connected layers with 512, 256 and 256 units, respectively. Between each layer, a dropout layer (rate of 0.5) prevents overfitting. The output of the MLP has 6 units, corresponding to the number of classes in the dataset (5 synthetic models and 1 real image class).
+
+The model can be used as a detector by either taking the class with the highest probability as the output or summing the probabilities of the synthetic classes and comparing them to the real class. The model can also be used as an recognition model by taking the class with the highest probability as the output.
+
+### Model Description
+
+- **Developed by:** Pablo Bernabeu, Enrique Lopez and Dario Garcia-Gasulla from [HPAI](https://hpai.bsc.es/)
+- **Model type:** Spatial-Based Synthetic Image Detection and Recognition Convolutional Neural Network
+- **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
+
+## Uses
+
+This model can be used to detect synthetic images in a scalable manner, thanks to its small size. Since it operates on patches of 224x224, a moving window should be implemented in inference when applied on larger inputs (the most likely scenario, and the one it was trained under). This also enables the capacity for synthetic content localization within a high resolution input.
+
+Any individual or organization seeking for support on the identification of synthetic content can use this model. However, it should not be used as the only source of evidence, particularly when applied to inputs produced by generative models not included in its training (see details in Training Data below).
+
+### Intended Uses
+
+Intended uses include the following:
+
+* Detection of authentic and synthetic images
+* Attribution of synthetic images to their generative model (if included in the training data)
+* Localization of image patches likely to be synthetic or tampered.
+
+### Out-of-Scope Uses
+
+Out-of-scope uses include the following:
+
+* Detection of manually edited images using traditional tools.
+* Detection of images automatically downscaled and/or upscaled. These are considered as non-synthetic samples in the model training phase. 
+* Detection of inpainted images.
+* Detection of synthetic vs manually crafted illustrations. The model is trained only on photorealistic samples.
+* Attribution of synthetic images to their generative model if the model was not included in the training data. Although some generalization capabilities are expected, reliability in this case cannot be estimated.
+
+## Bias, Risks, and Limitations
+
+The model may be biased in the following ways:
+
+* The model may be biased towards the training data, which may not be representative of all authentic and synthetic images. Particularly for the class of real world images, which were obtained from a single source.
+* The model may be biased towards the generative models included in the training data, which may not be representative of all possible generative models. Particularly new ones, since all models included were released between 2022 and 2023.
+* The model may be biased towards certain type of images or contents. While it is trained using roughly 18K synthetic images, no assessment was made on which domains and profiles are included in those.
+
+The model has the following technical limitations:
+
+* The performance of the model may be influenced by transformations and editions performed on the images. While the model was trained on some alterations (JPEG compression, downscaling, and downscaling+upscaling) there are other alterations applicable to images that could reduce the model accuracy.
+* The model will not be able to attribute synthetic images to their generative model if the model was not included in the training data.
+* The model is trained on patches with high gray-level contrast. For images composed entirely by low contrast regions, the model may not work as expected.
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+```python
+import torch
+from PIL import Image
+from torchvision import transforms
+
+# Load the model
+model = torch.jit.load("SuSy.pt")
+
+# Load patch
+patch = Image.open("midjourney-images-example-patch0.png")
+
+# Transform patch to tensor
+patch = transforms.PILToTensor()(patch).unsqueeze(0) / 255.
+
+# Predict patch
+model.eval()
+with torch.no_grad():
+    preds = model(patch)
+
+print(preds)
+```
+
+See `test_image.py` and `test_patch.py` for other examples on how to use the model.
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+|      Dataset      | Year |  Test | Train |
+|:-----------------:|:----:|------:|------:|
+|        COCO       | 2017 | 1,234 | 4,201 |
+|   dalle-3-images  | 2023 |   330 | 1,317 |
+|    diffusiondb    | 2022 | 1,234 | 4,201 |
+| midjourney-images | 2023 |   246 |   980 |
+|   midjourney-tti  | 2022 |   906 | 3,624 |
+|   realisticSDXL   | 2023 | 1,234 | 4,201 |
+
+#### Authentic Images
+
+- [COCO](https://cocodataset.org/)
+
+We use a random subset of the COCO dataset, containing 5,435 images, for the authentic images in our training dataset. The partitions are made respecting the original COCO splits, with 4,201 images in the training partition and 1,234 in the test partition.
+
+#### Synthetic Images
+
+- [dalle-3-images](https://huggingface.co/datasets/ehristoforu/dalle-3-images)
+- [diffusiondb](https://poloclub.github.io/diffusiondb/)
+- [midjourney-images](https://huggingface.co/datasets/ehristoforu/midjourney-images)
+- [midjourney-texttoimage](https://www.kaggle.com/datasets/succinctlyai/midjourney-texttoimage)
+- [realistic-SDXL](https://huggingface.co/datasets/DucHaiten/DucHaiten-realistic-SDXL)
+
+For the diffusiondb dataset, we use a random subset of 5,435 images, with 4,201 in the training partition and 1,234 in the test partition. We use only the realistic images from the realisticSDXL dataset, with images in the realistic-2.2 split in our training data and the realistic-1 split for our test partition. The remaining datasets are used in their entirety, with 80% of the images in the training partition and 20% in the test partition.
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing
+
+**Patch Extraction**
+
+To prepare the training data, we extract 240x240 patches from the images, minimizing the overlap between them. We then select the most informative patches by calculating the gray-level co-occurrence matrix (GLCM) for each patch. Given the GLCM, we calculate the contrast and select the five patches with the highest contrast. These patches are then passed to the model in their original RGB format and cropped to 224x224.
+
+**Data Augmentation**
+
+| Technique                | Probability | Other Parameters                                                                                        |
+|--------------------------|:-----------:|---------------------------------------------------------------------------------------------------------|
+| HorizontalFlip           |     0.35    | -                                                                                                      |
+| RandomBrightnessContrast |     0.50    | brightness\_limit=0.2 contrast\_limit=0.2                                                               |
+| RandomGamma              |     0.50    | gamma\_limit=(80, 120)                                                                                  |
+| CoarseDropout            |     0.50    | min\_holes=1, max\_holes=3 min\_height=64, max\_height=100, min\_width=64, max\_width=100 fill\_value=0 |
+
+
+#### Training Hyperparameters
+
+- Loss Function: Cross-Entropy Loss
+- Optimizer: Adam
+- Learning Rate: 0.0001
+- Weight Decay: 0
+- Scheduler: ReduceLROnPlateau
+- Factor: 0.1
+- Patience: 4
+- Batch Size: 128
+- Epochs: 50
+- Early Stopping: 8
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+- Test Split of our Training Dataset
+- Synthetic Images in the Wild: Dataset containing 210 Authentic and Synthetic Images obtained from Social Media Platforms
+- [Flickr 30k Dataset](https://www.kaggle.com/datasets/hsankesara/flickr-image-dataset)
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+- Accuracy: The proportion of correctly classified images.
+- F1 Score: The harmonic mean of precision and recall.
+
+### Results
+
+<!-- This section provides the results of the evaluation. -->
+
+#### Test Split
+
+Task | Detection F1 Score | Recognition F1 Score
+--- | --- | ---
+Original Images | 0.9867 | 0.9041
+JPEG Compressed Images | 0.9918 | 0.9141
+Downscaled Images | 0.9761 | 0.7470
+Downscaled+Upscaled Images | 0.9868 | 0.8266
+
+#### Synthetic Images in the Wild
+
+79.55% Detection Accuracy
+
+#### Flickr30k Dataset
+
+99.19% Detection Accuracy
+
+### Summary
+
+The model obtains performs well in the test split, with high detection and recognition F1 scores. The model shows robustness to the JPEG compressed images for both tasks while the performance in the downscaled and rescaled images suffers in the recognition task, but the detection task remains stable.
+
+The model is also evaluated in our Synthetic Images in the Wild dataset, which contains 220 images obtained from social media platforms, with 121 real images and 99 AI-generated images. The difficuly of this dataset lies in the fact that the images are uploaded to social media by a wide range of users, so the images may have different resolutions, lighting conditions and quality, additionally they may have been edited or compressed. Regarding the synthetic images, the generation process is unknown, so the model has to generalize to unseen generative models. The dataset was tested by 10 human evaluators, which achieved an average detection accuracy of 72.22% and a best detection accuracy of 78.73%. The model achieves a detection accuracy of 79.55% in this dataset at its best threshold.
+
+Finally, the model shows excellent performance in the Flickr30k dataset. This dataset contains authentic images, so it serves the purpose of testing the number of false positives generated by the model. The model achieves a detection accuracy of 99.19% in this dataset at its best threshold.
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+- **Hardware Type:** 2xH100
+- **Hours used:** 15
+- **Hardware Provider:** Barcelona Supercomputing Center (BSC)
+- **Compute Region:** Spain
+- **Carbon Emitted:** 2.11kg
+
+## Citation
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+```bibtex
+@thesis{bernabeu2024stair,
+    title={Detecting and Attributing AI-Generated Images with Machine Learning},
+    author={Bernabeu Pérez, Pablo},
+    school={UPC, Facultat d'Informàtica de Barcelona, Departament de Ciències de la Computació},
+    year={2024},
+    month={06}
+}
+```
+
+## Model Card Authors
+
+[Pablo Bernabeu](https://huggingface.co/pabberpe) and [Dario Garcia-Gasulla](https://huggingface.co/dariog)
+
+## Model Card Contact
+
+For further inquiries, please contact [HPAI](mailto:[email protected])

SuSy.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0f2916d1fe0967380340860a0d046f625fd4bb34359157bc318707770a002ce9
+size 50810328

test_image.py

@@ -0,0 +1,57 @@
+import numpy as np
+import pandas as pd
+import torch
+from PIL import Image
+from skimage.feature import graycomatrix, graycoprops
+from torchvision import transforms
+
+# Load the model
+model = torch.jit.load("SuSy.pt")
+
+# Load the image
+image = Image.open("midjourney-images-example.jpg")
+
+# Set Parameters
+top_k_patches = 5
+patch_size = 224
+
+# Get the image dimensions
+width, height = image.size
+
+# Calculate the number of patches
+num_patches_x = width // patch_size
+num_patches_y = height // patch_size
+
+# Divide the image in patches
+patches = np.zeros((num_patches_x * num_patches_y, patch_size, patch_size, 3), dtype=np.uint8)
+for i in range(num_patches_x):
+    for j in range(num_patches_y):
+        x = i * patch_size
+        y = j * patch_size
+        patch = image.crop((x, y, x + patch_size, y + patch_size))
+        patches[i * num_patches_y + j] = np.array(patch)
+
+# Compute the most relevant patches (optional)
+dissimilarity_scores = []
+for patch in patches:
+    transform_patch = transforms.Compose([transforms.PILToTensor(), transforms.Grayscale()])
+    grayscale_patch = transform_patch(Image.fromarray(patch)).squeeze(0)
+    glcm = graycomatrix(grayscale_patch, [5], [0], 256, symmetric=True, normed=True)
+    dissimilarity_scores.append(graycoprops(glcm, "contrast")[0, 0])
+
+# Sort patch indices by their dissimilarity score
+sorted_indices = np.argsort(dissimilarity_scores)[::-1]
+
+# Extract top k patches and convert them to tensor
+top_patches = patches[sorted_indices[:top_k_patches]]
+top_patches = torch.from_numpy(np.transpose(top_patches, (0, 3, 1, 2))) / 255.0
+
+# Predict patches
+model.eval()
+with torch.no_grad():
+    preds = model(top_patches)
+
+# Print results
+classes = ['authentic', 'dalle-3-images', 'diffusiondb', 'midjourney-images', 'midjourney_tti', 'realisticSDXL']
+result = pd.DataFrame(preds.numpy(), columns=classes)
+print(result)

test_patch.py

@@ -0,0 +1,23 @@
+import pandas as pd
+import torch
+from PIL import Image
+from torchvision import transforms
+
+# Load the model
+model = torch.jit.load("SuSy.pt")
+
+# Load patch
+patch = Image.open("midjourney-images-example-patch0.png")
+
+# Transform patch to tensor
+patch = transforms.PILToTensor()(patch).unsqueeze(0) / 255.
+
+# Predict patch
+model.eval()
+with torch.no_grad():
+    preds = model(patch)
+
+# Print results
+classes = ['authentic', 'dalle-3-images', 'diffusiondb', 'midjourney-images', 'midjourney_tti', 'realisticSDXL']
+result = pd.DataFrame(preds.numpy(), columns=classes)
+print(result)