Interpolation can make an image larger, but it cannot know whether a blurred patch contains hair, brick, fabric, or nothing at all. A neural network can learn visual patterns from examples and use them to produce a more plausible reconstruction. That still does not mean it can recover details that were never present in the input.

To explore the difference, I compared three small convolutional networks with a bicubic interpolation baseline on DIV2K. The task was to reconstruct 256 × 256 image patches from either 64 × 64 inputs (4× upscaling) or 32 × 32 inputs (8× upscaling).

The short version: all three networks improved the average metrics over bicubic interpolation, and the residual/PixelShuffle model, Netv3, performed best in both settings. But the gains were measured against the exact high-resolution target—not against an objective measure of whether generated details are “true.”

The experiment

Data and split

The source images came from DIV2K. High-resolution images were cropped into 256 × 256 patches. For each patch, a low-resolution input was made at either 32 × 32 or 64 × 64 using TorchVision resizing with bilinear interpolation and antialiasing.

DIV2K provided training and validation images, but no separate test split for this experiment. I divided the 100 validation source images by original image ID: 20 images for validation and the remaining 80 for testing. Keeping the split at source-image level prevents patches from one image appearing in both validation and test.

That distinction matters: the 240 test patches came from 80 source images. Patch-level measurements are useful for comparing reconstructions, but they should not be mistaken for 240 fully independent scenes.

Models

Each model first extracts features from the RGB input, then progressively upsamples those features to 256 × 256.

• BaseNet uses three 3 × 3 convolution layers for feature extraction, followed by transposed convolutions to double the resolution at each stage.
• Netv2 adds a convolution after each upsampling stage to further refine the features.
• Netv3 adds three residual blocks before upsampling and uses sub-pixel convolution with PixelShuffle to double the spatial resolution.

The number of ×2 upsampling stages is three for 32 → 256 and two for 64 → 256. The models have different capacities:

The baseline was OpenCV bicubic interpolation (INTER_CUBIC). It receives the same low-resolution input and does not learn from the training set.

Training and evaluation

All six model/scale combinations used the same training setup: Adam, a batch size of 32, an initial learning rate of 1e-3, and L1 pixel loss. Training ran for up to 100 epochs, with validation every two epochs and a ReduceLROnPlateau learning-rate scheduler. The checkpoint with the lowest validation L1 loss was used for test evaluation.

I compared the learned models and bicubic interpolation with four metrics:

• PSNR: pixel-level reconstruction quality, higher is better.
• SSIM: similarity in local image structure, higher is better.
• LPIPS: learned perceptual distance, lower is better.
• L1 / MAE: average absolute pixel error, lower is better.

Metrics were calculated per test patch and then averaged. PSNR was averaged over patches; it was not calculated from one pooled dataset-wide MSE. LPIPS used the AlexNet feature network. LPIPS was an evaluation metric, not part of the training loss.

Results

8× upscaling: 32 × 32 → 256 × 256

All three networks improve on the bicubic averages. Netv3 has the best mean score in each metric for the 8× task.

Netv3’s mean PSNR gain over bicubic was 0.411 dB. Its SSIM was higher by 0.0283, while LPIPS fell by 0.0887 (about 15%) and L1 fell by 0.00452 (about 9%).

This is the harder setting. The network must expand each input dimension by a factor of eight, so much of the fine structure in the target is no longer directly visible in the input.

4× upscaling: 64 × 64 → 256 × 256

Netv3 again has the best average scores. The 4× task also starts from more information, so every method scores better than in the 8× task.

Here Netv3 improved mean PSNR by 0.715 dB and SSIM by 0.0439. LPIPS decreased by 0.1043 (about 25%), and L1 decreased by 0.00373 (about 11%).

There is an interesting architecture detail: BaseNet outperformed Netv2 at 64 × 64 despite having fewer parameters. Adding layers did not automatically make the model better. Netv3’s skip connections and upsampling design appear to have helped in this experiment, though this comparison alone does not isolate which architectural choice caused the gain.

How consistent were the gains?

Netv3 beat bicubic in mean metrics, but a mean does not describe every patch. Across the 240 test patches, Netv3 had higher PSNR on 216 patches (90.0%) in the 8× setting and 217 patches (90.4%) in the 4× setting. Its LPIPS was lower on 227 patches (94.6%) and 231 patches (96.3%), respectively.

Most patches favor Netv3, but the distribution also contains patches where bicubic wins.

One notable exception was an almost uniform patch. Bicubic interpolation reconstructed it extremely closely and achieved around 89 dB PSNR, while Netv3’s small residual error led to a much lower score. Networks can be strong on average and still lose on particular image content.

What do the reconstructions look like?

The following figure shows example patches at both input resolutions. The low-resolution column is enlarged with nearest-neighbor interpolation only to make its pixelation visible; the model and bicubic columns are their actual 256 × 256 outputs.

The learned models tend to produce sharper-looking edges than bicubic. They still do not match all of the fine texture in the original high-resolution patch.

Here are additional examples for Netv3:

64 × 64 → 256 × 256.

32 × 32 → 256 × 256.

Visual sharpness is not the same as recovered truth. If an input has lost the texture of a fabric weave, the network has no direct way to know the exact weave pattern in the target. It can learn a plausible pattern from training examples, and that prediction may look convincing while still differing from the original.

Training behavior and cost

The Netv3 training curves show a rapid early reduction in L1 loss followed by smaller improvements. The best validation checkpoint for both scales occurred at epoch 100, the final epoch in this run.

Netv3 at 32 × 32 input. Validation is performed every two epochs.

Netv3 at 64 × 64 input.

The recorded training times on the experiment’s CUDA environment were:

These are timings from one run and one GPU environment, not hardware-independent benchmarks. Netv3 has the most parameters, yet trained faster than Netv2 in these runs; parameter count alone does not determine wall-clock training time.

What this experiment does—and does not—show

The results support a limited conclusion: on this DIV2K split, with this training setup, the learned models improved average reconstruction metrics over bicubic interpolation, and Netv3 was the strongest of the three tested architectures.

They do not establish that a neural network can recover the exact information discarded during downsampling. Nor do they prove that Netv3 is generally the best super-resolution architecture. This was a small comparison using one training seed, a single train/validation/test split, and relatively compact networks trained with pixel-wise L1 loss.

The next experiments I would run are:

1. Train each architecture with multiple seeds and report variation across runs, not only across image patches.
2. Compare more training crops and datasets, while keeping source-image-level separation between splits.
3. Test perceptual or adversarial objectives if perceptual realism is the goal, while being explicit that these can trade pixel fidelity for plausible texture.
4. Evaluate inference speed and memory on the target hardware.
5. Inspect failures as carefully as the average scores.

Final takeaway

Can a neural network recover detail? It can learn to make a better-informed guess than a fixed interpolation rule. In this experiment, that guess improved both average pixel metrics and perceptual distance—especially for 4× upscaling.

But “more detailed” does not always mean “more accurate.” The high-resolution target is still the ground truth, and any detail absent from the low-resolution input remains uncertain. Super-resolution is not just enlargement; it is reconstruction guided by learned prior experience.

Proponowane podejście

Zastosowanie architektury U-Net, ze wstępnie wytrenowanym koderem ResNet34.

U-Net to architektura typu koder-dekoder stworzona w 2015 roku, w celu segmentacji obrazów biomedycznych. Istotą tej architektury są połączenia (skip-connections) pomiędzy odpowiednimi warstwami kodera i dekodera.

Architektura U-Net znajduje szerokie zastosowanie w zagadnieniach segmentacji, stosowano ją na przykład do detekcji dróg w obrazach lotnicznych, przetwarzania obrazów satelitarnych czy segmentacji otoczenia w pojazdach autonomicznych.

W proponowanym modelu U-Net, koder składający się ze zwkyłych warstw splotowych, jest zastąpiony przez bloki sieci ResNet. W sumie koder będzie miał 33 warstwy (z sieci ResNet34 odrzucono ostatnią, w pełni połączoną warstwę). Koder zostanie zainicjalizowany wagami wstępnie wytrenowanymi na zbiorze ImageNet (wykorzystane wagi pochodzą z repozytrium udostępnionego przez PyTorch).

Proponowany model na wejściu przyjmuje kolorowy obraz z kanałami RGB. Aby zapewnić, że wstępnie wytrenowany koder działa poprawnie, zastosowany zostanie preprocessing zgodny ze standardem ImageNet. Obejmuje on skalowanie wartości pikseli do zakresu [0, 1] oraz normalizację przy użyciu średniej i odchylenia standardowego dla zbioru ImageNet.

Wyjście modelu ma taki sam rozmiar jak obraz wejściowy, ale z jednym kanałem. Na wyjściu modelu zaaplikowana zostanie funkcja aktywacji sigmoid wartości wyjściowe mogą być interpetowane jako prawdopodbieństwo przynależności danego piksela, do klasy, w tym przypadku wody. Funkcja straty modelu oparta jest o współczynnik Sorensena i nosi nazwę Dice Loss

$$\mathcal{L}_{\text{Dice}} = 1 - \frac{2 \sum_i y_i \hat{y}_i + \epsilon}{\sum_i y_i + \sum_i \hat{y}_i + \epsilon}$$

Strategia uczenia tak zmodyfikowanego modelu u-net moze przebiegać następująco:

• W pierwszym etapie wagi enkodera są zamrożone (frozen), uczone są jedynie wagi dekodera.
• W następnym etapie kolejne warstwy wag enkodera (zaczynając od góry) mogą być odblokowane i wytrenowane od nowa na zbiorze treningowym, w celu lepszej adaptacji do zadania.

Zbiór danych

Proponowane rozwiązanie zostanie przetestowane na zbiorze danych LaRS (Lakes, Rivers, Seas). Jest to zbiór przeznaczony do panoptycznej segmentacji oraz detekcji obiektów w środowiskach wodnych. Zawiera sceny rejestrowane na jeziorach, rzekach i morzach, dzięki czemu dobrze odzwierciedla zmienne warunki, takie jak różne typy zbiorników, oświetlenie czy pogoda.

Implementacja

Model został zaimplementowany z wykorzystaniem języka Python oraz biblioteki Segmentation Models PyTroch (SMP), która stanowi nakładke na PyTorch, dodającą modele segmentacji. Biblioteka ta jest często wykorzystywana w badaniach czy konkursach dotyczących uczenia maszynowego.

Kod zawiera klase generująca obiekt Dataset

class WaterBinaryDataset(Dataset):
    def __init__(self, root, split, image_size, augmentations=None):
        self.split = split
        self.image_size = image_size
        self.augmentations = augmentations

        self.image_dir = Path(root) / split / "images"
        self.mask_dir = Path(root) / split / "semantic_masks"

        self.image_paths = sorted(self.image_dir.glob("*.jpg"))

        self.has_masks = self.mask_dir.exists() and len(list(self.mask_dir.glob("*.png"))) > 0

        if self.has_masks:
            self.mask_paths = {p.stem: p for p in self.mask_dir.glob("*.png")}
            self.pairs = [
                (img, self.mask_paths[img.stem])
                for img in self.image_paths
                if img.stem in self.mask_paths
            ]
        else:
            # Test images
            self.pairs = self.image_paths

        self.resize = T.Resize(
            size=(image_size, image_size),
            interpolation=InterpolationMode.BILINEAR,
            antialias=True,
        )

        print(f"{split}: {len(self.pairs)} samples | masks: {self.has_masks}")

    def __len__(self):
        return len(self.pairs)

    def __getitem__(self, idx):

        # Test Images
        if not self.has_masks:
            img_path = self.pairs[idx]
            image = read_image(str(img_path)).float() / 255.0
            image = tv_tensors.Image(image)

            image = self.resize(image)

            if self.augmentations is not None:
                image = self.augmentations(image)

            return image.as_subclass(torch.Tensor)

        # Train and val images
        img_path, mask_path = self.pairs[idx]

        image = read_image(str(img_path)).float() / 255.0
        mask = read_image(str(mask_path))[0].long()

        mask[mask == 255] = 0
        mask = (mask == 1).float()
        mask = mask.unsqueeze(0)

        image = tv_tensors.Image(image)
        mask = tv_tensors.Mask(mask)

        image, mask = self.resize(image, mask)

        if self.augmentations is not None:
            image, mask = self.augmentations(image, mask)

        return image.as_subclass(torch.Tensor), mask.as_subclass(torch.Tensor)

Definicje modelu w PyTroch, korzystającego dodatkowo z bibilioteki PyLighting

class WaterSegmentationModel(pl.LightningModule):
    def __init__(
        self,
        arch,
        encoder_name,
        in_channels,
        out_classes,
        t_max,
        lr=2e-4,
        trainable_backbone_layers=0,
        **kwargs
    ):
        super().__init__()
        self.save_hyperparameters()

        self.model = smp.create_model(
            arch,
            encoder_name=encoder_name,
            in_channels=in_channels,
            classes=out_classes,
            **kwargs,
        )

        params = smp.encoders.get_preprocessing_params(encoder_name)
        self.register_buffer("std", torch.tensor(params["std"]).view(1, 3, 1, 1))
        self.register_buffer("mean", torch.tensor(params["mean"]).view(1, 3, 1, 1))

        self.t_max = t_max
        self.loss_fn = smp.losses.DiceLoss(smp.losses.BINARY_MODE, from_logits=True)
        self.lr = lr

        self.trainable_backbone_layers = trainable_backbone_layers


        for param in self.model.encoder.parameters():
            param.requires_grad = False

        if trainable_backbone_layers > 0:
            for layer in list(self.model.encoder.children())[-self.trainable_backbone_layers:]:
                for param in layer.parameters():
                    param.requires_grad = True

        self.training_step_outputs = []
        self.validation_step_outputs = []
        self.test_step_outputs = []

    def forward(self, image):
        image = (image - self.mean) / self.std
        return self.model(image)

    def shared_step(self, batch, stage):
        image = batch[0]
        mask = batch[1]

        logits_mask = self.forward(image)
        loss = self.loss_fn(logits_mask, mask)

        prob_mask = logits_mask.sigmoid()
        pred_mask = (prob_mask > 0.5).float()

        tp, fp, fn, tn = smp.metrics.get_stats(
            pred_mask.long(), mask.long(), mode="binary"
        )

        self.log(f"{stage}_loss", loss, on_step=(stage == "train"), on_epoch=True, prog_bar=True)

        return {
            "loss": loss,
            "tp": tp,
            "fp": fp,
            "fn": fn,
            "tn": tn,
        }

    def shared_epoch_end(self, outputs, stage):
        tp = torch.cat([x["tp"] for x in outputs])
        fp = torch.cat([x["fp"] for x in outputs])
        fn = torch.cat([x["fn"] for x in outputs])
        tn = torch.cat([x["tn"] for x in outputs])

        # IoU (you already had this)
        per_image_iou = smp.metrics.iou_score(
            tp, fp, fn, tn, reduction="micro-imagewise"
        )
        dataset_iou = smp.metrics.iou_score(
            tp, fp, fn, tn, reduction="micro"
        )

        # Precision, recall, F1 (global / micro)
        tp_sum = tp.sum().float()
        fp_sum = fp.sum().float()
        fn_sum = fn.sum().float()
        eps = 1e-7

        precision = tp_sum / (tp_sum + fp_sum + eps)
        recall = tp_sum / (tp_sum + fn_sum + eps)
        f1 = 2 * precision * recall / (precision + recall + eps)

        self.log_dict(
            {
                f"{stage}_per_image_iou": per_image_iou,
                f"{stage}_dataset_iou": dataset_iou,
                f"{stage}_precision": precision,
                f"{stage}_recall": recall,
                f"{stage}_f1": f1,
            },
            prog_bar=True,
        )

    def training_step(self, batch, batch_idx):
        out = self.shared_step(batch, "train")
        self.training_step_outputs.append(out)
        return out["loss"]

    def on_train_epoch_end(self):
        self.shared_epoch_end(self.training_step_outputs, "train")
        self.training_step_outputs.clear()

    def validation_step(self, batch, batch_idx):
        out = self.shared_step(batch, "valid")
        self.validation_step_outputs.append(out)
        return out

    def on_validation_epoch_end(self):
        self.shared_epoch_end(self.validation_step_outputs, "valid")
        self.validation_step_outputs.clear()

    def test_step(self, batch, batch_idx):
        out = self.shared_step(batch, "test")
        self.test_step_outputs.append(out)
        return out

    def on_test_epoch_end(self):
        self.shared_epoch_end(self.test_step_outputs, "test")
        self.test_step_outputs.clear()

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr=self.lr)
        scheduler = lr_scheduler.CosineAnnealingLR(
            optimizer,
            T_max=self.t_max,
            eta_min=1e-5
        )
        return {
            "optimizer": optimizer,
            "lr_scheduler": {
                "scheduler": scheduler,
                "interval": "step",
                "frequency": 1,
            },
        }