, , , , , , , , , , , , ,

TMM’s involvement of Raytheon implies that TMM could not figure out how to make their own technology work. This is understandable. Many smart people have tried and failed, and mathematics has not evolved to reveal any new directions.

An interesting trend these days is a resurgence in AI (artificial intelligence). Companies with vast databases like Google and IBM use large scale data mining and statistical techniques to make intelligent — or at least practical — guesses about things, like voice recognition or playing Jeopardy.

What we really want an image or video codec to do is upscale creatively. If we gave a person a photo of a landscape, he could enlarge it and make it geniuinely have higher resolution. Maybe he would start by using bicubic resampling or even Perfect Resize, but then he would make sensible adjustments to add detail. Or maybe he would surmise the gist of the original image, and create a whole new one using it as a guide.

Obviously a human would take a long time to do so. So too would a computer. And to what end? We could shoot photos and movies with low-resolution cameras and then upscale them after, or we could just use high-resolution cameras in the first place.

A computer would need to extract geometry and texture from a video and then rerender. This is still well beyond the capabilities of machine vision. The computer would actually need to know what it was looking at, and for small features, it would need to make logical guesses based on context.

How much could we upscale? At two to four times, the details we need to add comprise three to fifteen times extra information. The higher we go, the more creative we need to be. If an astronomer gave us a picture of a blue dot and wanted it to be upscaled to a poster, we could draw a near-infinite set of richly detailed blue planets. Since each one has an equal probability of being right (they can all be downsampled to match the blue dot), upscaling beyond a certain level creates a total breakdown in certainty and consistency.

If creative upscaling could be automated, it is unlikely to be deployed for realtime decoding due to the computational demands. Instead, it would be a tool with which to create footage prior to distribution. We could take legacy film, enlarge it to fit the highest-resolution TVs and cinemas, and then just encode it with ordinary codecs.

In fact, you can do this today: extract frames from a raw format video as individual bitmap files, batch process them with Perfect Resize, and then stitch the resulting larger bitmaps into a new, higher-resolution movie.