AI algorithms in GIMP

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

AI algorithms in GIMP

Developers mailing list
Dear devs,

I have recently joined the mailing list because I wanted to contribute my
two pennies to GIMP development (since I use it for my work). I had a look
at the proposed plan for GIMP and wondered if people would be interested in
including some popular AI algorithms for several image processing tasks.

I would be interested in writing implementations of some of these
algorithms into gimp if someone can commit to writing a frontend/GUI for it.

It would be great if we can make a list of these algorithms to implement
and rank them according to priority.

As for my background, I am a theoretical physicist making simulations for
HPCs (in C/C++) and interpreting their data (in Python). I have a
reasonable workstation to train neural nets, if necessary. Be warned that I
have never written a GUI software in my life and I don't know the GIMP
codebase at all. I envision these to be standalone scripts which can be
called in from the GIMP interface.

Please let me know what you think.

Cheerio,
Maitraya.

Senior Research Fellow
Center for Excellence in Space Sciences India
Indian Institute of Science Education and Research Kolkata
_______________________________________________
gimp-developer-list mailing list
List address:    [hidden email]
List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list
List archives:   https://mail.gnome.org/archives/gimp-developer-list
Reply | Threaded
Open this post in threaded view
|

Re: AI algorithms in GIMP

Developers mailing list
On Thu, Jan 17, 2019 at 2:14 PM Maitraya Bhattacharyya wrote:

>
> Dear devs,
>
> I have recently joined the mailing list because I wanted to contribute my
> two pennies to GIMP development (since I use it for my work). I had a look
> at the proposed plan for GIMP and wondered if people would be interested in
> including some popular AI algorithms for several image processing tasks.
>
> I would be interested in writing implementations of some of these
> algorithms into gimp if someone can commit to writing a frontend/GUI for it.
>
> It would be great if we can make a list of these algorithms to implement
> and rank them according to priority.
>
> As for my background, I am a theoretical physicist making simulations for
> HPCs (in C/C++) and interpreting their data (in Python). I have a
> reasonable workstation to train neural nets, if necessary. Be warned that I
> have never written a GUI software in my life and I don't know the GIMP
> codebase at all. I envision these to be standalone scripts which can be
> called in from the GIMP interface.
>
> Please let me know what you think.

Hello and welcome :)

What algorithms in particular are you thinking of?

Alex
_______________________________________________
gimp-developer-list mailing list
List address:    [hidden email]
List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list
List archives:   https://mail.gnome.org/archives/gimp-developer-list
Reply | Threaded
Open this post in threaded view
|

Re: AI algorithms in GIMP

Developers mailing list
Hi Alexandre,

I have yet to compile an exhaustive list of algorithms but I am going to
mention two at the moment. We can compile a list later too:

1. Assisted background removal: The paper on this from Facebook is
https://arxiv.org/pdf/1703.06870.pdf . The ability to identify different
objects in an image including foreground objects can assist users in
removing the background object. I am specifically interesting in the
removal of hair (which poses to be a really big problem). I believe there
are already a few trained models on this already available in python
libraries (TensorFlow).

2. Deep Style transfer: The relevant paper is at
https://arxiv.org/pdf/1508.06576.pdf . This can be used to create 'Prisma
like' transfer of style from one painting to another. I believe we may also
have pretrained models already available here but we could try training the
network specific to our needs.

I think the idea is to do something along the lines of what Adobe is doing
with Photoshop or in the wider context for their entire product line.
Although we lack the funding or infrastructure to be a serious competitor,
there are several researchers in the image processing/Machine Learning
community who've written amazing papers over the years and open sourced
their datasets. We should still be able to get something respectable using
their research as reference.

If there is nobody on the Machine Learning side in GIMP, I suggest you let
me read a few papers and make some implementations but specific to what the
GIMP users want. I am not an ML guy but there are people in my community
who are using ML for science and there is a possibility I will be able to
talk to some of them (who do image processing, but with sunspots instead of
pictures of puppies).

ML is a strange and wonderful thing, and people have tried doing everything
with it. Sometimes it works, sometimes it doesn't. But we have to know what
the GIMP community wants, then I will try to be organize our collective
thoughts in a more technical document.

Thanks,
Maitraya.

On Thu, Jan 17, 2019 at 5:01 PM Alexandre Prokoudine via
gimp-developer-list <[hidden email]> wrote:

> On Thu, Jan 17, 2019 at 2:14 PM Maitraya Bhattacharyya wrote:
> >
> > Dear devs,
> >
> > I have recently joined the mailing list because I wanted to contribute my
> > two pennies to GIMP development (since I use it for my work). I had a
> look
> > at the proposed plan for GIMP and wondered if people would be interested
> in
> > including some popular AI algorithms for several image processing tasks.
> >
> > I would be interested in writing implementations of some of these
> > algorithms into gimp if someone can commit to writing a frontend/GUI for
> it.
> >
> > It would be great if we can make a list of these algorithms to implement
> > and rank them according to priority.
> >
> > As for my background, I am a theoretical physicist making simulations for
> > HPCs (in C/C++) and interpreting their data (in Python). I have a
> > reasonable workstation to train neural nets, if necessary. Be warned
> that I
> > have never written a GUI software in my life and I don't know the GIMP
> > codebase at all. I envision these to be standalone scripts which can be
> > called in from the GIMP interface.
> >
> > Please let me know what you think.
>
> Hello and welcome :)
>
> What algorithms in particular are you thinking of?
>
> Alex
> _______________________________________________
> gimp-developer-list mailing list
> List address:    [hidden email]
> List membership:
> https://mail.gnome.org/mailman/listinfo/gimp-developer-list
> List archives:   https://mail.gnome.org/archives/gimp-developer-list
>
_______________________________________________
gimp-developer-list mailing list
List address:    [hidden email]
List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list
List archives:   https://mail.gnome.org/archives/gimp-developer-list
Reply | Threaded
Open this post in threaded view
|

Re: AI algorithms in GIMP

Developers mailing list
Hi again,

Something that I forgot to mention in the last email, here is a link for
fun: https://ai.googleblog.com/2017/04/teaching-machines-to-draw.html

M.

On Thu, Jan 17, 2019 at 6:43 PM Maitraya Bhattacharyya <
[hidden email]> wrote:

> Hi Alexandre,
>
> I have yet to compile an exhaustive list of algorithms but I am going to
> mention two at the moment. We can compile a list later too:
>
> 1. Assisted background removal: The paper on this from Facebook is
> https://arxiv.org/pdf/1703.06870.pdf . The ability to identify different
> objects in an image including foreground objects can assist users in
> removing the background object. I am specifically interesting in the
> removal of hair (which poses to be a really big problem). I believe there
> are already a few trained models on this already available in python
> libraries (TensorFlow).
>
> 2. Deep Style transfer: The relevant paper is at
> https://arxiv.org/pdf/1508.06576.pdf . This can be used to create 'Prisma
> like' transfer of style from one painting to another. I believe we may also
> have pretrained models already available here but we could try training the
> network specific to our needs.
>
> I think the idea is to do something along the lines of what Adobe is doing
> with Photoshop or in the wider context for their entire product line.
> Although we lack the funding or infrastructure to be a serious competitor,
> there are several researchers in the image processing/Machine Learning
> community who've written amazing papers over the years and open sourced
> their datasets. We should still be able to get something respectable using
> their research as reference.
>
> If there is nobody on the Machine Learning side in GIMP, I suggest you let
> me read a few papers and make some implementations but specific to what the
> GIMP users want. I am not an ML guy but there are people in my community
> who are using ML for science and there is a possibility I will be able to
> talk to some of them (who do image processing, but with sunspots instead of
> pictures of puppies).
>
> ML is a strange and wonderful thing, and people have tried doing
> everything with it. Sometimes it works, sometimes it doesn't. But we have
> to know what the GIMP community wants, then I will try to be organize our
> collective thoughts in a more technical document.
>
> Thanks,
> Maitraya.
>
> On Thu, Jan 17, 2019 at 5:01 PM Alexandre Prokoudine via
> gimp-developer-list <[hidden email]> wrote:
>
>> On Thu, Jan 17, 2019 at 2:14 PM Maitraya Bhattacharyya wrote:
>> >
>> > Dear devs,
>> >
>> > I have recently joined the mailing list because I wanted to contribute
>> my
>> > two pennies to GIMP development (since I use it for my work). I had a
>> look
>> > at the proposed plan for GIMP and wondered if people would be
>> interested in
>> > including some popular AI algorithms for several image processing tasks.
>> >
>> > I would be interested in writing implementations of some of these
>> > algorithms into gimp if someone can commit to writing a frontend/GUI
>> for it.
>> >
>> > It would be great if we can make a list of these algorithms to implement
>> > and rank them according to priority.
>> >
>> > As for my background, I am a theoretical physicist making simulations
>> for
>> > HPCs (in C/C++) and interpreting their data (in Python). I have a
>> > reasonable workstation to train neural nets, if necessary. Be warned
>> that I
>> > have never written a GUI software in my life and I don't know the GIMP
>> > codebase at all. I envision these to be standalone scripts which can be
>> > called in from the GIMP interface.
>> >
>> > Please let me know what you think.
>>
>> Hello and welcome :)
>>
>> What algorithms in particular are you thinking of?
>>
>> Alex
>> _______________________________________________
>> gimp-developer-list mailing list
>> List address:    [hidden email]
>> List membership:
>> https://mail.gnome.org/mailman/listinfo/gimp-developer-list
>> List archives:   https://mail.gnome.org/archives/gimp-developer-list
>>
>
_______________________________________________
gimp-developer-list mailing list
List address:    [hidden email]
List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list
List archives:   https://mail.gnome.org/archives/gimp-developer-list
Reply | Threaded
Open this post in threaded view
|

Re: AI algorithms in GIMP

Developers mailing list
In reply to this post by Developers mailing list
Hi!

On Thu, Jan 17, 2019 at 12:14 PM Maitraya Bhattacharyya via
gimp-developer-list <[hidden email]> wrote:

> Dear devs,
>
> I have recently joined the mailing list because I wanted to contribute my
> two pennies to GIMP development (since I use it for my work). I had a look
> at the proposed plan for GIMP and wondered if people would be interested in
> including some popular AI algorithms for several image processing tasks.
>

We are definitely interested by any AI algorithms. At least I am.

I would be interested in writing implementations of some of these
> algorithms into gimp if someone can commit to writing a frontend/GUI for
> it.
>

The hard point here is "if someone can commit to […]". It's a bit hard to
commit without knowing much (unless you are paid, then you have no choice
;p). Usually it's the other way around: you propose something. It doesn't
have to be with a great GUI or whatever. Then if we like what we see, we
will definitely add our own 2 cents to the pool. This is usually how most
features are done around here, when someone contributes a patch with a very
cool idea, then we review and often fix/change what we think is needed
(sometimes just a bit, sometimes deeply).

I often wrote crap GUI myself and others came to the rescue with ideas and
code. :-)


> It would be great if we can make a list of these algorithms to implement
> and rank them according to priority.
>

I would suggest to *not do that*. :-)
Basically we are not a company, we don't sell GIMP and don't have huge
plans for the next decade. Well "officially", we do have a roadmap and
such, but if you follow GIMP development, you'd see it is more flexible and
experimental than some rigid plan.
Making a huge list with big plans for the future "might" be just the way to
spend a lot of time and kill your project in the end.

Instead I would propose **you** just select **one** such algorithm which
you find is great and even irrefusable since it would be so fucking awesome
and useful! Then you implement and propose it and we will be so amazed that
we just have to include it and do a proper GUI for it. That sounds like the
best plan.

From there, you can go on with more awesome ideas. :-)
You may even be able to start doing more organized work with a list of
algorithms after, etc. But for a first patch, I would suggest you just take
the lead.


> As for my background, I am a theoretical physicist making simulations for
> HPCs (in C/C++) and interpreting their data (in Python). I have a

reasonable workstation to train neural nets, if necessary. Be warned that I
>

There is only a single very important part about AI algorithms which need
training: we will want the code to train, the data, etc. everything as free
software/Open Data and properly documented.
I have worked with trained algorithms in Free Software where the trained
data is just dropped as-is, and once the original author disappears, this
is unmaintainable. In particular it cannot be improved or fixed or nothing,
because we don't have the code to re-generate the neural networks (or
alike). This can only be a recipe for failure long-term.

So AI in GIMP? Yeah definitely! But it has to be reproducible generated
data, with the whole training infrastructure available and properly
documented and the input data under Libre license as well.


> have never written a GUI software in my life and I don't know the GIMP
> codebase at all. I envision these to be standalone scripts which can be
> called in from the GIMP interface.
>

Cool. Depending on the idea, it may be interesting to rather implement it
directly as a GEGL operation (GEGL is our graphics engine). Maybe you don't
even need to make a GUI then. Just make a GEGL op, and when it is done, run
it with examples to show us how good it is, and we might just get into the
game to make it a proper GUI.


> Please let me know what you think.
>

And here you are! I hope we will see soon a lot of baby robots in our code.
;-)

Jehan



> Cheerio,
> Maitraya.
>
> Senior Research Fellow
> Center for Excellence in Space Sciences India
> Indian Institute of Science Education and Research Kolkata
> _______________________________________________
> gimp-developer-list mailing list
> List address:    [hidden email]
> List membership:
> https://mail.gnome.org/mailman/listinfo/gimp-developer-list
> List archives:   https://mail.gnome.org/archives/gimp-developer-list
>


--
ZeMarmot open animation film
http://film.zemarmot.net
Liberapay: https://liberapay.com/ZeMarmot/
Patreon: https://patreon.com/zemarmot
Tipeee: https://www.tipeee.com/zemarmot
_______________________________________________
gimp-developer-list mailing list
List address:    [hidden email]
List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list
List archives:   https://mail.gnome.org/archives/gimp-developer-list
Reply | Threaded
Open this post in threaded view
|

Re: AI algorithms in GIMP

Developers mailing list
Hi guys,

I think Casey's suggestion of super resolution is an excellent idea and
would be useful to a lot of people using GIMP. The two algorithms which
have been doing quite well are:

1. SRGAN: The relevant paper is at https://arxiv.org/abs/1609.04802 . This
has yielded some very nice results so far and is now considered to be
state-of-the-art in the computer vision community. Implementing this will
be my first priority.

2. SRResNet: The paper is at https://arxiv.org/abs/1609.04802 . This
algorithm also gives quite good results.

Now to train these neural nets, I will have to prepare a dataset consisting
of low resolution and high resolution images and divide them into training,
testing and validation sets. I'll try to write a script which pulls
high-resolution CC-A/CC0/CC-A-SA images from the internet and downscale
them in python to prepare their low resolution counterparts. I only hope
that my GPU has enough memory to train the model, otherwise I am done for.

What I would like to do is create a python script (most of these libraries
are available in Python) and then pass the image along with the scaling
factor into this from inside GIMP. I've never worked with GEGL before but I
will try finding out how to use this.

Since I am working on this in my free time and have not implemented neural
nets much, this might take some time for me to learn. I'll try keeping you
guys informed.

Any suggestions are welcome.

Thanks,
Maitraya.

On Sun, Jan 20, 2019 at 12:41 AM Jehan Pagès <[hidden email]>
wrote:

> Hi!
>
> On Thu, Jan 17, 2019 at 12:14 PM Maitraya Bhattacharyya via
> gimp-developer-list <[hidden email]> wrote:
>
>> Dear devs,
>>
>> I have recently joined the mailing list because I wanted to contribute my
>> two pennies to GIMP development (since I use it for my work). I had a look
>> at the proposed plan for GIMP and wondered if people would be interested
>> in
>> including some popular AI algorithms for several image processing tasks.
>>
>
> We are definitely interested by any AI algorithms. At least I am.
>
> I would be interested in writing implementations of some of these
>> algorithms into gimp if someone can commit to writing a frontend/GUI for
>> it.
>>
>
> The hard point here is "if someone can commit to […]". It's a bit hard to
> commit without knowing much (unless you are paid, then you have no choice
> ;p). Usually it's the other way around: you propose something. It doesn't
> have to be with a great GUI or whatever. Then if we like what we see, we
> will definitely add our own 2 cents to the pool. This is usually how most
> features are done around here, when someone contributes a patch with a very
> cool idea, then we review and often fix/change what we think is needed
> (sometimes just a bit, sometimes deeply).
>
> I often wrote crap GUI myself and others came to the rescue with ideas and
> code. :-)
>
>
>> It would be great if we can make a list of these algorithms to implement
>> and rank them according to priority.
>>
>
> I would suggest to *not do that*. :-)
> Basically we are not a company, we don't sell GIMP and don't have huge
> plans for the next decade. Well "officially", we do have a roadmap and
> such, but if you follow GIMP development, you'd see it is more flexible and
> experimental than some rigid plan.
> Making a huge list with big plans for the future "might" be just the way
> to spend a lot of time and kill your project in the end.
>
> Instead I would propose **you** just select **one** such algorithm which
> you find is great and even irrefusable since it would be so fucking awesome
> and useful! Then you implement and propose it and we will be so amazed that
> we just have to include it and do a proper GUI for it. That sounds like the
> best plan.
>
> From there, you can go on with more awesome ideas. :-)
> You may even be able to start doing more organized work with a list of
> algorithms after, etc. But for a first patch, I would suggest you just take
> the lead.
>
>
>> As for my background, I am a theoretical physicist making simulations for
>> HPCs (in C/C++) and interpreting their data (in Python). I have a
>
> reasonable workstation to train neural nets, if necessary. Be warned that I
>>
>
> There is only a single very important part about AI algorithms which need
> training: we will want the code to train, the data, etc. everything as free
> software/Open Data and properly documented.
> I have worked with trained algorithms in Free Software where the trained
> data is just dropped as-is, and once the original author disappears, this
> is unmaintainable. In particular it cannot be improved or fixed or nothing,
> because we don't have the code to re-generate the neural networks (or
> alike). This can only be a recipe for failure long-term.
>
> So AI in GIMP? Yeah definitely! But it has to be reproducible generated
> data, with the whole training infrastructure available and properly
> documented and the input data under Libre license as well.
>
>
>> have never written a GUI software in my life and I don't know the GIMP
>> codebase at all. I envision these to be standalone scripts which can be
>> called in from the GIMP interface.
>>
>
> Cool. Depending on the idea, it may be interesting to rather implement it
> directly as a GEGL operation (GEGL is our graphics engine). Maybe you don't
> even need to make a GUI then. Just make a GEGL op, and when it is done, run
> it with examples to show us how good it is, and we might just get into the
> game to make it a proper GUI.
>
>
>> Please let me know what you think.
>>
>
> And here you are! I hope we will see soon a lot of baby robots in our
> code. ;-)
>
> Jehan
>
>
>
>> Cheerio,
>> Maitraya.
>>
>> Senior Research Fellow
>> Center for Excellence in Space Sciences India
>> Indian Institute of Science Education and Research Kolkata
>> _______________________________________________
>> gimp-developer-list mailing list
>> List address:    [hidden email]
>> List membership:
>> https://mail.gnome.org/mailman/listinfo/gimp-developer-list
>> List archives:   https://mail.gnome.org/archives/gimp-developer-list
>>
>
>
> --
> ZeMarmot open animation film
> http://film.zemarmot.net
> Liberapay: https://liberapay.com/ZeMarmot/
> Patreon: https://patreon.com/zemarmot
> Tipeee: https://www.tipeee.com/zemarmot
>
_______________________________________________
gimp-developer-list mailing list
List address:    [hidden email]
List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list
List archives:   https://mail.gnome.org/archives/gimp-developer-list
Reply | Threaded
Open this post in threaded view
|

Re: AI algorithms in GIMP

Elle Stone
On 1/21/19 8:53 AM, Maitraya Bhattacharyya via gimp-developer-list wrote:
>   super resolution

I don't know if this is the same "super resolution", but FWIW the topic
has come up several times on discuss.pixls.us:

https://discuss.pixls.us/search?q=%20superresolution

Best regards,
Elle Stone
--
https://ninedegreesbelow.com
Color management and free/libre photography
_______________________________________________
gimp-developer-list mailing list
List address:    [hidden email]
List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list
List archives:   https://mail.gnome.org/archives/gimp-developer-list
Reply | Threaded
Open this post in threaded view
|

Re: AI algorithms in GIMP

Developers mailing list
Hi,

I already tried the first one, but it has not the quality I expected. Did
anybody tried the second one? The second one looks quite good :)

[1] https://github.com/alexjc/neural-enhance
[2] https://github.com/tensorlayer/srgan

Best regards,

Martin


Am Mo., 21. Jan. 2019 um 16:07 Uhr schrieb Elle Stone <
[hidden email]>:

> On 1/21/19 8:53 AM, Maitraya Bhattacharyya via gimp-developer-list wrote:
> >   super resolution
>
> I don't know if this is the same "super resolution", but FWIW the topic
> has come up several times on discuss.pixls.us:
>
> https://discuss.pixls.us/search?q=%20superresolution
>
> Best regards,
> Elle Stone
> --
> https://ninedegreesbelow.com
> Color management and free/libre photography
> _______________________________________________
> gimp-developer-list mailing list
> List address:    [hidden email]
> List membership:
> https://mail.gnome.org/mailman/listinfo/gimp-developer-list
> List archives:   https://mail.gnome.org/archives/gimp-developer-list
>
_______________________________________________
gimp-developer-list mailing list
List address:    [hidden email]
List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list
List archives:   https://mail.gnome.org/archives/gimp-developer-list
Reply | Threaded
Open this post in threaded view
|

Re: AI algorithms in GIMP

Developers mailing list
Hi Martin,

Thanks for the second link on Tensorlayer. if the license of the library is
compatible with GPL, there is no reason we cannot used their VGG19 network
but I don't think it is! I will still need a few weeks to create the image
dataset and then train this.

Thanks,
Maitraya.


On Mon, Jan 21, 2019 at 8:51 PM Martin Marmsoler <[hidden email]>
wrote:

> Hi,
>
> I already tried the first one, but it has not the quality I expected. Did
> anybody tried the second one? The second one looks quite good :)
>
> [1] https://github.com/alexjc/neural-enhance
> [2] https://github.com/tensorlayer/srgan
>
> Best regards,
>
> Martin
>
>
> Am Mo., 21. Jan. 2019 um 16:07 Uhr schrieb Elle Stone <
> [hidden email]>:
>
>> On 1/21/19 8:53 AM, Maitraya Bhattacharyya via gimp-developer-list wrote:
>> >   super resolution
>>
>> I don't know if this is the same "super resolution", but FWIW the topic
>> has come up several times on discuss.pixls.us:
>>
>> https://discuss.pixls.us/search?q=%20superresolution
>>
>> Best regards,
>> Elle Stone
>> --
>> https://ninedegreesbelow.com
>> Color management and free/libre photography
>> _______________________________________________
>> gimp-developer-list mailing list
>> List address:    [hidden email]
>> List membership:
>> https://mail.gnome.org/mailman/listinfo/gimp-developer-list
>> List archives:   https://mail.gnome.org/archives/gimp-developer-list
>>
>
_______________________________________________
gimp-developer-list mailing list
List address:    [hidden email]
List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list
List archives:   https://mail.gnome.org/archives/gimp-developer-list
Reply | Threaded
Open this post in threaded view
|

Re: AI algorithms in GIMP

Elle Stone
In reply to this post by Elle Stone
On 1/21/19 10:01 AM, Elle Stone wrote:
>
> I don't know if this is the same "super resolution", but FWIW the topic
> has come up several times on discuss.pixls.us:
>
> https://discuss.pixls.us/search?q=%20superresolution

My apologies, I wasn't very clear. discuss.pixls.us is a discussion
forum for users and developers of free/libre software.

Possibly there is already some sort of super-resolution algorithm
implementation for one or another free-libre softwares, in which case
there is also free/libre code available. But I didn't read any of the
links to see what they are actually about.

Pat David (the person who runs the forum) might know whether there's any
actual implementation with code that might overlap with/be useful for
what Maitraya Bhattacharyya is proposing to do.

--
https://ninedegreesbelow.com
Color management and free/libre photography
_______________________________________________
gimp-developer-list mailing list
List address:    [hidden email]
List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list
List archives:   https://mail.gnome.org/archives/gimp-developer-list
Reply | Threaded
Open this post in threaded view
|

Re: AI algorithms in GIMP

Developers mailing list
Hi Elle,

I looked at the posts on discuss.pixls.us, and it seems that their approach
is quite different to what we are trying to achieve. They are mostly trying
to use multiple images or use traditional methods which don't yield very
good results.

Thanks,
Maitraya.

On Mon, Jan 21, 2019 at 9:12 PM Elle Stone <[hidden email]>
wrote:

> On 1/21/19 10:01 AM, Elle Stone wrote:
> >
> > I don't know if this is the same "super resolution", but FWIW the topic
> > has come up several times on discuss.pixls.us:
> >
> > https://discuss.pixls.us/search?q=%20superresolution
>
> My apologies, I wasn't very clear. discuss.pixls.us is a discussion
> forum for users and developers of free/libre software.
>
> Possibly there is already some sort of super-resolution algorithm
> implementation for one or another free-libre softwares, in which case
> there is also free/libre code available. But I didn't read any of the
> links to see what they are actually about.
>
> Pat David (the person who runs the forum) might know whether there's any
> actual implementation with code that might overlap with/be useful for
> what Maitraya Bhattacharyya is proposing to do.
>
> --
> https://ninedegreesbelow.com
> Color management and free/libre photography
>
_______________________________________________
gimp-developer-list mailing list
List address:    [hidden email]
List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list
List archives:   https://mail.gnome.org/archives/gimp-developer-list
Reply | Threaded
Open this post in threaded view
|

Re: AI algorithms in GIMP

Liam R. E. Quin
In reply to this post by Developers mailing list
On Mon, 2019-01-21 at 19:23 +0530, Maitraya Bhattacharyya via gimp-
developer-list wrote:
>
[...]
> Now to train these neural nets, I will have to prepare a dataset
> consisting
> of low resolution and high resolution images and divide them into
> training,
> testing and validation sets.

If it helps, i have high resolution versions of most of the images on
https://www.fromoldbooks.org/ - usually i sell them, but i can donate
some if would be of use. Most are engravings, up to 20,000 pixels on a
side in many cases.

This leads me to wonder whether we could make a “libre” image training
set.

It also leads me to wonder whether a neural net could be trained to
clean up scanned images.

Liam (slave ankh)


--
Liam Quin, https://www.delightfulcomputing.com/
Available for XML/Document/Information Architecture/XSLT/
XSL/XQuery/Web/Text Processing/A11Y training, work & consulting.
Web slave for vintage clipart http://www.fromoldbooks.org/

_______________________________________________
gimp-developer-list mailing list
List address:    [hidden email]
List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list
List archives:   https://mail.gnome.org/archives/gimp-developer-list
Reply | Threaded
Open this post in threaded view
|

Re: AI algorithms in GIMP

Liam R E Quin
In reply to this post by Developers mailing list
On Mon, 2019-01-21 at 19:23 +0530, Maitraya Bhattacharyya via gimp-
developer-list wrote:
[...]
> Now to train these neural nets, I will have to prepare a dataset
> consisting
> of low resolution and high resolution images and divide them into
> training,
> testing and validation sets.

If it helps, i have high resolution versions of most of the images on
https://www.fromoldbooks.org/ - usually i sell them, but i can donate
some if would be of use. Most are engravings, up to 20,000 pixels on a
side in many cases.

This leads me to wonder whether we could make a “libre” image training
set.

It also leads me to wonder whether a neural net could be trained to
clean up scanned images.

Liam (slave ankh)


--
Liam Quin, https://www.delightfulcomputing.com/
Available for XML/Document/Information Architecture/XSLT/
XSL/XQuery/Web/Text Processing/A11Y training, work & consulting.
Barefoot Web-slave for vintage clipart http://www.fromoldbooks.org/

_______________________________________________
gimp-developer-list mailing list
List address:    [hidden email]
List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list
List archives:   https://mail.gnome.org/archives/gimp-developer-list
Reply | Threaded
Open this post in threaded view
|

Re: AI algorithms in GIMP

Developers mailing list
In reply to this post by Developers mailing list
Hi guys,

I have started working on the super-resolution plugin after going through
some tutorials on GANs. I will try to put some kind of update on my blog in
a month or so.

Thanks,
Maitraya.

On Mon, Jan 21, 2019 at 9:17 PM Maitraya Bhattacharyya <
[hidden email]> wrote:

> Hi Elle,
>
> I looked at the posts on discuss.pixls.us, and it seems that their
> approach is quite different to what we are trying to achieve. They are
> mostly trying to use multiple images or use traditional methods which don't
> yield very good results.
>
> Thanks,
> Maitraya.
>
> On Mon, Jan 21, 2019 at 9:12 PM Elle Stone <[hidden email]>
> wrote:
>
>> On 1/21/19 10:01 AM, Elle Stone wrote:
>> >
>> > I don't know if this is the same "super resolution", but FWIW the topic
>> > has come up several times on discuss.pixls.us:
>> >
>> > https://discuss.pixls.us/search?q=%20superresolution
>>
>> My apologies, I wasn't very clear. discuss.pixls.us is a discussion
>> forum for users and developers of free/libre software.
>>
>> Possibly there is already some sort of super-resolution algorithm
>> implementation for one or another free-libre softwares, in which case
>> there is also free/libre code available. But I didn't read any of the
>> links to see what they are actually about.
>>
>> Pat David (the person who runs the forum) might know whether there's any
>> actual implementation with code that might overlap with/be useful for
>> what Maitraya Bhattacharyya is proposing to do.
>>
>> --
>> https://ninedegreesbelow.com
>> Color management and free/libre photography
>>
>
_______________________________________________
gimp-developer-list mailing list
List address:    [hidden email]
List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list
List archives:   https://mail.gnome.org/archives/gimp-developer-list
Reply | Threaded
Open this post in threaded view
|

Re: AI algorithms in GIMP

Developers mailing list
In reply to this post by Liam R. E. Quin
Hi,

[Disclaimer : 1)my knowledge of AI os limited 2) Just another Gimp user.
Not a Gimp dev]

just dropping a couple of ideas here

1) Back and white to colour:

If we train a NN on black and white images and their colour equivalents,
will it be able to generate colour photos fro, black and white?

Would it be a specialized form of style transfer?

2) Do you know of any algorithms which can separate different objects in
the image to different layers?
Probable detect and separate humans.
Another related separation is separating different parts of the face
Yet another separation could be simply based on the guessed depth of the
object

3) That brings us to :

Is it possible to use AI to get(guess) a depth map of the image? This could
be useful in making the depth of focus a bit shallower(limited capability,
of course) , place objects in-front of/behind objects, etc.. , Shift camera
perspective a little, etc.  3d scene aware object cut and paste (with
perspective change of object) anyone??!


4) Noise removal:

Is it better to do noise removal using AI?


On Wed, Mar 27, 2019, 11:41 PM Liam R. E. Quin <[hidden email] wrote:

> On Mon, 2019-01-21 at 19:23 +0530, Maitraya Bhattacharyya via gimp-
> developer-list wrote:
> >
> [...]
> > Now to train these neural nets, I will have to prepare a dataset
> > consisting
> > of low resolution and high resolution images and divide them into
> > training,
> > testing and validation sets.
>
> If it helps, i have high resolution versions of most of the images on
> https://www.fromoldbooks.org/ - usually i sell them, but i can donate
> some if would be of use. Most are engravings, up to 20,000 pixels on a
> side in many cases.
>
> This leads me to wonder whether we could make a “libre” image training
> set.
>
> It also leads me to wonder whether a neural net could be trained to
> clean up scanned images.
>
> Liam (slave ankh)
>
>
> --
> Liam Quin, https://www.delightfulcomputing.com/
> Available for XML/Document/Information Architecture/XSLT/
> XSL/XQuery/Web/Text Processing/A11Y training, work & consulting.
> Web slave for vintage clipart http://www.fromoldbooks.org/
>
> _______________________________________________
> gimp-developer-list mailing list
> List address:    [hidden email]
> List membership:
> https://mail.gnome.org/mailman/listinfo/gimp-developer-list
> List archives:   https://mail.gnome.org/archives/gimp-developer-list
>
_______________________________________________
gimp-developer-list mailing list
List address:    [hidden email]
List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list
List archives:   https://mail.gnome.org/archives/gimp-developer-list
Reply | Threaded
Open this post in threaded view
|

Re: AI algorithms in GIMP

Liam R E Quin
On Sat, 2019-03-30 at 07:40 +0530, Laxminarayan Kamath via gimp-
developer-list wrote:
> just dropping a couple of ideas here

[...]

Something like waifu2x would be fabulous to have in GIMP (a neural
network-based image upscaling algorithm).

slave liam (ankh on IRC)


--
Liam Quin - web slave for https://www.fromoldbooks.org/
with fabulous vintage art and fascinating texts to read.
Click here to have the slave rewarded with cold gruel.

_______________________________________________
gimp-developer-list mailing list
List address:    [hidden email]
List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list
List archives:   https://mail.gnome.org/archives/gimp-developer-list
Reply | Threaded
Open this post in threaded view
|

Re: AI algorithms in GIMP

Developers mailing list
All,

When people say "AI" here do they mean Neural Networks?

"Intelligent" algorithms have been implemented in GIMP for many years.
About 15 years ago, this algorithm got integrated into GIMP:
http://www.siox.org

Now, the reason I bring this is up is that any machine learning algorithm
needs extensive empirical testing and we setup frameworks for
that during the integration of the above algorithm. So whatever machine
learning you want to integrate in GIMP needs to come with an
independent benchmark dataset that is annotated for what you want to
achieve. This benchmark dataset is used AFTER you think you
are ready with building your algorithm.
This is, after you have trained and tested your machine learner (using yet
completely different datasets) and minimized parameters
for generalization, the benchmark dataset does not only further test
accuracy and generalization but also computational efficiency
and user experience. Question like how can the user correct errors of the
AI need to be answered too.

On a further note, if you did this benchmark testing, you would find that
waifu2x is scam. This is, it only works with a very specialized
set of images. The first thing to investigate would be what set of images
that is and how to explain that to a GIMP user. In general, images
cannot be upscaled. The data processing inequality mathematically proofs
that information cannot be created by processing an image.
So, while many TV crime shows suggest this, algorithmically zooming into a
blurry license plate or face to make it recognizable is physically
and mathematically impossible. An independent benchmark set would show the
limits of this algorithm.

Just my two cents,
Gerald
--
http://www.gerald-friedland.org


On Fri, Mar 29, 2019 at 8:58 PM Liam R E Quin <[hidden email]> wrote:

> On Sat, 2019-03-30 at 07:40 +0530, Laxminarayan Kamath via gimp-
> developer-list wrote:
> > just dropping a couple of ideas here
>
> [...]
>
> Something like waifu2x would be fabulous to have in GIMP (a neural
> network-based image upscaling algorithm).
>
> slave liam (ankh on IRC)
>
>
> --
> Liam Quin - web slave for https://www.fromoldbooks.org/
> with fabulous vintage art and fascinating texts to read.
> Click here to have the slave rewarded with cold gruel.
>
> _______________________________________________
> gimp-developer-list mailing list
> List address:    [hidden email]
> List membership:
> https://mail.gnome.org/mailman/listinfo/gimp-developer-list
> List archives:   https://mail.gnome.org/archives/gimp-developer-list
>
_______________________________________________
gimp-developer-list mailing list
List address:    [hidden email]
List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list
List archives:   https://mail.gnome.org/archives/gimp-developer-list
Reply | Threaded
Open this post in threaded view
|

Re: AI algorithms in GIMP

Developers mailing list
Hi people,

1. Laxminarayan has some very good suggestions. I think all of them are
possible with varying degree of success.
2. Liam, I am implementing a quite popular paper for image upscaling called
SRGAN.
3. Gerald, there would be a testing and validation datasets to gauge the
accuracy of the neural network. The goal would be to minimize 'overfitting'
of the neural network to the training dataset. You're going to lose quality
when you upscale, but the upscaled images will still be better than ones
produced using traditional algorithms.

Sorry I haven't been able to get back to you guys before, I am caught up in
work and will not be able to write code properly till something important
gets over. However, I have studied generative adversarial networks (GANs)
and am now learning a python ML library called PyTorch. Once this is done,
I'll be able to start writing code. My main problem here seems to be the
computational infraastructure, I fear that my 8 GB Nvidia Quadro M4000 will
not be good enough to train on the data. If anybody had better resources
they can share, please let me know.

Once we have the toy GAN code set up in Python, people can play around with
it for various types of projects.

Cheerio,
Maitraya

On Sat, Mar 30, 2019 at 8:44 PM Gerald Friedland via gimp-developer-list <
[hidden email]> wrote:

> All,
>
> When people say "AI" here do they mean Neural Networks?
>
> "Intelligent" algorithms have been implemented in GIMP for many years.
> About 15 years ago, this algorithm got integrated into GIMP:
> http://www.siox.org
>
> Now, the reason I bring this is up is that any machine learning algorithm
> needs extensive empirical testing and we setup frameworks for
> that during the integration of the above algorithm. So whatever machine
> learning you want to integrate in GIMP needs to come with an
> independent benchmark dataset that is annotated for what you want to
> achieve. This benchmark dataset is used AFTER you think you
> are ready with building your algorithm.
> This is, after you have trained and tested your machine learner (using yet
> completely different datasets) and minimized parameters
> for generalization, the benchmark dataset does not only further test
> accuracy and generalization but also computational efficiency
> and user experience. Question like how can the user correct errors of the
> AI need to be answered too.
>
> On a further note, if you did this benchmark testing, you would find that
> waifu2x is scam. This is, it only works with a very specialized
> set of images. The first thing to investigate would be what set of images
> that is and how to explain that to a GIMP user. In general, images
> cannot be upscaled. The data processing inequality mathematically proofs
> that information cannot be created by processing an image.
> So, while many TV crime shows suggest this, algorithmically zooming into a
> blurry license plate or face to make it recognizable is physically
> and mathematically impossible. An independent benchmark set would show the
> limits of this algorithm.
>
> Just my two cents,
> Gerald
> --
> http://www.gerald-friedland.org
>
>
> On Fri, Mar 29, 2019 at 8:58 PM Liam R E Quin <[hidden email]> wrote:
>
> > On Sat, 2019-03-30 at 07:40 +0530, Laxminarayan Kamath via gimp-
> > developer-list wrote:
> > > just dropping a couple of ideas here
> >
> > [...]
> >
> > Something like waifu2x would be fabulous to have in GIMP (a neural
> > network-based image upscaling algorithm).
> >
> > slave liam (ankh on IRC)
> >
> >
> > --
> > Liam Quin - web slave for https://www.fromoldbooks.org/
> > with fabulous vintage art and fascinating texts to read.
> > Click here to have the slave rewarded with cold gruel.
> >
> > _______________________________________________
> > gimp-developer-list mailing list
> > List address:    [hidden email]
> > List membership:
> > https://mail.gnome.org/mailman/listinfo/gimp-developer-list
> > List archives:   https://mail.gnome.org/archives/gimp-developer-list
> >
> _______________________________________________
> gimp-developer-list mailing list
> List address:    [hidden email]
> List membership:
> https://mail.gnome.org/mailman/listinfo/gimp-developer-list
> List archives:   https://mail.gnome.org/archives/gimp-developer-list
>
_______________________________________________
gimp-developer-list mailing list
List address:    [hidden email]
List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list
List archives:   https://mail.gnome.org/archives/gimp-developer-list