Have you ever thought that your images don't have enough emoji in them? Well let me tell you, those dark days are over. My latest hackathon project, Emojitizer, takes an image and turns it into a bunch of emojis. It can render as a new image, a webpage, and a list of slack-paste-able shortcodes.

The first step in this project is to get all the emoji into one place. At Carbon Black we have added an inordinate number of custom emojis to our slack workspace (1292 at the time of this writing... up 24 during the hackathon week!). There's a convenient slack API method for listing all emoji. The only complication is that slack lets users create aliases for emoji, so :+1: and :thumbs_up: can both map to the same image. For now, the aliases are skipped so we don't download the same image multiple times. In the future we could use the shortest emoji alias as the real name to help pack more emoji into one slack message.

client = SlackClient(os.environ.get('SLACK_TOKEN'))
emoji = client.api_call("emoji.list")

for name in emoji['emoji']:
    url = emoji['emoji'][name]
    if url.startswith("alias"):
        # skip the emoji aliases
        continue
    else:
        download_file(url, name)

With the emoji ready to use, we need some sort of emoji analysis engine. The Pillow library is incredible for working with images in python. There's even an ImageStat module that can do basically all of the analysis! These stats let us boil an emoji down into a single "color coordinate", represented as (red, green, blue). This coordinate is used when mapping parts of  an input image to a corresponding emoji. Each part of the input image is run through the same stats function, returning the color coordinate of  that part. Thinking about these color coordinates in a three-dimensional space, the colors that are physically the closest are the most similar. This means if we find the emoji with the closest coordinate, we can use it to represent that chunk of the image.

Each tuple can be thought of as a point in a 3d coordinate system

In order to actually figure out which emoji is closest to the desired color, I needed a data structure that allows for efficient queries of a 3D space. Sure, I could have written a for loop to check all 2581 emoji each time to find the closest match, but I knew there must be a better way. After some googling I learned about spatial indexing and R-trees.  An R-tree allows us to ask the question "What are the closest objects to a given coordinate?" without needing to look at each object in the system. Fortunately, someone already wrote an r-tree library in python. Using this library I was able to build a database of emojis mapped by their color.

from PIL import Image, ImageStat
from rtree import index

properties = index.Property()
properties.dimension = 3  # Our index needs 3 dimensions: red, green, blue
idx = index.Index('emoji', properties=properties)

# Set up the bounds of the index
rmin = gmin = bmin = 0
rmax = gmax = bmax = 255
idx.insert(0, (rmin, gmin, bmin, rmax, gmax, bmax))

def analyze_image(emoji, name):
    im = Image.open(emoji)
    if emoji.endswith(".gif") and im.is_animated:
        return  # skip animated emoji

    rgb = im.convert("RGB")
    stats = ImageStat.Stat(rgb)
    name = name.split(".")[0]

    record = {
        "path": emoji,
        "name": name,
        "median": stats.median,
        "mean": stats.mean,
        "index": len(data) + 1
    }
    
    # Add the emoji data into the spatal index
    # Insert takes in an area (rmin, gmin, bmin, rmax, gmax, bmax), there's no shortcut for adding a single point
    # doubling the stat function result with `median + median` gives us a point-sized area.
    idx.insert(len(data) + 1, stats.median + stats.median, obj=record)

With the main pieces ready we just need a little bit of glue to hold things together. This snippet loops over the image and uses the same techniques from above to analyze the image. It then reads from the spatial index to find the closest point and adds its :shortcode: to the output.

for y in range(im.height // chunk_size):
        for x in range(im.width // chunk_size):
            piece = im.crop(box=(x * chunk_size, y * chunk_size, (x + 1) * chunk_size, (y + 1) * chunk_size))
            stats = ImageStat.Stat(piece)
            coords = stats.median + stats.median
            nearest = idx.nearest(coords, num_results=3, objects=True)
            
            emoji = random.choice(list(filter(lambda x: x.object is not None, nearest))).object
            slack += ":{}:".format(emoji['name'])
        slack += "\n"

The full conversion script came together quickly with... mediocre results. Can you guess what this image was supposed to be?

Spoliers (don't worry, I can't tell either 🙂)

I tried increasing the size of the image, but that introduced its own problems... Turns out the max message length for slack is 4000 characters, which you blow through pretty quickly with emojis like :blob_slightly_smiling_face:. This one turned out a bit better, but obviously still not great.

A 🙂, obviously

The best image from this version of the converter was definitely the classic Windows XP Bliss background

Going back to the image analysis part, I found my first problem: using the median is not as effective of an analysis method as mean. Switching from median to mean yielded a huge quality improvement. I also made it easier to paste the emoji into slack by splitting the output into 4k character chunks. This brought me closer to the dream of rendering full size images directly in slack!

This looked way better than the previous images, but it just didn't work on a lot of image types. The taco sample, for example, had really weird artifacts in the semi-transparent shadow in the image.

Eww. Gross.

To rectify this we need to remove the transparency from the input images. I also added a render-as-html function, since copy-pasting into slack to test everything was becoming too cumbersome. This also allows for much larger images to be rendered.

from PIL import Image


def rgbtize(image):
    """ Convert an image to the RGB color space, removing any transparency"""
    if len(image.split()) == 4:
        return pure_pil_alpha_to_color(image)
    else:
        return image.convert("RGB")


def pure_pil_alpha_to_color(image, color=(255, 255, 255)):
    """Alpha composite an RGBA Image with a specified color.
    Simpler, faster version than the solutions above.
    Source: http://stackoverflow.com/a/9459208/284318
    Keyword Arguments:
    image -- PIL RGBA Image object
    color -- Tuple r, g, b (default 255, 255, 255)
    """
    image.load()  # needed for split()
    background = Image.new('RGB', image.size, color)
    background.paste(image, mask=image.split()[3])  # 3 is the alpha channel
    return background
We're really getting there!

Adding more emoji also (unsurprisingly) helps the final render quality. In the previous image, each emoji is 32x32 px. Here it is at 16x16 px:

It looks better, but there is small still a transparency issue. This time it's in the emoji selection itself. There are a lot of lighter sections in the output image because emojis with transparent backgrounds are being inappropriately selected. Tweaking the analysis script to use the same transparency removal gives us this:

The colors suddenly became much more vibrant and well defined. At this point the output looked solid. It was still far from finished though – a manual screenshot of a static webpage is not feasible. It was finally time to bite the bullet and write the image render code.

out_img = Image.new("RGB", ((im.width // chunk_size) * display_size, (im.height // chunk_size) * display_size), "white")

# [...]

emoji_im = rgbtize(Image.open(emoji['path'])).resize((display_size, display_size))
out_img.paste(emoji_im, (x * display_size, y * display_size, (x + 1) * display_size, (y + 1) * display_size))

To help get even greater accuracy I tried supplementing the emoji collection with solid colored squares from an online color palette. It didn't make too much of a difference, but I think it looks nice, especially in the flower image in the header. Without it the stamen don't look that great.

```python
from PIL import Image

colors = ["#060608", "#141013", ...]

for color in colors:
    img = Image.new('RGB', (64, 64), color=color)
    img.save("custom_emoji/new/c_{}.png".format(color.replace("#", "")))
```
Those brownish spots in the center are a :go: board emoji

The full source code, in all its rushed-hackathon-style glory, is on github.

The hackathon submission video