Goal¶

The goal of this tutorial is to build a search app using Django and Haystack You will learn how to use Django commands to initialize a database with emoji data. You will also learn how to add search to a Django project using Haystack.

Upon completion, you will have a built an app that allows you to search for over a thousand emojis. This app also gives you the ability to copy any emoji to your clipboard with one click.

Before you start¶

Make sure you meet the following prerequisites before starting the tutorial steps:

Pipenv - version 2018.11.26 (run pipenv --version to confirm)
Elasticsearch - version 5.5.3

This project depends on Pipenv. Pipenv allows you to download and install versions of packages in a virtual environment.

Another prerequisite is Elasticsearch. An Elasticsearch instance needs to run separate from the app.

Installing packages¶

The app depends on the following packages:

Django - to build the website
Haystack - to add search capability
Elasticsearch client - low-level client required by Haystack (different than the Elasticsearch instance)
Requests - to retrieve emoji data

Open up a terminal prompt and create a directory called emoji-in-the-haystack:

mkdir emoji-in-the-haystack
cd emoji-in-the-haystack

Install the packages:

pipenv install django==3.0.7
pipenv install git+https://github.com/django-haystack/django-haystack.git#egg=django-haystack
pipenv install elasticsearch==5.5.3
pipenv install requests==2.24.0

You’ll see a bunch of colorful output and a couple of 🐍 emojis. In this directory, you should now see the files Pipfile and Pipfile.lock.

You’re ready to create a Django project.

Setting up a Django project and app¶

After installing the packages, the next step is to create a Django project.

Activate your virtual environment:

pipenv shell

You should now see your terminal prompt prefixed with (emoji-in-the-haystack).

Create a Django project called emoji_haystack:

django-admin startproject emoji_haystack .

The directory should now look like this:

├── Pipfile
├── Pipfile.lock
├── manage.py
└── emoji_haystack
   ├── __init__.py
   ├── asgi.py
   ├── settings.py
   ├── urls.py
   └── wsgi.py

Create a Django app called search:

python manage.py startapp search

The directory should now look like this:

├── Pipfile
├── Pipfile.lock
├── manage.py
├── emoji_haystack
│   ├── __init__.py
│   ├── asgi.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
└── search
   ├── __init__.py
   ├── admin.py
   ├── apps.py
   ├── migrations
   │   └── __init__.py
   ├── models.py
   ├── tests.py
   └── views.py

You need to enable the newly created app.

Update the INSTALLED_APPS setting in settings.py:

INSTALLED_APPS = [
   'django.contrib.admin',
   'django.contrib.auth',
   'django.contrib.contenttypes',
   'django.contrib.sessions',
   'django.contrib.messages',
   'django.contrib.staticfiles',

   'search.apps.SearchConfig',
]

To test that everything is working, run the app:

python manage.py runserver

Navigate to http://127.0.0.1:8000/ and confirm that the app is working.

Note: You can run python manage.py migrate to get rid of the Django warnings when running the app.

Emoji data¶

The next step is to create a Django model class to represent the emoji data.

Update models.py:

from django.db import models


class Emoji(models.Model):
    name = models.CharField(
        max_length=50,
    )
    code = models.CharField(
        max_length=50,
    )

You need to store the name for each emoji. For example, “grimacing face” is the name given to 😬. You also need to store the code for an emoji. These code points are unique for every emoji. Django handles rendering emojis in the browser using these codes.

After creating the model, run a migration to apply these changes to the database:

python manage.py makemigrations --name add_emoji_model search
python manage.py migrate

The next step is to create a new directory for the Django command. Django commands are special scripts registered in Django projects.

The command in this app retrieves emoji data and saves it to the database using the Emoji model class. This commands must live in the new directory.

Create the new directory:

cd search
mkdir management
cd management
mkdir commands
cd commands

Inside this commands directory, create the initemojidata command:

touch initemojidata.py

The directory should now look like this:

├── Pipfile
├── Pipfile.lock
├── db.sqlite3
├── emoji_haystack
│   ├── __init__.py
│   ├── asgi.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── manage.py
└── search
   ├── __init__.py
   ├── admin.py
   ├── apps.py
   ├── management
   │   └── commands
   │       └── initemojidata.py
   ├── migrations
   │   ├── 0001_add_emoji_model.py
   │   └── __init__.py
   ├── models.py
   ├── tests.py
   └── views.py

Here is the code to retrieve and save emoji data:

import json
import requests

from django.core.management.base import BaseCommand, CommandError

from search.models import Emoji


EMOJI_JSON_URL = 'https://raw.githubusercontent.com/iamcal/emoji-data/master/emoji.json'


class Command(BaseCommand):
    help = 'Initialize database with emoji data'

    def add_arguments(self, parser):
        parser.add_argument(
            '--dry-run',
            action='store_true',
            default=False)

    def execute(self, *args, **options):
        self.count = 0

        try:
            super().execute(*args, **options)
        except KeyboardInterrupt:
            self.stdout.write('')

        self.stdout.write(self.style.SUCCESS(
            'Emojis created: {}'.format(self.count)))

    def handle(self, *args, **options):
        self.dry_run = options['dry_run']

        emojis = self.get_emojis()

        for emoji in emojis:
            if not emoji.get('name'):
                continue

            code = self.handle_code(emoji)
            name = emoji['name'].lower()
            self.stdout.write(
                '{} - {}'.format(name, code))

            if not self.dry_run:
                emoji = Emoji(
                    name=name,
                    code=code)

                emoji.save()

            self.count += 1

    def get_emojis(self):
        response = requests.get(
            url=EMOJI_JSON_URL)

        emojis = json.loads(response.content)

        return emojis

    def handle_code(self, emoji):
        """
        U+1F1EC, U+1F1FE - > &#x1F1EC&#x1F1FE
        """
        unified = emoji.get('non_qualified') or emoji.get('unified')
        unified = unified.split('-')

        codes = []
        for code in unified:
            _code = '&#x' + code
            codes.append(_code)

        return ''.join(codes)

The syntax for Django commands may take some time getting used to. Django commands require a Command class definition that subclasses BaseCommand. This class requires a handle() method. Your logic goes in here.

I use the execute() method to define some variables to count and output the number of items updated when a command finishes running.

On line 35, the get_emojis() method defined on the class gets called using the self property. The method makes a request to the URL defined on line 9. This endpoint is a JSON file hosted on GitHub.

It may not include the newest emojis but it’s the best option for this app. The Emojipedia API is no longer available for public use. Typically you need to handle errors when making API requests but it’s fine to leave out here.

The command retrieves the emoji data and begins to process each data item on line 37. It ignores data items with no name field. On line 41, the command calls the handle_code(). This method transforms the emoji unicode data into a string that gets stored in the database. The transformation of this unicode data makes it possible to render emojis in HTML. More on this later.

You can run this command with an optional dry_run argument. Providing this argument means you can test your Django command logic without saving anything to the database. If this argument is not passed in when running the command, the command creates an Emoji object with name and code set and saves it to the database.

Django commands are ran from the root of the project.

Run the Django command (--dry-run option):

python manage.py initemojidata --dry-run

Run the Django command (no regrets option):

python manage.py initemojidata

The emoji data is now stored in the database.

Haystack setup¶

Haystack makes it easy to add custom search to Django apps. You write your search code once and can go back and forth between search backends as you please. You can choose to use different search backends like Elasticsearch, Solr, and others. This tutorial uses Elasticsearch.

Integrating Haystack consists of creating a search index model and updating a couple of Django settings.

The search index model corresponds to the database model defined earlier. Haystack requires this file to know what data to place in the search index.

Inside the search app directory, create a search_indexes.py file:

cd search
touch search_indexes.py

Here’s what the code for that looks like:

import datetime

from haystack import indexes
from search.models import Emoji


class EmojiIndex(indexes.SearchIndex, indexes.Indexable):
    text = indexes.CharField(document=True, use_template=True)

    def get_model(self):
        return Emoji

When you make search a query, Haystack searches the text field. This field corresponds to the name field defined in the Emoji model.

Next, include the urls provided by Haystack in urls.py. Django implicitly calls a custom Haystack view that handles search requests and returning responses. This response uses an HTML template that you need to create and configure. More on this later.

from django.contrib import admin
from django.urls import include, path

urlpatterns = [
    path('admin/', admin.site.urls),
    path('search/', include('haystack.urls')),
]

You need to enable the Haystack app.

Update the INSTALLED_APPS setting in settings.py:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    'search.apps.SearchConfig',

    'haystack',
]

Add a connection to Elasticsearch in settings.py:

# Haystack configuration
# https://haystacksearch.org

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.elasticsearch5_backend.Elasticsearch5SearchEngine',
        'URL': 'http://127.0.0.1:9200/',
        'INDEX_NAME': 'haystack',
    },
}

Haystack setup continued¶

The following steps are cumbersome but they are essential in getting Haystack to work.

In settings.py, update the TEMPLATES setting:

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [os.path.join(BASE_DIR, 'templates')],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

From the root of the project, create a templates directory:

mkdir templates
cd templates

Creating a single project-level templates directory is a recognized Django pattern.

In the templates directory, create a search directory and a file called search.html:

mkdir search
cd search
touch search.html

In the search directory, create an indexes directory:

mkdir indexes
cd indexes

In the indexes directory, create a search directory and a file called emoji_text.txt:

mkdir search
cd search
touch emoji_text.txt

Here’s what emoji_text.txt should look like:

{{ object.name }}

Haystack uses this data template to build the document used by the search engine.

The final directory structure should look like this:

├── Pipfile
├── Pipfile.lock
├── db.sqlite3
├── emoji_haystack
│   ├── __init__.py
│   ├── asgi.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── manage.py
├── search
│   ├── __init__.py
│   ├── admin.py
│   ├── apps.py
│   ├── management
│   │   └── commands
│   │       └── initemojidata.py
│   ├── migrations
│   │   ├── 0001_add_emoji_model.py
│   │   └── __init__.py
│   ├── models.py
│   ├── search_indexes.py
│   ├── tests.py
│   └── views.py
└── templates
   └── search
      ├── indexes
      │   └── search
      │       └── emoji_text.txt
      └── search.html

Search template¶

Now it’s time to update search.html. This template contains a text field to type in a search query, a button that fires a search request and some template variables. Use the template example found here.

Note: Remove {% extends 'base.html' %} at the top of the file.

The main differences in the template for this tutorial are the following two lines:

{% for result in page.object_list %}
   <p>{{ result.object.code|safe }}</p>
   <p>{{ result.object.name }}</p>
{% empty %}

object_list is a list of search results. For each search result, display the emoji and its name. result.object provides direct access to the Emoji model and its database fields.

Displaying the emoji requires using the safe Django filter. It does not require further HTML escaping.

Running Elasticsearch¶

Navigate to the location of your Elasticsearch installation and start an instance. For example, say you downloaded Elasticsearch in your Downloads folder:

cd Downloads
cd elasticsearch-5.5.3
cd bin
elasticsearch

Haystack ships with a set of Django commands that handle indexing the emoji data stored in the database. This tutorial uses the rebuild_index command. This command rebuilds the search index by first clearing it and then updating it. Have a look at the source code for more info.

From the root of the project, run the command:

python manage.py rebuild_index

Run the app:

python manage.py runserver

Navigate to http://127.0.0.1:8000/search and confirm that the app is working.

If you query for “cat,” you get back a list of results. If you query for “flag,” you get back results for flag emojis.

If you scroll to the bottom, you’ll see a Previous button and Next button. Haystack returns at most 20 results per page. This out of the box feature is awesome. The layout needs a little bit of work though.

Bootstrap + clipboard.js¶

You can use Bootstrap to clean up the design. Another feature is to copy an emoji to your clipboard by clicking on it - clipboard.js can help here.

Load Bootstrap and clipboard.js from CDN in search.html:

<script src="https://cdn.jsdelivr.net/npm/clipboard@2/dist/clipboard.min.js"></script>

<!-- Bootstrap CSS -->
<link rel="stylesheet"
href="https://stackpath.bootstrapcdn.com/bootstrap/4.1.3/css/bootstrap.min.css"
integrity="sha384-MCw98/SFnGE8fJT3GXwEOngsV7Zt27NXFoaoApmYm81iuXoPkFOJwJ8ERdknLPMO"
crossorigin="anonymous">

{% block content %}

A couple of Bootstrap <div> elements and some styling updates go a long way in improving the look of the app.

Including the data-clipboard-text attribute on the emoji button lets you copy emojis to your clipboard:

        {% if query %}
            <h3>Results</h3>

            <div class="container">
            <div class="row">
            {% for result in page.object_list %}
                <div class="col-sm">
                    <button type="button" class="btn" data-clipboard-text="{{ result.object.code|safe }}" style="font-size:90px;">{{ result.object.code|safe }}</button>
                    <p style="text-align: center">{{ result.object.name }}</p>
                </div>
            {% empty %}
                <p>No results found.</p>
            {% endfor %}
            </div>
            </div>

The last thing to do is to initialize clipboard.js in search.html:

{% endblock %}

<script>
    var clipboard = new ClipboardJS('.btn');

    clipboard.on('success', function(e) {
        console.log(e);
    });

    clipboard.on('error', function(e) {
        console.log(e);
    });
</script>

Run the app with these new changes:

python manage.py runserver

Navigate to http://127.0.0.1:8000/search and confirm the changes. This looks much better. The emojis are more prominent and the click-to-copy feature is the 🍒 on top.

What you’ve learned¶

Rejoice and show your friends how to find the emoji in the haystack. If you’re up for the challenge, see if you can make the following app improvements:

Load a subset of emojis on the homepage before a user searches
Add a navigation bar to filter by emoji category
Support for newer emojis

Santos Solorzano

Where do I submit my Djangonaut application?

Tag elasticsearch

Building a search app with Django and Haystack

Goal¶

Before you start¶

Installing packages¶

Setting up a Django project and app¶

Emoji data¶

Haystack setup¶

Haystack setup continued¶

Search template¶

Running Elasticsearch¶

Bootstrap + clipboard.js¶

What you’ve learned¶