Kafka consumers

Create a new Kafka topic

1. Add the topic to the KAFKA_TOPIC_TO_CLUSTER in src/sentry/conf/server.py:

• e.g. subscription-results-eap-items

2. Add the topic to Topic in src/sentry/conf/types/kafka_definition.py

Define or re-use a processing strategy

In most cases a Streaming Factory is what you want to when defining a consumer (see next section). You can find examples of it in Sentry's code base.

Define a new Kafka consumer

1. Add a new entry in the KAFKA_CONSUMERS key in src/sentry/consumers/init.py:

KAFKA_CONSUMERS = {
    "<your_topic_str_here>": {
        "topic": Topic.YOUR_TOPIC,
        "strategy_factory": "sentry_package_defining_your_strategy_factory_class",
    }
}

2. You may need optional properties (e.g. click_options, you will need to research them by looking at ConsumerDefinition's code.

3. Make sure you can run it: sentry run consumer <your_topic>

4. You may need to add some devserver options here.

5. Add tests for your consumer

Run the consumer in production

1. Discuss it with SRE during their office hours
2. Create ops PR for adding your consumer to k8s/services/getsentry/_consumer-deployment.yaml.j2.
3. In the Sentry Consumers metrics dashboard, add a new saved view for your consumer.