nostr_core

Learn about the nostr protocol and how we call it from python with python-nostr

This repository relies heavily on python-nostr for underlying functionality. This notebook will walk you through the key characteristics of the nostr protocol and also make some changes to the classes from python-nostr that will be used in the basic client implementation.

What is nostr?

This notebook will attempt to describe nostr via python code. There are many resources dedicated to describing nostr in words. See:

the readme on the nostr protocol github
the Nostr Improvement Proposal (NIPs) nostr protocol github has detailed descriptions of technical implementations and is very well-documented.
usenostr.org for a basic and approachable description
awesome-nostr for a complete list of tools and resources.

In short, nostr is a communication protocol that provides a relatively simple framework for events and metadata to be published to servers (relays) after being cryptographically signed using a private key. In orther words, the data published to the relays is visible by anyone, but provably created by the person who holds the private key associated with the event. A technical description of how the events are created can be found here and a current list of the different event types here.

Because of the simplicity and openness of the nostr protocol a long list of tools and clients with different purposes have already been built on the nostr protocol as can be seen on awesome-nostr. There will continue to be more as new people are exposed to the protocol and the accepted event types continue to expand.

Fundamentally, nostr relays and clients are built on a relatively simple set of tools - most of which can be described nicely by looking at the modules of python-nostr, which is what we’ll do here. Specifically, covered topics in this notebook can be accessed quickly via the sidebar and include:

PrivateKey and PublicKey
Event
Relay, RelayManager, and the MessagePool
Subscription

Private and Public Keys

In nostr, the private key is the root of “account” ownership. The public key is derived from a private key using the same eliptic curve cryptography that is used for bitcoint private and public key pairs. nostr events are then signed using Schnorr Signatures, which prove “ownership” of the event. We will cover events in more detail later.

In this documentation you will see two versions of the PublicKey and PrivateKey classes, one version loaded directly from the python-nostr.key module as key.Class and one with added hex and bech32 functionality as denoted without key. in front.

First, let’s look at the PrivateKey classes

By default the class is instantiated with raw bytes as shown below

PrivateKey

 PrivateKey (raw_secret:bytes=None)

Initialize self. See help(type(self)) for accurate signature.

Instantiation with no argument gives a new random private key

private_key = PrivateKey()
print(private_key)

PrivateKey(npub1j8l27...zthqnwmn00)

more commonly you will likely instantiate the private key from a hex string

source

PrivateKey.from_hex

 PrivateKey.from_hex (hex:str)

private_key_hex = private_key.hex()
the_same_private_key = PrivateKey.from_hex(private_key_hex)
assert private_key.hex() == the_same_private_key.hex()

or after the introduction of bech32 entities in NIP-19 (private keys denoted as nsec...) like so

PrivateKey.from_nsec

 PrivateKey.from_nsec (nsec:str)

Load a PrivateKey from its bech32/nsec form

private_key_bech32 = private_key.bech32()
the_same_private_key = PrivateKey.from_nsec(private_key_bech32)
assert private_key.bech32() == the_same_private_key.bech32()

The PublicKey class from python-nostr has very similar characteristics to the private key class, but with the key difference that the public key can be generated from the private key and the reverse is not possible by design. Below are the docs for instantiating a PublicKey object in python-nostr

PublicKey

 PublicKey (raw_bytes:bytes)

Initialize self. See help(type(self)) for accurate signature.

Unlike the PrivateKey class, the PublicKey class requires an input to be created. A random public key would mean that no one knows the associated private key and so that is not allowed. Instead, we will get our public key from the private key we already generated

public_key = private_key.public_key
print(public_key)
assert isinstance(public_key, PublicKey)

PublicKey(npub1j8l27...zthqnwmn00)

The public can can also be retrieved from a public hex

source

PublicKey.from_hex

 PublicKey.from_hex (hex:str)

public_key_hex = public_key.hex()
the_same_public_key = PublicKey.from_hex(public_key_hex)
assert public_key.hex() == the_same_public_key.hex()

or an npub bech32 encoded public key

source

PublicKey.from_npub

 PublicKey.from_npub (npub:str)

Load a PublicKey from its bech32/npub form

public_key_bech32 = public_key.bech32()
the_same_public_key = PublicKey.from_npub(public_key_bech32)
assert public_key.bech32() == the_same_public_key.bech32()

Nostr Events

Events are the key substance through which user communication happens over Nostr. Right now, the most common event type is a text_note, but there are many other event kinds that can be published to a nostr relay. Events are generally made up of the following:

id - the hash of the event content - if the content of the a created event changes on a relay the old id would no longer match the hash of the content
pubkey - the public key of the author of the event
created_at - a time stamp given to the event at time of creation
kind - integer labeling the kind of event
tags - a list of arrays with tags such as e for related events or p for related people (like mentions)
content - the content of an event - the format of the content will vary depending on the kind of event. For a text_note (kind 1) this would be a string
signature - a signature generated from the private key associated with the public key and the event ID field - a valid signature ensures that anyone can check that a specific event was in fact signed by the private key of the author and that the content has not changed from the time the event was signed.

Note: the above all applies specifically to event creation. An event can be republished across many relays by any party as long as the event data has not changed. Some relays have limits on the age of an event, but where the event lives and who put it there is completely irrelevant to the validity of an event. This has many interesting implications, some of which will be explored elsewhere in this module.

In python-nostr we create events like so:

from nostr.event import Event, EventKind
import time
import pprint
from fastcore.test import test_fail

Event

 Event (public_key:str, content:str, created_at:int=1674361640,
        kind:int=<EventKind.TEXT_NOTE: 1>, tags:list[list[str]]=[],
        id:str=None, signature:str=None)

Initialize self. See help(type(self)) for accurate signature.

As example, we can create a new event

event = Event(
    public_key=public_key.hex(),
    content='this is a test',
    kind=EventKind.TEXT_NOTE,
    created_at=int(time.time(),)
)
pprint.pprint(event.to_json_object())

{'content': 'this is a test',
 'created_at': 1674322515,
 'id': '589ad714c2a36929abd4a817a2177c594d31e5c373948af9c22890f5590c17f6',
 'kind': <EventKind.TEXT_NOTE: 1>,
 'pubkey': '91feaf1b03c1dc0ed61163f77b4cf4d1821cce392983d4c68d82207a773f12ee',
 'sig': None,
 'tags': []}

The id hash is automatically generated from the event content, but the signature is blank because the event has not been signed with a private key. Below we can test signing an event with our real private key and also a random private key. We see that one event verifies that the public key, signature, and event id all match, while the event signed with the wrong private key does not verify.

event.sign(PrivateKey().hex())
assert not event.verify()
event.sign(private_key.hex())
assert event.verify()
pprint.pprint(event.to_json_object())

{'content': 'this is a test',
 'created_at': 1674322515,
 'id': '589ad714c2a36929abd4a817a2177c594d31e5c373948af9c22890f5590c17f6',
 'kind': <EventKind.TEXT_NOTE: 1>,
 'pubkey': '91feaf1b03c1dc0ed61163f77b4cf4d1821cce392983d4c68d82207a773f12ee',
 'sig': '23bfb21c8f9594ca068e52d7d752af05041fc710dcf3cf9c63064890c1687b8271cd0d36c874fed13dc5caa849b5cfae17c9969d182b0f36612f6585eb7e3ef3',
 'tags': []}

Additionally, if we change the content and don’t recompute both the ID and the signature then the event will not verify.

event.created_at = time.time()
event.id = event.compute_id(public_key=event.public_key, created_at=event.created_at, kind=event.kind, tags=event.tags, content=event.content)
assert not event.verify()

event.created_at = time.time()
event.sign(private_key.hex())
assert not event.verify()

As long as we do both of these things the event verifies event with changed data.

event.created_at = time.time()
event.id = event.compute_id(public_key=event.public_key, created_at=event.created_at, kind=event.kind, tags=event.tags, content=event.content)
event.sign(private_key.hex())
assert event.verify()

python-nostr also supports other event types. The basic client module in this library can help build the content and tags of other event types in the correct format.

for kind in EventKind:
    display(kind)

<EventKind.SET_METADATA: 0>

<EventKind.TEXT_NOTE: 1>

<EventKind.RECOMMEND_RELAY: 2>

<EventKind.CONTACTS: 3>

<EventKind.ENCRYPTED_DIRECT_MESSAGE: 4>

<EventKind.DELETE: 5>

Relays, Relay Management, and the Message Pool

Though this package is primarily focused on the tools needed to run a nostr client - we will talk a bit about only enough to understand how they serve the client. More detailed information on relays can always be found in the NIPs section of the nostr github repository. For testing in this package we are going to use a python package called nostr-relay, which is installed alongside this package. It will run a local server that we can connect to instead of pinging public servers over and over with tests.

Relays act as a simple data server for clients to interact with. Clients can publish events a relay and make requests (in the form of subscriptions) to get events back from a relay.

Connections between clients and relays are made over a ‘websocket’ (typically handled by websockets in Python). A single websocket between a client and a relay can handle all data requests and typically should remain open as long as the client is live. Repeated connection and disconnections of websockets can put extra, unneeded strain on relays.

The message types that can be exchanged between a client and a relay are outlined here:

client to relay
- EVENT - A nostr event being published to a relay
- REQ - a client requesting a subscription to events from a relay
- CLOSE - a client closing a subscription
relay to client
- EVENT - an incoming event from a relay
- NOTICE - human-readable text with an explanation of any issues
- EOSE - a notice that all past stored messages have been transferred and that all future messages are newly published
- OK - a True/False response if an EVENT was published successfully

python-nostr uses a MessagePool class to Queue the incoming stream of messages from the individial Relay websockets that are all running on different threads.

This package implements slightly modified versions of the Relay, RelayManager, and MessagePool classes found in python-nostr the basic usage is nearly identical, with a few added features that make building a the Client class easier. Added features that aren’t yet included in python-nostr:

a Connection context manager that is used to open connections on a Relay or RelayManager object using a with statement
Relay
- an open_connections method on a single relay to mirror the thread-based connection opening used in the RelayManager class.
- an is_connected property to see if the websocket is open
RelayManager
- additional connection management methods to make relay manager more robust to inactive connections
- an __iter__ method that allows the client to access the Relay objects by iterating directly on the RelayManager object instead of RelayManager.relays.values()

from nostr import relay, relay_manager

Relay

The documentation below shows how a Relay is instantiated using python-nostr. The relay policy is defined by the RelayPolicy class

Relay

 Relay (url:str, policy:nostr.relay.RelayPolicy,
        message_pool:nostr.message_pool.MessagePool,
        subscriptions:dict[str,nostr.subscription.Subscription]={})

Initialize self. See help(type(self)) for accurate signature.

Remember that in this package we will use a Relay class that inherits the python-nostr relay class. The same will go for the relay manager and the message pool.

assert issubclass(Relay, relay.Relay) and \
       issubclass(RelayManager, relay_manager.RelayManager) and \
       issubclass(MessagePool, message_pool.MessagePool)

Below we test using the context manager from the Relay class in this package to connect and disconnect to a relay.

import ssl

url='ws://127.0.0.1:6969'

a_relay = Relay(
    url=url,
    policy=RelayPolicy(),
    message_pool=MessagePool()
    )

assert not a_relay.is_connected

with a_relay.connection(ssl_options={'cert_reqs': ssl.CERT_NONE}):
    time.sleep(.5)
    print(f'Is the relay connected? {a_relay.is_connected}')
assert not a_relay.is_connected
time.sleep(.5)

INFO:     ('127.0.0.1', 51211) - "WebSocket /" [accepted]

2023-01-21 09:35:27,716 - nostr_relay.web - INFO - Accepted 127.0.0.1-2a7e from Origin: http://127.0.0.1:6969

INFO:     connection open

2023-01-21 09:35:27,725 - websocket - WARNING - websocket connected
Is the relay connected? True
2023-01-21 09:35:28,212 - nostr_relay.web - INFO - Done 127.0.0.1-2a7e. sent: 0 bytes. duration: 0sec

INFO:     connection closed

and make sure that when it errors out it properly reports the error and proceeds to safely close the connection to the relay

from fastcore.test import test_fail

def error_in_context():
    with a_relay.connection(ssl_options={'cert_reqs': ssl.CERT_NONE}):
        time.sleep(.5)
        raise Exception()
test_fail(error_in_context)
assert not a_relay.is_connected

INFO:     ('127.0.0.1', 51216) - "WebSocket /" [accepted]

2023-01-21 09:35:29,804 - nostr_relay.web - INFO - Accepted 127.0.0.1-b577 from Origin: http://127.0.0.1:6969

INFO:     connection open

2023-01-21 09:35:29,817 - websocket - WARNING - websocket connected
2023-01-21 09:35:30,303 - nostr_relay.web - INFO - Done 127.0.0.1-b577. sent: 0 bytes. duration: 0sec

INFO:     connection closed

Let’s try to publish an event to a our local relay - typically you would do this through the relay manager as to publish content across multiple relays, but for testing purposes we will only use one.

Relay.publish

 Relay.publish (message:str)

from nostr import message_type

We need to get our message to the relay in the right format as specified by NIP-01

message = [message_type.ClientMessageType.EVENT, event.to_json_object()]
message = json.dumps(message)
print(message)

["EVENT", {"id": "290b51301f55bf2ad51dba58e935cd6fbef8b9780c5e557a706c9403ae12b345", "pubkey": "91feaf1b03c1dc0ed61163f77b4cf4d1821cce392983d4c68d82207a773f12ee", "created_at": 1674322516.899612, "kind": 1, "tags": [], "content": "this is a test", "sig": "0c205776f1470aa1391d2556032b0a666fc9f88c5dfb1ddf968851a1186da8ea5ba3ca05c7202d68ee1b6b010323fd221220e13f16342cf319e7b347b12ed591"}]

And now we can publish it.

with a_relay.connection(ssl_options={'cert_reqs': ssl.CERT_NONE}):
    time.sleep(.5)
    a_relay.publish(message)

INFO:     ('127.0.0.1', 51222) - "WebSocket /" [accepted]

2023-01-21 09:35:35,170 - nostr_relay.web - INFO - Accepted 127.0.0.1-3be4 from Origin: http://127.0.0.1:6969

INFO:     connection open

2023-01-21 09:35:35,176 - websocket - WARNING - websocket connected

INFO:     connection closed

2023-01-21 09:35:35,679 - nostr_relay.web - INFO - 127.0.0.1-3be4 added 290b51301f55bf2ad51dba58e935cd6fbef8b9780c5e557a706c9403ae12b345 from 91feaf1b03c1dc0ed61163f77b4cf4d1821cce392983d4c68d82207a773f12ee
2023-01-21 09:35:35,685 - nostr_relay.web - INFO - Done 127.0.0.1-3be4. sent: 0 bytes. duration: 1sec

We will try to read this later.

Relay Manager

Generally, most interactions will happen through the RelayManager class. The relay manager allows easy interaction across multiple relays. Key methods are documented below

source

RelayManager.add_relay

 RelayManager.add_relay (url:str, read:bool=True, write:bool=True,
                         subscriptions=None)

source

RelayManager.remove_relay

 RelayManager.remove_relay (url:str)

RelayManager.publish_message

 RelayManager.publish_message (message:str)

You can also access connection statuses of the relays with RelayManager.connection_statuses. Below we use that property to check connections after making a RelayManager object, adding a Relay and connecting with the context manager.

manager = RelayManager()
urls=[url]
for url in urls:
    manager.add_relay(url=url)

with manager.connection():
    print('opening')
    print(manager.connection_statuses)
    assert all(manager.connection_statuses.values())
    print('closing')
print(manager.connection_statuses)
assert not any(manager.connection_statuses.values())

INFO:     ('127.0.0.1', 51225) - "WebSocket /" [accepted]

2023-01-21 09:35:40,202 - nostr_relay.web - INFO - Accepted 127.0.0.1-ef13 from Origin: http://127.0.0.1:6969

INFO:     connection open

2023-01-21 09:35:40,211 - websocket - WARNING - websocket connected
opening
{'ws://127.0.0.1:6969': True}
closing
{'ws://127.0.0.1:6969': False}
2023-01-21 09:35:42,207 - nostr_relay.web - INFO - Done 127.0.0.1-ef13. sent: 0 bytes. duration: 2sec

Relay objects that don’t connect will be remove automatically and throw a warning.

manager = RelayManager()
urls=['ws://this-relay-doesnt-exist.com']
for u in urls:
    manager.add_relay(url=u)

with manager.connection():
    pass
    assert len(manager.relays) == 0

INFO:     connection closed

2023-01-21 09:35:42,308 - websocket - ERROR - [Errno 8] nodename nor servname provided, or not known - goodbye

/var/folders/3x/w083pw853418fqm2blj59w480000gn/T/ipykernel_10817/1720938854.py:40: UserWarning: ws://this-relay-doesnt-exist.com is not connected... removing relay.
  warnings.warn(

Message Pool

Because we have multiple open websockets, the connections are threaded in Python. The MessagePool class acts as a shared Queue where different types of messages from the relay can all be accessed. The only significant change from the python-nostr MessagePool to this package is allowing the client to chose whether it should only keep the first event or whether it should keep multiples of event messages across all relays. Keeping all responses from all relays will allow the client to check how well distributed the user content is across the relays.

source

MessagePool

 MessagePool (first_response_only:bool=True)

Initialize self. See help(type(self)) for accurate signature.

The key events to access the MessagePool in python-nostr are listed below. We will add more functionality at the client level to modify how these events are handled upon receipt.

MessagePool.get_event

 MessagePool.get_event ()

MessagePool.get_notice

 MessagePool.get_notice ()

MessagePool.get_eose_notice

 MessagePool.get_eose_notice ()

In order to populate the message pool we need subscriptions.

Subscriptions

The nostr protocol requires clients to make requests (as subscriptions) to the relays in order to get events sent back (in our case, into the MessagePool). The REQ (request or subscription) query is also clearly outlined in NIP-01. A request must be labeled with a subscription_id and a series of filters that the relay will use to determine which events it should return. Current options for a single filter are as follow:

ids - a list of event IDs
authors - a list of authors (pubkeys), which could be something like a contact list
kinds - a list of kinds, such as text_notes (1) or set_metadata (0)
tags - a list of tags, like e event or p people tags
since - a time stamp that defines the earliest created_at that should be sent
until - a time stamp that defines the latest created_at that should be sent
limit - a limit of events to return upon initial request to keep the client from being overwhelmed if the query is wide

A list of filters is provided in a REQ message and the following happens:

the relay returns all events in past storage that meet the union of all the filter specifications (i.e. each filter is handled separately)
the relay would then send an End Of Stored Events notice with the subscription_id
the relay will continue to send newly published events that meet the criteria until 1 of the following occurs:
- the websocket is closed (by any measure - intentional closure or client crash)
- no more events are being published because the until clause of a subscription is met
- the client sends a request with the same subscription_id and it overwrites the initial behavior
- the client sends a CLOSE message to close the subscription_id

from nostr import filter
from nostr import subscription
import uuid

Filter

 Filter (ids:list[str]=None, kinds:list[int]=None, authors:list[str]=None,
         since:int=None, until:int=None, tags:dict[str,list[str]]=None,
         limit:int=None)

Initialize self. See help(type(self)) for accurate signature.

Subscription

 Subscription (id:str, filters:nostr.filter.Filters=None)

Initialize self. See help(type(self)) for accurate signature.

We can create a subscription to request the event we created earlier

filters = filter.Filters([filter.Filter(ids=[event.id])])
subscription_id = str(uuid.uuid4())
request = [message_type.ClientMessageType.REQUEST, subscription_id]
request.extend(filters.to_json_array())
message = json.dumps(request)
print(message)

["REQ", "10000", {"ids": ["290b51301f55bf2ad51dba58e935cd6fbef8b9780c5e557a706c9403ae12b345"]}]

Now we can get that from the relay we published it to

manager = RelayManager()
manager.add_relay(url)

with manager.connection(ssl_options={'cert_reqs': ssl.CERT_NONE}):
    time.sleep(.5)
    manager.add_subscription(subscription_id, filters)
    manager.publish_message(message)
    time.sleep(.5)
    while manager.message_pool.has_events():
        event_msg = manager.message_pool.get_event()
        print(f'event received from {event_msg.url} '
                f'with subscription id {event_msg.url}\n\t'
                f'{event_msg.event.content}')

INFO:     ('127.0.0.1', 51244) - "WebSocket /" [accepted]

2023-01-21 09:36:08,568 - nostr_relay.web - INFO - Accepted 127.0.0.1-eb74 from Origin: http://127.0.0.1:6969

INFO:     connection open

2023-01-21 09:36:08,577 - websocket - WARNING - websocket connected
2023-01-21 09:36:11,075 - nostr_relay.db - INFO - 127.0.0.1-eb74/10000 query – events:1 duration:1ms
event received from ws://127.0.0.1:6969 with subscription id ws://127.0.0.1:6969
    this is a test
2023-01-21 09:36:11,577 - nostr_relay.web - INFO - Done 127.0.0.1-eb74. sent: 400 bytes. duration: 3sec

INFO:     connection closed

We will use the Client class in this package to make subscriptions and publishing events even easier.