Openstack: How to debug neutron with pdb

When I was much younger and foolish (last month), I tried to debug neutron network creation, by inserting ‘import pdb; pdb.set_trace()’ to  /usr/lib/python2.7/site-packages/neutron/db/ and simply create network, waiting for pdb to take action. But that didn’t happen. I had to guess it will not work since neutron-server starts as a daemon, and not owned by the terminal. So using the pdb.set_trace() had no effect.

The right way would be to start neutron-server directly and not by starting the service. Don’t worry, we’ll do it together. step by step.

Stop neutron-server

I’m assuming you are debugging live environment. In my case I debugged environment installed with packstack.

So before running neutron-server directly, we need to ensure the current process is not active. Do it with the following comamnd:

sudo systemctl stop neutron-server

Make sure it really stopped using one of the following commands:

sudo systemctl status neutron server;
ps -ef | grep neutron-server

You should inactive/dead if using systemctl. Using the ps command, you should see nothing.

Add pdb.set_trace()

Quick note: this step could be the first one. The order here is not important.

In order to debug network creation, we need to add the pdb in the desired code block.

The network creation resides in /usr/lib/python2.7/site-packages/neutron/db/, in the function:  def create_network(self, context, network).

Let’s add there ‘import pdb; pdb.set_trace()”:

def create_network(self, context, network):
        """Handle creation of a single network."""
        import pdb; pdb.set_trace()
        # single request processing
        n = network['network']
        # NOTE(jkoelker) Get the tenant_id outside of the session to avoid
        #                unneeded db action if the operation raises
        tenant_id = self._get_tenant_id_for_create(context, n)
        with context.session.begin(subtransactions=True):

Don’t forget to save the file =)

Run neutron-server

pdb is in place, all that left now is to run neutron-server directly so all the server output spilled right to our terminal.

Check what command used to to run the server. If you are using distribution with systemd, you should check in ‘/usr/lib/systemd/system/neutron-server.service’. If your system is not using systemd, it’s probably in /etc/init.d/neutron-server.

You should find something similar to the following command:

/usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf
--config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log

Run it from your terminal. You should see a lot of output starting to get spilled.

Let the debugging begin!

You are ready to go! Open anther terminal and connect the same server in which you started neutrons-server directly.

Now create new network:

neutron net-create yay

Hop in to the second terminal. You should see the output flood stopped and pdb is waiting for you to debug this code 🙂

Back to origins

You’ve completed the debugging and you would like to revert changes and run server in normal state.

All you have to do is to stop the running of neutron-server, remove pdb from source code and run neutron-server with:

sudo systemctl start neutron-server

One thought on “Openstack: How to debug neutron with pdb

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s