Basically everyone gets infected.

That red shape is the curve people talk about when they say #FlattenTheCurve.

It doesn’t look like a lot of people died.

You can barely see the dark blue bit showing up at the top of the plot. If you look at the dots, you might not even have noticed the five dots that turned dark blue and stopped moving. But those five dots represent 2.5% of the population. If we scale this model up to the US population of 330 million, 2.5% is 8.5 million people dead!

Luckily, this model probably doesn’t scale up perfectly to the whole US. But that’s a worst case scenario of “what if we did nothing” and it’s terrible.

Social Distancing

So what can we do? One idea is social distancing or social isolation. That’s the idea that you just stay at home and don’t interact with people. How much can that help? Let’s follow the Washington Post article and see what happens if 25% of people isolate.

%%capture
u = Universe(npeople,distancing=0.25,
             initial_infection_chance=initial_infection_chance,
            )
u.run(nsteps)
anim25 = getanim(u,show_capacity=False)

u.data.i[0],u.data.d[-10:]

(1.0, array([5., 5., 5., 5., 5., 5., 5., 5., 5., 5.]))

HTML(anim25.to_html5_video())

That made almost no perceptible difference!

A bit about models like this: you should look at the general trends. Here, you can see that the shape of the red curve is about the same. The model uses a lot of randomness, so running it 100 times will give 100 different results. Looking for exact values of small numbers (e.g. the exact number of deaths) isn’t going to be as informative. For better estimates of that, you want to run this with a larger population, and you want to run it for longer. We’ll stick to qualitative results.

%%capture
u = Universe(npeople,distancing=0.5,
             initial_infection_chance=initial_infection_chance,
            )
u.run(nsteps)
anim50 = getanim(u,show_capacity=False)

u.data.i[0],u.data.d[-10:]

(4.0, array([5., 5., 5., 5., 5., 5., 5., 5., 5., 5.]))

HTML(anim50.to_html5_video())

%%capture
u = Universe(npeople,distancing=0.75,
             initial_infection_chance=initial_infection_chance,
            )
u.run(nsteps)
anim75 = getanim(u,show_capacity=False)

u.data.i[0],u.data.d[-10:]

(2.0, array([2., 2., 2., 2., 2., 2., 2., 2., 2., 2.]))

HTML(anim75.to_html5_video())

%%capture
u = Universe(npeople,distancing=0.9,
             initial_infection_chance=initial_infection_chance,
            )
u.run(nsteps)
anim = getanim(u,show_capacity=False)

u.data.i[0],u.data.i[-10:],u.data.d[-10:]

(2.0,
 array([15., 15., 15., 15., 14., 14., 14., 14., 14., 14.]),
 array([2., 2., 2., 2., 2., 2., 2., 2., 2., 2.]))

HTML(anim.to_html5_video())

%%capture
u = Universe(npeople,distancing=0.9,
             initial_infection_chance=initial_infection_chance,
            )
u.run(nsteps*2)
anim = getanim(u,show_capacity=False)

u.data.i[0],u.data.i[-10:],u.data.d[-10:]

(4.0,
 array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]),
 array([4., 4., 4., 4., 4., 4., 4., 4., 4., 4.]))

HTML(anim.to_html5_video())

What about hospital capacity?

What if we run out of hospital beds? In that case, obviously more people are going to die. The estimates of mortality rates when we have hospital capacity seem to range quite a bit, but 2% seems like a good estimate (I’m not linking to any sources here because things keep changing. Feel encouraged to download the code and re-run things yourself if you want a different estimate!).

The overall mortality rate without hospitals seems more like 5%. That’s not a factor of 10 or anything, but it’s quite a bit. And it becomes a bigger difference if hospitals get better at treating patients.

What’s our capacity? Googling around, it looks like we have a hospital capacity of around 3 beds per 1000 people in the US.

Bigger population: in order to see this, we’re going to want a bigger population! Both in terms of having a large enough population to see the capacity matter, and so differences of 1-2 people don’t make such a difference. That means things will take longer to compute. Drawing the awesome animation actually takes a long time, so I’m just going to show the charts you saw on top of those figures, pre-drawn, with no animations for the rest of this. Hopefully that makes sense at this point, since you’ve seen several animations.

So here’s the nobody-does-anything plot, original size. The dashed line at the bottom is the hospital capacity. In this case, here are the exact numbers: * with 3 beds per 1000 people, that’s 0.6 beds for our 200 people. * We expect 5% of sick people to need beds. That means that, out of 20 sick people, one of them is expected to need a bed. So, if we’re just talking about corona virus, we need one bed for every 20 sick people. One bed means we can handle 20 sick people. Two beds means we can handle 40 sick people. 0.6 beds means we can handle 20*0.6 = 12 sick people. So the dashed line in the chart below is at 12 people

npeople = 200
initial_infection_chance = 1.5/100

u = Universe(npeople=npeople,
             initial_infection_chance=initial_infection_chance,
             hospital_beds_fraction=0.003,
            )
u.run(nsteps)
draw_stacked_plot(u,show_capacity=True,print_description=True)

4.0% of people died in the figure below

NOTE if you want to know the percentage of people who died in the above simulation, you can see it in text right below the Python code/right above the figure. This one says “4% of people died in the figure below”. You’ll see a similar bit of text on all of the next figures, but it’s kind of hard to see, and I’m not great at text formatting.

And now with 1000 people

npeople = 1000
initial_infection_chance = 1.5/100

u = Universe(npeople=npeople,
             initial_infection_chance=initial_infection_chance,
             hospital_beds_fraction=0.003,
            )
u.run(nsteps)
draw_stacked_plot(u,show_capacity=True,print_description=True)

5.6% of people died in the figure below

You can see the awful trend there: the number of deaths starts going up dramatically once everyone is sick and the hospitals are over capacity. This only becomes more pronounced when hospitals get better at treating people!

Let’s say we start a bit later, with 5% of the population infected. That’s probably not a terrible guess. Given our lack of testing, it could even be higher than that.

initial_infection_chance = 0.05

u = Universe(npeople=npeople,
             initial_infection_chance=initial_infection_chance,
             hospital_beds_fraction=0.003,
            )
u.run(nsteps)
draw_stacked_plot(u,show_capacity=True,print_description=True)

5.0% of people died in the figure below

That’s bad.

u = Universe(npeople=npeople,distancing=0.5,
             initial_infection_chance=initial_infection_chance,
             hospital_beds_fraction=0.003,
            )
u.run(nsteps)
draw_stacked_plot(u,show_capacity=True,print_description=True)

4.1% of people died in the figure below

u = Universe(npeople=npeople,distancing=0.9,
             initial_infection_chance=initial_infection_chance,
             hospital_beds_fraction=0.003,
            )
u.run(nsteps)
draw_stacked_plot(u,show_capacity=True,print_description=True)

3.5% of people died in the figure below

What if we build more beds

We could certainly do something like mobilize the Army Corps of Engineers to build a bunch of hospitals. That’s what other countries did a while ago. What if we increased our hospital capacity by a factor of 5? I’m guessing that’s reasonable because we don’t need great multi-purpose hospitals. We need temporary stuff for one particular purpose. Maybe this includes the navy hospitals? Maybe it includes repurposed dorm rooms? I’m hoping. But I’m not a civil engineer; I could be wrong here.

u = Universe(npeople=npeople,distancing=0.9,
             initial_infection_chance=initial_infection_chance,
             hospital_beds_fraction=0.003*5,
            )
u.run(nsteps)
draw_stacked_plot(u,show_capacity=True,print_description=True)

2.2% of people died in the figure below

So that’s way better. Not perfect, but way better.

What if people stop distancing?

u = Universe(npeople=npeople,distancing=0.9,
             initial_infection_chance=initial_infection_chance,
             hospital_beds_fraction=0.003*5,
            )
u.run(nsteps,stop_distancing_at = int(nsteps*0.5))
draw_stacked_plot(u,show_capacity=True,print_description=True)

3.7% of people died in the figure below

u = Universe(npeople=npeople,distancing=0.0,
             initial_infection_chance=initial_infection_chance,
             hospital_beds_fraction=0.003*2,
            )
u.run(nsteps,stop_distancing_at = int(nsteps*0.5))
draw_stacked_plot(u,show_capacity=True,print_description=True)

5.9% of people died in the figure below

u = Universe(npeople=npeople,distancing=0.9,
             initial_infection_chance=initial_infection_chance,
             hospital_beds_fraction=0.003*2,
            )
u.run(nsteps)
draw_stacked_plot(u,show_capacity=True,print_description=True)

3.4% of people died in the figure below

u = Universe(npeople=npeople,distancing=0.9,
             initial_infection_chance=initial_infection_chance,
             hospital_beds_fraction=0.003*2,
            )
u.run(nsteps,stop_distancing_at=nsteps*0.75)
draw_stacked_plot(u,show_capacity=True,print_description=True)

3.6% of people died in the figure below

u = Universe(npeople=npeople,distancing=0.9,
             initial_infection_chance=initial_infection_chance,
             hospital_beds_fraction=0.003*2,
            )
u.run(nsteps*2,stop_distancing_at=nsteps*0.75)
draw_stacked_plot(u,show_capacity=True,print_description=True)

4.9% of people died in the figure below

Conclusions

Can I make any conclusions from this? Even though this isn’t a quantitatively correct model, I think a few things are probably true:

1. We need to increase our hospital capacity. This means mobilizing the Army Corps of Engineers, using the navy hospital ships, maybe repurposing college dorms, etc. It also means obtaining more personal protective equipment, so that our health care workers can continue to work with proper protection! If they get sick, our capacity decreases hugely!
2. We need to look aggressively for cures or vaccines. I see reports that some malaria drugs might treat coronavirus within a few days in a hospital setting. That’s huge.
3. #FlattenTheCurve, by itself, is only a complete solution if you can do it super aggressively, and basically forever (until everyone has been exposed). But flattening the curve now while we do things like building more beds and looking for cures is a huge win!

Seems like we’d be best off flattening the curve for as long as society can stand, while pushing hard at capacity and cures.

Running it yourself

If you want to run it yourself, here’s the code. Instead of

from covid import social_distancing
from covid.social_distancing import Universe, getanim, draw_stacked_plot

up top, just run the cell below.

If you’re playing around with the code, here are some things to note:

• Generating the animation is much slower than generating the data. So maybe just make the stacked plots if you’re looking for trends.
• Units and parameterization are tricky. I made very little effort to get the physical distances/movement right.
• I had to re-run the cells a bunch to get similar starting conditions. That’s what you get with randomness. You could think about modifying things so that we had a fixed number of infected people at the beginning. Or a fixed number of days those folks had been infected. You could also explicitly seed the random number generator for consistency.
• One that you might want make if you play around with the model is letting people transition from R back to S, since there have been some reports of people getting reinfected with the disease. Eek!

import numpy as np, scipy as sp, pandas as pd
from matplotlib import pyplot as plt
from sklearn.metrics.pairwise import euclidean_distances
from matplotlib import animation
from IPython.display import HTML
from itertools import combinations
from collections import namedtuple

"""
Units: time is measured in 12 hour chunks, but there's no strict correspondence to units here.
"""

class EffectiveArea:
    """Tells us the boundaries of the area that people can live in.
    """
    def __init__(self,):
        self.xmin, self.xmax = 0, 20
        self.ymin, self.ymax = 0, 20

class Person:
    """
    In this version, after 14 days, you either recover or die
    """
    def __init__(self,effectivearea,state='succeptible',distancing=False):
        self.r = 0.15
        self.days_infected = 0
        self._typical_recovery_days = 14
        self.mortality_rate = {True:0.05,False:0.02} # index is whether we're over capacity
        self.distancing = distancing
        if self.distancing:
            self.m = 1000
        else:
            self.m = 1
        self.state = state
        if self.state == 'infected':
            self.days_infected = np.random.randint(self._typical_recovery_days)
        self.ea = effectivearea
        self.x, self.y = np.random.uniform(self.ea.xmin,self.ea.xmax), np.random.uniform(self.ea.ymin,self.ea.ymax)
        if self.distancing:
            self.distance(force=True)
        else:
            self.undistance(force=True)
    def distance(self,force=False):
        if force or not self.distancing:
            self.m = 1000
            self.vx, self.vy = 0,0
            self.distance = True
    def undistance(self,force=False):
        if force or self.distancing:
            self.m = 1
            self.vx, self.vy = np.random.normal(size=2) # Maxwell Boltzmann?
            self.distancing = False
    def move(self,dt,ea,overcapacity=False):
        self.x, self.y = self.x + self.vx*dt, self.y + self.vy*dt
        
        """People don't recover instantly. Once it's been two weeks, 
        they start getting a chance to recover. We could also give 
        them a chance to die here.
        """
        if self.state == 'infected':
            if self.days_infected < self._typical_recovery_days:
                self.days_infected += dt
            else:
                if np.random.uniform() < self.mortality_rate[overcapacity]:
                    self.state = 'dead'
                    self.m = 1000
                    self.vx, self.vy = 0,0
                else:
                    self.days_infected = 0
                    self.state = 'recovered'
    
def collide(p1,p2):
    """This is the 2D elastic collision problem from your intro physics book.
    """
    if p1.state == 'infected' and p2.state == 'succeptible':
        p2.state = 'infected'
    elif p2.state == 'infected' and p1.state == 'succeptible':
        p1.state = 'infected'

    m1, m2 = p1.m, p2.m
    r1, r2 = np.array([p1.x,p1.y]), np.array([p2.x,p2.y])
    v1, v2 = np.array([p1.vx,p1.vy]), np.array([p2.vx,p2.vy])
    M = m1 + m2
    d = np.linalg.norm(r1 - r2)**2
    u1 = v1 - 2*m2 / M * np.dot(v1-v2, r1-r2) / d * (r1 - r2)
    u2 = v2 - 2*m1 / M * np.dot(v2-v1, r2-r1) / d * (r2 - r1)
    p1.vx,p1.vy = u1
    p2.vx,p2.vy = u2

    
class Universe:
    def __init__(self,
                 npeople, # How many people there are in the world
                 initial_infection_chance=0.1, # Initial fraction of population which is infected
                 distancing=0.0, # Fraction of population which practices distancing
                 hospital_beds_fraction = 1.0 # Better is 0.003 which is closer to reality
                ):
        self.npeople = npeople
        self.ea = EffectiveArea()
        self.dt = 0.1
        self.hospital_beds_fraction = hospital_beds_fraction
        self.hospital_beds = npeople*hospital_beds_fraction
        self.data = None # gets set in self.run
        def _state():
            if np.random.uniform() < initial_infection_chance:
                return 'infected'
            return 'succeptible'
        def _distancing():
            return np.random.uniform() < distancing
        self.people = [Person(self.ea,_state(),_distancing()) for i in range(self.npeople)]
        # self.color = {'succeptible':0.5,'infected':0.0,'recovered':0.7} # old color scheme
        self.color = {'succeptible':'lightblue','infected':'red','recovered':'green','dead':'midnightblue'}
        
    def _step(self):
        """iterate through one timestep
        """
        points = list(zip([p.x for p in self.people],[p.y for p in self.people]))
        dists = euclidean_distances(points,points)
        close = dists < 2*self.people[0].r
        close = close.tolist()
        for (i,j) in combinations(range(self.npeople),2):
            if close[i][j]: # a bit faster than numpy indexing once things get big.
                collide(self.people[i],self.people[j])
                
        # Are we over capacity?
        ninfected = len([p for p in self.people if p.state == 'infected'])
        # about 5% need hospitalization
        overcapacity = 0.05 * ninfected > self.hospital_beds
        for p in self.people:
            p.move(self.dt,self.ea,overcapacity)
            if p.x <= self.ea.xmin or p.x >= self.ea.xmax: 
                p.vx = -p.vx
            if p.y <= self.ea.ymin or p.y >= self.ea.ymax:
                p.vy = -p.vy
    
    def run(self,steps,stop_distancing_at = None):
        """Run a simulation
        
        Internal data looks like
        
        ```
        x_coords[frame,particle_number]
        ```
        """
        x_coords = np.zeros((steps,len(self.people)))
        y_coords = np.zeros((steps,len(self.people)))
        state = np.zeros((steps,len(self.people)),dtype='object')
        
        # SIR model. `i` is a questionable variable name TBH
        s,i,r,d = np.zeros(steps),np.zeros(steps),np.zeros(steps),np.zeros(steps)
        def pop_count(people):
            s,i,r,d = 0,0,0,0
            for p in people:
                if p.state == 'succeptible':
                    s += 1
                elif p.state == 'infected':
                    i += 1
                elif p.state == 'recovered':
                    r += 1
                elif p.state == 'dead':
                    d += 1
                    
            return s,i,r,d
        s[0],i[0],r[0],d[0] = pop_count(self.people)
        x_coords[0] = [p.x for p in self.people]
        y_coords[0] = [p.y for p in self.people]
        state[0] = [p.state for p in self.people]
        
        for step in range(1,steps):
            if step == stop_distancing_at:
                for p in self.people:
                    p.undistance()
            self._step()
            s[step],i[step],r[step],d[step] = pop_count(self.people)
            x_coords[step] = [p.x for p in self.people]
            y_coords[step] = [p.y for p in self.people]
            state[step] = [p.state for p in self.people]
        dtype = namedtuple('RunData',['x','y','state','s','i','r','d','steps'])
        self.data = dtype(x_coords,y_coords,state,s,i,r,d,steps)
        
    def draw(self,ax=None):
        """
        A very simple method to draw the current state. Better graphing comes in the
        animation functions.
        """
        if ax is None:
            fig,ax = plt.subplots(figsize=(5,5))
        scat = ax.scatter([p.x for p in self.people],[p.y for p in self.people],
                   c = [self.color[p.state] for p in self.people],
                   marker='.')
        return scat,



def getanim(u,show_capacity=True):
    fig,ax = plt.subplots(figsize=(6,6))
    ax.axis('off')

    left, width = 0.1, 0.85
    bottom, height = 0.1, 0.65
    spacing = 0.02

    rect_universe = [left, bottom, width, height]
    rect_trend = [left, bottom + height + spacing, width, 0.2]

    ax_universe = plt.axes(rect_universe)
    ax_trend = plt.axes(rect_trend)
    #ax_universe.axis('off') # I want the box around it.
    ax_universe.axes.get_xaxis().set_ticks([])
    ax_universe.axes.get_yaxis().set_ticks([])    
    ax_trend.axis('off')

    s,i,r,d = u.data.s,u.data.i,u.data.r,u.data.d

    line, = draw_stacked_plot(u,ax_trend,show_time=True,show_capacity=show_capacity)

    #ax_trend.stackplot(range(len(s)), i, s, r, d, labels=['sick','healthy','recovered','dead'],
    #                   colors=[u.color['infected'],u.color['succeptible'],u.color['recovered'],u.color['dead']])

    scat, = u.draw(ax_universe)
    
    def drawframe(i):
        data = np.column_stack(([u.data.x[i],u.data.y[i]]))
        scat.set_offsets(data)
        colors = np.array([u.color[_] for _ in u.data.state[i]])
        scat.set_color(colors)

        #_s,_i,_r = np.zeros(len(u.data.s)),np.zeros(len(u.data.s)),np.zeros(len(u.data.s))
        #_s[:i],_i[:i],_r[:i] = u.data.s[:i],u.data.i[:i],u.data.r[:i]
        #ax_trend.stackplot(range(len(_s)), _s, _i, _r, labels=['s','i','r'],colors=['blue','green','yellow'])
        #ax_trend.legend(loc='upper left')
        line.set_data([i,i],[0,u.npeople])

        return scat,line


    anim = animation.FuncAnimation(fig, drawframe, frames=u.data.steps,
                                  interval=20, blit=False, repeat=False)
    return anim



def draw_stacked_plot(u,ax=None,show_time=False,show_capacity=True,print_description=False):
    if ax is None:
        fig,ax = plt.subplots()
        ax.axis('off')
    s,i,r,d = u.data.s,u.data.i,u.data.r,u.data.d
    if print_description:
        print(f'{100*d[-1]/u.npeople}% of people died in the figure below')
    ax.stackplot(range(len(s)), i, s, r, d, labels=['sick','healthy','recovered','dead'],
                     colors=[u.color['infected'],u.color['succeptible'],u.color['recovered'],u.color['dead']])
    ax.legend(loc='center left')
    if show_capacity:
        ax.hlines(y=u.hospital_beds*20,xmin=0,xmax=len(s),linestyle='dashed')
    if show_time is True:
        line, = ax.plot([0,0],[0,s[0]+i[0]+r[0]+d[0]],linestyle='dashed',color='black')
        return line,
    return

Orienting

We’re in New Zealand for a semester. Recently, Ash Patea was telling us some Māori lore about local mountains. We’re based in Whanganui, learning about the mountains Tongariro, Taranaki, and Ruapehu, all of which are to our north. Ash starts all of these sessions off by telling us that we’re learning his iwi’s stories, but that they’re not the only stories. That other stories are different, and none is “right.” We then get to the part where we learn that the name “Tongariro” tells you it’s a mountain to the south, and Ash says that’s how they know the story we’re learning isn’t their story, and all of a sudden I have to write a little math post.

I don’t want to spoil the punchline too much (yes I do), but there’s a really neat connection between Ash’s language and mathematics, about how you orient yourself in the world, and about how things can’t always be consistent.

Math time. A “chart” is essentially your local set of coordinates, or how you tell yourself how to move around in the world. You can think of it like a street map in a city, or the idea of north, south, east, and west.

makeone()

You could describe different regions with different charts. You need to “glue” them together in the areas where they overlap.

makethree()

When you glue all of those charts together to cover an object, the result is called an “atlas”

makelots()

So, obviously, the connection I want point out is that each of these charts is like the stories of an individual iwi. They tell that iwi how to understand the world; how to get around in the world; how to orient themselves in the world. Understanding the local map tells you where you are; and knowing how to glue those local maps together tells you how to get around.

Sometimes, you can glue everything together into one consistent set of coordinates. Like in the above pictures, we could just have called the same direction “north” everywhere. Or, maybe every iwi has the same stories. Back to math language, maybe you can just have one chart to make up your whole atlas.

make_half_sphere_chart()

make_north_example()

from matplotlib.patches import Rectangle
from matplotlib.path import Path
import matplotlib.patches as patches
import numpy as np, matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
plt.xkcd()

def plt1(ax,offsetx=0,offsety=0,color='orange',text=True):
    # Mostly taken from
    # https://matplotlib.org/users/path_tutorial.html
    verts = [
        (0. + offsetx, 0. + offsety), # left, bottom
        (0. + offsetx, 1. + offsety), # left, top
        (1. + offsetx, 1. + offsety), # right, top
        (1. + offsetx, 0. + offsety), # right, bottom
        (0. + offsetx, 0. + offsety), # ignored
        ]

    codes = [Path.MOVETO,
             Path.LINETO,
             Path.LINETO,
             Path.LINETO,
             Path.CLOSEPOLY,
             ]

    path = Path(verts, codes)

    patch = patches.PathPatch(path, facecolor=color, lw=2, alpha=0.5)
    ax.add_patch(patch)

    #arrow( x, y, dx, dy, **kwargs )
    if text:
        alpha=1
    else:
        alpha=0.2
    plt.arrow( 0.5 + offsetx, 0.2 + offsety, 0.0, 0.4, fc="k", ec="k",head_width=0.05, head_length=0.1,alpha=alpha)
    if text:
        plt.text(0.6 + offsetx,0.3 + offsety,'North')
    ax.set_xlim(-2,2)
    ax.set_ylim(-2,2)
    ax.set_yticklabels([])
    ax.set_xticklabels([])
    
def makeone():
    fig = plt.figure()
    ax = fig.add_subplot(111)

    plt1(ax)
    
def makethree():
    fig = plt.figure()
    ax = fig.add_subplot(111)

    plt1(ax)
    plt1(ax,-0.9,-0.1,'green')
    plt1(ax,-0.5,-0.9,'blue')    

def makelots():
    fig = plt.figure()
    ax = fig.add_subplot(111)

    for i in np.arange(-2,2,.8):
        for j in np.arange(-2,2,.8):
            plt1(ax,i,j,'grey',text=False)

    plt1(ax)
    plt1(ax,-0.9,-0.1,'green')
    plt1(ax,-0.5,-0.9,'blue')
    
def plot_top(ax,n_theta,n_phi,r,fixz=False,alpha=0.5):
    # Largely taken from
    # https://stackoverflow.com/questions/41105754/heat-map-half-sphere-plot
    
    #theta inclination angle
    #phi azimuthal angle

    theta, phi = np.mgrid[0.0:0.5*np.pi:n_theta*1j, 0.0:2.0*np.pi:n_phi*1j]

    x = r*np.sin(theta)*np.cos(phi)
    y = r*np.sin(theta)*np.sin(phi)
    z = r*np.cos(theta)
    if fixz:
        z = np.ones_like(z)*2

    ax.plot_surface(
        x,y,z,  rstride=1, cstride=1, 
        color='red',
        alpha=alpha, linewidth=1) 

def plot_bottom(ax,n_theta,n_phi,r,alpha=0.5):
    # Largely taken from
    # https://stackoverflow.com/questions/41105754/heat-map-half-sphere-plot
    
    #theta inclination angle
    #phi azimuthal angle

    theta, phi = np.mgrid[0.0:0.5*np.pi:n_theta*1j, 0.0:2.0*np.pi:n_phi*1j]

    x = r*np.sin(theta)*np.cos(phi)
    y = r*np.sin(theta)*np.sin(phi)
    z = -r*np.cos(theta)

    ax.plot_surface(
        x,y,z,  rstride=1, cstride=1, 
        color='blue',
        alpha=alpha, linewidth=1) 

def plot_circles(ax,includetop=True,color='k'):
    # Largely taken from
    # https://stackoverflow.com/questions/32424670/python-matplotlib-drawing-3d-sphere-with-circumferences
    u = np.linspace(0, 2 * np.pi, 100)
    v = np.linspace(0, np.pi, 100)

    elev = 10.0
    rot = 80.0 / 180 * np.pi

    #calculate vectors for "vertical" circle
    a = np.array([-np.sin(elev / 180 * np.pi), 0, np.cos(elev / 180 * np.pi)])
    b = np.array([0, 1, 0])
    b = b * np.cos(rot) + np.cross(a, b) * np.sin(rot) + a * np.dot(a, b) * (1 - np.cos(rot))
    horiz_front = np.linspace(0, np.pi, 100)
    ax.plot(np.sin(u),np.cos(u),0,color=color, linestyle = 'dashed')
    ax.plot(np.sin(horiz_front),np.cos(horiz_front),0,color=color)

    if includetop:
        ax.plot(np.sin(u),np.cos(u),2.0,color=color, linestyle = 'dashed')
        ax.plot(np.sin(horiz_front),np.cos(horiz_front),2.0,color=color)

    
    vert_front = np.linspace(np.pi / 2, 3 * np.pi / 2, 100)
    ax.plot(a[0] * np.sin(u) + b[0] * np.cos(u), b[1] * np.cos(u), a[2] * np.sin(u) + b[2] * np.cos(u),
            color=color, linestyle = 'dashed')
    ax.plot(a[0] * np.sin(vert_front) + b[0] * np.cos(vert_front), 
            b[1] * np.cos(vert_front), a[2] * np.sin(vert_front) + b[2] * np.cos(vert_front),
            color=color)

def plot_lines(ax):
    ax.plot(xs=[0.75,.75],ys=[-0.75,-0.75],zs=[0,2],color='k',alpha=1.0)
    ax.plot(xs=[0.75,.75],ys=[0.75,0.75],zs=[0,2],color='k',alpha=1.0)

    ax.plot(xs=[0,0],ys=[0,0],zs=[1,1.65],color='k')
    ax.plot(xs=[0,0],ys=[0,0],zs=[1.65,2.0],color='k',linestyle='dashed',alpha=0.75)
    
    ax.plot(xs=[-0.75,-.75],ys=[-0.5,-0.5],zs=[0.5,1.5],color='k',alpha=1.0)
    ax.plot(xs=[-0.75,-.75],ys=[-0.5,-0.5],zs=[1.5,2.0],color='k',linestyle='dashed',alpha=0.75)
    ax.plot(xs=[-0.75,-.75],ys=[-0.5,-0.5],zs=[0,0.5],color='k',linestyle='dashed',alpha=0.75)

    ax.plot(xs=[-0.75,-.75],ys=[0.5,0.5],zs=[0.5,1.5],color='k',alpha=1.0)
    ax.plot(xs=[-0.75,-.75],ys=[0.5,0.5],zs=[1.5,2.0],color='k',linestyle='dashed',alpha=0.75)
    ax.plot(xs=[-0.75,-.75],ys=[0.5,0.5],zs=[0,0.5],color='k',linestyle='dashed',alpha=0.75)
    

def make_half_sphere_chart():
    fig = plt.figure(figsize=(10, 8))
    ax = fig.add_subplot(111, projection='3d')


    plot_top(ax,30,30,1,alpha=0.25)
    plot_bottom(ax,30,30,1,alpha=0.25)
    plot_circles(ax,color='darkgrey')
    plot_top(ax,30,30,1,fixz=True,alpha=0.7)
    plot_lines(ax)

    ax.set_xlim([-2.2,2.2])
    ax.set_ylim([-2.2,2.2])
    ax.set_zlim([-2.2,2.2])

    ax.set_aspect('equal')
    ax.view_init(elev = 20, azim = 0)
    ax.set_yticklabels([])
    ax.set_xticklabels([])
    ax.set_zticklabels([])

def make_north_example():
    fig = plt.figure(figsize=(10, 8))
    ax = fig.add_subplot(111, projection='3d')


    plot_top(ax,30,30,1,alpha=0.25)
    plot_bottom(ax,30,30,1,alpha=0.25)
    plot_circles(ax,includetop=False,color='darkgrey')

    offsetx = 0
    offsety = 0
    alpha=1.0
    # These arrows aren't mathematically correct ... but they look reasonable!
    ax.quiver([0],[-.13],[-0.9],[ 0],[-.1],[.75],color='k')
    ax.text(0,0,-.75,'North',fontsize=18)
    
    ax.quiver([0],[-.16],[0.5], [0],[.1],[.75],color='k')
    ax.text(0,0,.75,'North',fontsize=18)
    

    ax.set_xlim([-1.5,1.5])
    ax.set_ylim([-1.5,1.5])
    ax.set_zlim([-1.5,1.5])

    ax.set_aspect('equal')
    ax.view_init(elev = 20, azim = 0)
    ax.set_yticklabels([])
    ax.set_xticklabels([])
    ax.set_zticklabels([])    

Say you’re going to flip 4 coins. What’s the most likely outcome? How likely is it?

How should we keep track of outcomes? One easy way is to ask “how many heads did we get?”

Well, intuitively, we’d all say you’re most likely to get two heads and two tails. Let’s check. If it’s a fair coin, we can assume that we’re equally likely to get a head or a tail on each flip. So, we flip the first coin, and make a table:

Great. Two outcomes, equally likely. One head 50% of the time, zero heads 50% of the time. What if we flip a second coin (students should work all of this out on their own, but hey, I’m just a blog post). If the first was heads, the second could be either heads or tails. Ditto for the second.

This time, it’s a bit more interesting. We had three possible outcomes. We got two heads once, one head twice, and zero heads once. So, our most likely outcome is half heads, but not by a huge amount.

If we add a third coin, and then a fourth coin, we get (again, students should work it all out)

And if we tabulate that, we have 16 possible outcomes. We get 4 heads 1/16 of the time, 3 heads 4/16 of the time, 2 heads 6/16 of the time, 1 head 4/16 of the time, and 0 heads 1/16 of the time.

We can visualize that with a histogram.

import numpy as np,itertools, pandas as pd, scipy as sp
import seaborn as sns
from matplotlib import pyplot as plt
from IPython.display import HTML
from ipywidgets import interact, interactive, fixed

heads = pd.Series(np.array([4,3,3,2,3,2,2,1,3,2,2,1,2,1,1,0]),name='Number of heads')
sns.distplot(heads,kde=False,bins=5);

Phrasing it with statistical mechanics

The fundamental postulate of statistical mechanics says that each of these different ways of flipping coins (HHHH, HHTH, HHHT, etc.) is equally likely. That makes intuitive sense here: each individual coin flip was equally likely. But, we can use that to figure out probabilities. We actually did that for the “flip three coins” example above. Here, we say that there were 4 heads in 1/16 of the flips, so we have a 1/16 = 6.25% chance of getting one head. Here’s the same thing, plotted with probabilities instead of “number of times this happened”:

sns.distplot(heads,kde=False,bins=5,norm_hist=True);

So, that’s cool. We now know how likely it is that we’re going to get any given number of heads. For four coin flips, our intuition was probably right: more likely to get two heads. But not that much more likely.

Flipping more coins

If we want to flip more coins, it’s going to be a pain in the neck to make that table over and over. Luckily, this is bundled up in a math/probability/stats concept called “combinations”. Remember how, when we flipped four coins, we had four ways to get one head?

sp.special.comb(4,1)

4.0

And how there were six ways to get two heads?

sp.special.comb(4,2)

6.0

We can get a list of all of those:

[sp.special.comb(4,i) for i in range(5)]

[1.0, 4.0, 6.0, 4.0, 1.0]

And we can wrap that whole thing up into an interactive plot

def plotflips(num_coins=4):
    flips_with_heads = [sp.special.comb(num_coins,i) for i in range(num_coins+1)]
    flips_with_heads = np.array(flips_with_heads)
    #plt.bar(range(num_coins+1),flips_with_heads/flips_with_heads.sum())
    plt.bar(range(num_coins+1),flips_with_heads)
    plt.show()
    
interact(plotflips,num_coins=list(range(1,100)) + list(range(100,200,10)) + list(range(200,1001,100)));

Back to some physics

How does this all connect to entropy? People usually say that “entropy” means things “want to be more disordered” … let’s use that to think about the air molecules in the room. If air molecules can be randomly located anywhere in a room, why don’t you ever have to worry about walking into a room and finding all of them on the other side? If that happened, you’d asphyxiate right away!

Well, in terms of entropy, stuffing them all on one side would certainly be a more ordered state. What can our coins tell us about that? A really simple model would be to say “heads” means a molecule is on the left side of the room, “tails” means it’s on the right. Then, flipping 0 heads would mean you asphyxiate. Look at the above interactive plot, and notice that even with 1000 “molecules” the probability of having things spread out evenly is WAY bigger than the alternative. Think about avogadro’s number.

Visualizing differential geometry in Jupyter notebooks

I taught a senior seminar on differential geometry last year. I’ll be honest: it was a selfish course. Markus Deserno writes all of these cool papers about membranes, and I didn’t know enough math to follow them. So, the goal of our course was basically to learn enough differential geometry to read several of his papers. It was fantastic. We used several of his papers, as well as Kreyszig’s textbook, as our core materials.

Here’s the thing: a lot of this was pretty foreign to my physics students. In particular, the discussion of surfaces and mappings was new. So, we wrote some tools in Jupyter Notebooks to help us visualize and solve problems. I particularly like the stuff we wrote to visualize a mapping, and I don’t know of a comparable resource elsewhere.

As a teaser, the above movies show you how to interpolate between a surface and a curve in the first case, and how the Monge gauge works for a membrane in the second case.

Let’s jump in.

Surfaces

We get our introduction to surfaces in Kreyszig Ch. 3. Specifically, In most of our classes, we’ve defined surfaces like

and we’ve been good at visualizing them. Here, we run into the definition of a surface in terms of a mapping that sends

and Kreyszig warns us that something like

is problematic because, since it can be parameterized by a single variable, it’s actually a curve rather than a surface. We also run into the Jacobian and some linear algebra. Our goal right now is to work with concrete curves and surfaces in So, let’s figure out how to visualize these sorts of mappings. First, some gunk to set up our python environment. For the experts, note that we could use %matplotlib notebook to make things prettier, but it seems really slow.

import numpy as np, scipy as sp, pandas as pd, seaborn as sns, matplotlib.pyplot as plt, matplotlib as mpl
import itertools
from mpl_toolkits.mplot3d import Axes3D, proj3d
import matplotlib.animation as animation
from matplotlib.patches import FancyArrowPatch
from IPython.display import HTML
from ipywidgets import interact, interactive, fixed

In class, we used interact to visualize things. However, I’m making this available on the web using the fact that Jake Vanderplas’ awesome JSAnimation package is built in to matplotlib 2.1. I like it way better than just an html5 video, because I like to be able to step through the frames. I still strongly recommend interact for playing around on your own server.

# You can change this to %matplotlib notebook if you want. It looks better, but is much slower for me.
#%matplotlib inline

# mpl.rcParams['animation.html'] = 'jshtml'
# Doesn't seem to work for me, so I make the HTML explicitly below

Now, let’s just try one of these out. One of the examples we want to look at in the book is

so let’s make a grid of points where and vary from -2 to 2, then plot the corresponding surface. We’ll plot in grey and the surface in blue.

u1 = np.linspace(-2,2,100)
u2 = np.linspace(-2,2,100)
U1, U2 = np.meshgrid(u1, u2)
X = U1 + U2
Y = np.sin(U1)
Z = np.cos(U1+U2)

ax = plt.gca(projection='3d')
surf2 = ax.plot_surface(U1, U2, np.zeros_like(U1), color='grey', alpha=0.5)
surf = ax.scatter(X, Y, Z)
plt.show()

You can play around with the above, using more or fewer points, etc. You can clearly see that the surface extends vertically and horizontally beyond the patch from and , although it’s kind of cheating to plot them both at the same time. So let’s wrap that up in a function and try it with different maps.

To make it cuter, we’ll draw red arrows from some specific points to see which points get mapped where. And we’ll plot a surface not a collection of points, by default, because it’s way faster. Most impoartantly, we’ll wrap all of the plotting stuff up in a function of its own so that we just need to define a map, and then ask to plot it.

class Arrow3D(FancyArrowPatch):
    #http://stackoverflow.com/questions/22867620/putting-arrowheads-on-vectors-in-matplotlibs-3d-plot
    def __init__(self, xs, ys, zs, *args, **kwargs):
        FancyArrowPatch.__init__(self, (0,0), (0,0), *args, **kwargs)
        self._verts3d = xs, ys, zs

    def draw(self, renderer):
        xs3d, ys3d, zs3d = self._verts3d
        xs, ys, zs = proj3d.proj_transform(xs3d, ys3d, zs3d, renderer.M)
        self.set_positions((xs[0],ys[0]),(xs[1],ys[1]))
        FancyArrowPatch.draw(self, renderer)

We played around with things quite a bit in class, so the function below has options for changing from surface to points and for changing urange (the range included in the patch)

def plot_surface(f,plotmode='surface',urange=2*np.pi):
    u1 = np.linspace(-urange,urange,100)
    u2 = np.linspace(-urange,urange,100)
    ax = plt.gca(projection='3d')
    U1, U2 = np.meshgrid(u1, u2)
    X,Y,Z = f(U1,U2)
    surf2 = ax.plot_surface(U1, U2, np.zeros_like(U1), color='grey', alpha=0.5)
    if plotmode == 'surface':
        surf = ax.plot_surface(X,Y,Z)
    elif plotmode == 'points':
        surf = ax.scatter(X,Y,Z)
    else:
        raise Exception('Unknown plot mode {p}'.format(p=plotmode))
    for (w1,w2) in itertools.permutations([-urange,-urange/2,0,urange/2,urange],2):
        x,y,z = f(w1,w2)
        a = Arrow3D([w1,x],[w2,y],[0,z], mutation_scale=20, 
                    lw=1, arrowstyle="-|>", color="r",
                   alpha=0.7,zorder=-1)
        ax.add_artist(a)
    plt.show()

And now let’s plot that same surface

def f(U1,U2):
    return U1+U2, np.sin(U1), np.cos(U1+U2)
plot_surface(f)

plot_surface(f,urange=2)

def f(U1,U2):
    return U1,np.log(U2),3*U1+5*U2
plot_surface(f)

/Users/mglerner/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:2: RuntimeWarning: invalid value encountered in log
  
/Users/mglerner/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:2: RuntimeWarning: divide by zero encountered in log
  
/Users/mglerner/anaconda3/lib/python3.6/site-packages/mpl_toolkits/mplot3d/proj3d.py:141: RuntimeWarning: invalid value encountered in true_divide
  txs, tys, tzs = vecw[0]/w, vecw[1]/w, vecw[2]/w

The curve

OK. A large portion of why we did this was because it was really not obvious to the class why mapped to a curve. We could all follow the mathematical arguments, but it didn’t really stick without a good visualization. So, let’s look:

so, what does the curve look like?

def f(U1,U2):
    return U1+U2, (U1+U2)**2, (U1+U2)**3
plot_surface(f, plotmode='points')

That’s a great start. We can see where representative points match, and it’s definitely useful to visualize things and “prove” to ourselves that you do indeed get a curve. It’s much more convincing when we make an interactive version (below). Before we do that, let’s also quickly check out a cylinder.

def f(U1,U2):
    return 2*np.cos(U1),2*np.sin(U1),U2
plot_surface(f, urange=np.pi)

In class, we played around with urange and the scale factors to make sure we understood what was going on. Speaking of which, Markus Deserno was kind enough to look at a preliminary version of this post, and he noticed that I was originally using urange=2*np.pi which got a “double covering” of the cylinder (i.e. the patch mapped two points to every point on the cylinder. Check it out:

plot_surface(f, urange=np.pi/2)
plot_surface(f, urange=np.pi)
plot_surface(f, urange=2*np.pi)

An interactive one

That was OK, and it convinced us that the maps really did go from, e.g., a plane to a cylinder. But (1) it still wasn’t quite so obvious exactly how/why it worked (2) it’s obvious that playing around with the input parameters can give a lot of insight.

Then I had the idea to make it interactive: you get a slider so you can watch the green surface interpolate linearly between the grey patch and the surface defined by . This is worth running interactively, and playing with, as it really cleared things up. If you run it locally and interactively, you get sliders that let you rotate the surface around in 3D, so you can look at it from all angles. If you’re runnign it locally, you wnat the interp step to be more like 0.01 and the degree step to be more like 1.

For notebooks to be viewable online, I’m JSAnimations with a coarser interp step and I’m skipping the controls for elevation and azimuth altogether.

** If you just want the results, skip the code here and look for the next pretty picture! **

def plot_surface_interpolate(f=f,plotmode='surface',urange=2*np.pi,interp=0.1,
                             elev=None,azim=None):
    u1 = np.linspace(-urange,urange,100)
    u2 = np.linspace(-urange,urange,100)
    #fig = plt.figure()
    ax = plt.gca(projection='3d')
    U1, U2 = np.meshgrid(u1, u2)
    X,Y,Z = f(U1,U2)
    U3 = np.zeros_like(Z)
    A = (1-interp)*U1 + interp*X
    B = (1-interp)*U2 + interp*Y
    C = (1-interp)*U3 + interp*Z
    surf2 = ax.plot_surface(U1, U2, np.zeros_like(U1), color='grey', alpha=0.5)
    if plotmode == 'surface':
        surf = ax.plot_surface(X,Y,Z,alpha=0.4,color='blue')
        surf3 = ax.plot_surface(A,B,C,color='green',zorder=-2,alpha=0.9)
    elif plotmode == 'points':
        surf = ax.scatter(X,Y,Z,alpha=0.4,color='blue')
        surf3 = ax.scatter(A,B,C,color='green',zorder=-2,alpha=0.9)
    else:
        raise Exception('Unknown plot mode {p}'.format(p=plotmode))
    
    for (w1,w2) in itertools.permutations([-urange,-urange/2,0,urange/2,urange],2):
        x,y,z = f(w1,w2)
        a = Arrow3D([w1,x],[w2,y],[0,z], mutation_scale=20, 
                    lw=1, arrowstyle="-|>", color="r",
                   alpha=0.7,zorder=-1)
        ax.add_artist(a)
    plt.title(f'interp = {interp}')        
    ax.view_init(elev=elev,azim=azim)
    plt.show()
def do_interact(f,interp_step=0.01,degree_step=1,urange=np.pi,azim=30,elev=30,):   
    def plot_this(f,interp=0.1,azim=azim,elev=elev,plotmode='surface',urange=np.pi):
        plot_surface_interpolate(f,interp=interp,azim=azim,elev=elev,plotmode=plotmode,urange=urange)
    interact(plot_this,interp=(0,1,interp_step),
             azim=(0,360,degree_step),
             elev=(0,360,degree_step),
             plotmode=['points','surface'],
             urange=fixed(urange),
             f=fixed(f));

def update_interp(i,f,surfs,ax,
                  U1,U2,U3,
                  X,Y,Z,
                  interpsteps,
                  elev,azim,urange):
    surf,surf2,surf3 = surfs
    ax.clear()
    interp = i/interpsteps
    A = (1-interp)*U1 + interp*X
    B = (1-interp)*U2 + interp*Y
    C = (1-interp)*U3 + interp*Z

    surf2 = ax.plot_surface(U1, U2, U3, color='grey', alpha=0.4)
    surf = ax.plot_surface(X,Y,Z,alpha=0.4,color='blue')
    surf3 = ax.plot_surface(A,B,C,color='green',zorder=-2,alpha=0.9)

    for (w1,w2) in itertools.permutations([-urange,-urange/2,0,urange/2,urange],2):
        x,y,z = f(w1,w2)
        a = Arrow3D([w1,x],[w2,y],[0,z], mutation_scale=20, 
                    lw=1, arrowstyle="-|>", color="r",
                   alpha=0.7,zorder=-1)
        ax.add_artist(a)

    plt.title(f'interp = {interp}')

    ax.view_init(elev=elev,azim=azim)
    return (surf,surf2,surf3)

def get_ani(f,urange=2*np.pi,interpsteps=10,elev=30,azim=30):
    u1 = np.linspace(-urange,urange,100)
    u2 = np.linspace(-urange,urange,100)
    U1, U2 = np.meshgrid(u1, u2)
    U3 = np.zeros_like(U1)

    X,Y,Z = f(U1,U2)

    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')

    surf2 = ax.plot_surface(U1,U2,U3, color='grey', alpha=0.5)
    surf = ax.plot_surface(U1,U2,U3,alpha=0.9,color='blue')
    surf3 = ax.plot_surface(U1,U2,U3,color='green',zorder=-2,alpha=0.9)
    surfs = (surf,surf2,surf3)

    ani = animation.FuncAnimation(fig, update_interp, interpsteps, fargs=(f,surfs,ax,U1,U2,U3,X,Y,Z,interpsteps,
                                                                         elev,azim,urange),
                                 interval=250, blit=False)
    return ani    

We can make this interactive in two ways. For live notebooks, we use interact and for the web we use JSAnimation. In both cases, once the boilerplate is done, we just need to define a function f and call either do_interact or get_ani, as seen below. To be clear, we expect a function f to be defined before each new call. f maps u1 and u2 into x, y, and z. See below for examples.

def f(X,Y):
    return 2*np.cos(X),2*np.sin(X),Y
do_interact(f)

def f(X,Y):
    return 2*np.cos(X),2*np.sin(X),Y
HTML(get_ani(f,interpsteps=10,urange=np.pi).to_jshtml())

def f(X,Y):
    return 2*np.cos(X),2*np.sin(X),Y
HTML(get_ani(f,urange=np.pi,elev=66,azim=30).to_jshtml())
#do_interact(f)

def f(X,Y):
    return X+Y, (X+Y)**2, (X+Y)**3
HTML(get_ani(f,urange=np.pi).to_jshtml())
#do_interact(f,urange=np.pi)

More examples

After that, we did a bunch of examples from the book, and from our own imagination. I don’t want this notebook to become too big, so I’m not going to make JSAnimations for all of them. Locally, I run them with interact … I’ve included JS animations below with a small number of frames so as to keep this post at least somewhat reasonably-sized, but you should (have I said this yet?) download the notebook and play with it interactively!

def f(U1,U2):
    return U1, U2, np.zeros_like(U2)
HTML(get_ani(f,urange=np.pi,elev=30,azim=113,interpsteps=5).to_jshtml())
#do_interact(f,urange=np.pi,elev=30,azim=113)

def f(U1,U2):
    return U1*np.cos(U2), U1*np.sin(U2), np.zeros_like(U2)
HTML(get_ani(f,urange=np.pi,elev=30,azim=113,interpsteps=5).to_jshtml())
#do_interact(f,urange=np.pi,elev=30,azim=113)

def f(U1,U2,r=2):
    return r*np.cos(U2)*np.cos(U1), r*np.cos(U2)*np.sin(U1), r*np.sin(U2)

HTML(get_ani(f,urange=np.pi,elev=30,azim=113,interpsteps=5).to_jshtml())
#do_interact(f,urange=np.pi,elev=30,azim=113)

def f(U1,U2,a=1):
    return U1*np.cos(U2), U1*np.sin(U2), a*U1
HTML(get_ani(f,urange=2,elev=30,azim=113,interpsteps=5).to_jshtml())
#do_interact(f,urange=np.pi,elev=30,azim=113)

def f(U1,U2):
    return 0, U1, U2
HTML(get_ani(f,urange=2,elev=30,azim=113,interpsteps=5).to_jshtml())
#do_interact(f,urange=np.pi,elev=30,azim=113)

def f(U1,U2):
    return U1+U2, U1+U2, U1
HTML(get_ani(f,urange=2,elev=30,azim=113,interpsteps=5).to_jshtml())
#do_interact(f,urange=np.pi,elev=30,azim=113)

def f(U1,U2):
    return np.cos(U1), np.sin(U1), U2
HTML(get_ani(f,urange=2,elev=30,azim=113,interpsteps=5).to_jshtml())
#do_interact(f,urange=np.pi,elev=30,azim=113)

def s(U1,U2):
    return U1+U2
def h1(s):
    return s
def h2(s):
    return s**2
def h3(s):
    return s

def f(U1,U2):
    S = s(U1,U2)
    return h1(S), h2(S), h3(S)
def f(U1,U2):
    return h1(U1), h2(U1), h3(U1) + U2
HTML(get_ani(f,urange=2,elev=30,azim=113,interpsteps=5).to_jshtml())
#do_interact(f,urange=np.pi,elev=30,azim=113)

def f(U1,U2):
    X = U1
    Y = U2
    def h(x,y):
        return 4+ np.cos(x) + np.sin(y)
    return X,Y,h(X,Y)
HTML(get_ani(f,urange=2,elev=30,azim=113,interpsteps=10).to_jshtml())
#do_interact(f,urange=np.pi,elev=30,azim=113)

Grabbing the site

Log in as an administrator, go to the Tools menu, click on Export, then download an Export File. Nice and easy. Thanks, WordPress!

Install Nikola

Note: pip doesn’t actually play nicely with the conda environments. It looks like it installs nikola’s stuff on top of everything else (e.g. the Nikola tools are available even when I’m not in the blog environment from below). That’s not great. I probably should use a virtualenv, but I don’t have time to learn what one of those is today.

I’m using Anaconda for my Python, and Nikola installs via pip, so I make a new environment just for the blog. Probably good practice anyway. Speaking of which, this is all Python 3. There’s some weird bug involved in having pip upgrade setuptools,

conda create --name blog
source activate blog
conda install setuptools
pip install --upgrade "Nikola[extras]"
mkdir blog
cd blog
mkdir posts
mkdir output

Now import the wordpress site. If I just say

nikola import_wordpress ~/Downloads/biophysicsandbeer.wordpress.2017-10-10.xml

I don’t actually get some of my content imported. In particular, the quoted Python code doesn’t show up. So,

nikola import_wordpress --download-auth=[my secret username]:[my secret password] --install-wordpress-compiler --transform-to-html ~/Downloads/biophysicsandbeer.wordpress.2017-10-10.xml

Tells it to be explicit about converting to html and downloading everything it can. It also produces some errors, specifically a bunch of lines like

[2017-10-11T19:49:10Z] WARNING: Nikola: Can't do a redirect for: 'https://mglerner.com/blog/?p=8/'

which are telling us that it can’t just spit out a new file in place of the old one because the question marks in the URL mean request data. We’ll deal with this in a minute.

We then fire up the site with

cd new_site
nikola build
nikola serve --browser

And it almost all works. Here are the missing things:

Comments are all missing.

1. For future stuff to work, I’ll get a Disqus ID and start using it. (biophysics-and-beer … check).
2. Add all of the old comments by hand. This isn’t taking a huge amount of time, because I don’t have a huge amount of comments, and I’m not going crazy with matching formatting. I’m just copying the comments div from the old blog. If you’re doing this to your own blog, don’t forget to get rid of the “response” part of the comments div. If I had it to do over, I’d write a script that did that for me, or I’d learn how to import comments into Disqus, but it didn’t take much time this way. On the plus side, I’m ditching some (not all) spam I had missed.

Cleaning up long posts

Some of the posts are really long. Like, to the point where the main page is something ridiculous like 80MB. I think WordPress deals with this automatically (only showing the first few paragraphs in the index), because I didn’t remember it being an issue. You can tell Nikola to only show a “teaser” by putting

INDEX_TEASERS = True

in conf.py

And then adding

<!-- more -->

to a Markdown cell in a Jupyter notebook, or directly to a Markdown post, or directly to an html post. Cool.

My pre-rendered Jupyter notebooks suck.

I used the pageview WordPress plugin (short code? I don’t know) to include pre-rendered Jupyter notebooks. That just breaks upon import. But the Nikloa posts are just straight HTML. I can just include the stuff directly.

That was pretty easy, and it worked, but it seemed lame and “Teasers” didn’t work when I stuck them in the middle of the pre-rendered HTML blocks. So, I switched everything over to Jupyter notebooks.

When I do that, I need to

• make sure I include comments explicitly in the Jupyter notebook. As mentioned above, I can just include the raw html for them. To do this in a Markdown cell in a Jupyter notebook, do something like

from IPython.core.display import HTML
HTML('''fancy pants html code for comments here''')

• I also need to remember to edit the meta data at the end of the imported Jupyter notebook so that Nikola gets the date right. You have to remember to do that after Nikola imports the post.

• I also need to remember to stick a Markdown cell near the top with a link to download the whole things as a Jupyter notebook directly. This part is pretty awesome: I can link directly to the Jupyter notebook on the Nikola site, rather than stuffing a copy in a gist, etc. Neat!

Getting a graphical header/theme fun time

I found the Nikola tutorial for theming … somewhat more than I was interested in. But, it had enough useful nuggets to figure out the basics. The basic idea is (1) put my header image somewhere (2) edit the header template. Let’s go.

First, I’m going to want to change the base_header.tmpl so I searched for themes that came with that on the Nikola Theme site. Of those, I like jidn the most. So,

nikola theme -i bootstrap3-gradients

Then edit conf.py to switch the theme to jidn and, as indicated in the text when you install the jidn theme, add a few lines to conf.py

THEME = "jidn"
...
GLOBAL_CONTEXT.update({
    "JIDN": {},  # Extra info about authors
    # "JIDN-theme": "theme-base-blue",
})
...

(Actually, as described in the jidn README.md, you can add more info to that dictionary, so I added a link to my twitter profile image, etc. Neat.)

The WordPress import stuck all of the versions of my header in files/blog/wp-content/wp-content/uploads/ and I’m just copying them to files to make them easy to find. Then I add the img tag right after the title in themes/jidn/templates/base_header.tmpl:

    <h2 id="brand" class="masthead-title">
      <a href="${abs_link(_link("root", None, lang))}" title="${blog_title}" rel="home">${blog_title}</a>
    </h2>
    <img src="/mglblogheader.png" />

I’m very far from up-to-date with modern web design, but that looks OK for now.

While I’m editing the templates, there’s a link to “blog” in the sidebar I don’t want, so I replaced it with a link to the about page.

Missing buttons

The code in post.tmpl looks like it should work, but the icons don’t show up. This means I’m missing the links to my twitter/github profile, the “share this on your favorite social media site” links, and the buttons that link to next/previous post.

It looks to me like it’s trying to grab stuff from FontAwesome, but I don’t see anywhere that the FontAwesome stuff is actually linked on my site. Following the FontAwesome directions,

• Download the font-awesome directory.
• Stick it in themes/jidn/assets/font-awesome
• Add the following to base.tmpl:

<link rel="stylesheet" href="path/to/font-awesome/css/font-awesome.min.css">

… Now, I added it right before </head>, which looks ot me like it’s OK, but it’s right after a place where it says “don’t change this block” so it’s possible that I screwed something up.

That fixes the lack of next/previous/social buttons.

Back to the redirects

Those redirects are kind of a pain. That’s Nikola saying it can’t just spit out a new file in place of the old file because the question marks mean request data. Thanks, wordpress! I wanted simple stuff like an .htaccess file saying

Redirect /blog/page8 /blog/new-and-fancy.html

but no … Now I need to look at a StackOverflow answer and remember why I stopped being a web developer years ago. Here’s my new .htaccess file

RewriteEngine on
RewriteBase /

RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /blog/\?p=5/ [NC]
RewriteRule ^ /blog/posts/the-blogging-begins.html [L,R=301]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /blog/\?p=5 [NC]
RewriteRule ^ /blog/posts/the-blogging-begins.html [L,R=301]

to handle people accessing each page both with and without a trailing slash. Now, where should I point the redirects?

Static site means why host it myself?

It made sense to host my own WordPress site. But I’m not sure I need to host my own static site. In fact, life would probably be easier if I just used github pages. Cool. Github has instructions that basically boil down to “set up a repo called mglerner.github.io and do some special things” … and Nikola means I don’t have to actually know what those special things are! After initializing my Nikola blog directory as a github repo and doing

git remote add origin git@github.com:mglerner/mglerner.github.io.git

I can make Nikola rebuild things like github wants, add, commit and push everything automatically with

nikola github_deploy

So, after having done that, I just make the .htaccess file listed above point everything from my old site to my fancy new github site.

… and it’s up and running!

Making it possible to post with Jupyter notebooks

This is documented in a few places, but sometimes things change, so you’re probably better off just looking at the Nikola site directly. At the moment, you need to add the last line here to each of POSTS and PAGES in /blog/conf.py, so that Nikola will recognize Jupyter notebooks as … you know … both posts and pages. You also need to add the trailing line in COMPILERS.

POSTS = (
    ("posts/*.rst", "posts", "post.tmpl"),
    ("posts/*.txt", "posts", "post.tmpl"),
    ("posts/*.md", "posts", "post.tmpl"),
    ("posts/*.html", "posts", "post.tmpl"),
    ("posts/*.ipynb", "posts", "post.tmpl"),
)

PAGES = (
    ("pages/*.rst", "pages", "story.tmpl"),
    ("pages/*.txt", "pages", "story.tmpl"),
    ("pages/*.md", "pages", "story.tmpl"),
    ("pages/*.html", "pages", "story.tmpl"),
    ("pages/*.ipynb", "pages", "story.tmpl"),
    )

COMPILERS = {
    "rest": ('.txt', '.rst'),
    "markdown": ('.md', '.mdown', '.markdown'),
    "html": ('.html', '.htm'),
    "ipynb": ('.ipynb',),
}

Making posts

Now comes the whole point: making new posts. Blog posts come with meta data (when it was created, etc.) that normally gets automatically handled when you create a new post via nikola new_post -e. If you want to start with other formats (e.g. Markdown or Jupyter notebooks), you just stick the relevant file in /posts and import it via nikola new_post -i.

I took the notes for this migration in Markdown, so I just stick that file in /posts/BlogMigration.md run nikola new_post -i posts/BlogMigration …

Or, at least, that’s how it should work. I’m a little disappointed to find that the new post import keeps breaking, saying it can’t find metadata like the date. It’s an easy workaround for markdown: I just make a new post, then copy over my markdown. Looking at the results, I find that Nikola wants the metadata imbedded in the markdown file. So, I should have started the markdown file with

<!--
.. title: Switching to Nikloa for Jupyter Notebooks and a static site
.. slug: switching-to-nikloa-for-jupyter-notebooks-and-a-static-site
.. date: 2017-10-12 13:50:23 UTC
.. tags: 
.. category: 
.. link: 
.. description: 
.. type: text
-->

Cool. I tested, and I can indeed just stick that at the top of my markdown file (which I’m storing in a folder called drafts) and then import. Almost. Of all things, it sticks that metadata in twice. So, for now, the clear answer is just to make a blank post and then copy in my markdown. Easy enough.

Meanwhile, importing Jupyter notebooks seems to just work!

then nikola build, then fire up a test server to check it out with nikola serve --browser, verify that all looks awesome, and push it to the blog with nikola github_deploy!

TODO

• The text is too big. I can apparently change that in jidn’s custom.css but I don’t quite get how.

Jupyter issues

The conversion of Jupyter notebooks doesn’t look perfect yet:

• Inline math (starting with a single dollar sign) doesn’t always get converted. Starting with two dollar signs does seem to consistently work.

• For JSAnimations, the “animation bar” doesn’t have images on the buttons. UPDATE: That seems to have magically fixed itself. The FontAwesome thing may have been the key, but I’m going full cargo-cult mentality here and just saying it worked.

Updated Quick Notes

To create a new environment

conda create -n blog python=3
conda activate blog

Then double-check to make sure you’re using the pip from the conda environment. Then

pip install --upgrade "Nikola[extras]"

Then

nikola build
nikola new_post -i drafts/thing.md
nikola serve --browser
nikola github_deploy

Writing a faster mean-squared-displacement function

I’m trying to calculate the Mean Squared Displacement (MSD) of a particle trajectory. In reality, I have an array of positions for numtrj particles and numpts timepoints and dim dimensions: pos[numtrj, numpts, dim]. I think my question has the same answer if I just have the x trajectory of a single particle, though.

In case you haven’t done MSD calculations before, there’s one cute way in which you get extra information out of a trajectory. Say I have just five time points, and my positions are

In [83]: x

Out[83]: array([ 0.        ,  1.74528704,  1.59639865,  2.59976219,  3.70852457])

You could just get squared displacement by looking at x**2. However, you could also say that you have 4 different values for the displacement at one timestep:

x[1] - x[0], x[2] - x[1], x[3] - x[2], x[4] - x[3]

Similarly, three values for displacement at two timesteps: x[2:] - x[:-2]. Etc. So, the way I’m calculating MSD at the moment is:

def msd_1d(x):
    result = np.zeros_like(x)
    for i in range(1,len(x)):
        result[i] = np.average((x[i:] - x[:-i])**2)
    return result

or

def get_msd_traj(pos):
    result = np.zeros_like(pos)
    for i in range(1,pos.shape[1]):
        result[:,i,:] = np.average((pos[:,i:,:] - pos[:,:-i,:])**2,axis=1)
    return result

(side note: often the data comes indexed like pos[numpts, numtrj, dim] for molecular dynamics trajectories, but that doesn’t change anything here)

So, I asked Joshua Adelman if he had any quick thoughts.

Josh cleared up some basic misconceptions for me and gave a couple of suggestions:

import numpy as np
from numba import jit

def msd_1d(x):
    result = np.zeros_like(x)
    for i in range(1,len(x)):
        result[i] = np.average((x[i:] - x[:-i])**2)
    return result

@jit(nopython = True)
def msd_1d_nb1(x):
    result = np.zeros_like(x)
    for delta in range(1,len(x)):
        thisresult = 0
        for i in range(delta,len(x)):
            thisresult += (x[i] - x[i-delta])**2
        result[delta] = thisresult / (len(x) - delta)
    return result
    
@jit(nopython = True)
def msd_1d_nb2(x):
    result = np.zeros_like(x)
    for delta in range(1,len(x)):
        for i in range(delta,len(x)):
            result[delta] += (x[i] - x[i-delta])**2
        result[delta] = result[delta] / (len(x) - delta)
    return result

%timeit msd_1d(np.random.randn(10))
%timeit msd_1d_nb1(np.random.randn(10))
%timeit msd_1d_nb2(np.random.randn(10))

The slowest run took 4.14 times longer than the fastest. This could mean that an intermediate result is being cached 
10000 loops, best of 3: 114 µs per loop
The slowest run took 11972.39 times longer than the fastest. This could mean that an intermediate result is being cached 
100000 loops, best of 3: 7.02 µs per loop
The slowest run took 14872.73 times longer than the fastest. This could mean that an intermediate result is being cached 
100000 loops, best of 3: 6.94 µs per loop

%timeit msd_1d(np.random.randn(100))
%timeit msd_1d_nb1(np.random.randn(100))
%timeit msd_1d_nb2(np.random.randn(100))

1000 loops, best of 3: 1.19 ms per loop
100000 loops, best of 3: 15.8 µs per loop
100000 loops, best of 3: 18.5 µs per loop

%timeit msd_1d(np.random.randn(1000))
%timeit msd_1d_nb1(np.random.randn(1000))
%timeit msd_1d_nb2(np.random.randn(1000))

100 loops, best of 3: 13.3 ms per loop
1000 loops, best of 3: 540 µs per loop
1000 loops, best of 3: 594 µs per loop

def get_msd_traj(pos):
    result = np.zeros_like(pos)
    for i in range(1,pos.shape[1]):
        result[:,i,:] = np.average((pos[:,i:,:] - pos[:,:-i,:])**2,axis=1)
    return result

@jit(nopython = True)
def get_msd_traj_nb1(pos):
    result = np.zeros_like(pos)
    deltastop = pos.shape[1]
    for traj in range(pos.shape[0]):
        for dim in range(pos.shape[2]):
            for delta in range(1,deltastop):
                thisresult = 0
                for i in range(delta,deltastop):
                    thisresult += (pos[traj,i,dim] - pos[traj,i-delta,dim])**2
                result[traj,delta,dim] = thisresult / (deltastop - delta)
    return result

a = np.random.randn(5,3,2)
np.all(get_msd_traj(a) == get_msd_traj_nb1(a))

True

%timeit get_msd_traj(np.random.randn(10,10,2))
%timeit get_msd_traj_nb1(np.random.randn(10,10,2))

1000 loops, best of 3: 200 µs per loop
100000 loops, best of 3: 14.6 µs per loop

%timeit get_msd_traj(np.random.randn(100,100,2))
%timeit get_msd_traj_nb1(np.random.randn(100,100,2))

10 loops, best of 3: 12.3 ms per loop
1000 loops, best of 3: 1.84 ms per loop

%timeit get_msd_traj(np.random.randn(512,2001,2))
%timeit get_msd_traj_nb1(np.random.randn(512,2001,2))

1 loops, best of 3: 27.8 s per loop
1 loops, best of 3: 2.47 s per loop

MMMC the 2015 update

See the original blog post for details and history. Here’s the short story: in my Statistical and Thermal Physics class, we want to use Monte Carlo simulations to generate brackets for March Madness. There are at least two obvious ways to go about this:

1. Make some function that tells us the chance that team A beats team B, then flip coins for each matchup. That gets you one bracket. Repeat 100,000 times, collect statistics. This is the way Nate Silver’s 538.com handles simulations for basketball, elections, etc, and I should probably implement it (note to self/motivated students: it’s as easy as just generating 100,000 new brackets at a given temperature).

2. Generate one bracket, then do a Monte-Carlo walk through bracket space. This is tougher. We have to figure out how to make a move in bracket space, which is part of the fun of Monte Carlo simulations in general. To see how this is done, check out the code in Bracket.swap and Brackets.simulate.

As you can tell, we take option 2 above. I’ve made things a bit nicer from a user standpoint this year; here’s a walkthrough. First, load up our standard IPython setup

from __future__ import division
%matplotlib inline
import seaborn as sns
sns.set_style('darkgrid')
import numpy as np
from matplotlib import pyplot as plt
from IPython.display import HTML

import MarchMadnessMonteCarlo as MMMC
teams = MMMC.teams['south']
b = MMMC.Bracket(teams=teams,T=0.5)
print b

Duke (1)                                                     
Robert Morris (16)        Duk (1)                            
San Diego St. (8)                                            
St. John's (9)            San (8)  Duk (1)                   
Utah (5)                                                     
Stephen F. Austin (12)    Ste (12)                           
Georgetown (4)                                               
Eastern Washington (13)   Geo (4)  Ste (12) Ste (12)         
SMU (6)                   UCL (11) UCL (11) UCL (11) Ste (12)
UCLA (11)                                                    
Iowa St. (3)              Iow (3)                            
UAB (14)                                                     
Iowa (7)                  Iow (7)  Iow (7)                   
Davidson (10)                                                
Gonzaga (2)               Gon (2)                            
North Dakota St. (15)                                        
Total bracket energy: -17.4533174222

sr = MMMC.simulate(1000,['Kentucky','Wisconsin','Villanova','Duke'],0.5)
MMMC.showstats(sr,newfig=True)

Lowest energy bracket
Kentucky (1)                               
Wisconsin (1)             Ken (1)          
Villanova (1)             Vil (1)  Ken (1) 
Duke (1)                                   
Total bracket energy: -3.05919025906

Most common bracket (159)
Kentucky (1)                               
Wisconsin (1)             Ken (1)          
Villanova (1)             Duk (1)  Ken (1) 
Duke (1)                                   
Total bracket energy: -3.04122415391

What should our temperature be?

If we had chosen option 1 at the top, we’d just flip coins with a given probability of winning. Here (and this may be a questionable decision), we need to set an overall temperature for our simulation. Intuitively, the higher the temperature, the closer we come to a random outcome. The lower the temperature, the closer we come to a “best seed always wins” bracket. If we’re going to make sense of temperature, we should pick a reasonable energy function.

We can use KenPom’s log5 defined as

def log5_energy_game(winner, loser):
    A,B = strength[winner],strength[loser]
    # see http://207.56.97.150/articles/playoff2002.htm
    win_pct = (A-A*B)/(A+B-2*A*B)
    return -win_pct

Conveniently, that’s coded up for you in MarchMadnessMonteCarlo.examples.

Later on, we could make a fancy energy function with, e.g., a weighted average of KenPom, Jeff Sagarin, and the NCAA rankings.

import MarchMadnessMonteCarlo.examples
MMMC.set_energy_function(MarchMadnessMonteCarlo.examples.log5_energy_game)

Now, what should our actual temperature be? Historically, we know that an 8 seed vs. a 9 seed should essentially be a tossup. So, as a proxy here, we could just look at the chance of an 8 seed winning over a range of temperatures, and pick the point where it’s pretty close to 0.5.

def winpct8(team8,team9,T,numtrials=10000):
    results = [MMMC.playgame(team8,team9,T)[0] == team8 for i in range(numtrials)]
    return np.average(results)
def plotwins(team8,team9,numtrials=10000):
    Ts = np.linspace(0,3,100)
    pct = [winpct8(team8,team9,T,numtrials) for T in Ts]
    plt.plot(Ts,pct,label='{t1} vs. {t2}'.format(t1=team8,t2=team9))
    plt.xlabel('T')
    plt.ylabel('winpct')

We’ll look at it for all four of the 8 vs. 9 matchups

plotwins('Cincinnati','Purdue')
plotwins('Oregon','Oklahoma St.')
plotwins('North Carolina St.','LSU')
plotwins('San Diego St.',"St. John's")
plt.legend()

A couple of things jump out. First, the green line: KenPom has Oregon ranked below Oklahoma State, so the curve approaches 50% from below instead of from above.

Second, just eyeballing that, it looks like we could make a legitimate argument for almost anything between T=1.0 and 2.0.

What does that do to “clear” matchups? Well, Kentucky is a big favorite over Kansas with this model:

MarchMadnessMonteCarlo.examples.log5_energy_game('Kentucky','Kansas')

-0.8176307825952608

plotwins('Kentucky','Kansas')

If I pick T=1.5, I get Kentucky, the favorite, winning 60% of the time. You can feel free to choose a “saner” temperature, but I’m rooting for KU here!

results = MMMC.runbracket2(ntrials1=10000,ntrials2=1000,T=1.5)

YOUR LOWEST ENERGY BRACKETS
LOWEST ENERGY BRACKET FOR REGION midwest
Kentucky (1)                                                 
Hampton (16)              Ken (1)                            
Cincinnati (8)                                               
Purdue (9)                Cin (8)  Ken (1)                   
West Virginia (5)                                            
Buffalo (12)              Wes (5)                            
Maryland (4)                                                 
Valparaiso (13)           Mar (4)  Wes (5)  Ken (1)          
Butler (6)                Tex (11) Not (3)  Not (3)  Ken (1) 
Texas (11)                                                   
Notre Dame (3)            Not (3)                            
Northeastern (14)                                            
Wichita St. (7)           Wic (7)  Kan (2)                   
Indiana (10)                                                 
Kansas (2)                Kan (2)                            
New Mexico St. (15)                                          
Total bracket energy: -10.5524230049


LOWEST ENERGY BRACKET FOR REGION west
Wisconsin (1)                                                
Coastal Carolina (16)     Wis (1)                            
Oregon (8)                                                   
Oklahoma St. (9)          Okl (9)  Wis (1)                   
Arkansas (5)                                                 
Wofford (12)              Ark (5)                            
North Carolina (4)                                           
Harvard (13)              Nor (4)  Nor (4)  Wis (1)          
Xavier (6)                Xav (6)  Bay (3)  Ari (2)  Ari (2) 
Mississippi (11)                                             
Baylor (3)                Bay (3)                            
Georgia St. (14)                                             
VCU (7)                   Ohi (10) Ari (2)                   
Ohio St. (10)                                                
Arizona (2)               Ari (2)                            
Texas Southern (15)                                          
Total bracket energy: -10.8592988759


LOWEST ENERGY BRACKET FOR REGION south
Duke (1)                                                     
Robert Morris (16)        Duk (1)                            
San Diego St. (8)                                            
St. John's (9)            San (8)  Duk (1)                   
Utah (5)                                                     
Stephen F. Austin (12)    Uta (5)                            
Georgetown (4)                                               
Eastern Washington (13)   Geo (4)  Uta (5)  Duk (1)          
SMU (6)                   SMU (6)  Iow (3)  Gon (2)  Gon (2) 
UCLA (11)                                                    
Iowa St. (3)              Iow (3)                            
UAB (14)                                                     
Iowa (7)                  Iow (7)  Gon (2)                   
Davidson (10)                                                
Gonzaga (2)               Gon (2)                            
North Dakota St. (15)                                        
Total bracket energy: -10.5015086048


LOWEST ENERGY BRACKET FOR REGION east
Villanova (1)                                                
Lafayette (16)            Vil (1)                            
North Carolina St. (8)                                       
LSU (9)                   Nor (8)  Vil (1)                   
Northern Iowa (5)                                            
Wyoming (12)              Nor (5)                            
Louisville (4)                                               
UC Irvine (13)            Lou (4)  Nor (5)  Vil (1)          
Providence (6)            Pro (6)  Okl (3)  Vir (2)  Vir (2) 
Dayton (11)                                                  
Oklahoma (3)              Okl (3)                            
Albany (14)                                                  
Michigan St. (7)          Mic (7)  Vir (2)                   
Georgia (10)                                                 
Virginia (2)              Vir (2)                            
Belmont (15)                                                 
Total bracket energy: -10.8839299775


LOWEST ENERGY BRACKET FOR FINAL FOUR
Kentucky (1)                               
Wisconsin (1)             Ken (1)          
Utah (5)                  Vir (2)  Ken (1) 
Virginia (2)                               
Total bracket energy: -1.95692907302

YOUR MOST COMMON BRACKETS
MOST COMMON BRACKET FOR REGION midwest
Kentucky (1)                                                 
Hampton (16)              Ken (1)                            
Cincinnati (8)                                               
Purdue (9)                Pur (9)  Ken (1)                   
West Virginia (5)                                            
Buffalo (12)              Wes (5)                            
Maryland (4)                                                 
Valparaiso (13)           Mar (4)  Mar (4)  Ken (1)          
Butler (6)                Tex (11) Not (3)  Kan (2)  Ken (1) 
Texas (11)                                                   
Notre Dame (3)            Not (3)                            
Northeastern (14)                                            
Wichita St. (7)           Wic (7)  Kan (2)                   
Indiana (10)                                                 
Kansas (2)                Kan (2)                            
New Mexico St. (15)                                          
Total bracket energy: -10.3577160931

number of times this bracket happened: 11


MOST COMMON BRACKET FOR REGION west
Wisconsin (1)                                                
Coastal Carolina (16)     Wis (1)                            
Oregon (8)                                                   
Oklahoma St. (9)          Ore (8)  Wis (1)                   
Arkansas (5)                                                 
Wofford (12)              Ark (5)                            
North Carolina (4)                                           
Harvard (13)              Nor (4)  Nor (4)  Wis (1)          
Xavier (6)                Mis (11) Mis (11) Ohi (10) Wis (1) 
Mississippi (11)                                             
Baylor (3)                Geo (14)                           
Georgia St. (14)                                             
VCU (7)                   Ohi (10) Ohi (10)                  
Ohio St. (10)                                                
Arizona (2)               Tex (15)                           
Texas Southern (15)                                          
Total bracket energy: -9.30614882152

number of times this bracket happened: 13


MOST COMMON BRACKET FOR REGION south
Duke (1)                                                     
Robert Morris (16)        Duk (1)                            
San Diego St. (8)                                            
St. John's (9)            St. (9)  Duk (1)                   
Utah (5)                                                     
Stephen F. Austin (12)    Uta (5)                            
Georgetown (4)                                               
Eastern Washington (13)   Geo (4)  Uta (5)  Uta (5)          
SMU (6)                   UCL (11) UCL (11) Gon (2)  Uta (5) 
UCLA (11)                                                    
Iowa St. (3)              Iow (3)                            
UAB (14)                                                     
Iowa (7)                  Dav (10) Gon (2)                   
Davidson (10)                                                
Gonzaga (2)               Gon (2)                            
North Dakota St. (15)                                        
Total bracket energy: -9.78926932204

number of times this bracket happened: 9


MOST COMMON BRACKET FOR REGION east
Villanova (1)                                                
Lafayette (16)            Vil (1)                            
North Carolina St. (8)                                       
LSU (9)                   LSU (9)  Vil (1)                   
Northern Iowa (5)                                            
Wyoming (12)              Nor (5)                            
Louisville (4)                                               
UC Irvine (13)            Lou (4)  Lou (4)  Vil (1)          
Providence (6)            Day (11) Okl (3)  Vir (2)  Vir (2) 
Dayton (11)                                                  
Oklahoma (3)              Okl (3)                            
Albany (14)                                                  
Michigan St. (7)          Geo (10) Vir (2)                   
Georgia (10)                                                 
Virginia (2)              Vir (2)                            
Belmont (15)                                                 
Total bracket energy: -10.5204279257

number of times this bracket happened: 11


MOST COMMON BRACKET FOR FINAL FOUR
Kentucky (1)                               
Wisconsin (1)             Ken (1)          
Utah (5)                  Uta (5)  Ken (1) 
Virginia (2)                               
Total bracket energy: -1.78538460124

number of times this bracket happened: 176

# Basic visualization
for region in 'midwest west east south'.split():
    MMMC.showstats(results[region],description=region,newfig=True)

h = HTML(MMMC.maketable(results))
h

results = MMMC.runbracket1(ntrials=10000,T=1.5)

Lowest energy bracket
Kentucky (1)                                                                   
Hampton (16)              Ken (1)                                              
Cincinnati (8)                                                                 
Purdue (9)                Cin (8)  Ken (1)                                     
West Virginia (5)                                                              
Buffalo (12)              Wes (5)                                              
Maryland (4)                                                                   
Valparaiso (13)           Mar (4)  Wes (5)  Ken (1)                            
Butler (6)                                                                     
Texas (11)                Tex (11)                                             
Notre Dame (3)                                                                 
Northeastern (14)         Not (3)  Not (3)                                     
Wichita St. (7)                                                                
Indiana (10)              Wic (7)                                              
Kansas (2)                                                                     
New Mexico St. (15)       Kan (2)  Kan (2)  Not (3)  Ken (1)                   
Duke (1)                                                                       
Robert Morris (16)        Duk (1)                                              
San Diego St. (8)                                                              
St. John's (9)            San (8)  Duk (1)                                     
Utah (5)                                                                       
Stephen F. Austin (12)    Uta (5)                                              
Georgetown (4)                                                                 
Eastern Washington (13)   Geo (4)  Uta (5)  Duk (1)                            
SMU (6)                                                                        
UCLA (11)                 SMU (6)                                              
Iowa St. (3)                                                                   
UAB (14)                  Iow (3)  Iow (3)                                     
Iowa (7)                                                                       
Davidson (10)             Iow (7)                                              
Gonzaga (2)                                                                    
North Dakota St. (15)     Gon (2)  Gon (2)  Gon (2)  Gon (2)  Ken (1)          
Wisconsin (1)             Wis (1)  Wis (1)  Wis (1)  Ari (2)  Ari (2)  Ken (1) 
Coastal Carolina (16)                                                          
Oregon (8)                Okl (9)                                              
Oklahoma St. (9)                                                               
Arkansas (5)              Ark (5)  Nor (4)                                     
Wofford (12)                                                                   
North Carolina (4)        Nor (4)                                              
Harvard (13)                                                                   
Xavier (6)                Xav (6)  Bay (3)  Ari (2)                            
Mississippi (11)                                                               
Baylor (3)                Bay (3)                                              
Georgia St. (14)                                                               
VCU (7)                   Ohi (10) Ari (2)                                     
Ohio St. (10)                                                                  
Arizona (2)               Ari (2)                                              
Texas Southern (15)                                                            
Villanova (1)             Vil (1)  Vil (1)  Vil (1)  Vir (2)                   
Lafayette (16)                                                                 
North Carolina St. (8)    Nor (8)                                              
LSU (9)                                                                        
Northern Iowa (5)         Nor (5)  Nor (5)                                     
Wyoming (12)                                                                   
Louisville (4)            Lou (4)                                              
UC Irvine (13)                                                                 
Providence (6)            Pro (6)  Okl (3)  Vir (2)                            
Dayton (11)                                                                    
Oklahoma (3)              Okl (3)                                              
Albany (14)                                                                    
Michigan St. (7)          Mic (7)  Vir (2)                                     
Georgia (10)                                                                   
Virginia (2)              Vir (2)                                              
Belmont (15)                                                                   
Total bracket energy: -44.697443432

Most common bracket (7)
Kentucky (1)                                                                   
Hampton (16)              Ham (16)                                             
Cincinnati (8)                                                                 
Purdue (9)                Cin (8)  Cin (8)                                     
West Virginia (5)                                                              
Buffalo (12)              Wes (5)                                              
Maryland (4)                                                                   
Valparaiso (13)           Mar (4)  Wes (5)  Cin (8)                            
Butler (6)                                                                     
Texas (11)                But (6)                                              
Notre Dame (3)                                                                 
Northeastern (14)         Nor (14) But (6)                                     
Wichita St. (7)                                                                
Indiana (10)              Wic (7)                                              
Kansas (2)                                                                     
New Mexico St. (15)       Kan (2)  Kan (2)  But (6)  But (6)                   
Duke (1)                                                                       
Robert Morris (16)        Rob (16)                                             
San Diego St. (8)                                                              
St. John's (9)            San (8)  San (8)                                     
Utah (5)                                                                       
Stephen F. Austin (12)    Uta (5)                                              
Georgetown (4)                                                                 
Eastern Washington (13)   Geo (4)  Geo (4)  Geo (4)                            
SMU (6)                                                                        
UCLA (11)                 SMU (6)                                              
Iowa St. (3)                                                                   
UAB (14)                  Iow (3)  Iow (3)                                     
Iowa (7)                                                                       
Davidson (10)             Iow (7)                                              
Gonzaga (2)                                                                    
North Dakota St. (15)     Gon (2)  Gon (2)  Gon (2)  Gon (2)  Gon (2)          
Wisconsin (1)             Wis (1)  Ore (8)  Ore (8)  Ari (2)  Ari (2)  Ari (2) 
Coastal Carolina (16)                                                          
Oregon (8)                Ore (8)                                              
Oklahoma St. (9)                                                               
Arkansas (5)              Ark (5)  Nor (4)                                     
Wofford (12)                                                                   
North Carolina (4)        Nor (4)                                              
Harvard (13)                                                                   
Xavier (6)                Xav (6)  Xav (6)  Ari (2)                            
Mississippi (11)                                                               
Baylor (3)                Bay (3)                                              
Georgia St. (14)                                                               
VCU (7)                   Ohi (10) Ari (2)                                     
Ohio St. (10)                                                                  
Arizona (2)               Ari (2)                                              
Texas Southern (15)                                                            
Villanova (1)             Laf (16) Nor (8)  Nor (5)  Mic (7)                   
Lafayette (16)                                                                 
North Carolina St. (8)    Nor (8)                                              
LSU (9)                                                                        
Northern Iowa (5)         Nor (5)  Nor (5)                                     
Wyoming (12)                                                                   
Louisville (4)            UC  (13)                                             
UC Irvine (13)                                                                 
Providence (6)            Pro (6)  Pro (6)  Mic (7)                            
Dayton (11)                                                                    
Oklahoma (3)              Okl (3)                                              
Albany (14)                                                                    
Michigan St. (7)          Mic (7)  Mic (7)                                     
Georgia (10)                                                                   
Virginia (2)              Bel (15)                                             
Belmont (15)                                                                   
Total bracket energy: -38.100050778

h = HTML(MMMC.maketable(results))
h

results = MMMC.runbracket1(ntrials=100000,T=1.5)

Lowest energy bracket
Kentucky (1)                                                                   
Hampton (16)              Ken (1)                                              
Cincinnati (8)                                                                 
Purdue (9)                Cin (8)  Ken (1)                                     
West Virginia (5)                                                              
Buffalo (12)              Wes (5)                                              
Maryland (4)                                                                   
Valparaiso (13)           Mar (4)  Wes (5)  Ken (1)                            
Butler (6)                                                                     
Texas (11)                Tex (11)                                             
Notre Dame (3)                                                                 
Northeastern (14)         Not (3)  Not (3)                                     
Wichita St. (7)                                                                
Indiana (10)              Wic (7)                                              
Kansas (2)                                                                     
New Mexico St. (15)       Kan (2)  Kan (2)  Not (3)  Ken (1)                   
Duke (1)                                                                       
Robert Morris (16)        Duk (1)                                              
San Diego St. (8)                                                              
St. John's (9)            San (8)  Duk (1)                                     
Utah (5)                                                                       
Stephen F. Austin (12)    Uta (5)                                              
Georgetown (4)                                                                 
Eastern Washington (13)   Geo (4)  Uta (5)  Duk (1)                            
SMU (6)                                                                        
UCLA (11)                 SMU (6)                                              
Iowa St. (3)                                                                   
UAB (14)                  Iow (3)  Iow (3)                                     
Iowa (7)                                                                       
Davidson (10)             Iow (7)                                              
Gonzaga (2)                                                                    
North Dakota St. (15)     Gon (2)  Gon (2)  Gon (2)  Gon (2)  Ken (1)          
Wisconsin (1)             Wis (1)  Wis (1)  Wis (1)  Ari (2)  Ari (2)  Ken (1) 
Coastal Carolina (16)                                                          
Oregon (8)                Okl (9)                                              
Oklahoma St. (9)                                                               
Arkansas (5)              Ark (5)  Nor (4)                                     
Wofford (12)                                                                   
North Carolina (4)        Nor (4)                                              
Harvard (13)                                                                   
Xavier (6)                Xav (6)  Bay (3)  Ari (2)                            
Mississippi (11)                                                               
Baylor (3)                Bay (3)                                              
Georgia St. (14)                                                               
VCU (7)                   Ohi (10) Ari (2)                                     
Ohio St. (10)                                                                  
Arizona (2)               Ari (2)                                              
Texas Southern (15)                                                            
Villanova (1)             Vil (1)  Vil (1)  Vil (1)  Vir (2)                   
Lafayette (16)                                                                 
North Carolina St. (8)    Nor (8)                                              
LSU (9)                                                                        
Northern Iowa (5)         Nor (5)  Nor (5)                                     
Wyoming (12)                                                                   
Louisville (4)            Lou (4)                                              
UC Irvine (13)                                                                 
Providence (6)            Pro (6)  Okl (3)  Vir (2)                            
Dayton (11)                                                                    
Oklahoma (3)              Okl (3)                                              
Albany (14)                                                                    
Michigan St. (7)          Mic (7)  Vir (2)                                     
Georgia (10)                                                                   
Virginia (2)              Vir (2)                                              
Belmont (15)                                                                   
Total bracket energy: -44.697443432

Most common bracket (9)
Kentucky (1)                                                                   
Hampton (16)              Ken (1)                                              
Cincinnati (8)                                                                 
Purdue (9)                Cin (8)  Ken (1)                                     
West Virginia (5)                                                              
Buffalo (12)              Buf (12)                                             
Maryland (4)                                                                   
Valparaiso (13)           Val (13) Buf (12) Ken (1)                            
Butler (6)                                                                     
Texas (11)                Tex (11)                                             
Notre Dame (3)                                                                 
Northeastern (14)         Nor (14) Nor (14)                                    
Wichita St. (7)                                                                
Indiana (10)              Wic (7)                                              
Kansas (2)                                                                     
New Mexico St. (15)       Kan (2)  Wic (7)  Wic (7)  Ken (1)                   
Duke (1)                                                                       
Robert Morris (16)        Rob (16)                                             
San Diego St. (8)                                                              
St. John's (9)            St. (9)  St. (9)                                     
Utah (5)                                                                       
Stephen F. Austin (12)    Ste (12)                                             
Georgetown (4)                                                                 
Eastern Washington (13)   Eas (13) Eas (13) St. (9)                            
SMU (6)                                                                        
UCLA (11)                 SMU (6)                                              
Iowa St. (3)                                                                   
UAB (14)                  Iow (3)  Iow (3)                                     
Iowa (7)                                                                       
Davidson (10)             Dav (10)                                             
Gonzaga (2)                                                                    
North Dakota St. (15)     Nor (15) Dav (10) Iow (3)  Iow (3)  Ken (1)          
Wisconsin (1)             Wis (1)  Wis (1)  Ark (5)  Ark (5)  Vir (2)  Vir (2) 
Coastal Carolina (16)                                                          
Oregon (8)                Okl (9)                                              
Oklahoma St. (9)                                                               
Arkansas (5)              Ark (5)  Ark (5)                                     
Wofford (12)                                                                   
North Carolina (4)        Nor (4)                                              
Harvard (13)                                                                   
Xavier (6)                Xav (6)  Bay (3)  Bay (3)                            
Mississippi (11)                                                               
Baylor (3)                Bay (3)                                              
Georgia St. (14)                                                               
VCU (7)                   Ohi (10) Ohi (10)                                    
Ohio St. (10)                                                                  
Arizona (2)               Ari (2)                                              
Texas Southern (15)                                                            
Villanova (1)             Vil (1)  Vil (1)  Lou (4)  Vir (2)                   
Lafayette (16)                                                                 
North Carolina St. (8)    LSU (9)                                              
LSU (9)                                                                        
Northern Iowa (5)         Nor (5)  Lou (4)                                     
Wyoming (12)                                                                   
Louisville (4)            Lou (4)                                              
UC Irvine (13)                                                                 
Providence (6)            Day (11) Alb (14) Vir (2)                            
Dayton (11)                                                                    
Oklahoma (3)              Alb (14)                                             
Albany (14)                                                                    
Michigan St. (7)          Mic (7)  Vir (2)                                     
Georgia (10)                                                                   
Virginia (2)              Vir (2)                                              
Belmont (15)                                                                   
Total bracket energy: -37.1483683417

h = HTML(MMMC.maketable(results))
h

MMMC.showstats(results['all'],description='full run',newfig=True)

goodbrackets = [i for i in results['all'].brackets if i.bracket[-1][0] == 'Kansas']

from MarchMadnessMonteCarlo.Brackets import Stats
lb, mcb, mcb_count, unique_brackets, lowest_sightings = Stats.gather_uniquestats(goodbrackets)
sr = MMMC.SimulationResults(goodbrackets,unique_brackets,lb,lowest_sightings,mcb,mcb_count)
trueresults = {'all':sr}

h = HTML(MMMC.maketable(trueresults))
h