Christian Long's Bloghttp://www.christianlong.com/blog/2015-06-23T13:35:00-05:00Van Lindberg’s List of Security Practices2015-06-23T13:35:00-05:00Christian Longtag:www.christianlong.com,2015-06-23:blog/van-lindbergs-list-of-security-practices.html<p><a href="https://twitter.com/VanL">Van Lindberg</a>, chair of the <a href="https://www.python.org/psf/">Python Software Foundation</a>, recently tweeted <a href="https://twitter.com/VanL/status/613395093165191168">1</a> <a href="https://twitter.com/VanL/status/613395274824744960">2</a> <a href="https://twitter.com/VanL/status/613395987646644226">3</a> <a href="https://twitter.com/VanL/status/613397346550812672">4</a> <a href="https://twitter.com/VanL/status/613397542491983873">5</a> a list of security best practices. For my information and future reference, I’m gathering them here.</p>
<ol>
<li>Thorough negative testing. Cause failures with test cases.</li>
<li>Fuzz with address checking and standard alloc</li>
<li>Compiling with address checking and standard memory alloc</li>
<li>Focused manual spotcheck validation of fields</li>
<li>Fuzzing w/ output examination</li>
<li>Context-sensitive source code analysis</li>
<li>Multi-implementation tests</li>
<li>Aggressive, not compiled out runtime assertions</li>
<li>Implementations in safer languages</li>
<li>Static analysis</li>
<li>Thorough human review/audit</li>
<li>Formal methods</li>
</ol>
<p>Use more than 1.</p>
<h4 id="update-2015-06-23-1411">Update 2015-06-23 14:11<a class="headerlink" href="#update-2015-06-23-1411" title="Permanent link">¶</a></h4>
<p>Van <a href="https://twitter.com/VanL/status/613423845114953728">mentionned</a> that this list was from David Wheeler. Thanks! David’s <a href="http://www.dwheeler.com/secure-programs/">Secure Programming <span class="caps">HOWTO</span></a> is an essential reference.</p>ipdb, the iPython debugger2015-06-15T15:32:00-05:00Christian Longtag:www.christianlong.com,2015-06-15:blog/ipdb-the-ipython-debugger.html<div class="toc">
<ul>
<li><a href="#introduction">Introduction</a></li>
<li><a href="#installation">Installation</a></li>
<li><a href="#basic-debugging">Basic debugging</a></li>
<li><a href="#basic-usage">Basic usage</a></li>
<li><a href="#ipython-additions">iPython additions</a></li>
<li><a href="#further-reading">Further reading</a></li>
<li><a href="#alternatives">Alternatives</a></li>
</ul>
</div>
<h4 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h4>
<p>Python comes with a very useful debugger called <code>pdb</code>. The <a href="http://ipython.org/">iPython</a> project provides an enhanced version of <code>pdb</code> called <code>ipdb</code>. The <code>ipdb</code> debugger provides the same standard debugging commands as <code>pdb</code>, and it includes some nice additional features such as tab completion.</p>
<h4 id="installation">Installation<a class="headerlink" href="#installation" title="Permanent link">¶</a></h4>
<p>Install <code>ipdb</code> from the <a href="https://pypi.python.org/pypi">Python Package Index</a>.</p>
<div class="highlight"><pre>pip install ipdb
</pre></div>
<p>On Windows, you have to install the <code>pyreadline</code> library as well</p>
<div class="highlight"><pre>pip install pyreadline
</pre></div>
<h4 id="basic-debugging">Basic debugging<a class="headerlink" href="#basic-debugging" title="Permanent link">¶</a></h4>
<p>Let’s step through a very simple Python program. Add this to a file called <code>example.py</code></p>
<div class="highlight"><pre><span class="nx">def</span> <span class="nx">main</span><span class="p">()</span><span class="o">:</span>
<span class="s2">"""</span>
<span class="s2"> Main function</span>
<span class="s2"> """</span>
<span class="nx">x</span> <span class="o">=</span> <span class="mi">34</span>
<span class="nx">y</span> <span class="o">=</span> <span class="mi">3</span>
<span class="nx">z</span> <span class="o">=</span> <span class="nx">y</span><span class="o">/</span><span class="nx">x</span>
<span class="nx">print</span><span class="p">(</span><span class="nx">x</span><span class="p">,</span><span class="nx">y</span><span class="p">,</span><span class="nx">z</span><span class="p">)</span>
<span class="k">if</span> <span class="nx">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="o">:</span>
<span class="nx">main</span><span class="p">()</span>
</pre></div>
<p>Run it, wrapped in the <code>ipdb</code> debugger.</p>
<div class="highlight"><pre>ipdb example.py
</pre></div>
<p>If you are using Python 3, the debugger script is called <code>ipdb3</code> instead of <code>ipdb</code></p>
<div class="highlight"><pre>ipdb3 example.py
</pre></div>
<p>The debugger starts up, and waits at the first line.</p>
<div class="highlight"><pre>> /home/example.py(1)<module>()
----> 1 def main():
2 """
3 Main function
</pre></div>
<p>First, let’s get help on what commands are available. Type <code>?</code> and press enter.</p>
<div class="highlight"><pre>ipdb> ?
Documented commands (type help <topic>):
========================================
EOF bt cont enable jump pdef psource run unt
a c continue exit l pdoc q s until
alias cl d h list pfile quit step up
args clear debug help n pinfo r tbreak w
b commands disable ignore next pinfo2 restart u whatis
break condition down j p pp return unalias where
Miscellaneous help topics:
==========================
exec pdb
Undocumented commands:
======================
retval rv
</pre></div>
<p>More help is available for each command.</p>
<div class="highlight"><pre>ipdb> ?s
s(tep)
Execute the current line, stop at the first possible occasion
(either in a function that is called or in the current function).
</pre></div>
<h4 id="basic-usage">Basic usage<a class="headerlink" href="#basic-usage" title="Permanent link">¶</a></h4>
<p>Step through the program with <code>s</code> until you get past the line where 34 is assigned to <code>x</code>.</p>
<p>Now that we’re here, let’s see what value <code>x</code> holds. Type <code>x</code> and <code>ipdb</code> will display the value of <code>x</code>.</p>
<p>Set a breakpoint</p>
<div class="highlight"><pre>ipdb> b 8
Breakpoint 1 at /home/example.py:8
</pre></div>
<p>Use <code>c</code> to continue execution until that breakpoint.</p>
<p>Use <code>q</code> to quit out of the debugger.</p>
<h4 id="ipython-additions">iPython additions<a class="headerlink" href="#ipython-additions" title="Permanent link">¶</a></h4>
<p><code>ipdb</code> adds some useful features above what plain <code>pdb</code> provides. The most useful is tab-completion. Any identifiers that are defined in locals() or in globals() can be tab-completed.</p>
<p>The “magic” commands (<code>%</code>, <code>??</code>) from the regular iPython shell are not available in <code>ipdb</code>. Instead, <code>ipdb</code> provides some extra debugger commands, prefixed with a “p”. <code>pinfo obj</code> is the same as <code>obj?</code> in the iPython shell.</p>
<p>Step in to <code>example.py</code>, past the definition of <code>main()</code>. Type <code>pinfo main</code></p>
<div class="highlight"><pre>ipdb> pinfo main
Signature: main()
Docstring: Main function
File: ~/example.py
Type: function
</pre></div>
<p>This shows us some useful information about the <code>main()</code> function.</p>
<p><code>psource</code> will display the source of an object.</p>
<div class="highlight"><pre>ipdb> psource main
def main():
"""
Main function
"""
x = 34
y = 3
z = y/x
print(x,y,z)
</pre></div>
<h4 id="further-reading">Further reading<a class="headerlink" href="#further-reading" title="Permanent link">¶</a></h4>
<p>We have just scratched the surface of <code>pdb</code> and <code>ipdb</code>. There is good information available right in the debugger,
available by typing <code>?pdb</code>.</p>
<p>For more information about plain <code>pdb</code> and its commands, the “Python Module of the Week” series did a <a href="http://pymotw.com/2/pdb/">nice introduction</a>.</p>
<h4 id="alternatives">Alternatives<a class="headerlink" href="#alternatives" title="Permanent link">¶</a></h4>
<p>The <a href="https://pypi.python.org/pypi/pudb"><code>pudb</code> debugger</a> presents a more visual display, with separate panes for breakpoints and local variables. It is available on Linux and <span class="caps">OS</span> X.</p>
<p>For more intractable problems, the Linux <code>strace</code> utility traces all system calls made by a program. It can attach to an already-running process, and can capture all the file and network input and output.</p>
<div class="highlight"><pre>strace -o output.txt python example.py
grep example output.txt
</pre></div>
<p>This shows us all the occurrences of the word ‘example’ in the system calls that happen when we run <code>python example.py</code></p>
<div class="highlight"><pre>execve("/home/.virtualenvs/test/bin/python", ["python", "example.py"], [/* 36 vars */]) = 0
readlink("example.py", 0x7ffe244a2fe0, 4096) = -1 EINVAL (Invalid argument)
lstat("/home/example.py", {st_mode=S_IFREG|0664, st_size=135, ...}) = 0
stat("example.py", {st_mode=S_IFREG|0664, st_size=135, ...}) = 0
open("example.py", O_RDONLY) = 3
stat("example.py", {st_mode=S_IFREG|0664, st_size=135, ...}) = 0
open("example.py", O_RDONLY) = 3
</pre></div>
<p>Let’s remove read permissions for <code>example.py</code></p>
<div class="highlight"><pre>chmod a-r example.py
</pre></div>
<p>If we run our <code>strace</code> again, we see that the <code>open</code> syscall now returns an error.</p>
<div class="highlight"><pre>execve("/home/virtualenvs/test/bin/python", ["python", "example.py"], [/* 36 vars */]) = 0
readlink("example.py", 0x7ffc15852b60, 4096) = -1 EINVAL (Invalid argument)
lstat("/home/example.py", {st_mode=S_IFREG|0220, st_size=135, ...}) = 0
stat("example.py", {st_mode=S_IFREG|0220, st_size=135, ...}) = 0
open("example.py", O_RDONLY) = -1 EACCES (Permission denied)
stat("example.py", {st_mode=S_IFREG|0220, st_size=135, ...}) = 0
open("example.py", O_RDONLY) = -1 EACCES (Permission denied)
write(2, "python: can't open file 'example"..., 67) = 67
</pre></div>
<p><a href="http://chadfowler.com/blog/2014/01/26/the-magic-of-strace/">Chad Fowler</a> and <a href="http://jvns.ca/blog/categories/strace/">Julia Evans</a> have written good introductions to <code>strace</code>.</p>pytest vs py.test2015-06-10T20:12:00-05:00Christian Longtag:www.christianlong.com,2015-06-10:blog/pytest-vs-pytest.html<div class="toc">
<ul>
<li><a href="#the-problem">The Problem</a></li>
<li><a href="#investigating-using-strace">Investigating using strace</a></li>
</ul>
</div>
<h4 id="the-problem">The Problem<a class="headerlink" href="#the-problem" title="Permanent link">¶</a></h4>
<p><code>pylint</code> installs a <code>pytest</code> script, and <code>pytest</code> installs a <code>py.test</code> script. Confusing.</p>
<p>The <a href="http://www.pylint.org/">pylint</a> code quality checker installs the <a href="https://www.logilab.org/project/logilab-common">logilab-common</a> package. <code>logilab-common</code> provides a module called <code>pytest</code>. That’s awfully close to the <code>py.test</code> script that the <a href="http://pytest.org"><code>pytest</code></a> project provides. I can never remember which one to use, when I have both <code>pylint</code> and <code>pytest</code> installed. Fortunately, the <a href="https://www.logilab.org/project/logilab-common">logilab-common</a> package has marked its <code>pytest</code> module as “to be deprecated”, so the confusion might be fixed someday.</p>
<h4 id="investigating-using-strace">Investigating using <code>strace</code><a class="headerlink" href="#investigating-using-strace" title="Permanent link">¶</a></h4>
<p>I knew the <code>pytest</code> project was supplying the <code>py.test</code> script, but I did not know where the unwanted <code>pytest</code> script was coming from. <code>strace</code> to the rescue! </p>
<div class="highlight"><pre><span class="nv">$ </span>strace -o trace.txt -e open pytest --help
<span class="nv">$ </span>grep pytest trace.txt
open<span class="o">(</span><span class="s2">"/home/me/.virtualenvs/dev/bin/pytest"</span>, O_RDONLY<span class="o">)</span> <span class="o">=</span> 3
open<span class="o">(</span><span class="s2">"/home/me/.virtualenvs/dev/bin/pytest"</span>, O_RDONLY<span class="o">)</span> <span class="o">=</span> 3
open<span class="o">(</span><span class="s2">"/home/me/.virtualenvs/dev/local/lib/python2.7/site-packages/logilab/common/pytest.x86_64-linux-gnu.so"</span>, O_RDONLY<span class="o">)</span> <span class="o">=</span> -1 ENOENT <span class="o">(</span>No such file or directory<span class="o">)</span>
open<span class="o">(</span><span class="s2">"/home/me/.virtualenvs/dev/local/lib/python2.7/site-packages/logilab/common/pytest.so"</span>, O_RDONLY<span class="o">)</span> <span class="o">=</span> -1 ENOENT <span class="o">(</span>No such file or directory<span class="o">)</span>
open<span class="o">(</span><span class="s2">"/home/me/.virtualenvs/dev/local/lib/python2.7/site-packages/logilab/common/pytestmodule.so"</span>, O_RDONLY<span class="o">)</span> <span class="o">=</span> -1 ENOENT <span class="o">(</span>No such file or directory<span class="o">)</span>
open<span class="o">(</span><span class="s2">"/home/me/.virtualenvs/dev/local/lib/python2.7/site-packages/logilab/common/pytest.py"</span>, O_RDONLY<span class="o">)</span> <span class="o">=</span> 3
open<span class="o">(</span><span class="s2">"/home/me/.virtualenvs/dev/local/lib/python2.7/site-packages/logilab/common/pytest.pyc"</span>, O_RDONLY<span class="o">)</span> <span class="o">=</span> 4
</pre></div>
<p>Here we see that the <code>pytest.py</code> file is in the <code>logilab/common</code> directory, so we know what package installed it. </p>
<p>For a fun intro to <code>strace</code>, see Julia Evans’ <a href="http://jvns.ca/blog/2015/04/14/strace-zine/">strace zine</a>.</p>Twisted on Windows, 2015 Edition: Part 32015-05-18T12:03:00-05:00Christian Longtag:www.christianlong.com,2015-05-18:blog/twisted-on-windows-2015-edition-part-3.html<div class="toc">
<ul>
<li><a href="#recap">Recap</a></li>
<li><a href="#windows-file-permissions">Windows file permissions</a></li>
<li><a href="#inheritance-mode">Inheritance mode</a></li>
<li><a href="#troubleshooting-permissions">Troubleshooting permissions</a></li>
<li><a href="#zip-files">Zip files</a></li>
<li><a href="#by-your-leave">By your leave</a></li>
</ul>
</div>
<h4 id="recap">Recap<a class="headerlink" href="#recap" title="Permanent link">¶</a></h4>
<p>In <a href="http://www.christianlong.com/blog/twisted-on-windows-2015-edition.html">Part 1</a>, we set up <a href="https://twistedmatrix.com">Twisted</a>. In <a href="http://www.christianlong.com/blog/twisted-on-windows-2015-edition-part-2.html">Part 2</a>, we configured Twisted to run as a Windows service. Now, we will assign the right permissions to the account that the service is using.</p>
<h4 id="windows-file-permissions">Windows file permissions<a class="headerlink" href="#windows-file-permissions" title="Permanent link">¶</a></h4>
<p>Windows implements file-level security through access control lists. Windows provides <a href="http://ss64.com/nt/icacls.html">the <code>icacls</code> utility</a> to query and set permissions at the command line.</p>
<p>For example</p>
<div class="highlight"><pre>icacls C:\PythonEnvs\Example /grant "nt service\my_service":(OI)(CI)RX
</pre></div>
<p>Let’s break this down:</p>
<ul>
<li><code>C:\PythonEnvs\Example</code> - The folder you want to change permissions on</li>
<li><code>/grant</code> - Grant rights additively</li>
<li><code>"nt service\my_service"</code> - The name of the virtual service account</li>
<li><code>(OI)(CI)</code> - Applies the grant recursively to this folder, subfolders, and files</li>
<li><code>RX</code> - Grants Read and Execute access</li>
</ul>
<h4 id="inheritance-mode">Inheritance mode<a class="headerlink" href="#inheritance-mode" title="Permanent link">¶</a></h4>
<p>Inherited folder permissions are specified as:</p>
<ul>
<li><span class="caps">OI</span> - Object inherit - This folder and files (no inheritance to subfolders)</li>
<li><span class="caps">CI</span> - Container inherit - This folder and subfolders</li>
<li><span class="caps">IO</span> - Inherit only - The <span class="caps">ACE</span> does not apply to the current file/directory</li>
</ul>
<p>These can also be combined:</p>
<ul>
<li>(<span class="caps">OI</span>)(<span class="caps">CI</span>) this folder, subfolders, and files</li>
<li>(<span class="caps">OI</span>)(<span class="caps">CI</span>)(<span class="caps">IO</span>) subfolders and files only</li>
<li>(<span class="caps">CI</span>)(<span class="caps">IO</span>) subfolders only</li>
<li>(<span class="caps">OI</span>) (<span class="caps">IO</span>) files only</li>
</ul>
<p>So <code>BUILTIN\Administrators:(OI)(CI)F</code> means that both files and subfolders will inherit ‘F’ (Full Control)</p>
<p>Similarly <code>(CI)R</code> means folders will inherit ‘R’ (Read folders only = List permission)</p>
<p><span class="caps">SS64</span> has a <a href="http://ss64.com/nt/icacls.html">good reference</a> for the <code>icacls</code> options.</p>
<h4 id="troubleshooting-permissions">Troubleshooting permissions<a class="headerlink" href="#troubleshooting-permissions" title="Permanent link">¶</a></h4>
<p>Many Windows admin tasks require the wonderful <a href="https://technet.microsoft.com/en-us/sysinternals/bb545021.aspx">Sysinternals</a> tools, written by Mark Russinonvich. The first thing I do on a new Windows server is <a href="https://technet.microsoft.com/en-us/sysinternals/bb842062.aspx">download the Sysinternals Suite</a> and put it on the system path. For this troubleshooting section, I’ll be using the <a href="https://technet.microsoft.com/en-us/sysinternals/bb897553"><code>psexec</code> tool</a>.</p>
<p>Whenever a service is not working, it’s helpful to try running it by hand, in the same user account and environment that the service is trying to run in. As far as I know, there is no way to run commands manually as a virtual service account (<code>NT SERVICE\my_service</code>). Put another way, there is no equivalent of <code>su - <user></code> that can be used for virtual service accounts. However, we can approximate it. We can use <code>psexec</code> to run commands as the LocalService account (<code>NT AUTHORITY\LocalService</code>). It’s far from perfect, but if we can get it working under the LocalService account, chances are we can then get it working under a virtual service account.</p>
<p>Let’s look at some steps we can take to investigate permissions problems. First, open a new admin command prompt. On Windows 8 and newer, “Win-x then a” is a quick way to open an admin command prompt.</p>
<p>Here’s how to execute a single command, under the LocalService account.</p>
<div class="highlight"><pre>psexec -u "NT AUTHORITY\LocalService" C:\PythonEnvs\Example\Scripts\pip freeze
</pre></div>
<p>The <code>-u</code> option tells <code>psexec</code> what user to run as. The rest of the line is the command that you want <code>psexec</code> to run. Notice that we pass the full path to the <code>pip</code> inside the virtualenv. <code>pip freeze</code> prints a list of the packages that are installed.</p>
<p>If you get a “Couldn’t install <span class="caps">PSEXESVC</span> service” error, try running <code>psexec</code> as administrator.</p>
<p>Here’s another example. We’re telling Python to import the pyodbc module, and print “ok” if it succeeds.</p>
<div class="highlight"><pre><span class="n">psexec</span> <span class="o">-</span><span class="n">u</span> <span class="s">"NT AUTHORITY\LocalService"</span> <span class="n">C</span><span class="p">:</span>\<span class="n">PythonEnvs</span>\<span class="n">Example</span>\<span class="n">Scripts</span>\<span class="n">python</span> <span class="o">-</span><span class="n">c</span> <span class="s">"import odbc; print 'ok'"</span>
</pre></div>
<p>Here’s how to start a new interactive shell in a new window, running as the LocalService user.</p>
<div class="highlight"><pre>psexec -u "NT AUTHORITY\LocalService" -i cmd /k
</pre></div>
<p>It should look something like this.</p>
<p><img alt="New interactive shell screenshot" class="no_round" src="http://www.christianlong.com/blog/images/cmd_i.png" /></p>
<p>In the screenshot above, I have used the <code>whoami</code> command to show that, in the new console window, we are running as <code>NT AUTHORITY\LocalService</code>.</p>
<p>If you want to start a new interactive session as a different user, but stay in your current command window, omit the <code>-i</code> argument to <code>psexec</code>.</p>
<div class="highlight"><pre>psexec -u "NT AUTHORITY\LocalService" cmd /k
</pre></div>
<p>Since Windows does not print the username in the shell prompt, it can be hard to know what user you are running as. Use the <code>whoami</code> command to see the current user.</p>
<p>The <code>whoami /all</code> command shows a lot of useful information about the user, groups and privileges.</p>
<p><img alt="whoami all screenshot" class="no_round" src="http://www.christianlong.com/blog/images/whoami_all.png" /></p>
<p>Two things to keep in mind when using <code>psexec</code> to troubleshoot. One, it must be called from an admin command prompt (Win-x then a). Two, you can not use <code>psexec</code> to run commands as a virtual service account.</p>
<p>When I’m setting up Windows service, I configure it to run under the LocalService account at first, and I use <code>psexec</code> to track down any permissions problems. Then I switch the service over to a virtual service account. I grant the virtual service account the same privileges that I granted to the LocalService account.</p>
<h4 id="zip-files">Zip files<a class="headerlink" href="#zip-files" title="Permanent link">¶</a></h4>
<p>You may find that the virtual service accounts you’re using to run the services don’t have permission to handle zip files. This may be due to a policy restriction set by your network administrator. The LocalService account may have the same restriction.</p>
<p>This is why, when I’m installing pyodbc, I have to pass the <code>--always-unzip</code> option to <code>easy_install</code>. Otherwise, <code>easy_install</code> installs it as a zip file, and the virtual service account doesn’t have permission to read it.</p>
<p>Another example: on Windows, distutils packages source distributions as zip files. Let’s say for example you’re running python’s <a href="https://docs.python.org/2/library/simplehttpserver.html">simpleHTTPServer</a> as a Windows service (using nssm), running it as account <code>nt service\package_server</code>. You can visit http://localhost:8000 and you get a nice file listing. However, if you try to download one of the zip files, you get a 404. Assigning permissions for <code>nt service\package_server</code> doesn’t work - the permissions apply to folders and text files but they don’t apply to zip files through inheritance. It only works if you assign read permission specifically for the zip file itself.</p>
<p>Here’s a command to specifically assign permissions for a zip file</p>
<div class="highlight"><pre> icacls "C:\PackageServer\www\downloads\My_Package-0.0.1.zip" /grant "nt service\package_server":R
</pre></div>
<h4 id="by-your-leave">By your leave<a class="headerlink" href="#by-your-leave" title="Permanent link">¶</a></h4>
<p>We’ve come to the end of Part 3. The virtual service account under which the service is running has been assigned the permissions it needs. </p>
<p>Note that these commands can be used for any service you are running on Windows, not only those using Twisted or <span class="caps">NSSM</span>.</p>
<p>Thanks for following along, and find me on Twitter at <a href="https://twitter.com/christianmlong">@christianmlong</a> if you have any suggestions or fixes.</p>Twisted on Windows, 2015 Edition: Part 22015-04-21T16:57:00-05:00Christian Longtag:www.christianlong.com,2015-04-21:blog/twisted-on-windows-2015-edition-part-2.html<div class="toc">
<ul>
<li><a href="#recap">Recap</a></li>
<li><a href="#install-nssm">Install <span class="caps">NSSM</span></a></li>
<li><a href="#nssm-command-line"><span class="caps">NSSM</span> command line</a></li>
<li><a href="#create-the-windows-service">Create the Windows service</a></li>
<li><a href="#configure-the-working-directory">Configure the working directory</a></li>
<li><a href="#set-display-name-and-description">Set display name and description</a></li>
<li><a href="#configure-startup">Configure startup</a></li>
<li><a href="#which-account-should-we-use">Which account should we use?</a></li>
<li><a href="#virtual-service-accounts">Virtual service accounts</a></li>
<li><a href="#set-the-account">Set the account</a></li>
<li><a href="#nssm-workaround"><span class="caps">NSSM</span> workaround</a></li>
<li><a href="#start-it-up">Start it up</a></li>
<li><a href="#update">Update</a></li>
<li><a href="#at-your-service">At your service</a></li>
</ul>
</div>
<h4 id="recap">Recap<a class="headerlink" href="#recap" title="Permanent link">¶</a></h4>
<p>In <a href="http://www.christianlong.com/blog/twisted-on-windows-2015-edition.html">Part 1</a>, we installed <a href="https://twistedmatrix.com">Twisted</a>, and set it up to run inside a virtualenv. Now, we will configure Twisted to run as a Windows service, under a virtual service account.</p>
<h4 id="install-nssm">Install <span class="caps">NSSM</span><a class="headerlink" href="#install-nssm" title="Permanent link">¶</a></h4>
<p>With <a href="http://twistedmatrix.com/pipermail/twisted-python/2011-October/024632.html">some</a> <a href="http://www.banquise.org/python/making-and-deploying-a-twisted-project-as-a-service-under-windows/">coding</a>, Twisted applications can run as a <a href="https://msdn.microsoft.com/en-us/library/windows/desktop/ms685141%28v=vs.85%29.aspx">Windows service</a>. However, instead of doing that, I’m running my application under <a href="http://nssm.cc/"><span class="caps">NSSM</span></a>, the Non-Sucking Serivce Manager. This handy application allows you to run any command-line application as a Windows service.</p>
<p><a href="https://nssm.cc/download">Download</a> <span class="caps">NSSM</span> and unzip it. In this example, I’m using <span class="caps">NSSM</span> version 2.24, so the folder name is <code>nssm-2.24</code>. Change these intructions as needed to match the version of <span class="caps">NSSM</span> you are installing. Move the <code>nssm-2.24</code> folder to <code>C:\Program Files</code>. Add this to the <a href="http://www.howtogeek.com/118594/how-to-edit-your-system-path-for-easy-command-line-access/">system Path</a>:</p>
<div class="highlight"><pre>C:\Program Files\nssm-2.24\win64
</pre></div>
<p>Note that we are using the 64-bit version of <span class="caps">NSSM</span>, even though the application we are installing is running on 32-bit Python. The 64-bit version of <span class="caps">NSSM</span> can manage both 64-bit and 32-bit applications.</p>
<p>The nssm.exe file might be marked as untrusted. You can unblock it by right-clicking on the file and choosing Properties. In the Properties dialog, click Unblock.</p>
<p><img alt="Properties screenshot" class="no_round" src="http://www.christianlong.com/blog/images/security_zone.png" /></p>
<p><a href="http://weblogs.asp.net/dixin/understanding-the-internet-file-blocking-and-unblocking">More information</a> about unblocking files in Windows.</p>
<h4 id="nssm-command-line"><span class="caps">NSSM</span> command line<a class="headerlink" href="#nssm-command-line" title="Permanent link">¶</a></h4>
<p>We will use the <span class="caps">NSSM</span> <a href="https://nssm.cc/commands">command line tools</a> to configure the new service. The <span class="caps">NSSM</span> commands look like this:</p>
<div class="highlight"><pre><span class="n">nssm</span> <span class="p">[</span><span class="n">nssm</span> <span class="n">command</span><span class="p">]</span> <span class="p">[</span><span class="n">service</span> <span class="n">name</span><span class="p">]</span> <span class="p">[</span><span class="n">arguments</span><span class="p">]</span>
</pre></div>
<p>Type <code>nssm help</code> to get an overview of the commands. If you get a “not recognized” error from the shell, add <span class="caps">NSSM</span> to your <a href="#install-nssm">path</a>. Changes to the path don’t apply to already-open command windows, so open a new command window after changing the path.</p>
<p>Running <code>nssm</code> commands requires admin privileges.</p>
<h4 id="create-the-windows-service">Create the Windows service<a class="headerlink" href="#create-the-windows-service" title="Permanent link">¶</a></h4>
<p>The following commands assume that you have created a virtualenv called “Example”, as described in <a href="http://www.christianlong.com/blog/twisted-on-windows-2015-edition.html">Part 1</a>. Change these commands as needed to match the name of your service and your virtualenv. You have to quote paths if they contain spaces.</p>
<p>Start a new admin command prompt (Win-x then a):</p>
<div class="highlight"><pre>nssm install my_service ^
C:\PythonEnvs\Example\Scripts\python.exe ^
"C:\PythonEnvs\Example\Scripts\twistd.py --python my_tacfile.tac"
</pre></div>
<p>For readability, I have broken this long command in to multiple lines using <code>^</code>, the dos line-continuation character.</p>
<p>This will install a service called my_service.</p>
<p>Look at the paths in the command above. Instead of specifying the system-wide <code>python.exe</code> (C:\Python27\python.exe), we give the path to a <code>python.exe</code> in the Scripts folder of our virtual environment. This has the same effect as calling <code>activate</code> in an interactive session. The Python interpreter will have access to all the packages installed in that virtual environment.</p>
<p>We also have to specify the full path to <code>twistd.py</code>. This file comes with Twisted; it starts the server process.</p>
<p>However, we don’t need to specify the full path to <code>my_tacfile.tac</code>. This is because we specify a working directory for the Windows service, as covered in the next section.</p>
<h4 id="configure-the-working-directory">Configure the working directory<a class="headerlink" href="#configure-the-working-directory" title="Permanent link">¶</a></h4>
<p>The Windows service has been created, but we still need to configure it. Stay in the admin command prompt, and type:</p>
<div class="highlight"><pre>nssm set my_service AppDirectory C:\PythonEnvs\Example\Lib\site-packages\my_app
</pre></div>
<p>In the above example, replace <code>my_app</code> with your app name (it’s the name you used when you did <code>pip install</code>). Check that directory; it should contain your <code>.tac</code> file.</p>
<p>By setting the <code>AppDirectory</code> config variable, we are telling <span class="caps">NSSM</span> to make that directory the current working directory before starting the service. That is why we did not need to specify the full path to <code>my_tacfile.tac</code> when we installed the service.</p>
<p>This is the equivalent of passing the <code>--chroot</code> option to <code>twistd</code>.</p>
<h4 id="set-display-name-and-description">Set display name and description<a class="headerlink" href="#set-display-name-and-description" title="Permanent link">¶</a></h4>
<div class="highlight"><pre>nssm set my_service DisplayName "My App"
nssm set my_service Description "My sweet application - running as a Windows service."
</pre></div>
<p>The display name and description will show up in the Windows service Manager console.</p>
<p><img alt="Services screenshot" src="http://www.christianlong.com/blog/images/services.png" /></p>
<h4 id="configure-startup">Configure startup<a class="headerlink" href="#configure-startup" title="Permanent link">¶</a></h4>
<div class="highlight"><pre>nssm set my_service Start SERVICE_DELAYED_AUTO_START
</pre></div>
<p>This setting tells the Windows service to start automatically when the server restarts. The <span class="caps">NSSM</span> docs have <a href="https://nssm.cc/commands">more information</a> about the possible startup options.</p>
<h4 id="which-account-should-we-use">Which account should we use?<a class="headerlink" href="#which-account-should-we-use" title="Permanent link">¶</a></h4>
<p>There are a number of accounts you can use to run your Windows service. It is a good idea to run network services under the least-privileged account possible. For that reason, a user account is not a good choice.</p>
<p>Windows provides some built-in accounts for this purpose:</p>
<ul>
<li>
<p>The <a href="https://msdn.microsoft.com/en-us/library/windows/desktop/ms684190%28v=vs.85%29.aspx">LocalSystem</a> account (<code>NT AUTHORITY\SYSTEM</code>) is still quite privileged.</p>
</li>
<li>
<p>The <a href="https://msdn.microsoft.com/en-us/library/windows/desktop/ms684272%28v=vs.85%29.aspx">NetworkService</a> account (<code>NT AUTHORITY\NetworkService</code>) allows the service to access network resources on the Windows network. However, we don’t need to run as the NetworkService account if we are just serving local resources.</p>
</li>
<li>
<p>The <a href="https://msdn.microsoft.com/en-us/library/windows/desktop/ms684188%28v=vs.85%29.aspx">LocalService</a> account (<code>NT AUTHORITY\LocalService</code>) has traditionally been the account Windows sysadmins used for services. It does, however, have <a href="https://social.technet.microsoft.com/Forums/en-US/57e8b300-b3f7-4005-927b-e7b7b6a7097b/are-virtual-accounts-more-secure-than-local-service-accounts?forum=winserversecurity">some drawbacks</a>. There is only one LocalService account per machine. Let’s say you want to set up multiple services per server (a database and a web server, for example). If you assign permissions to the LocalService account for the benefit of one service, those permissions are shared by all the services that use that account.</p>
</li>
</ul>
<p>None of these are ideal. We’re going to be using a different account type, covered in the next section.</p>
<h4 id="virtual-service-accounts">Virtual service accounts<a class="headerlink" href="#virtual-service-accounts" title="Permanent link">¶</a></h4>
<p>In Windows Server 2008 R2, Microsoft introduced a new kind of account for this purpose, called a “Virtual Service Account”. These accounts are automatically created, one for each Windows service. By default, they have few privileges. And, if you assign privileges to a virtual service account, those privileges apply only to that service and that account. Other services on the same machine do not get those privileges.</p>
<p>Virtual service accounts have the same name as the service they apply to. So, for a service called my_service, the account name would be <code>NT SERVICE\my_service</code>.</p>
<p>Virtual service accounts are not especially well documented. This <a href="https://technet.microsoft.com/library/dd548356%28v=ws.10%29.aspx">Microsoft TechNet article</a> is a good reference. There’s no conveniently-located anchor tag in that document, so scroll down to the section titled “Using virtual accounts”. Here’s <a href="http://weblogs.asp.net/owscott/managed-service-accounts-msa-and-virtual-accounts">another article</a>. Again, scroll down to the section titled “Virtual Accounts”.</p>
<p><strong>Ignore everything about “Managed service accounts”.</strong> Virtual service accounts and Managed service accounts are often lumped together in the documentation.</p>
<h4 id="set-the-account">Set the account<a class="headerlink" href="#set-the-account" title="Permanent link">¶</a></h4>
<p>Because of the advantages listed above, I use virtual service accounts to run my Twisted services on Windows 2012 R2. Virtual service accounts are available on Windows Server 2008 and later.</p>
<p><span class="caps">NSSM</span> can not currently configure services to use virtual service accounts. I contacted the developer, and he said he is interested in adding support.</p>
<p>However, we can use good old <code>sc</code> to configure the service to use the virtual service account. <code>sc</code> is the service manager command line utility that comes with Windows.</p>
<div class="highlight"><pre>sc config my_service obj= "NT SERVICE\my_service"
</pre></div>
<p>Yes, that is a space after the equals sign. <code>sc</code> is very particular. <span class="caps">SS64</span> has a good <a href="http://ss64.com/nt/sc.html">reference</a> for <code>sc</code>.</p>
<h4 id="nssm-workaround"><span class="caps">NSSM</span> workaround<a class="headerlink" href="#nssm-workaround" title="Permanent link">¶</a></h4>
<p><span class="caps">NSSM</span> doesn’t know about virtual service accounts, so it will complain if you use it to configure a service that has been set to use a virtual service account. As a workaround, use <code>sc</code> to temporarily set the account back to LocalSystem.</p>
<p>For example:</p>
<p>Use <code>sc</code> to set the user to something <span class="caps">NSSM</span> understands</p>
<div class="highlight"><pre>sc config my_service obj= "LocalSystem"
</pre></div>
<p>Use <code>nssm</code> to configure your service</p>
<div class="highlight"><pre>nssm set my_service Description "A better description"
</pre></div>
<p>Use <code>sc</code> to set the user back to the virtual service account</p>
<div class="highlight"><pre>sc config my_service obj= "NT SERVICE\my_service"
</pre></div>
<p>You don’t have to do it this way. Once you create the Windows service with <span class="caps">NSSM</span>, you can configure that service with the traditional Windows service configuration tools.</p>
<h4 id="start-it-up">Start it up<a class="headerlink" href="#start-it-up" title="Permanent link">¶</a></h4>
<div class="highlight"><pre>nssm start my_service
</pre></div>
<p>This is what it should look like</p>
<div class="highlight"><pre>C:\WINDOWS\system32>nssm start my_service
my_service: START: The operation completed successfully.
</pre></div>
<p>The service prorperties dialog should look like this:</p>
<p><img alt="Services screenshot" class="no_round" src="http://www.christianlong.com/blog/images/service2.png" /></p>
<p>Notice our service is running, and is set to start automatically.</p>
<h4 id="update">Update<a class="headerlink" href="#update" title="Permanent link">¶</a></h4>
<p>Updated 2015-05-19</p>
<p>Via email, Michael Schlenker points this out:</p>
<blockquote>
<p>Virtual Service Accounts can have the ‘Logon as a Service’ right removed via
Group Policy, which leads to weird errors. It is actually a recommended step
for security hardening certain Windows servers (as you don’t want to have
random services running).</p>
</blockquote>
<p>Good point, and thanks for the feedback.</p>
<h4 id="at-your-service">At your service<a class="headerlink" href="#at-your-service" title="Permanent link">¶</a></h4>
<p>We’ve come to the end of Part 2. We have a Twisted application running as Windows service. It is running under a virtual service account. </p>
<p>Of course, virtual service accounts can be used for any service you are running on Windows, not only those using Twisted or <span class="caps">NSSM</span>.</p>
<p>In the next part, we will configure the right privileges for the account. Thanks for following along, and find me on Twitter at <a href="https://twitter.com/christianmlong">@christianmlong</a> if you have any suggestions or fixes.</p>Pelican2015-04-21T16:22:50-05:00Christian Longtag:www.christianlong.com,2015-04-21:blog/first-post.html<p>With a bit of wrangling, I got Pelican working to manage my blog. <a href="http://blog.getpelican.com/">Pelican</a> is a static site generator written in Python. It can handle reStructuredText, Markdown, or AsciiDoc formats.</p>
<p>The next question is, what theme to use? The <a href="http://pelican-themes-gallery.place.org/">Pelican theme gallery</a> is helpful here. I debated between these themes:</p>
<ul>
<li><a href="http://pelican-themes-gallery.place.org/irfan/">Irfan</a></li>
<li><a href="http://pelican-themes-gallery.place.org/tuxlite_tbs/">Tuxlite <span class="caps">TBS</span></a></li>
<li><a href="http://pelican-themes-gallery.place.org/pelican-mockingbird/">Pelican Mockingbird</a></li>
<li><a href="http://pelican-themes-gallery.place.org/built-texts/">Built Texts</a></li>
</ul>
<p>For now, I’m using <a href="http://pelican-themes-gallery.place.org/built-texts/">Built Texts</a>. It’s as easy as setting </p>
<p><code>THEME = /path/to/built-texts</code> </p>
<p>in your pelicanconf.py. </p>
<p>Also, make sure that you set </p>
<p><code>SITEURL = ''</code></p>
<p>in pelicanconf.py. Otherwise, your local preview site will try to load resoureces (css, etc.) from the url of your published site. </p>Pelican Blog Workflow2015-04-21T16:22:50-05:00Christian Longtag:www.christianlong.com,2015-04-21:blog/pelican-blog-workflow.html<p>This blog is running on <a href="http://docs.getpelican.com">Pelican</a>. It’s pretty slick, and here are a few tips I use to make writing and publishing faster and easier.</p>
<h4 id="new-post">New post<a class="headerlink" href="#new-post" title="Permanent link">¶</a></h4>
<p>I have this shell script accessible via an alias <code>nbp</code>.</p>
<table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17</pre></div></td><td class="code"><div class="highlight"><pre><span class="c">#!/bin/bash</span>
<span class="nb">set</span> -eu
<span class="nv">DRAFTS_DIR</span><span class="o">=</span>~/projects/personal/blog/drafts
<span class="k">if</span> <span class="o">[</span> -z <span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span> <span class="o">]</span> <span class="p">;</span> <span class="k">then</span>
<span class="nv">FILE_NAME</span><span class="o">=</span><span class="s1">'new_post'</span>
<span class="k">else</span>
<span class="nv">FILE_NAME</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
<span class="k">fi</span>
<span class="nv">SKEL_FILE</span><span class="o">=</span><span class="nv">$DRAFTS_DIR</span>/skeleton_post
<span class="nv">FILE_PATH</span><span class="o">=</span><span class="nv">$DRAFTS_DIR</span>/<span class="s2">"</span><span class="nv">$FILE_NAME</span><span class="s2">"</span>.md
cp --no-clobber <span class="nv">$SKEL_FILE</span> <span class="nv">$FILE_PATH</span>
vi <span class="nv">$FILE_PATH</span>
</pre></div>
</td></tr></table>
<p>This copies my posting template to a new file in the drafts directory, and opens it for editing in vim.</p>
<h4 id="dev-server">Dev server<a class="headerlink" href="#dev-server" title="Permanent link">¶</a></h4>
<p>Run this command to start the Pelican development server.</p>
<div class="highlight"><pre>make devserver
</pre></div>
<p>This will automatically regenerate the site when files are changed. It will also serve the site at <a href="http://localhost:8000">http://localhost:8000</a>.</p>
<p>Run <code>develop_server.sh stop</code> to stop the development server.</p>
<h4 id="browser-auto-refresh">Browser auto refresh<a class="headerlink" href="#browser-auto-refresh" title="Permanent link">¶</a></h4>
<p>There are many auto-refresh solutions for the browser, but they don’t work well when you are editing a file on a remote server. <a href="http://www.livejs.com/">live.js</a> is a nice solution that takes care of auto-refreshing in javascript. However, we don’t want to include the live.js javascript file in the published version, just in the local development version. We can modify our theme to include it only when developing.</p>
<p>First edit pelicanconf.py. Add this</p>
<div class="highlight"><pre>IS_DEVELOPMENT_VERSION = True
</pre></div>
<p>Also edit publishconf.py. Add this</p>
<div class="highlight"><pre>IS_DEVELOPMENT_VERSION = False
</pre></div>
<p>Now, change the theme so that every article page includes the live.js javascript, if we are in development. Find your theme’s template directory. It’s probably at <code>themes/<theme name>/templates</code>. Edit <code>article.html</code>. Look for the head block in the template <code>{% block head %}</code>. Add this to it:</p>
<div class="highlight"><pre><span class="cp">{%</span> <span class="k">if</span> <span class="nv">IS_DEVELOPMENT_VERSION</span> <span class="cp">%}</span>
<span class="nt"><script</span> <span class="na">type=</span><span class="s">"text/javascript"</span> <span class="na">src=</span><span class="s">"http://livejs.com/live.js"</span><span class="nt">></script></span>
<span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span>
</pre></div>
<p>Run <code>make devserver</code>, and open <a href="http://localhost:8000">http://localhost:8000</a>. Edit one of your articles, and see if it reloads in the browser automatically. Neat!</p>
<p>The nice thing about the live.js solution is that it works even if the files you are editing are on a remote server. It polls the page by making a head request every few seconds. Obviously, you don’t want this polling to happen on your published pages.</p>Twisted on Windows, 2015 Edition2015-03-25T16:43:00-05:00Christian Longtag:www.christianlong.com,2015-03-25:blog/twisted-on-windows-2015-edition.html<div class="toc">
<ul>
<li><a href="#intro">Intro</a></li>
<li><a href="#install-python">Install Python</a></li>
<li><a href="#install-a-compiler">Install a compiler</a></li>
<li><a href="#upgrade-pip-and-virtualenv">Upgrade pip and virtualenv</a></li>
<li><a href="#set-up-a-virtual-environment">Set up a virtual environment</a></li>
<li><a href="#the-state-of-python-packaging-in-2015">The state of Python packaging in 2015</a></li>
<li><a href="#install-twisted">Install Twisted</a></li>
<li><a href="#install-dependencies-from-pypi">Install dependencies from PyPI</a></li>
<li><a href="#install-dependencies-that-are-not-on-pypi">Install dependencies that are not on PyPI</a></li>
<li><a href="#twisted-application">Twisted Application</a></li>
<li><a href="#install-your-application">Install your application</a></li>
<li><a href="#run-it">Run it</a></li>
<li><a href="#up-and-running">Up and running</a></li>
</ul>
</div>
<h4 id="intro">Intro<a class="headerlink" href="#intro" title="Permanent link">¶</a></h4>
<p>Installing and configuring <a href="https://twistedmatrix.com">Twisted</a> on Windows has gotten easier over the years. Here is how to do it in 2015, using the latest packaging tools.</p>
<p>Twisted runs well on Windows, in my experience. I moved some Twisted applications from Linux to Windows for a client, and they have been running happily for months now without issue.</p>
<p>These instructions have been tested with Windows 8.1 and Windows Server 2012 R2. The applications I run on Windows use <a href="http://twistedmatrix.com/trac/wiki/TwistedWeb">Twisted.Web</a> to serve single-page web apps, talking to a database using <a href="https://code.google.com/p/pyodbc/">pyodbc</a>. My <span class="caps">ODBC</span> driver is 32-bit, so I’m using 32-bit Python for these instructions. However, Twisted does not depend on pyodbc, so don’t install it if you don’t need it.</p>
<p>These instructions assume that you already have an application that runs on Twisted. The Twisted documentation has a <a href="https://twistedmatrix.com/documents/current/core/howto/application.html">good explanation</a> of how to set up your project so it runs as a Twisted Applicaion. It also includes an <a href="https://twistedmatrix.com/documents/current/core/examples/">nice selection</a> of example applications, if you don’t have an existing Twisted application.</p>
<h4 id="install-python">Install Python<a class="headerlink" href="#install-python" title="Permanent link">¶</a></h4>
<p>Twisted requires Python 2. Install the <a href="https://www.python.org/downloads/release/python-279/">latest version</a> of Python 2.7 (<a href="https://www.python.org/ftp/python/2.7.9/python-2.7.9.msi">direct link to the 32-bit installer</a>). Run the installer, and in the installer dialog, choose the option to add python to the path.</p>
<p><img alt="Python installer screenshot" class="no_round" src="http://www.christianlong.com/blog/images/python_install.png" /></p>
<h4 id="install-a-compiler">Install a compiler<a class="headerlink" href="#install-a-compiler" title="Permanent link">¶</a></h4>
<p>Some of Twisted’s dependencies have C extensions, and are not available from the <a href="https://pypi.python.org/pypi">Python Package Index</a> in the binary <code>wheel</code> format. So, we need a compiler to compile them from source. This <a href="http://stackoverflow.com/questions/26140192/microsoft-visual-c-compiler-for-python-2-7/28618559#28618559">used</a> <a href="http://stackoverflow.com/questions/11405549/how-do-i-install-pycrypto-on-windows">to</a> <a href="http://stackoverflow.com/questions/3047542/building-lxml-for-python-2-7-on-windows">be</a> <a href="http://stackoverflow.com/questions/2817869/error-unable-to-find-vcvarsall-bat">tricky</a>, but fortunately, Microsoft now provides a free download that makes it easy. Download the <a href="http://www.microsoft.com/en-us/download/details.aspx?id=44266">Microsoft Visual C++ Compiler for Python 2.7</a>. It may have to be installed as admin, to get around policy restrictions on compilers.</p>
<h4 id="upgrade-pip-and-virtualenv">Upgrade pip and virtualenv<a class="headerlink" href="#upgrade-pip-and-virtualenv" title="Permanent link">¶</a></h4>
<p>The Python 2.7.9 installer now includes pip and virtualenv, and sets them up for you by default. However, it does not come with the very latest pip and virtualenv. Here’s how to upgrade them to the latest versions.</p>
<p>Start an admin command prompt. On Windows 8 and newer, Win-x then a is a quick keyboard shortcut to open an admin command prompt.</p>
<p>Upgrade pip to the latest version.</p>
<div class="highlight"><pre>python -m pip install --upgrade pip
</pre></div>
<p>Upgrade virtualenv to the latest version.</p>
<div class="highlight"><pre>pip install --upgrade virtualenv
</pre></div>
<p>Now close the admin command prompt. We will be installing the rest of the packages in to a <a href="http://docs.python-guide.org/en/latest/dev/virtualenvs/">virtualenv</a>, and that does not require admin access. The great advantage of using a virtualenv is that it keeps the packages we install isolated from each other, and from the system-wide packages.</p>
<h4 id="set-up-a-virtual-environment">Set up a virtual environment<a class="headerlink" href="#set-up-a-virtual-environment" title="Permanent link">¶</a></h4>
<p>Start a regular (non-admin) command prompt. Win-x then c is a quick keyboard shortcut for a non-admin command prompt.</p>
<div class="highlight"><pre>mkdir C:\PythonEnvs
virtualenv C:\PythonEnvs\Example
C:\PythonEnvs\Example\Scripts\activate.bat
</pre></div>
<p>This makes a new directory on the C: drive, makes a new virtualenv, and then activates the new virtualenv. You should see the name of the virtualenv in parentheses at the start of your command prompt, something like this:</p>
<div class="highlight"><pre>(Example) C:\Users\Me>
</pre></div>
<p>When a virtualenv is activated, it looks for installed Python packages in its own site-packages directory <code>C:\PythonEnvs\Example\Lib\site-packages</code>, instead of looking in the system wide site-packages directory <code>C:\Python27\Lib\site-packages</code>. Note that we don’t need to be in the virtualenv directory for it to be active.</p>
<p>The <a href="https://virtualenvwrapper.readthedocs.org/en/latest/">virtualenvwrapper</a> project is full of useful shortcuts for working with virtualenvs. For simplicity, I will be using only virtualenv, and not virtualenvwrapper, in this writeup. However, if you’re interested in setting up virtualenvwrapper, <a href="https://github.com/christianmlong/virtualenvwrapper-win">this patched version</a> works on Windows with the latest version of virtualenv.</p>
<h4 id="the-state-of-python-packaging-in-2015">The state of Python packaging in 2015<a class="headerlink" href="#the-state-of-python-packaging-in-2015" title="Permanent link">¶</a></h4>
<p>Installing Python packages on Windows has gotten a lot easier over the years. The <a href="https://pypi.python.org/pypi">Python Package Index</a> (PyPI) now provides pre-compiled binary installers in the <a href="https://wheel.readthedocs.org/en/latest/">wheel</a> format for many packages.</p>
<p>When a wheel is not available, pip can automatically compile C extensions using <a href="http://www.microsoft.com/en-us/download/details.aspx?id=44266">this compiler</a> that Microsoft provides at no cost.</p>
<p>However, there are still packages that are not available on PyPI. Many are distributed for Windows in the Windows installer format (.msi or .exe). Pip can not install these packages, but there <a href="#install-dependencies-that-are-not-on-pypi">is a way</a> to install them in to a virtualenv.</p>
<h4 id="install-twisted">Install Twisted<a class="headerlink" href="#install-twisted" title="Permanent link">¶</a></h4>
<div class="highlight"><pre>pip install Twisted
</pre></div>
<p>This will pull the latest version from PyPI. It will also install its dependencies. One dependency, <code>zope.interface</code>, will use the compiler to compile a C extension.</p>
<div class="highlight"><pre>. . .
Installing collected packages: zope.interface, Twisted
Running setup.py install for zope.interface
building 'zope.interface._zope_interface_coptimizations' extension
. . .
</pre></div>
<p>If you get a vcvarsall error, <a href="#install-a-compiler">install the Microsoft Visual C++ compiler</a>.</p>
<h4 id="install-dependencies-from-pypi">Install dependencies from PyPI<a class="headerlink" href="#install-dependencies-from-pypi" title="Permanent link">¶</a></h4>
<p>Install pywin32.</p>
<div class="highlight"><pre>pip install pypiwin32
</pre></div>
<p>As of March 2015, pywin32 <a href="https://pypi.python.org/pypi/pypiwin32">is available on PyPI</a> in the wheel format. That means it can be installed by pip. Note that in order to get the PyPI version, we must tell pip to install package <code>pypiwin32</code>, not <code>pywin32</code>.</p>
<h4 id="install-dependencies-that-are-not-on-pypi">Install dependencies that are not on PyPI<a class="headerlink" href="#install-dependencies-that-are-not-on-pypi" title="Permanent link">¶</a></h4>
<p>For packages that are not on PyPI, the installation steps are different. If the package is distributed using a Windows binary installer (.msi or .exe) we can use the older <code>easy_install</code> command to install it in to a virtualenv.</p>
<p>One such package is <a href="https://code.google.com/p/pyodbc/">pyodbc</a>, which my application uses to talk to the database. Twisted itself does not depend on pyodbc, so there is no need to install it if your application doesn’t use it.</p>
<p>As of March 2015, pyodbc is not available in wheel format from PyPI.
<a href="https://code.google.com/p/pyodbc/downloads/detail?name=pyodbc-3.0.7.win32-py2.7.exe&can=2&q=">Download</a> the Windows installer. Make sure to get the installer that matches your version of Python and your architecture. I am using this one “3.0.7 32-bit Windows Installer for Python 2.7”.</p>
<p>Use <code>easy_install</code> to install pyodbc in to the virtualenv from the executable Windows installer file.</p>
<div class="highlight"><pre>easy_install --always-unzip C:\Path\to\pyodbc-3.0.7.win32-py2.7.exe
</pre></div>
<p>I’ll talk about why we need <code>--always-unzip</code> in Part 3.</p>
<p>Not all installers will work with <code>easy_install</code> this way. See this <a href="http://stackoverflow.com/questions/25984095/install-pysvn-in-a-virtualenv/25984096#25984096">Stack Overflow question</a> for more details.</p>
<h4 id="twisted-application">Twisted Application<a class="headerlink" href="#twisted-application" title="Permanent link">¶</a></h4>
<p>In this series, I’m assuming you have your project structured as a Twisted Application. In that structure, you have a <code>.tac</code> file that is the connection point between your code and the Twisted server, <code>twistd</code>. </p>
<p>Here a basic <code>.tac</code> file, taken from the <a href="https://twistedmatrix.com/documents/current/core/howto/application.html">documentation</a> for Twisted Application.</p>
<p><code>service.tac</code></p>
<div class="highlight"><pre><span class="c"># You can run this .tac file directly with:</span>
<span class="c"># twistd -ny service.tac</span>
<span class="sd">"""</span>
<span class="sd">This is an example .tac file which starts a webserver on port 8080 and</span>
<span class="sd">serves files from the current working directory.</span>
<span class="sd">The important part of this, the part that makes it a .tac file, is</span>
<span class="sd">the final root-level section, which sets up the object called 'application'</span>
<span class="sd">which twistd will look for</span>
<span class="sd">"""</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">from</span> <span class="nn">twisted.application</span> <span class="kn">import</span> <span class="n">service</span><span class="p">,</span> <span class="n">internet</span>
<span class="kn">from</span> <span class="nn">twisted.web</span> <span class="kn">import</span> <span class="n">static</span><span class="p">,</span> <span class="n">server</span>
<span class="k">def</span> <span class="nf">getWebService</span><span class="p">():</span>
<span class="sd">"""</span>
<span class="sd"> Return a service suitable for creating an application object.</span>
<span class="sd"> This service is a simple web server that serves files on port 8080 from</span>
<span class="sd"> underneath the current working directory.</span>
<span class="sd"> """</span>
<span class="c"># create a resource to serve static files</span>
<span class="n">fileServer</span> <span class="o">=</span> <span class="n">server</span><span class="o">.</span><span class="n">Site</span><span class="p">(</span><span class="n">static</span><span class="o">.</span><span class="n">File</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getcwd</span><span class="p">()))</span>
<span class="k">return</span> <span class="n">internet</span><span class="o">.</span><span class="n">TCPServer</span><span class="p">(</span><span class="mi">8080</span><span class="p">,</span> <span class="n">fileServer</span><span class="p">)</span>
<span class="c"># this is the core part of any tac file, the creation of the root-level</span>
<span class="c"># application object</span>
<span class="n">application</span> <span class="o">=</span> <span class="n">service</span><span class="o">.</span><span class="n">Application</span><span class="p">(</span><span class="s">"Demo application"</span><span class="p">)</span>
<span class="c"># attach the service to its parent application</span>
<span class="n">service</span> <span class="o">=</span> <span class="n">getWebService</span><span class="p">()</span>
<span class="n">service</span><span class="o">.</span><span class="n">setServiceParent</span><span class="p">(</span><span class="n">application</span><span class="p">)</span>
</pre></div>
<p>If you don’t already have a project that you run under Twisted, the documentation has a nice set of <a href="https://twistedmatrix.com/documents/current/core/examples/index.html">examples</a> to get you started.</p>
<h4 id="install-your-application">Install your application<a class="headerlink" href="#install-your-application" title="Permanent link">¶</a></h4>
<p>Whatever Twisted application you are going to be running on this server, install it as you normally would. For example:</p>
<div class="highlight"><pre>pip install my_app
</pre></div>
<p>Now that everything has been installed, check it.</p>
<div class="highlight"><pre>pip freeze
</pre></div>
<p>should look something like this:</p>
<div class="highlight"><pre>my_app==4.14.2
pypiwin32==219
Twisted==14.0.2
zope.interface==4.1.2
other dependencies here
. . .
</pre></div>
<p>Newer versions of some of these packages may since have been released.</p>
<p>If you are installing using <code>pip</code>, make sure your <code>.tac</code> file is included in your distribution file. For exapmple, put this line in your <code>MANIFEST.in</code> file:</p>
<div class="highlight"><pre>include *.tac
</pre></div>
<p><a href="https://docs.python.org/2/distutils/sourcedist.html#the-manifest-in-template">Documentation</a> on <code>MANIFEST.in</code>.</p>
<h4 id="run-it">Run it<a class="headerlink" href="#run-it" title="Permanent link">¶</a></h4>
<p>Try it out. Make sure your virtualenv is <a href="#set-up-a-virtual-environment">activated</a>, and type:</p>
<div class="highlight"><pre>python C:\PythonEnvs\Example\Scripts\twistd.py --help
</pre></div>
<p>The virtualenv picks up the right Python path, but on Windows we have to specify the full path to the <code>twistd.py</code> file. This command should give you a nice help message and no errors.</p>
<p>Now try running your app under Twisted.</p>
<div class="highlight"><pre>python C:\PythonEnvs\Example\Scripts\twistd.py ^
--rundir "C:\PythonEnvs\Example\Lib\site-packages\my_app" ^
--python "my_tacfile.tac"
</pre></div>
<p>For readability, I have broken this long command in to multiple lines using <code>^</code>, the dos line-continuation character.</p>
<p>This command should print out some lines showing the <code>twistd</code> server starting up. Again, on Windows, we have to specify the full path to the app install directory when starting the <code>twistd</code> server. Go try out your app, and press Ctrl-c to shut down the server when you’re done.</p>
<h4 id="up-and-running">Up and running<a class="headerlink" href="#up-and-running" title="Permanent link">¶</a></h4>
<p>That’s it for Part 1. We have installed Python, set up a virtualenv, and gotten your Twisted app up and running. In <a href="http://www.christianlong.com/blog/twisted-on-windows-2015-edition-part-2.html">Part 2</a>, we will set up a Windows service to run the app, using the virtual service account that was introduced in Windows Server 2008. In Part 3, we will look at setting the right privileges for the app. In Part 4, we will package the app and its dependencies for deployment to test and production servers. Thanks for reading, and if you have any questions or suggestions, let me know. I’m on Twitter at <a href="https://twitter.com/christianmlong">@christianmlong</a>.</p>More on Pelican themes2015-03-05T16:03:00-06:00Christian Longtag:www.christianlong.com,2015-03-04:blog/more-on-pelican-themes.html<p>The <a href="https://github.com/getpelican/pelican-themes">Pelican Themes</a> project gathers together a bunch of nice themes for the <a href="http://docs.getpelican.com">Pelican</a> static blog generator.</p>
<p>This is a good case for using git’s <a href="http://git-scm.com/book/en/v2/Git-Tools-Submodules">submodule</a> feature.</p>
<p>Move to where your Pelican is installed (where your pelicanconf.py file is).</p>
<div class="highlight"><pre><span class="nv">$ </span><span class="nb">cd</span> ~/personal/blog
</pre></div>
<p>Make sure you are in a git working copy and the status is clean.</p>
<div class="highlight"><pre><span class="nv">$ </span>git status
</pre></div>
<p>Add the pelican-themes repository as a git submodule, and commit the change.</p>
<div class="highlight"><pre><span class="nv">$ </span>git submodule add git@github.com:getpelican/pelican-themes.git themes
<span class="nv">$ </span>git commit -am <span class="s2">"Add pelican-themes as a submodule"</span>
</pre></div>
<p>Now we should see the submodule listed</p>
<div class="highlight"><pre><span class="nv">$ </span>git submodule status
</pre></div>
<p>The pelican-themes project is itself made up of git submodules. Let’s take a look.
Change to the newly-created themes directory, and look at the submodules defined in there</p>
<div class="highlight"><pre><span class="nv">$ </span>git submodule status
-656296ab29a76d980155427f1f1ffe1892966a2a BT3-Flat
-a74606061d62be0f8508ca840375abe52ae24786 Responsive-Pelican
-bd337dffaa8aca10a1757d17385030e3a9d6b835 alchemy
-4ea9f35b517e67488f330799e8637e2e045d657e blue-penguin
. . . etc.
</pre></div>
<p>Here <code>git submodule status</code> prints all the submodules that make up the pelican-themes project, one line for each theme.
See the little minus sign before the commit hash on each line? That means that the submodule for that theme
is not initialized. We could initialize all the themes, but that would pull down a lot of code I’m not interested
in. I just want a few themes.</p>
<div class="highlight"><pre>git submodule init blue-penguin
git submodule init pelican-mockingbird
git submodule update
</pre></div>
<p><code>git submodule init</code> initializes the blue-penguin and pelican-mockingbird themes. Then, <code>git submodule update</code> clones the missing submodules.</p>
<p>Then edit your <code>pelicanconf.py</code> file, and add this line, giving Pelican the appropriate path to your theme.</p>
<div class="highlight"><pre>THEME = 'path/to/your/theme'
</pre></div>
<p>I’m using the <a href="https://github.com/jody-frankowski/blue-penguin">Blue Penguin</a> theme. I made <a href="https://github.com/christianmlong/blue-penguin">a few modifications</a>.
I’m not justifying the text, and I replaced the dark solarized code formatting with my own format based on the <a href="http://pygments.org/docs/styles">Pygments</a> ‘friendly’ style.</p>
<p>Here’s some code, to show off the syntax highlighting.</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">pygments.style</span> <span class="kn">import</span> <span class="n">Style</span>
<span class="kn">from</span> <span class="nn">pygments.token</span> <span class="kn">import</span> <span class="n">Keyword</span><span class="p">,</span> <span class="n">Name</span><span class="p">,</span> <span class="n">Comment</span><span class="p">,</span> <span class="n">String</span><span class="p">,</span> <span class="n">Error</span><span class="p">,</span> \
<span class="n">Number</span><span class="p">,</span> <span class="n">Operator</span><span class="p">,</span> <span class="n">Generic</span>
<span class="k">class</span> <span class="nc">YourStyle</span><span class="p">(</span><span class="n">Style</span><span class="p">):</span>
<span class="n">default_style</span> <span class="o">=</span> <span class="s">""</span>
<span class="n">styles</span> <span class="o">=</span> <span class="p">{</span>
<span class="n">Comment</span><span class="p">:</span> <span class="s">'italic #888'</span><span class="p">,</span>
<span class="n">Keyword</span><span class="p">:</span> <span class="s">'bold #005'</span><span class="p">,</span>
<span class="n">Name</span><span class="p">:</span> <span class="s">'#f00'</span><span class="p">,</span>
<span class="n">Name</span><span class="o">.</span><span class="n">Function</span><span class="p">:</span> <span class="s">'#0f0'</span><span class="p">,</span>
<span class="n">Name</span><span class="o">.</span><span class="n">Class</span><span class="p">:</span> <span class="s">'bold #0f0'</span><span class="p">,</span>
<span class="n">String</span><span class="p">:</span> <span class="s">'bg:#eee #111'</span>
<span class="p">}</span>
</pre></div>Looking for a new editor2014-12-19T10:35:00-06:00Christian Longtag:www.christianlong.com,2014-12-19:blog/looking-for-a-new-editor.html<p>I’m looking for a new code editor. I have been using <a href="http://komodoide.com/">Komodo</a> for ten years, and it has been very solid. I’m on Komodo 4 now; I like the project tree, which makes it easy to organize live and static folders, and local and remote files. It offers simple incremental search, and a good, clean interface for global find and replace in all open files.</p>
<p>Unfortunately, later versions of Komodo have introduced some features I can’t get used to. The Project pane was split in to Projects and Places, and it does not work well for managing big projects on remote servers. Also, the incremental search was changed for the worse.</p>
<p>I have been looking at <a href="https://www.jetbrains.com/pycharm/">PyCharm</a>. I like its code completion features, and I figured out how to do find and replace in just open files (when doing a find, choose a custom Scope, and choose Open Files). The multiple-cursor editing is better than in Komodo 9 beta: for example alt-click adds a new cursor to the current group. The refactoring support looks great. I’m not a big fan of the way remote files are handled. They are copied down to the local system, and then the editor syncs changes back to the remote system. Also, the backspace key eats up whole indents, not just one space character <a href="https://youtrack.jetbrains.com/issue/IDEA-87318">(bug report)</a>. If I wanted tab-like behavior, I’d use tabs!</p>
<p>One other possibility I’m looking at is using cygwin. A full tmux, vim, fish shell and powerline setup would be very nice.</p>