Bristle Software Python Tips

This page is offered as a service of Bristle Software, Inc.  New tips are sent to an associated mailing list when they are posted here.  Please send comments, corrections, any tips you'd like to contribute, or requests to be added to the mailing list, to tips@bristle.com.

Table of Contents:

  1. Django Framework
    1. Django Project Layout
    2. Django Coding Guidelines
    3. Django ORM
      1. defer() -- Defer loading some fields from the DB
      2. Unicode in SQLite
    4. DjangoCon Conference
      1. DjangoCon US 2016
        1. DjangoCon US 2016 Links
        2. DjangoCon US 2016 Notes
  2. PyCharm IDE
    1. PyCharm Shortcut Keys
  3. Python Language Tips
    1. Beware calling format() on long strings
    2. Debugging import statements
    3. Unicode
      1. Unicode Literals
      2. Unicode in SQLite
      3. Unicode CSV Reader
  4. Pip (Package Installer for Python)
    1. Use pip for installing packages
    2. Checking for updates

Details of Tips:

  1. Django Framework

    1. Django Project Layout

      Original Version: 11/22/2014
      Last Updated: 12/6/2014

      The default project layout created by the django-admin.py startproject command is fine for small projects, but here's an article that improves it in several ways:

      I use that layout in my projects, but with one additional improvement.  The article suggests using different settings files in DEV, TEST, and PROD, and accomplishes this by using different values for the DJANGO_SETTINGS_MODULE environment variable, or different values for the --settings option on the manage.py command.

      I use different settings files, but I specify them via a different environment variable that is not specific to Django.  Since my project is called HHL, I created an environment variable called HHL_ENVIRONMENT.  It typically has values like "prod", "test", "fred_dev", "joe_dev", "bob_dev", etc.  Anywhere that I need to do things differently on different servers, whether related to Django or not, I test the environment variable.  So how do I get Django to use different settings files based on this non-Django environment variable?  Read on...

      My settings/__init__.py file looks like this: 

      import os
env = os.environ['HHL_ENVIRONMENT']
cmd = 'from %s import *' % env
exec cmd

      My environment-specific settings files all reside in the settings folder and are named to exactly match the values of HHL_ENVIRONMENT: "prod.py", "test.py", "fred_dev.py", "joe_dev.py", "bob_dev.py", etc.  Each of these files looks like those described in the article: 

      # Start with common values
from common import *

# Put environment-specific overrides here

      My settings/common.py file contains the default values for all settings.

      The net effect is that Django loads settings.py as usual, but __init__.py imports an environment-specific file specified by HHL_ENVIRONMENT, and each of those files imports common.py and then overrides bits and pieces as needed.

      This has all the same advantages as the article above:

      1. Single set of static files (no editing required) for all environments
      2. Can hide the PROD settings by not deploying them to any server except PROD
      3. Can define as many different environments as you like
      4. etc.

      It also has these advantages:

      1. Single environment variable to define the environment for Django and non-Django components
      2. Now that we've taken advantage of the ability to put code in __init__.py instead of just using the existence of the empty file as a flag to define a Python module, we can easily add default values, logging and error handling to the file.  See below.

      The __init__.py file shown above is a greatly simplified "happy path" version of my real file, which looks more like: 

      
"""
__init__.py

Import the settings file for the desired environment, as specified by the
HHL_ENVIRONMENT environment variable, typically, "prod", "test", "dev", 
etc.  Defaults to "test".

Advantages:
- No need to edit any source files when we deploy to PROD
  vs TEST vs DEV.  Just set a single environment variable
  in each environment to identify it.
- Can define as many additional environments as we like:
  - Joe has joe_win_dev for his Windows environment, for example, 
    and joe_mac_dev for his Mac environment
- If we want, for ultra security, we can limit deployment
  of the prod.py file to the PROD server only, for less
  visibility of the PROD passwords, etc., hiding them even
  from some developers.
- Works equally well when the app is invoked via:
  - Local command line
  - PyCharm
  - Apache WSGI
"""

# Echo filename BEFORE importing anything since imported files may echo
# stuff too, so this looks more sensible in the log file.
print 'Executing ' + __file__

import os
import traceback

# Must catch exceptions here, or the app fails to start, reporting only
# the cryptic message:
#    Unknown command: 'runserver'
#    Type 'manage.py help' for usage.
try:
    env = os.environ['HHL_ENVIRONMENT']
except KeyError as e:
    print 'WARNING: HHL_ENVIRONMENT not defined.  Defaulting to test.'
    env = 'test'
except BaseException as e:
    print 'ERROR: Unexpected error getting HHL_ENVIRONMENT.  Aborting.'
    print traceback.format_exc()
    raise
if env == '':
    print 'WARNING: HHL_ENVIRONMENT defined as "".  Defaulting to test.'
    env = 'test'

# Ignore exceptions here because they get logged clearly with a useful
# error message:
#   ImportError: Could not import settings 'hhl.settings' (Is it on sys.path?):
#   No module named xxx
# and a stack trace.
cmd = 'from %s import *' % env
print cmd
exec cmd

# Echo to log file, just to be sure nothing is wrong
# Note: Explicit str() call is required for some types, like INSTALLED_APPS
#       when it is still the tuple from common.py, not replaced with the
#       list from one of the other settings/*.py files.  Othwerwise, we
#       get error:
#          TypeError: not all arguments converted during string formatting
try:
    print '. DATABASE_ROUTERS: %s'                % str(DATABASE_ROUTERS)
    if 'default' in DATABASES:
        if 'NAME' in DATABASES['default']:
            print '. Default DB name: %s'         % str(DATABASES['default']['NAME'])
        else:
            print '. Default DB name: %s'              % 'None'
    print '. INSTALLED_APPS: %s'                  % str(INSTALLED_APPS)
    print '. DEBUG: %s'                           % str(DEBUG)
    print '. ALLOWED_HOSTS: %s'                   % str(ALLOWED_HOSTS)
    print '. PYTHONPATH env var: %s'              % str(os.environ.get('PYTHONPATH',None))
    print ''
except BaseException as e:
    print 'ERROR: Unexpected error in settings/__init__.py.  Aborting.'
    print traceback.format_exc()
    raise

      11/23/2014 Update:
      Frank Wiles, the author of the article I cited above, suggests this improvement to my improvement.

      As of Python 2.7, there is a better way to do the dynamic import.  Instead of building an import command as a string and passing it to exec, use importlib(), as: 

      import importlib
importlib.import_module(env)
      One advantage of importlib(): it's more limited than exec, which makes it safer.

      Importlib can only be used to import the module named in HHL_ENVIRONMENT.  Exec can be used to execute arbitrary code, so if an attacker managed to create a value of HHL_ENVIRONMENT like the following, he could trick your Django code into deleting your entire /etc tree: 
      prod; from shutil import rmtree; rmtree("/etc/");

      Thanks, Frank!

      --Fred

    2. Django Coding Guidelines

      Original Version: 10/3/2012
      Last Updated: 10/3/2012

      Here are some guidelines I recommend for your Django code:

      • Use reverse() for URLs in Python code
        Instead of hardcoding the URLs for app pages in your Python code, use the reverse() function, to cause them to be looked up in your URLconf.  This allows you to easily change the URLs in your URLconf without breaking anything.   It also allows Python to detect and report typos.  For example, use: 
          return HttpResponseRedirect(
              reverse('mobile.views.donate_to_patient', args=(patient.id,))
              )
        instead of: 
          return HttpResponseRedirect(
              '/mobile/' + str(patient.id) + '/donate/'
              )
      • Use url for URLs in Django templates
        Instead of hardcoding the URLs for app pages in your Django templates, use the url tag, to cause them to be looked up in your URLconf.  This allows you to easily change the URLs in your URLconf without breaking anything.   It also allows Python to detect and report typos.  For example, use:
          <a href="{% url mobile.views.donate_to_patient patient.id %}">Donate</a>
        instead of:
          <a href="/mobile/{{ patient.id }}/donate/">Donate</a>

      --Fred

    3. Django ORM

      1. defer() -- Defer loading some fields from the DB

        Original Version: 12/23/2014
        Last Updated: 12/23/2014

        Need to get the Django ORM to not load data quite so automatically?

        You can use .defer() and/or .only() to tell it which fields to load at first.  Other fields are only loaded when you actually refer to them.  So, none of the rest of your code has to change, but you can delay the loading of certain fields until you actually need them.  This is useful to:

        1. Avoid loading fields that are expensive to load from the DB. For example, fields that are large, or that require lots of processing to convert to Python objects.
        2. Delay loading a field that may cause an error, until you enter an exception handler.

        Here's an example of the 2nd case.  On my current project, we have some bad data in a legacy DB that claims to be Unicode, but is actually encoded as Windows-1252.  We haven't get taken the time to find and fix all such data, and there's a legacy system that allows more bad data to be created.  So, when we write simple code like: 

          query_set = MyClass.objects.filter()
for item in query_set:
              do_something_here()
        the exception UnicodeDecodeError is sometimes raised as Django loads its cache for one of the loop iterations. To catch the exception, we'd have to write: 
          query_set = MyClass.objects.filter()
try:
    for item in query_set:
                  do_something_here()
except UnicodeDecodeError:
    deal_with_error_but_loop_has_already_aborted()
        Using defer() we can instead write: 
          query_set = MyClass.objects.filter().defer("risky_field")
for item in query_set:
    try:
                  do_something_here(item.risky_field)
    except UnicodeDecodeError:
        deal_with_error_but_then_loop_continues()

        For more info, see:

        --Fred

      2. Unicode in SQLite

        Original Version: 5/1/2016
        Last Updated: 6/7/2016

        Warning: SQLite always does case-sensitive comparisons of strings that contain Unicode chars.

        MySQL and SQLite act pretty much the same via Django.  That's why we can get way with using SQLite for speed when running regression tests, but using MySQL in our TEST and PROD environments.  However, I found one critical difference:

        For any string that contains non-ASCII chars, SQLite insists on doing a case-sensitive query, even if you explicitly use Django "__icontains" instead of just "__contains".

        On my project, we lowercase user-specified search strings before doing a search.  So, the regression tests (using SQLite) of our search function failed unless the data in the DB was lowercase.   A surprising bug, since the interactive web site (using MySQL) worked fine.  To fix it, we changed the test data in the DB to be lowercase.

        This includes lowercase versions of Unicode chars themselves, so I had to change:

          'Sample\xc3\xa1'

        not just to:

          'sample\xc3\xa1' ('\xc3' is uppercase 'A' with tilde)

        but to:

          'sample\xe3\xa1' ('\xe3' is lowercase 'a' with tilde)

        This is documented in the Django docs. See:

        --Fred

    4. DjangoCon Conference

      1. DjangoCon US 2016

        1. DjangoCon US 2016 Links

          Original Version: 7/25/2016
          Last Updated: 8/3/2016

          Here are all the links I gathered from DjangoCon US 2016.  Lots of links still missing because I didn't attend the talk, or didn't trip across the link on Twitter, Slack, etc.   Feel free to send me any missing ones and I'll add them.

          --Fred

        2. DjangoCon US 2016 Notes

          Original Version: 7/25/2016
          Last Updated: 8/2/2016

          Here are my raw notes from DjangoCon US 2016.  Basically just my scribblings from the talks I attended.  I think I've copied all the links to talks and such into the links tip above.

          
Sun 7/17 - 9 - DjangoCon US 2016, Philly, Wharton School, Penn -- Tutorials
---------------------------------------------------------------------------

  - Jon M. Huntsman Hall, The Wharton School
    3730 Walnut St, Philadelphia, PA 19104

  - Downloaded and installed DjangoCon app from:
    - https://guidebook.com/g/djangoconus2016/
    - Can use it on laptop also at:
      - https://guidebook.com/guide/66834/

  - 9-12:30 - An Introduction to Secure Web Development with Python/Django
              James Bennett
              Classroom 1 (F-45)
    - Slides at:
      - http://media.b-list.org/presentations/2016/djangocon/secure-web-dev.pdf
    - 34 attendees
    - Injection attack:  '); DROP TABLE slides;--
    - No absolute security
    - OWASP top 10
      ------------
      - https://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project
      1.  Injection attacks
        - SQL 
          - ''; UPDATE users SET is_superuser = true WHERE username = 'xxx';--
          - How to fix:
            - Parameterized query
            - Django ORM does this automatically
              - Unless you use extra() or raw(), in which case you should use
                params argument, not string concat
        - Mail header
          - Headers are separated by newlines.  Can fake them.
          - How to fix:
            - Django mail support does this automatically
              - BadHeaderError when you embed a newline, etc.
              - You should still sanitize to prevent BadHeaderError
              - James' "contact form" library
        - Command injection
          - Inject into executed shell commands
          - How to fix:
            - Python subprocess module
              - Never use shell=True
        - XML injection
          - Scripts,
          - Entity defs that:
            - Expand to consume all memory
            - Read a local (server) file into the XML stream
          - How to fix:
            - Use the defusedxml Python library for safe XML handling
      2.  Broken authentication and session mgmt
          - Fail to protect credentials
          - Transmit unencrypted
          - Credentials reset/overwritten (password reset)
          - Expose ids to public view (session in a GET param)
          - Session hijacking/fixation
          - How to fix:
            - Just use what Django gives you.  Very robust.  Battle-tested
              for over a decade.  Especially django-allauth
            - Use SSL
            - Use HSTS (HTTP Strict Transport Security)
              - Forces encryption
            - Make cookies secure and inaccessible to JS
            - NEVER expose a session ID
            - Use password validation (Django 1.9) to prevent easily 
              guessed credentials
            - Don't commit API keys to GIT
              - Bandit app uses a password checker to search codebase 
                for secure-password-looking strings
      3.  XSS (Cross Site Scripting)
          - JS in user input
            - Echo "Hello <name>", where user enters name as:
                <script>alert('oops');</script>
          - How to fix:
            - Never use |safe
              - Bleach tool
              - He's sometimes written his own DTD and validated explicitly 
                against it
            - Don't use innerHTML
            - Use |escapejs filter, but not for security
      4.  Insecure Direct Object References
          - Guessable PKs in URLs
            - Scrape ids 1-1,000,000 to see info on all users at a site
            - Use slug instead
              - No, only security through obscurity
          - How to fix:
            - Use a natural key, not a system-generated surrogate key
              - Instead of: http://example.com/users/23
                use:        http://example.com/users/janedoe
              - Not sequential or enumerable
            - Use a UUID instead of a PK or MD5, etc.
              - Hacker can't tie it to a user w/o having already hacked 
                the DB lookup table that 
      5.  Misconfiguration
          - Default admin password on router
          - Default accounts
          - Default master passwords
          - Auth bypasses
          - Debug mode
          - Security settings
          - Default error-handling behaviors
            - Don't allow it to show a traceback
          - How to fix:
            % python manage.py check --deploy
              - Added in Django 1.8
      6.  Exposed Sensitive Data
          - Insecure (non-encrypted) connections
          - Transmitted logs to log server
          - DB secure connections
          - Monitoring services
          - Not only logged info, but also credentials for logging system
          - How to fix:
            - Decorators:
              - @sensitive_post_parameters('username', 'password')
              - @sensitive_variables
                - Specified vars are scrubbed from all reported tracebacks
                - Also scrubs name like API, SECRET, PASS
            - Avoid getting sensitive data in GET query string
              - No control over what sees and logs it.  Apache server may 
                log it, for example.
              - Loggers are going to log (haters are going to hate)
      7.  Missing Function Level Access Control
          - How to fix:
            - Decorators:
              - @login_required
              - @permission_required
                - Specific, perhaps custom, permission
              - @user_passes_test
                - Custom authorization test
              - For CBV:
                - LoginRequiredMixin
                - PermissionRequiredMixin
                - UserPassesTestMixin
              - @require_GET
              - @require_POST
              - @require_http_methods
              - @require_safe
              - For CBV;
                - http_method_names = ['GET', 'POST']
            - Custom access control:
              - If you raise django.core.exceptions.PermissionDenied anywhere 
                in your code, Django will convert it to an HTTP 403 Forbidden 
                response
          - Idea:
            - Add a honeypot URL to end of urls.py?
      8.  CSRF (Cross-Site Request Forgery)
          - Legit user tricked into making an unintended request
          - How to fix:
            - Use Django {% csrf_token %}
              - Goes in form and cookie, and checks them on the fly
              - csrf_token is NOT tied to the session, just to the cookie
            - For Ajax:
              - https://docs.djangoproject.com/en/1.9/ref/csrf/#ajax
      9.  Components w/Known Vulnerabilities
          - Do security patches
          - Pip/yum/apt-get/aptitude install dependencies
            - Don't do yum install -y
            - Leftpad problem
          - How to fix:
            - Subscribe to django-announce for Django security advisories
            - Do OS/package updates regularly (at least weekly)
            - https://requires.io
              - Checks the requirements.txt file at your Git repo.  Free 
                for open source projects.
            - Keep your own package repo
      10. Unvalidated redirects and forwards
          - next=http://evilsite.com/...
          - How to validate redirection targets?  Hard problem.
          - How to fix:
            - Validate via django.utils.http.is_safe_url()
    - Beyond OWASP top 10
      -------------------
      - Force HTTPS
        - Add django.middleware.security.SecurityMiddleware to 
          MIDDLEWARE_CLASSES
        - SECURE_SSL_REDIRECT = True
          - Django setting to Force HTTPS
        - Use HSTS (HTTP Strict Transport Security)
          - Browser headers that forces encryption even on the first hit
            - Browser remembers timeout for delay below and refuses to even
              try HTTP during that time.  If you get locked out, use a 
              different browser.
            - Better than just SECURE_SSL_REDIRECT or Apache redirect
              because it prevents a deep link from connecting w/cookies etc.
              all unencrypted before the first redirect
          - SECURE_HSTS_SECONDS = 31536000 (one year, may want less)
          - SECURE_HSTS_INCLUDE_SUBDOMAINS = True
      - Content Sniffing by Browser
        - Some browsers will ignore Content-Type and sniff the first few 
          bytes to decide the type
          - SECURE_CONTENT_TYPE_NOSNIFF = True
            - Browsers take this as a directive to not sniff
      - Cookies
        - Cookies are useful but:
          - JS can access them
          - Get sent w/secure and non-secure requests
          - So, cookies can be exposed
          - How to fix:
            - CSRF_COOKIE_SECURE = True
            - SESSION_COOKIE_SECURE = True
              - Sends only over HTTPS
            - CSRF_COOKIE_HTTPONLY = True
            - SESSION_COOKIE_HTTPONLY = True
              - Prevents JS access to the cookies
              - May prevent Ajax access via JS, unles you go to the trouble
                of storing the CSRF value somewhere that JS can access it.
      - Frames
        - Clickjacking attacks
          - Overlay hidden frame with my content over a more expected/
            tempting site, and trick user into clicking on my site's buttons
          - Framebusting JS does not work.  DO not use it
          - django.,middleware.clickjacking.XFrameOptionsMiddleware
          - X_FRAME_OPTIONS = "DENY"
          - X_FRAME_OPTIONS = "SAMEORIGIN"
          - Per view:
            - django.views.decorators.clickjacking contains decorators to 
              let you do this on a per-view basis:
              - @xframe_options_exempt
              - @xframe_options_deny
              - @xframe_options_sameorigin
      - XSS
        - Autoescaping HTML is a good start, but can still sneak in JS
        - 6 chars allow all of JS: "{}[]+1"
        - How to fix:
          - SecurityMiddleware
          - SECURE_BROWSER_XSS_FILTER = True
            - Finds instances of query string embedded verbatim in HTML
            - Causes browser to refuse to render the page
          - Want a way to say only my original JS is allowed to be executed
            - CSP (Content Security Policy) HTTP header
            - Browser will refuse to load/execute any resource not 
              permitted by the CSP.  Can be told to send a report to a URL
              you specify, so you can see blocked things.
            - Supported by Chrome, Safari, FF, not IE
            - https://django-csp.readthedocs.io/
            - Can block inline JS, JS in CSS, etc.
      - Sandboxes
        - JavaScript Same-origin sandbox
        - Can override via CORS (Cross-Origin Resource Sharing), which uses
          an HTTP header to specify access-control policies
        - Similar features in Flash, Silverlight, etc, but often require 
          you to serve an XML file (not HTTP header) specifying the 
          restrictions:
          - Flash and Silverlight: crossdomain.xml
          - Silverlight:           clientaccesspolicy.xml
        - http://django-flashpolicies.readthedocs.io
    - Brief History of Django and Security
      ------------------------------------
      - Positives:
        - 2006: Django relese
        - 2007: First security issue (pre 1.0)
        - 2008: Django 1.0
                - Template auto HTML-escaping of template vars
        - 2010: Django 1.2
                - CSRF token
        - 2012: Django 1.4
                - Better password strage, vetted crypto, signed cookies,
                  clickjacking protection, error scrubbing, formal security 
                  process
        - 2013: Django 1.5, 1.6
                - Host header hardening, more password-storage improvements
                - Password storage
                  - Ideally, don't store them
                  - Else, use a non-reversible encryption
                    - PBKDF2 (Django default), BCrypt, SCrypt, Argon
                      - Can make them slower on purpose, by adding more levels
                        or re-hashing
                    - Not MD5, not SHA
                      - Too fast, can brute force them
                      - Bitcoin miners are good brute force crackers
                  - Django stores all of these explicitly for each password
                    - Salt
                    - Hashed result
                    - Algorithm
                    - Tuning params
                  - PASSWORD_HASHERS list in Django config file
                    - Can arrange for passwords to be updated as people use 
                      them since they type in the plaintext password), but 
                      can't process DB, converting them all
                  - Django has a special password value that is always 
                    invalid, to force a password reset for a user
                  - Timing attacks
                    - Can detect that Python aborted the string compare sooner
                      or later and know how much of the leading string was a 
                      match
                    - How to fix:
                      - Django password checking used a fixed-time comparison 
                        algorithm.  Python == operator does not.
        - 2014: Django 1.7
                - System check framework (manage.py check)
        - 2015: Django 1.8
                - Security middleware, deployment check
        - 2016: Django 1.9
                - Password validation, new permission mixins for CBV
      - Negatives
        - July 2005: First Django release
        - Aug 16, 2006: 1st security issue
        - Mar 1, 2016: 57th security issue
        - There's a page that lists all 57:
          - https://docs.djangoproject.com/en/releases/security/
      - Security Policy
        - Protect users, encourage responsible reporting/disclosure of
          security issues
          - Bug bounties, etc.
            - Done through hacker1 third-party service
          - security@djangoproject.com
            - Small team sees it and responds w/i 36-48 hours
            - Can send encrypted email via PGP
            - Tracked in private security-only GitHub repo 
            - Request CVE id after patching it
            - Pre-notify the limited number of people on the security 
              notification list (OS vendors, major Django sites, etc.)
              a week in advance
        - pip install now uses SSL.  Didn't always.
          % pip install rickroll
        - Checksum files of all releases
        - Can tell pip to check checksums of all packages, but you have to 
          list all recursive dependencies (from pip freeze) and specify 
          checksum of each.  May have to generate a checksum manually first
          from each package
        - Git signed tags
    - Patterns in Django Security Issues
      ----------------------------------
      - Not vulnerable to buffer overflow atacks
      - Very vulnerable to DoS attacks:
        - 2007: Caching the Accept-Language header
                - DoS via arbitrarily large Accept-Language header
        - 2010: One time base36 token that expires quickly
                - DoS via lots of large tokens
        - 2013: Dynamic growing number of forms in formset
                - max_num defaulted to unlimited; now 1000
                - DoS via large number of forms
        - 2013: Forcing long password
                - DoS via very long password
        - How to fix:
          - Check lengths of inputs before you start expensive processing
        - Format checking:
          - URLField "verify_exists"
          - EmailField regexp was too slow and timeable
          - Compressed image files
            - DoS to unzip a zip bomb
          - Large image files
            - TIFF doesn't store type in first few bytes
            - Now reads 4, then 8, then 16, etc.
      - HTTP Host header problems bit Django 5 times
        - Now allowed-hosts much be specified if DEBUG is off
    - Trust no one
    - But, there is no such thing as "secure"
    - URL for slides will be posted later today at:
      - https://twitter.com/ubernostrum

  - 2:00
    - Swag bag stuffing

  - 1:30-5 - Demystifying The Django Rest Framework
             Haris Ibrahim K V
             Classroom 1 (F-45)
             Sold out.  Bummer!
  - 1:30-5 - Supercharge Your Next Project with the Cookiecutter Django 
             Framework
             Audrey Roy Greenfeld, Daniel Roy Greenfeld
             Classroom 3
             Canceled?
  - 5:15-6 - Orientation Event
             Classroom 3 (F-55)
  - 3:30
    - LiveStream podcast w/Trey Hunner:
      - https://www.crowdcast.io/e/djangocon
        - Lacey Williams @laceynwilliams
        - Fred Stluka: 9:06 - 16:18
        - Lucie Daeye, Django Girls
        - Kojo Idrissa @Transition
          - 2nd DjangoCon
          - Programmer as of 12/1/2015, previously accountant and university 
            instructor
          - Taught in China for 2.5 years
          - From Houston Texas
        - Tim Graham and others
          - Tim Graham says Google "patch review checklist" to find:
            - https://docs.djangoproject.com/en/dev/internals/contributing
              /writing-code/submitting-patches/


Mon 7/18 - DjangoCon US 2016, Philly, Wharton School, Penn -- Talks
-------------------------------------------------------------------

  - 9-10 - Saron Yitbarek, CodeNewbie (Keynote)
           Auditorium (G-06)
    - Jeff Triplett 
    - Lacey Williams @laceynwilliams
    - Kojo Idrissa
    - Saron Yitbarek
      - Ruby dev
      - Lucky: Diversity vs Inclusion
      - Only privileged people have the luxury to be able to contribute 
        to FOSS.
      - saron@codenewbie.org

  - 10:30-11 - READABILITY COUNTS, Trey Hunner
    - http://treyhunner.com/readability-counts
    - Text width vs line width
    - for s,c,*_ in state_capitals
    - Blur code to focus on structure
    - Use list comprehensins to make one list from another
      - Like thevalue of a ternary operator
    - Write your own context manager with:
      - __init__()
      - __enter__()
      - __exit__()
    - Operator overloading instead of:
      - contains()    in           __contains__()
      - set()         =            __setitem__()
      - add()         +=           
      - remove()      del          __delitem__()
      - count()       len          __len__()
      - is_empty()    not          __bool__()
    - Uses classes to reduce same params to all functions
    - Have an explicit style guide

  - 11-11:30 BUILDING DYNAMIC DASHBOARDS WITH DJANGO AND D3, Clinton Dreisbach
    - http://www.dreisbach.us/blog/building-dashboards-with-django-and-d3/
    - @cndreisbach
      http://dreisbach.us
      http://git.io.cfs
      http://cfsdemo.rticds.org
    - django-url-filter
      - Maps GET params to queryset filters
    - Watches for changes to URL and goes to server if necessary
    - Ractive.js, not React.js
      - Re-reneders when data changes
        - Ractive.observe(path, function(newData) {})
    - D3 = toolkit for building visualizations, not a chart library
      - Plotly.js may be better, but 2MB of code
    - Reactive programming good for data visualization
    - Use higher level ibs on topfo D3
    - Use webpack and django-webpack-loader for serious frontend work      

  - 11:30-12 DJANGO FOR IOT: FROM HACKATHON TO PRODUCTION, Anna Schneider
    - http://www.slideshare.net/annarschneider
      /django-for-iot-from-hackathon-to-production-djangocon-us
    - @windupanna
    - @wattTime
    - https://github.com/aschn/cookiecutter-django-iot.git
    - Hackathon = "a really fun way to write really fragile code really quickly"
    - Steps:
      - Models
      - Views?  No, tasks.
        - Not just browser / server -- browser / server / IoT device
      - Apps
      - Deploy
      - Scheduled updates (cron)   
        - Hackathon way:
          - Heroku + mamanegemtn commands
            - 10 minutes, hourly, daily, skips some occurrences if 
              they run too long  
        - PROD way:
          - Celery
            - Complicates architecture a lot, but needed for PROD
            - Event-drive and perioidc tasks
            - Message broker transport queue
              - Redis good for this
            - @shared_task
        - If you have access to it:
          - Cron

  - 12-12:45 Lightning Talks
    - Emma, donate to the Django Software Foundation
    - Russell Keith-Magee, "I am the very model of a modern Django ModelForm"
      - https://youtu.be/D1FpYhDDjvQ
      - https://github.com/timb07/modern-django-modelform/blob/master
        /Modern_Django_ModelForm.pdf
    - Tom Christie, author of DRF, Funding DRF, @_tomchristie
    - Trey Hunner, "Give a Lightning Talk"
    - Paul Logston, @PaulLogston, http://pytube.org, written in Python 
      and Pelican, not Django
    - Tim Allen, @flipperpa, ENIAC 4K memory  vacuum tubes, 3300 Walnut St
    
  - 1:30-2:20 WEBSOCKETS: INTRO TO MESSAGING, Josue Balandrano Coronel
    - https://docs.google.com/presentation/d
      /1YMVkAY80T4c4ygLVDBJZmKAqp3myugnX2lXFCTjvENo
    - jcoronel@tacc.utexas.edu
    - @eusoj_xirdneh

  - 2:20-2:50 DJANGO AND POSTGRESQL: AN EVER-CLOSER UNION, Christophe Pettus
    - cpettus@pgexperts.com
    - PostgreSQL has:
      - Arrays (homogeneous)
        - 1-based
        - Use for M:M fields
      - Ranges
        - [1,3), etc.
      - GIN and GIST indexes
      - Constraint Exclusion
        - btree_gist extension to PostgreSQL
      - hstore fields (boring, not discussed)
      - JSON fields
      - JSONB fields
      - Full text search 
        - tsvector = block of text encoded for search

  - 2:50-3:20 SOLVING PROBLEMS WITH DJANGO FORMS, Kirt Gittens
              Dealertrack Technologies
    - https://docs.google.com/presentation/d
      /1geTFUcoFM9BtnIodTwefkvCVfjeWq1jZ2XyYORRnDQM/edit#slide=id.g35f391192_00
    - http://dmg-control.com/blog/#djangocon2016
    - Django Forms
      - Structure
      - Rendering
      - Validation/processing
    - DealerTrack needed to make them all dynamic:
      - Structure
        - Change fields on the fly
          - Can change fields[] dict on the fly
            - Doesn't scale well when lots of conditionally existing fields
          - Solution:
            - Treat fields as data.  Dict instead of object.
            - Could use OrderedDict or list
      - Rendering
        - Want the rendering to update w/dynamic field defs
        - Beware crispyforms
          - Forces you to tie forms to layout
      - Validation/processing
        - May want a field to call a 3rd party service to validate itself
          - Can use self.add_error(field, errors) internally
      - User-defined fields
    - DealerTrack moved to ReactJS/Redux
      - HTML forms rendered from JSON by ReactJS
      - Creating an SPA
      - Redux for state storage
    - Questions:
      - How to show server-side validation errors
        - Errors sent as JSON from React on server to JS on front end
      - Wagtail does similar stuff for forms

  - 3:50-4:20 HOW WE USED NLP AND DJANGO TO BUILD A MOVIE SUGGESTION 
            WEBSITE & TWITTERBOT, Vince Salvino
  - https://www.coderedcorp.com/media/documents/Djangocon_2016_Vince_Salvino.pdf
  - salvino@coderedcorp.com
  - @vincesalvino
  - Twython
    from twython import Twython
    twitter = Twython (key, secret, token, secret)
    twitter.update_status("blah, blah, blah")
  
  - 4:40-5:10 BUILDING JSON APIS WITH DJANGO / PINAX, Brian Rosner
    - https://speakerdeck.com/brosner/pinax 

  - 5:10-5:40 A NEW LOOK INTO APIS - GRAPHENE, Syrus Akbary
    - https://speakerdeck.com/syrusakbary/graphene-a-new-look-into-apis
    - REST API weaknesses
    - GraphQL
      - Created a year ago
      - JSON formatted QBE (Query By Example)
      - Query validations
      _ Strictly typed
      - Client specified desired fields, so no under/over-fetching
      - Introspection
      - Resolver Context
      - Only one round trip per query
      - Lots of language findings
    - Graphene
      - Pythonic binding to GraphQL
      - http://graphene-python.org/playground
      - Map from Django models
    - Demo was a pre-recorded screencast.  Smart!


Tue 7/19 - DjangoCon US 2016, Philly, Wharton School, Penn -- Talks
-------------------------------------------------------------------

  - 9:30-10:30 Andrew Godwin, Channels (Keynote)
    - https://speakerdeck.com/andrewgodwin/architecting-with-channels
    - http://github.com/andrewgodwin/channels-examples
    - http://channels.readthedocs.io
    - Works at EventBrite
    - Channels inspired by Meteor
    - Guaranteed "at most once" delivery not "at least once"

  - 11-11:30 CONFIDENT ASSET DEPLOYMENTS WITH WEBPACK & DJANGO, Scott Burns
    - https://speakerdeck.com/sburns/
      confident-asset-deployments-with-webpack-and-django
    - Resource packer/minifier

  - 11:30-12 SPICING UP DJANGO: AN INTRODUCTION TO MEZZANINE CMS, Ed Rivas
    - https://docs.google.com/presentation/d
      /1uGfuk1e8UTiveo89GuStPjzurybNjx916SwL9Z5Oa_8
    - http://github.com/jerivas/mezzanine-poll-demo
    - http://bit.ly/mezz-polls
    - @je92rivas
    - Add to standard Django polls app:
      - Publishing workflow (drafts, etc.)
      - Most recent first
      - Slugs, not PKS
      - Choices sortable
    - Use Mezzanine model: Displayable
      - Draft/Published status
      - Scheduled publishing/expiring
      - Meta fields
      - Status (published or not, etc.)
      - etc.
      - To use it:
        - class Poll(Displayable)
        - Refer to objects.published() , not objects.all()
        - Add new fields:
          - inlines
    - Admin features added:
      - Search
      - Date hierarchy
      - Filter
    - Orderable model for lists of choices
      - Admin can order the choices
    - Templates

  - 12-12:45 Lightning Talks
    - https://drive.google.com/file/d/0B4sJQVq6JHWwUy1ETW9rR1lSWUE/view
      Small Improvements to Django
      Or: a Super Easy Way to improve the lives of other developers
      Tobias McNulty - Caktus Group

  - 1:30-2:20 DJANGO SUPPORTING VIRTUAL REALITY GAME DEVELOPMENT, Rudy Mutter
    - http://djangocon-vr.surge.sh/
    - Google Cardboard prize to highest quiz score
    - Screen door effect = Too close to face, seeing pixels
    - MLP = Minimum Lovable Product
    - Unity game platform (C#)
      - Pokemon Go written in Unity
    - Django DRF API back end
      - Add new levels, level packs, etc.
      - Change levels
      - Store asset bundles (picures, images, sounds loaded JIT) 
      - In app purchases
      - Future proof

  - 2:20-2:50 HIGH-AVAILABILITY DJANGO, Frankie Dintino
    - http://github.com/hadjango
    - frankie@theatlantic.com
    - Monitoring/profiling
    - Performance improvements:
      - Low hanging fruit first
        - Query optimization
        - Caching
      - Front end also
      - Profilers: New Relic, Opbeat, django-debug-toolbar, django-silk
      - Monitoring: Alerta, new Relic, Opbeat, Chartbeat, Nagios, Zabbix
      - Caching:
        - CDN
        - Proxy cache (Nginx, Squid, Varnish)
        - {% cache %}
        - @cached_property
        - CachedStaticFilesStorage
        - Cached template loader
        - Cache for 2 secs in case site goes down briefly
        - Page caching frameworks
        - ORM caching
          - Mixed bag.  Why?
            - DB server is local and well tuned for reads
      - Query optimization
        - .prefetch_related(), .selct_related()
        - prefetch_related_objects()
        - .values(), .values_list()
          - Model instantiation/hydration missed by profilers
        - DB tuning, judiciious indexing
      - More hardware
    - Request queueing
    
  - 2:50-3:20 MAKING A SPLASH WITH YOUR OPEN SOURCE PROJECT Russell Keith-Magee
    - Russell@Keith-Magee.com
    - @freakboy3742
    - Django core developer for 11 years
    - Doing BeeWare also
    - Risks of Open Source projects:
      - "Cargo Cult"
        - Trying to lure planes via landing strips
      - Don't be an Underpants Gnome
        - Steal underpants, step 2?, profit
      - "You might find this useful" vs "I want everyone to use this"
        - Communicate your intentions
          - Everything changes
    - Setting up for success
      - Out of the box experience matters
        - Good tutorial
      - Time from zero to kick-ass
      - Low suck threshold
    - Getting people involved:
      - Potential users
      - New Users
      - User
      - Community member
      - Contributor
      - Core team
      - Leader
    - Backward compatibility
      - How much does it matter:
        - Time is relative (short transient projects don't care)
    - Boring is a virtue in production, exciting does not belong in PROD
    - Tools vs ecosystems
      - Jacob's Tractors (Jacob Kaplan-Moss' tractor is "Tinkerbell")
        - Same 3-point linkage for connecting add-ons for 90 years!
      - Metcalfe's Law = Utility of a network increases with the square of 
        participants
      - Sturgeon's Law = 90% of everyhing is crap
    - Communication is key to success
    - 10 years ago competitors to Django:
      - CherryPy
      - TurboGears
      - Repoze.BFG
    - "Django Evolution" was failed predecssor of South
    - Sales and Marketing
    - BeeWare:
      - All that matters is whether the code change makes one more test
        pass, deals with one more edge case, or improves the code in some 
        small way.  "Architecture can be cleaned up later" !?!??!
    - Plan ahead: Fortune favors the prepared mind

  - 3:50-4:40 WALKING DOWN THE A11Y ROAD - LESSONS LEARNT FROM WORKING ON 
              ACCESSIBILITY OF A DJANGO PROJECT, Radina Matic
    - radina@learningequality.org
    - Add alt attribute to images so Google can index them
    - Add hidden text to be read by screen readers

  - 3:50-4:40 THE FRAUD POLICE ARE COMING: WORK, LEADERSHIP, AND IMPOSTER 
              SYNDROME
    - Amanda Clark, Briana Morgan
    - https://speakerdeck.com/brianalmorgan/the-fraud-police-are-coming-work-leadership-and-imposter-syndrome
    - amandajclark.com
    - brianalmorgan.com
    - "People who have Imposter Syndrome are less likely to raise their hands"
    - "Superman is fictional. You are not."
    - "Heroes don't limit themselves to fights they know they can win"
    - "Shame cannot survive empathy"

  - 4:40-5:10 MIGHTY MODEL MANAGERS, Shawn Inman
    - Worked in banking, then healthcare
    - Multiple managers for readability:
      - Person.people.all()
      - Person.editors.all()
      - Person.authors.all()
    - Subclass QuerySet instead for more flexibility.  Refer as:
      - Person.people.all()
      - Person.people.editors.all()
      - Person.people.authors.all()
      - Person.people.editors.part_timers()
      - Person.people.authors.full_timers()
    - Get rid of the explicit custom manager:
      - PersonQueryset.as_manager()
    - Protect future-you from present-you:
      - Manager to deal with is_active flag
        - Add _active() and call it from authors(), editors, etc.
        - .all() would still return all, not only active.  Good!
    - Less duplicate code --> better test coverage
    - Do all the complex thungs from the next talk (querysets) but hide
      all the details in the custom model manager.
    - Django does NOT automatically give you a .object if you define a 
      custom manager, but you can define it yourself.
    - Can also do:
      - Caching
      - Logging (though need access to user to log username)

  - 5:10-5:40 I DIDN'T KNOW QUERYSETS COULD DO THAT, Charlie Guo
    - https://dl.dropboxusercontent.com/u/1807435/reveal/index.html
    - @CharlieRGuo
    - Basics
      - annotate() -- Returns new field for each item
      - aggregate() -- Returns single field for all items
      - etc...
    - Unbasics
      - select_related() -- SQL join
      - prefetch_related() -- Python join
      - only() -- Certain fields only (but will lazy load others if needed)
      - defer() -- Opposite of only -- list those not wanted
      - in_bulk()
    - Conjunction Dysfunction
      - How to do OR instead of AND in SELECT clause?
        - Q() with AND (&) OR (|) and NOT (~)
    - Query Arithmetic
      - F()
        - Implicit field reference
        item_sum = Sum(F('unit_price' * F('quantity'))
        aggregate(amount = item_sum)
      - F()
        .update(purchases=F('purchases')+1)
    - Query Expressions (like F())
      - Avg, Count, Max, Min, StdDev, Sum, Varianbce
      - Coalesce, Concat, Inverse, Func, Conditional, ExpressionWrapper, 
        Value, F, etc.
      - producte.annoate - Func(F('nane'), function='LOWER'))
      - Can easily define your own custom Query Expressions by 
        overloading 4-5 methods
      - ExpressionWrapper example
    - Deeper:
      - Raw SQL
        - Not usually a good idea, but sometimes...
        - extra() -- Planned for deprecation
          - select, where, tables, order_by, select_params, params
        - .raw("SELECT xyz FROM pdq ...")


Wed 7/20 - DjangoCon US 2016, Philly, Wharton School, Penn -- Talks
-------------------------------------------------------------------

  - Missed morning of DjangoCon
    X - 10-11 Breaking through the Glass Walls of Tech, Janice Levenhagen, 
    X         ChickTech (Keynote)
    X - 11:30-12 THIS OLD PONY: WORKING WITH LEGACY DJANGO APPS, Ben Lopatin
    X - 12-12:45 Lightning Talks

  - 1:30-2:20 FROG AND TOAD LEARN ABOUT DJANGO SECURITY, Philip James
    - http://bit.ly/djangotoad
    - https://www.wordfugue.com/talks/frog-and-toad-learn-about-django-security/
    - http://wordfugue.s3.amazonaws.com/talks/FrogAndToadLearnDjangoSecurity.pdf
    - @phildini
    - #djangotoad
    - mark_safe(), |n, |safe
    - @csrf_exempt()
    - @method_decorator(csrf_exempt, dispatch)
    - Clickjacking, iframe of our site in someone else's
    - XFrameOptionsMiddleware
    - @method_decorator(xframe_options_exempt, dispatch)
    - HOST header validation
      - get_host()
      - ALLOWED_HOSTS
    - Passwords
      - Uses old hasher, updates to new hasher
    - Constant Vigilance
      - Code reviews, standards checkers, security audits
    - HTTPS only
    - CSP (Content Security Policy) Reporting
      - Browser will enforce loading only from listed domains, or log when
        loaded from other domains.
    - django-encrypted_fields
      - https://github.com/defrex/django-encrypted_fields
        - Protects data at rest, like HTTPS does in transit
    - django-secure
      http://django-secure.readthedocs/en/v0.1.2
    - http://ponycheckup.com
    - "Making Django ridiculously secure", Kelsey Innis
    - Don't push SECRET_KEY to Git
    - Questions/Answers:
      - DoS
        - DDoS 
        - Rate-limiting at the API level (DRF) and the site
      - Key mgmt policies
        - Google keyczar
      - Encrypted settings via SECRET_KEY
        - Can load a new random one each time, if not to be stored anywhere
        - Create salted HMACs
        - Used for secure cookie signer
        - Marcus (in the front row) is a good security guy
      - No such thing as perfect security.  Use good practices, and maintain
        constant vigilance.  Be aware of attack surface.  Who are likely 
        attackers, from where and why?  Subscribe to security feeds in your 
        Slack
      - Tools:
        - BruteXSS
        - Scan for "|n", "|safe", "csrf"

  - 2:20-2:50 ENTOMOLOGY 101: EFFECTIVE BUG HUNTING, Frank Wiles
    - Try to not "enbug" (add bugs)
    - django-debug-toolbar
      - SQL queries done, from where, query plan, etc.
      - Template context
      - Signals fired
      - Logging done
    - pdb
      - He never uses debuggers.  ??
      - pdb.set_trace()
      - pdb.run()
      - pdb.run_eval()
      - Type "interact" to get an ipython shell (or ipdb?)
    - ipython embed()
      - As good as, or better than pdb?
    - Logging
      - Standard Django logging
      - logbook
      - structlog (like fred logging)
    - Divide and conquer
      - Rule out places where bug is not, then dig deeper
    - Start with a clean slate
      - Take a break, start over.  Fresh data, fresh perseptive
      - Fresh set of eyes ((rubber-ducky debugging)
    - Change one thing at a time
    - Uses Sublime Text
    - Questions:
      - ipdb vs pdb vs ipython?
      - How to teach junior devs how to debug?
        - Walk them through your thought process
      - How to debug distributed micro services?
        - Shared log file
      - How to debug a JS SPA making REST calls to Django?
        - React/Redux debug tools
      - New/cool features of RevSys Spectrum? (Tim Allen)

    - Fred's Notes from Frank Wiles' security talk at PhillyPUG in 2014:
    X Thu 10/16 - 6:30 - PhillyPUG, python.org goes Django on Python3
    X   Rm F-60, Huntsman Hall, Wharton, 3730 Walnut St, Philadelphia PA 19104
    X   - http://www.meetup.com/phillypug/events/206061252
    X   - Tim Gross, organizer
    X   - Timothy Michael, organizer
    X   - Tim Allen, IT Director, Advanced Initiatives, Wharton
    X     - @flipperpa
    X     - flipper@peregrinesalon.com
    X   - Frank Wiles frank@revsys.com
    X     - Ported python.org to Django
    X     - Python/Django trainer, helping Wharton move to Python/Django from CF
    X       and a mix of other technologies
    X     - Lives in Kansas, travels to Philly clients for a week or 2 5-6 times 
    X       per year
    X     - Uses Mac, Time Machine
    X     - Uses AWS, Rackspace and other Cloud providers
    X     - 6-7 years exprience with Python, 3 years w/Django
    X   - Notes:
    X     - ipython 
    X       - Better interactive shell
    X       - embed
    X         - Enters debugger shell
    X             from IPython import embed
    X             def some_complicated_code():
    X                 """  Maybe a daemon or other console process """
    X                 if some_failure_condition:  
    X                     embed() 
    X     - pip
    X       - requirements/base.txt
    X       - requirements/dev.txt, requirements/test.txt, requirements/prod.txt
    X         - Include base.txt with line:  -r base.txt
    X         - Add env specific lines after that  
    X       - pip install piptools
    X         - pip-review
    X           - Shows what versions you are using and could update to
    X     - PYTHONSTARTUP=/path/to/some/code.py
    X       - http://revsys.com/blog/2014/jul/12/python-tip-shell-pythonstartup/
    X     - virtualenvwrapper
    X       - workon env_name (no need for full path)
    X         - Does activate and sets up post-deactivate hook
    X         - Runs these files from <venv>/bin
    X           - preactivate
    X           - postactivate
    X           - predeactivate
    X           - postdeactivate
    X       - Frank uses it, even when not doing python, just for Linux shell 
    X         environment setup/teardown with separate env for each working 
    X         directory that he does a workon to
    X       - http://revsys.com/blog/2014/jul/12/python-tip-shell-pythonstartup/
    X     - Fabric
    X       - Python script for sys admin, even to remote hosts
    X       - local("shell command")
    X       - Use its "roles" instead of Puppet, Chef, Ansible, etc.
    X     - py.test --reuse-db
    X       - For speed, don't re-create the DB each time
    X     - Sphinx docs
    X       - sphinx-apidoc uses "autodoc" extension to generate Javadoc-like
    X         docs from Python code
    X     - coverage
    X     - Use a Git tag to record date/time of a PROD deployment
    X     - logging_tree
    X       - To analyze/debug logging config
    X     - Tools for streaming logs to for filtering and event notification
    X       - loggly.com
    X       - logentries.com
    X       - PaperTraillApp.com
    X     - honcho
    X        - Like Foreman or Heroku
    X        - Configure a ProcFile
    X     - ack
    X       - Better grep
    X       - ack --python 
    X         - Searches only .py files
    X     - jq
    X       - Cmd line tool to parse/process/filter JSON
    X     - django-debug-toolbar
    X       - Run in browser, but server must be in debug mode
    X       - Reports timing, SQL queries, etc
    X       - Good for profiling, performance, SQL, Django server, etc.
    X       - What line in a template did DB access?
    X       - Templates used
    X       - Data passed to template
    X       - Caching
    X       - Signals
    X       - Python logging messages generated, etc.
    X       - Intercept and debug browser redirects
    X       - [Tue 11/11/2014]
    X         Researched:
    X         - http://django-debug-toolbar.readthedocs.org/en/1.2.2/index.html
    X           - Describes what Frank told us
    X           % manage.py debugsqlshell
    X             - Shows SQL generated by Django calls you make from the shell
    X     - Gulp
    X       - gulpjs.com
    X       - livereload.listen()
    X         - Can do SASS preprocessing, minify, etc
    X         - Watch for templates that change, and do a live reload via a
    X           browser plugin.
    X           - Edit template file, browser refreshes automatically
    X           - Can have 3 different screen sizes and have them all autorefresh
    X             whenever you edit a template
    X           - See also (per Collin Anderson of django-users list):
    X             - http://livereload.com/
    X       - http://revsys.com/blog/2014/oct/21/ultimate-front-end-development-setup
    X     - See what's new occasionally
    X       - Mailing lists:
    X         - python-weekly
    X         - django-weekly
    X         - Do Google searches occasionally, for past month only, for tips
    X           on your favorite tools
    X     - ssh has a ControlMaster feature to reuse same SSH connection from
    X       another process
    X       - In .ssh/config
    X           ControlMaster auto
    X           ControlPath /tmp/localhost-%l-remotehost-%h-port-%p-user-%r
    X           host names, ports, users, etc.
    X     - Celery alternatives
    X       - rq
    X       - Use redis as a lightweight broker
    X     - Use Celery for simple task scheduling?
    X       - No.  Use cron to run django-admin.py or manage.py
    X     - IDE
    X       - Until recently, Frank always used vi and shell
    X       - Switched to sublime in vi mode
    X       - Doesn't like slow startup time of an IDE
    X       - Doesn't need IDE's debuggers
    X       - Has clients that use PyCharm
    X     - Speed
    X       - C and Go may be faster for some workloads, like video editing
    X       - For websites, developer time is much faster in Python, and speed
    X         is cheap: caching, server farm, etc.
    X       - Drupal is much slower than Django
    X       - Don't often need to go to Java
    X       - C libraries callable from Python:  numpy, scipy 
    X     - pypy - JIT compiler
    X     - yslow
    X       - Chrome plugin for why a Web site is slow
    X     - pagespeed
    X       - Same as yslow, but by Google
    X     - Typical performance opportunities:
    X       - DB connection pooling, as of Django 1.6, or with an add-on previously
    X         From:
    X         - http://media.revsys.com/media/talks/slides/performance-slides.pdf
    X         DATABASES = {
    X             'default': {
    X                 'ENGINE':
    X                 'django.db.backends.postgresql_psycopg2',
    X                 'NAME': 'rando',
    X                 'USER': 'rando',
    X                 'CONN_MAX_AGE': 300,
    X             }
    X         }
    X       - DB caching
    X       - Page caching
    X         - Varnish
    X       - More server processes/threads per box
    X         - One for Apache, one for memcached, one per remaining core for WSGI.
    X       - Template caching
    X         - django.template.loaders.cached.Loader
    X           - Like collectstatic for static files
    X           - To avoid searching all template subdirs all the time for each
    X             template file.
    X           - Details from:
    X             - http://media.revsys.com/media/talks/slides/performance-slides.pdf
    X             Change this:
    X                 TEMPLATE_LOADERS = (
    X                     'django.template.loaders.filesystem.Loader',
    X                     'django.template.loaders.app_directories.Loader',
    X                 )
    X             Do this, in PROD settings.py only, so template changes will 
    X             still be picked up on the fly in DEV and TEST:
    X                 TEMPLATE_LOADERS = (
    X                     ('django.template.loaders.cached.Loader', (
    X                         'django.template.loaders.filesystem.Loader',
    X                         'django.template.loaders.app_directories.Loader',
    X                     )),
    X                 )
    X       - Avoid include, use {% block %} of parent template instead
    X       - Far future "expires" headers (at least an hour)
    X         - Find a way to allow browser-side caching, but still update when
    X           it actually changes
    X           - We're doing that now by putting version number in 
    X             collected_static files.
    X           - Should set far future expires headers on those files?  How?
    X       - django-compress
    X         - {% compress css %}
    X            - Generates combined CSS file with a hashed name.
    X            - Fewer separate CSS file requests.
    X            - Can use a far future expires header because any change to 
    X              any CSS file generates a different hashed composite name
    X       - Session caching, from:
    X         - http://media.revsys.com/media/talks/slides/performance-slides.pdf
    X         Change this:
    X             SESSION_ENGINE = 'django.contrib.sessions.backends.db'
    X         To this:
    X             CACHES = {
    X                 'default': {
    X                     'BACKEND': '
    X                          django.core.cache.backends.memcached.MemcachedCache',
    X                     'LOCATION': '127.0.0.1:11211',
    X                 }
    X             }
    X             SESSION_ENGINE = "django.contrib.sessions.backends.cache"         
    X         You should really be caching your data at some level anyway, so 
    X         hopefully you have memcached or redis setup already but just in 
    X         case here is a simple memcached setup. Just change your 
    X         SESSION_ENGINE to store your sessions there as well. If you are 
    X         using session data, you can use cached_db to only write to the DB 
    X         when the session data actually changes. 
    X       - Switch from Apache to nginx
    X       - Switch from MySQL to PostgreSQL
    X       - Switch from standard Django templates to Jinja2 templates
    X       - See also:
    X         - https://speakerdeck.com/jacobian
    X           /django-minus-django-djangocon-eu-2014
    X     - disqus is built with Django
    X     - Flask is like Django, but lighter weight, less capable
    X     - Bottle is similar to Flask
    X     - No concerns about Python 3.  He's dropping Python 2.
    X   - Notes I gathered later from Frank Wile's blog posts
    X     - bookmarklets:
    X       - http://revsys.com/blog/2014/feb/19/django-debugging-bookmarklet-trick/
    X   - How to watch this presentation remotely?  Some people are.
    X     - Saw email link to "bluejeans" teleconf when I got home

  - 2:50-3:20 GIT IN CONTROL, VERSION CONTROL AND HOW IT COULD SAVE YOUR LIFE,
              Rachell Calhoun
    - https://speakerdeck.com/rachellcalhoun/git-in-control-version-control-and-how-it-can-save-your-life
    - Fork at GitHub to your GitHub repo, then git clone to local computer
    - Git push to your GitHub repo, then issue GitHub pull request
    - % git remote add upstream path/to/oroginal/repo
    - % git pull upstream master
    - Tricky stuff:
      - Recover via delete and re-clone
      - % git stash -u (--include-untracked)
        - % git stash apply + drop = pop
        - git stash branch branch-name (creates branch and applies stach to it)
      - % git reset HEAD~2 (ignore 2 bad commits)
      - % git reset --hard
      - % git rm
        - % git reset filename (does not delete local file)
      - Edit commit message
        - % git commit --amend
      - % git diff --cached 

  - 3:50-4:40 DJANGO AND REACT: PERFECT TOGETHER, Jack McCloy
    - https://docs.google.com/presentation/d/1TRgDA-fImXTj88eeNW37iokt4ehuK8Q6Y-7hZnBqW4g
    - Uses components -- *.jsx (Like HTML w/JS snippets)
    - Uses ES6 (classes, constructors, modules, iterators, generators, 
      promises, fat arrows)
      - Requires explicit exports to be able to import
    - Replace Django template w/React to:
      - Make it an SPA
      - Share code for Web, iOS, Android (React Native)
      - Manage/manipulate complicated state (biggest advantage)
    - View layer becomes REST API
    - React principles:
      1. Agnostic about rest of your tech stack
      2. Declarative views
         - Data down, actions up [like event bubbling]
      3. Component based
    - react-router does URL routing inside React SPA
    - Setup
      - Node.js and npm
      - Init npm inside Django project
        - Creates package.json (like requirements.txt)
      - Install Node packages:
        - % npm install pkg_name --options
        - Contents of package.json
          - babel-* packages are JS --> ES6 shim
          - radium-*
          - webpack-*
          - react-*
          - redux-*
            - Package and pattern for state management
      - Learn Redux
      - Make server.js 
        % node server.js (like manage.py runserver)
        - webpack.config.jsx
          - Define React web trees here
        - Index.jsx
      - In Django template:
        - {% load renderBundel from...%}
    - Questions
    - More stuff:
      - Component Lifecycle
      - Flux/Redux
        - Flux:
          - Manage state BETWEEN components.  Store state outside components
            so other components can access it indirectly as property and send 
            back actions
        - Redux
          - Like Flux, but cnetralizes state changes in Redux store
          - Dispatch an action to Redux store, which preserves old state, 
            and transitions to new state
      - State vs Props
      - React-router / Redux-router
        - Redux-router more powerful, but not always needed
      - Authentication / access restriction (using Django)
      - React as a tool to eliminate CSS
      - Learning React is hard perhaps because you have to learn a lot of 
        new patterns, but easy to use once you get it.

  - 4:40-5:10 THE CITY AS CYBORG: A HISTORY OF CIVIC TECHNOLOGY IN THE 
              FIRST QUARTER OF THE 21ST CENTURY, Mjumbe Poe
    - 2025: Andrew Godwin: You are a networked biosensitive app
    - https://github.com/mjumbewu/cyborg-city-presentation
    - https://mjumbewu.github.io/cyborg-city-presentation
    - https://github.com/mjumbewu/cyborg-city-presentation/blob/gh-pages/SCRIPT.md

  - 5:10-5:40 Closing


Thu 7/21 - DjangoCon US 2016, Philly, Wharton School, Penn -- Sprints
---------------------------------------------------------------------

  - Volunteered to be a sprint volunteer for Django Sprint, helping orient
    other sprinters, even if I get less sprinting done myself

  - Tim Graham: Life of a Django Ticket
    - https://docs.google.com/presentation/d
      /1bEtU6nYm-ZrjyFGZNBA8HaU9BNJBkPwQDwGOKA4fQS0/
    - http://code.djangoproject.com/wiki/LittleEasyImprovements
    - "Easy" tickets at dashboard.djangoproject.com
    - 5 tickets per day, triaged by Tim Graham
    - 26,921+ over 10 years
    - Write test case.  Confirm still exists
    - git bisect
    - ticketnnn branch name
    
    - Commit comment
      - Fixed #nnnn -- fixed a bug
      
        More details

        Thanks John Doe for reporting it...
    - Create pull request
      - Add pull reauest link to ticket,  Check "has patch"
    - "Brace yourself.  Tim's review is coming."
    - Flake8 for coding style
    - Jenkins CI
    - Add yourself to AUTHORS and submit licensing agreement
    - Squash smaller commits
    - Put in Git comment:
      "buildbot, test this please"
    - Ticket number in comment to auto close the ticket    
    - IRC #django-sprint
    - Slack
    - Discuss on django-dev
    - Core team: Tim Graham, Markus, Russell, etc.
    - Committer: Tobias

Fri 7/22 - DjangoCon US 2016, Philly, Wharton School, Penn -- Sprints
- ENIAC Tour w/Tim Allen and ~20 others
- Core committers in Django sprint room:
  - Baptiste Mispelon
  - Markus Holtermann

- Contributing to Django Framework
  - Steps below are pulled from a combination of:
    - https://docs.google.com/presentation/d
      /1bEtU6nYm-ZrjyFGZNBA8HaU9BNJBkPwQDwGOKA4fQS0/
    - https://docs.djangoproject.com/en/dev/internals/contributing/
    - https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/working-with-git/
    - https://docs.djangoproject.com/en/dev/intro/contributing/
  % mcd ~/fred/dsf
  - https://github.com/django/django
    - Fork (not "Clone or download")
  % git clone git@github.com:bristle-com/django.git
    - Could have done:
      % git clone https://github.com/bristle-com/django.git
      but then it would use https instead of ssh (less secure, per Marcus) 
      for future interactions until I edited the .git/config file.
  % o https://github.com/settings/ssh/465988
    - Clicked "Approve"
  % git clone git@github.com:bristle-com/django.git
      Cloning into 'django'...
      remote: Counting objects: 361407, done.
      remote: Compressing objects: 100% (22/22), done.
      remote: Total 361407 (delta 3), reused 0 (delta 0), pack-reused 361384
      Receiving objects: 100% (361407/361407), 139.23 MiB | 5.32 MiB/s, done.
      Resolving deltas: 100% (260593/260593), done.
      Checking connectivity... done.
      Elapsed time: 0:45.33  CPU seconds: 14.652/5.229
    - Success!
  % cd django
  % git remote add upstream https://github.com/django/django
    - Note: Use HTTPS syntax here, not SSH syntax, because I don't have
            rights to login or push, just to pull.
  % git config --global core.excludesfile ~/.gitignore_global
  % e ~/.gitignore_global
    - Added rules to exclude .idea, etc.
  % git fetch upstream
  % git checkout master
  % git rebase upstream/master
  % cd tests
    % pip install -r requirements/py3.txt
    % ./runtests.py
  - Looked over existing open tickets
  - There's a doc ticket to re-write the instructions to use the new
    Git workflow.
  - Good explanations of forking a GitHub repo and issuing a pull request:
    - http://blog.scottlowe.org/2015/01/27/using-fork-branch-git-workflow/
    - http://www.eqqon.com/index.php/Collaborative_Github_Workflow

          --Fred

  2. PyCharm IDE

    1. PyCharm Shortcut Keys

      Original Version: 10/3/2012
      Last Updated: 10/3/2012

      Here is a list of some of the more useful shortcut keys for the PyCharm IDE:


      Did I miss any good ones?  Let me know.

      --Fred

  3. Python Language Tips

    1. Beware calling format() on long strings

      Original Version: 1/9/2015
      Last Updated: 1/9/2015

      Ever have trouble with the .format() method of a Python string not inserting its values?

      Here's a easy way to accidentally cause that:

        print (
    "first part of long string that I purposely split into" +
    " multiple lines to keep all lines under {0} chars" +
    " but I still want the number 80 inserted above".format(80)
)

      This fails because of the precedence of operators. The .format() method is called for the 3rd line of text before the 3 strings are concatenated. So it prints "{0}" instead of changing it "80".  To fix this, add parentheses around the 3 strings:

        print (
    ("first part of long string that I purposely split into" +
     " multiple lines to keep all lines under {0} chars" +
     " but I still want the number 80 inserted above").format(80)
)

      Alternatively, since Python automatically concatenates any 2 adjacent string literals, before calling .format(), you can also fix this by omitting the plus signs:

        print (
    "first part of long string that I purposely split into"
    " multiple lines to keep all lines under {0} chars"
    " but I still want the number 80 inserted above".format(80)
)

      This is "cleaner" (less clutter), but less obvious and more fragile.  If someone later adds the plus signs, it will silently break.

      --Fred

    2. Debugging import statements

      Original Version: 1/14/2016
      Last Updated: 1/14/2016

      Ever have trouble debugging import statements in Python?

      It's easy to create an accidental import loop, or other combination of imports that causes an error but doesn't report any useful message, which can be a very difficult to debug.  Here's a tip from Carl Meyer on the django-users mailing list:

      Add the statement "assert False" to the top of any source file.  When the file is imported, an AssertionError will be raised, giving you a stack trace that shows the import chain that led to the file being imported.

      See:

      Very handy!  Thanks, Carl!

      --Fred

    3. Unicode

      1. Unicode Literals

        Original Version: 4/15/2016
        Last Updated: 5/12/2016

        Having trouble with UnicodeEncodeError exceptions in Python 2.x?  Here's how I fixed them:

        Problem:
        The basic problem was that I was allowing web users to enter Unicode chars, and was trying to insert them into ASCII strings as:

          'Embed the string here: {0}'.format(the_string)

        Solution:
        The simplistic solution was to tell Python 2 to use a Unicode string literal, instead of an ASCII one, via the "u" prefix as:

          u'Embed the string here: {0}'.format(the_string)

        Better Solution:
        But there were thousands of places in the code where I would need to do that.  A better solution was to tell Python 2 to use Unicode for all string literals, like Python3 does.  Add this to the top of each source file:

          from __future__ import unicode_literals

        Exceptions:
        If you need some string literals to still be ASCII instead of Unicode, you can override the default (in both Python 2 and 3) with the "b" (bytecodes) prefix as:

          b'Embed the string here: {0}'.format(the_string)

        For more info:

        Thanks to Stephen Butler for telling me about the __future__ import!

        Thanks to Tim Graham, the Django "fellow" (coordinates the pull requests of the Django committers), for pointing me to the documentation link above!

        And thanks to the members of the "django-users" mailing list (including Stephen and Tim) for being an extraordinarily helpful community!  They have a strict policy of being helpful and supportive.  Never an RTFM -- only links to specific relevant pages in the docs.

        --Fred

      2. Unicode in SQLite

        Original Version: 5/1/2016
        Last Updated: 5/12/2016

        Warning: SQLite always does case-sensitive comparisons of strings that contain Unicode chars.

        See:

        --Fred

      3. Unicode CSV Reader

        Original Version: 5/1/2016
        Last Updated: 6/8/2016

        Having problems reading UTF-8 data from a CSV file in Python?

        On my current project, we generate CSV files from the Web server for download by the browser.  We generate them as UTF-8, so they can support international chars and be opened on any computer, laptop, phone, etc.

        However, we also need to read and parse the downloaded files in the Python code of our regression tests.  This was a problem because Python's native csv.DictReader does not support UTF-8.  Here's my workaround:

          import csv
def get_csv_dict_reader(self, utf8_csv):
    """
    Create and return a csv.DictReader from a UTF-8 encoded string of CSV.
    Useful for handling CSV files streamed from our server, since we have
    it configured to send UTF-8, not standard Django Unicode.
    """
    # Convert the string from UTF-8 to Unicode, split it at newlines,
    # build a list of lines, and convert each line back to UTF-8.
    # Clunky, but necessary to be able to feed it to DictReader.
    unicode_string = utf8_csv.decode('utf-8')
    unicode_lines = unicode_string.split("\n")
    utf8_lines = []
    for unicode_line in unicode_lines:
        utf8_line = unicode_line.encode('utf-8')
        utf8_lines.append(utf8_line)

    # Pass the list of UTF-8 lines of CSV data to DictReader for parsing.
    reader = csv.DictReader(utf8_lines)
    return reader

        --Fred

  4. Pip (Package Installer for Python)

    1. Use pip for installing packages

      Original Version: 11/3/2012
      Last Updated: 11/3/2012

      Use the pip command to install Python packages, complete with all dependencies.  Also to uninstall, update, list, search, etc.  Much like yum for Linux.  See:

      --Fred

    2. Checking for updates

      Original Version: 5/6/2016
      Last Updated: 5/6/2016

      How to know which Python packages have newer versions available?

      The following command should show you info about a package, including whether there's a newer version available than what you have installed: 

        pip search package-name

      But, at least recently, it shows an older version as the newest available.  There's a bug reported here:

      Meanwhile, here's a Python script that lists all of your installed packages and their version numbers, each with either the newer available version number or the text "up to date":

      And in case that ever vanishes, here's my local copy of it:

      You can get much the same behavior out of the following command, but it shows only those that are outdated, not also those that are current: 

        pip list --outdated

      Thanks to Vincent Catalano for pointing out the bug report!

      --Fred

©Copyright 2012-2021, Bristle Software, Inc.  All rights reserved.