To subscribe to this RSS feed, copy its address and paste it into your favorite feed reader.
Greg Burekhttps://www.gregburek.com/Recent content on Greg BurekHugo -- gohugo.ioen-usTue, 23 Jul 2019 16:23:17 -0700Ed25519 SSH Keys Are Great, But Barriers Remainhttps://www.gregburek.com/2019/07/23/ed25519-ssh-keys-are-great-but-barriers-remain/Tue, 23 Jul 2019 16:23:17 -0700https://www.gregburek.com/2019/07/23/ed25519-ssh-keys-are-great-but-barriers-remain/
<p>Last year, I read a blog post that urged me to <a href="https://medium.com/risan/upgrade-your-ssh-key-to-ed25519-c6e8d60d3c54">Upgrade Your SSH Key to
Ed25519</a>
and so I did. Ed25519 keys have been available since OpenSSH 6.5 (OpenSSH 8.0
was released on 2019-04-17), and they are smaller, faster and better than RSA,
it seems. More info is in the blog post.</p>
<p>However, many months later, I found that ed25519 keys are not well supported for
a few key systems:</p>
<ol>
<li>Unifi network devices allow you to <a href="https://help.ubnt.com/hc/en-us/articles/235247068-UniFi-Adding-SSH-Keys-to-UniFi-Devices">provide SSH keys in the CloudKey
UI</a>
to be distributed to your network devices, but the UI only accepts some types
of keys. At the moment, <a href="https://community.ui.com/questions/UCK-Firmware-GUI-SSH-Key-Minor-Feature-Request-/b888e182-a029-460d-941d-91de3812829c#answer/1910a856-123d-4a57-91ea-286d98740959">Unifi CloudKey, AP, and USG all support ed25519
keys on the hardware, but the CloudKey UI rejects
them</a>.</li>
<li><a href="https://matt.ucc.asn.au/dropbear/dropbear.html">Dropbear</a>, which you can run
inside of <a href="https://packages.debian.org/buster/dropbear-initramfs">initramfs</a>
to <a href="https://hamy.io/post/0009/how-to-install-luks-encrypted-ubuntu-18.04.x-server-and-enable-remote-unlocking/">remotely unlock encrypted Linux
filesystems</a>,
<a href="https://hamy.io/post/0009/how-to-install-luks-encrypted-ubuntu-18.04.x-server-and-enable-remote-unlocking/#fn:3">does not seem to support
ed25519</a>.
An <a href="https://news.ycombinator.com/item?id=17765549">HN comment</a> from 11 months
ago suggests a fix is in the works, but nothing about ed25519 has appeared in
the <a href="https://matt.ucc.asn.au/dropbear/CHANGES">changelog</a>.</li>
</ol>
<p>Admittedly, these issues are not total show stoppers and I could use ed25519
keys for normal access and RSA keys for network device access and when my linux
box reboots. It’s just that juggling ed25519 and RSA keys for all my iOS devices
(<a href="https://www.blink.sh/">Blink is great</a>), a linux workstation, raspberry pis
and several laptops, seeding them correctly, managing passphrases and
configuring clients to use the right ones is complicated.</p>
<p>There already is an open feature request for ed25519 keys in the <a href="https://community.ui.com/questions/UCK-Firmware-GUI-SSH-Key-Minor-Feature-Request-/b888e182-a029-460d-941d-91de3812829c#answer/1910a856-123d-4a57-91ea-286d98740959">Unifi
UI</a>
and an unreviewed PR for <a href="https://github.com/mkj/dropbear/pull/75">Dropbear</a>,
but we build systems with what we have, not what is on the roadmap.</p>
<p>If you are looking at ed25519 keys for your infra, they are fine and good,
except for the unifi and Dropbear edge cases. You could probably work around
them by deploying a
<a href="https://help.ubnt.com/hc/en-us/articles/215458888-UniFi-USG-Advanced-Configuration">config.gateway.json</a>
and applying the <a href="https://github.com/mkj/dropbear/pull/75">Dropbear patch</a>
manually, but that sounds as exhausting as managing RSA and ed25519 keys, so we
have a cure which becomes an ailment.</p>
<p>So, for now, I’m following <a href="https://help.github.com/en/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent">Github’s
docs</a>
and using 4096 bit RSA keys, as well as Github’s public key “feature”
(<a href="https://github.com/username.keys">https://github.com/username.keys</a>) to seed <a href="https://docs.ansible.com/ansible/latest/modules/authorized_key_module.html">authorized_keys from
Ansible</a>
through out my infra.</p>
<p>It’s fine.</p>
<h3 id="postscript">Postscript</h3>
<p><a href="https://edovia.com/en/screens-ios/">Screens for iOS</a> can connect to VNC via a
<a href="https://help.edovia.com/hc/en-us/articles/115011943907-Configuring-a-Secure-Connection-in-Screens">secure
connection</a>
which appears to use an SSH tunnel.
While you can use <a href="https://help.edovia.com/hc/en-us/articles/115005876368-SSH-Keys">SSH
keys</a> for
authentication, ed25519 keys not supported and one can only use
<a href="https://help.edovia.com/hc/en-us/articles/115005876368-SSH-Keys">”…RSA keys of 2048-bits or less; 4096-bits or greater are
unsupported”</a>.
This is less than ideal and I’ll revisit Screens, if I get a desktop on linux
that is worth using VNC for.</p>
<h3 id="post-postscript-august-5-2019">Post-postscript (August 5, 2019)</h3>
<p><a href="https://docs.microsoft.com/en-us/azure/virtual-machines/linux/mac-create-ssh-keys">Linux VMs in
Azure</a>
also do not support ed25519 keys.</p>
<blockquote>
<p>Azure currently supports SSH protocol 2 (SSH-2) RSA public-private key pairs
with a minimum length of 2048 bits. Other key formats such as ED25519 and
ECDSA are not supported.</p>
</blockquote>
<p><a href="https://twitter.com/danfarina/status/1154855746628009984">A friend alleges</a>
that Azure Devops also does not support ed25519 keys, but the product name gives
me heartburn and makes it difficult to find docs that confirm this.</p>Running Node.js inside Postgres, Part Mea Culpahttps://www.gregburek.com/2017/09/01/running-node.js-inside-postgres-part-mea-culpa/Fri, 01 Sep 2017 00:00:00 +0000https://www.gregburek.com/2017/09/01/running-node.js-inside-postgres-part-mea-culpa/<p><a href="https://www.gregburek.com/2016/10/08/running-node-js-inside-postgres-part-1">Previously</a>, I wrote about
trying to run node.js code in PLV8 in Postgres. Unfortunately, that effort
sputtered out once I
<a href="https://stackoverflow.com/questions/12666148/does-plv8-support-making-http-calls-to-other-servers">found</a>
<a href="https://github.com/plv8/plv8/issues/190">out</a>
that PLV8 is a ‘trusted’ languagefor postgres. As documented for
<a href="https://www.postgresql.org/docs/current/static/plperl-trusted.html">PL/Perl</a>:</p>
<blockquote>
<p>Normally, PL/Perl is installed as a “trusted” programming language named
plperl. In this setup, certain Perl operations are disabled to preserve
security. In general, the operations that are restricted are those that
interact with the environment. This includes file handle operations, require,
and use (for external modules). There is no way to access internals of the
database server process or to gain OS-level access with the permissions of
the server process, as a C function can do. Thus, any unprivileged database
user can be permitted to use this language.</p>
</blockquote>
<p>So, no filesystem access means no unix socket access, which means no network
access. <code>¯\_(ツ)_/¯</code></p>
<p>If <a href="https://github.com/plv8/plv8/issues/222">plv8u</a> lands, I can revisit my
previous goal, but until then…</p>Running Node.js inside Postgres, Part 1https://www.gregburek.com/2016/10/08/running-node.js-inside-postgres-part-1/Sat, 08 Oct 2016 00:00:00 +0000https://www.gregburek.com/2016/10/08/running-node.js-inside-postgres-part-1/
<p><a href="https://github.com/plv8/plv8">PLV8</a> is a procedural language for Postgres,
that runs JavaScript, powered by the
<a href="https://en.wikipedia.org/wiki/V8_(JavaScript_engine)">V8</a> runtime. This allows
generic JavaScript code to be executed on a Postgres host, with the same
runtime that the web browser Chrome uses. So, it follows that most code that
could run in a browser should also be able to be run in Postgres.</p>
<p>PLV8 has been an extension of Postgres since version 9.2, and has been
available on <a href="https://devcenter.heroku.com/articles/heroku-postgres-extensions-postgis-full-text-search#languages">Heroku
Postgres</a>
professional tier databases for about as long.</p>
<h1 id="this-is-probably-a-bad-idea">This is probably a bad idea</h1>
<p>While potentially powerful, I personally have not seen much use of PLV8 other
than causing Out of Memory errors on busy Postgres dbs.</p>
<p>This series of posts aims to document my attempts to use node modules and npm
in PLV8 to develop a simple js app that uses Postgres as a runtime for the V8
runtime.</p>
<p>To be fair, this is one of my first attempts at using node and npm, so these
posts will be about that as much as anything.</p>
<h1 id="why">Why?</h1>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Why use node when you can use Postgres as your JavaScript runtime?</p>— Optimization Fence (@t_crayford) <a href="https://twitter.com/t_crayford/status/784529763603976192?ref_src=twsrc%5Etfw">October 7, 2016</a></blockquote>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Why not?</p>
<p><img src="https://i.imgur.com/S0YQYR4.jpg" alt="Why not Postgres?" /></p>
<h1 id="goal">Goal</h1>
<p>To have a Postgres function that logs into twitter and downloads my home feed to
the database it runs in. I’ll limit myself to a Heroku Postgres db, out of my
own convenience. I’ll attempt to use
<a href="https://github.com/BoyCook/TwitterJSClient">twitter-node-client</a> for this
purpose.</p>
<p>EDIT: <a href="https://www.gregburek.com/2017/09/01/running-node-js-inside-postgres-part-mea-culpa/">This goal is not currently possible.</a></p>
<h1 id="first-thing-to-attempt">First thing to attempt</h1>
<p>A google search for <code>node plv8</code> led me to the
<a href="https://github.com/clkao/plv8x">plv8x project</a>, which provides a cli and
allows importing of npm modules into the db. I’ll start here.</p>
<h2 id="install-node">Install node</h2>
<p>I downloaded and installed the <a href="https://nodejs.org/en/download/">official node
packages</a> for my system.</p>
<h2 id="create-a-heroku-app-and-postgres-db">Create a Heroku app and Postgres DB</h2>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">gburek@gburek-ltm2:~/code
> mkdir pg-twitter
gburek@gburek-ltm2:~/code
> cd pg-twitter
gburek@gburek-ltm2:~/code/pg-twitter
> git init
Initialized empty Git repository in /Users/gburek/code/pg-twitter/.git/
gburek@gburek-ltm2:~/code/pg-twitter
> heroku create pg-twitter
Creating ⬢ pg-twitter... done
https://pg-twitter.herokuapp.com/ | https://git.heroku.com/pg-twitter.git
gburek@gburek-ltm2:~/code/pg-twitter
> heroku addons:create heroku-postgresql:standard-0
Creating heroku-postgresql:standard-0 on ⬢ pg-twitter... $50/month
Created postgresql-triangular-47265 as DATABASE_URL
The database should be available in 3-5 minutes.
! The database will be empty. If upgrading, you can transfer
! data from another database with pg:copy.
Use `heroku pg:wait` to track status
Use heroku addons:docs heroku-postgresql to view documentation</pre></div>
<p>After a few minutes, our new db is available and has plv8 available for use:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">gburek@gburek-ltm2:~/code/pg-twitter
> heroku pg:wait
Waiting for database postgresql-triangular-47265... available
gburek@gburek-ltm2:~/code/pg-twitter
> heroku pg:psql
---> Connecting to DATABASE_URL
Timing is on.
Expanded display is used automatically.
psql (9.5.3, server 9.5.4)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.
=> CREATE EXTENSION plv8;
CREATE EXTENSION
Time: 1109.898 ms</pre></div>
<p>Now, on to working with plv8x.</p>
<h2 id="plv8x">plv8x</h2>
<p>First, I’m going to create a new npm project:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">gburek@gburek-ltm2:~/code/pg-twitter
> npm init --yes
Wrote to /Users/gburek/code/pg-twitter/package.json:
{
"name": "pg-twitter",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"keywords": [],
"author": "",
"license": "ISC"
}</pre></div>
<p>Now, to install plv8x. Surely, <code>npm install plv8x --save</code> will work.</p>
<p><details>
<summary>Click here to see 400+ lines of woe, errors and false starts. </summary></p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">gburek@gburek-ltm2:~/code/pg-twitter
> npm install plv8x --save
npm WARN prefer global LiveScript@1.2.0 should be installed with -g
> libpq@1.8.5 install /Users/gburek/code/pg-twitter/node_modules/libpq
> node-gyp rebuild
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
CXX(target) Release/obj.target/addon/src/connection.o
CXX(target) Release/obj.target/addon/src/connect-async-worker.o
CXX(target) Release/obj.target/addon/src/addon.o
SOLINK_MODULE(target) Release/addon.node
clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9
pg-twitter@1.0.0 /Users/gburek/code/pg-twitter
└─┬ plv8x@0.6.6
├── async@0.9.2
├─┬ js-yaml@3.0.2
│ ├─┬ argparse@0.1.16
│ │ ├── underscore@1.7.0
│ │ └── underscore.string@2.4.0
│ └── esprima@1.0.4
├─┬ LiveScript@1.2.0
│ └── prelude-ls@1.0.3
├─┬ one@2.5.2
│ ├── boxcars@2.0.0
│ ├─┬ debug@2.2.0
│ │ └── ms@0.7.1
│ ├── flatten-array@1.0.0
│ ├── functools@1.4.0
│ ├─┬ glob@7.1.1
│ │ ├── fs.realpath@1.0.0
│ │ ├─┬ inflight@1.0.5
│ │ │ └── wrappy@1.0.2
│ │ ├── inherits@2.0.3
│ │ ├─┬ minimatch@3.0.3
│ │ │ └─┬ brace-expansion@1.1.6
│ │ │ ├── balanced-match@0.4.2
│ │ │ └── concat-map@0.0.1
│ │ ├── once@1.4.0
│ │ └── path-is-absolute@1.0.1
│ └── hogan.js@2.0.0
├─┬ optimist@0.6.1
│ ├── minimist@0.0.10
│ └── wordwrap@0.0.3
├─┬ pg@4.5.6
│ ├── buffer-writer@1.0.1
│ ├── generic-pool@2.4.2
│ ├── packet-reader@0.2.0
│ ├── pg-connection-string@0.1.3
│ ├─┬ pg-types@1.11.0
│ │ ├── ap@0.2.0
│ │ ├── postgres-array@1.0.0
│ │ ├── postgres-bytea@1.0.0
│ │ ├── postgres-date@1.0.3
│ │ └─┬ postgres-interval@1.0.2
│ │ └── xtend@4.0.1
│ ├─┬ pgpass@0.0.3
│ │ └─┬ split@0.3.3
│ │ └── through@2.3.8
│ └── semver@4.3.6
├─┬ pg-native@1.10.0
│ ├─┬ libpq@1.8.5
│ │ ├── bindings@1.2.1
│ │ └── nan@2.4.0
│ ├── pg-types@1.6.0
│ └─┬ readable-stream@1.0.31
│ ├── core-util-is@1.0.2
│ ├── isarray@0.0.1
│ └── string_decoder@0.10.31
├── resolve@0.6.3
└─┬ tmp@0.0.29
└── os-tmpdir@1.0.2
npm WARN pg-twitter@1.0.0 No description
npm WARN pg-twitter@1.0.0 No repository field.</pre></div>
<p>ok, now to try and run it against my new db:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">gburek@gburek-ltm2:~/code/pg-twitter
> export PLV8XDB=$(h config:get DATABASE_URL)
gburek@gburek-ltm2:~/code/pg-twitter
> find . -name plv8x
./node_modules/.bin/plv8x
./node_modules/plv8x
gburek@gburek-ltm2:~/code/pg-twitter
> ./node_modules/.bin/plv8x -l
module.js:457
throw err;
^
Error: Cannot find module 'boxcars'
at Function.Module._resolveFilename (module.js:455:15)
at Function.Module._load (module.js:403:25)
at Module.require (module.js:483:17)
at require (internal/module.js:20:19)
at Object.<anonymous> (/Users/gburek/code/pg-twitter/node_modules/one/lib/templating/coll.js:1:77)
at Module._compile (module.js:556:32)
at Object.Module._extensions..js (module.js:565:10)
at Module.load (module.js:473:32)
at tryModuleLoad (module.js:432:12)
at Function.Module._load (module.js:424:3)</pre></div>
<p>welp, this sucks. Maybe I have to install it globally? The README seems to
suggest that.</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">gburek@gburek-ltm2:~/code/pg-twitter
> npm install plv8x -g
/Users/gburek/.nvm/versions/node/v6.7.0/bin/plv8x -> /Users/gburek/.nvm/versions/node/v6.7.0/lib/node_modules/plv8x/bin/cmd.js
...
gburek@gburek-ltm2:~/code/pg-twitter
> plv8x -l
>
module.js:457
throw err;
^
Error: Cannot find module 'boxcars'
at Function.Module._resolveFilename (module.js:455:15)
at Function.Module._load (module.js:403:25)
at Module.require (module.js:483:17)
at require (internal/module.js:20:19)
at Object.<anonymous>
(/Users/gburek/.nvm/versions/node/v6.7.0/lib/node_modules/plv8x/node_modules/one/lib/templating/coll.js:1:77)
at Module._compile (module.js:556:32)
at Object.Module._extensions..js (module.js:565:10)
at Module.load (module.js:473:32)
at tryModuleLoad (module.js:432:12)
at Function.Module._load (module.js:424:3)</pre></div>
<p>Uh ok. Maybe I need to install from github?</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">gburek@gburek-ltm2:~/code/pg-twitter
> npm uninstall plv8x -g
...
gburek@gburek-ltm2:~/code/pg-twitter
> npm install https://github.com/clkao/plv8x --save
npm WARN deprecated minimatch@2.0.10: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
- boxcars@2.0.0 node_modules/boxcars
- flatten-array@1.0.0 node_modules/flatten-array
- fs.realpath@1.0.0 node_modules/fs.realpath
- functools@1.4.0 node_modules/functools
- hogan.js@2.0.0 node_modules/hogan.js
- ms@0.7.1 node_modules/ms
- debug@2.2.0 node_modules/debug
- one@2.5.2 node_modules/one
- path-is-absolute@1.0.1 node_modules/path-is-absolute
pg-twitter@1.0.0 /Users/gburek/code/pg-twitter
└─┬ plv8x@0.7.0 (git+https://github.com/clkao/plv8x.git#3c19d57adfa5050c27715699d2369d2c441c817d)
...
npm WARN pg-twitter@1.0.0 No description
npm WARN pg-twitter@1.0.0 No repository field.
npm install https://github.com/clkao/plv8x --save 10.96s user 2.93s system 85% cpu 16.253 total
gburek@gburek-ltm2:~/code/pg-twitter
> find . -name plv8x
./node_modules/.bin/plv8x
./node_modules/plv8x
./node_modules/plv8x/bin/plv8x
gburek@gburek-ltm2:~/code/pg-twitter
> ./node_modules/plv8x/bin/plv8x -l
module.js:457
throw err;
^
Error: Cannot find module '../lib/cli.js'
at Function.Module._resolveFilename (module.js:455:15)
at Function.Module._load (module.js:403:25)
at Module.require (module.js:483:17)
at require (internal/module.js:20:19)
at Object.<anonymous> (/Users/gburek/code/pg-twitter/node_modules/plv8x/bin/plv8x:2:1)
at Module._compile (module.js:556:32)
at Object.Module._extensions..js (module.js:565:10)
at Module.load (module.js:473:32)
at tryModuleLoad (module.js:432:12)
at Function.Module._load (module.js:424:3)</pre></div>
<p>Ok. Different error. Progress! Try again, globally, from github.</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">gburek@gburek-ltm2:~/code/pg-twitter
> npm install https://github.com/clkao/plv8x -g
npm WARN deprecated minimatch@2.0.10: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
/Users/gburek/.nvm/versions/node/v6.7.0/bin/plv8x -> /Users/gburek/.nvm/versions/node/v6.7.0/lib/node_modules/plv8x/bin/plv8x
> libpq@1.8.5 install /Users/gburek/.nvm/versions/node/v6.7.0/lib/node_modules/plv8x/node_modules/libpq
> node-gyp rebuild
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
CXX(target) Release/obj.target/addon/src/connection.o
CXX(target) Release/obj.target/addon/src/connect-async-worker.o
CXX(target) Release/obj.target/addon/src/addon.o
SOLINK_MODULE(target) Release/addon.node
clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9
/Users/gburek/.nvm/versions/node/v6.7.0/lib
└─┬ plv8x@0.7.0 (git+https://github.com/clkao/plv8x.git#3c19d57adfa5050c27715699d2369d2c441c817d)
...
npm install https://github.com/clkao/plv8x -g 13.35s user 5.16s system 90% cpu 20.531 total
gburek@gburek-ltm2:~/code/pg-twitter
> plv8x -l
module.js:457
throw err;
^
Error: Cannot find module '../lib/cli.js'
at Function.Module._resolveFilename (module.js:455:15)
at Function.Module._load (module.js:403:25)
at Module.require (module.js:483:17)
at require (internal/module.js:20:19)
at Object.<anonymous> (/Users/gburek/.nvm/versions/node/v6.7.0/lib/node_modules/plv8x/bin/plv8x:2:1)
at Module._compile (module.js:556:32)
at Object.Module._extensions..js (module.js:565:10)
at Module.load (module.js:473:32)
at tryModuleLoad (module.js:432:12)
at Function.Module._load (module.js:424:3)</pre></div>
<p>What am I doing with my life?</p>
<p>OK So let’s actually follow the <a href="https://github.com/clkao/plv8x#install-plv8x">README on github</a></p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">gburek@gburek-ltm2:~/code
> git clone git://github.com/clkao/plv8x.git; cd plv8x
Cloning into 'plv8x'...
remote: Counting objects: 1091, done.
remote: Total 1091 (delta 0), reused 0 (delta 0), pack-reused 1091
Receiving objects: 100% (1091/1091), 188.83 KiB | 0 bytes/s, done.
Resolving deltas: 100% (564/564), done.
Checking connectivity... done.
gburek@gburek-ltm2:~/code/plv8x
> npm i -g .
> plv8x@0.7.0 prepublish /Users/gburek/code/plv8x
> env PATH="./node_modules/.bin:$PATH" lsc -cj package.ls &&
env PATH="./node_modules/.bin:$PATH" lsc -bc -o lib src
env: lsc: No such file or directory
npm ERR! addLocal Could not install /Users/gburek/code/plv8x
npm ERR! Darwin 15.6.0
npm ERR! argv "/Users/gburek/.nvm/versions/node/v6.7.0/bin/node" "/Users/gburek/.nvm/versions/node/v6.7.0/bin/npm" "i" "-g" "."
npm ERR! node v6.7.0
npm ERR! npm v3.10.3
npm ERR! file sh
npm ERR! code ELIFECYCLE
npm ERR! errno ENOENT
npm ERR! syscall spawn
npm ERR! plv8x@0.7.0 prepublish: `env PATH="./node_modules/.bin:$PATH" lsc -cj package.ls &&
npm ERR! env PATH="./node_modules/.bin:$PATH" lsc -bc -o lib src`
npm ERR! spawn ENOENT
npm ERR!
npm ERR! Failed at the plv8x@0.7.0 prepublish script 'env PATH="./node_modules/.bin:$PATH" lsc -cj package.ls &&
npm ERR! env PATH="./node_modules/.bin:$PATH" lsc -bc -o lib src'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the plv8x package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! env PATH="./node_modules/.bin:$PATH" lsc -cj package.ls &&
npm ERR! env PATH="./node_modules/.bin:$PATH" lsc -bc -o lib src
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs plv8x
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! npm owner ls plv8x
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR! /Users/gburek/code/plv8x/npm-debug.log</pre></div>
<p>Welp. Maybe it needs to be installed locally?</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">gburek@gburek-ltm2:~/code/plv8x
> npm install
npm WARN deprecated minimatch@2.0.10: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN deprecated to-iso-string@0.0.2: to-iso-string has been deprecated, use @segment/to-iso-string instead.
npm WARN deprecated jade@0.26.3: Jade has been renamed to pug, please install the latest version of pug instead of jade
npm WARN deprecated minimatch@0.3.0: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
> libpq@1.8.5 install /Users/gburek/code/plv8x/node_modules/libpq
> node-gyp rebuild
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
CXX(target) Release/obj.target/addon/src/connection.o
CXX(target) Release/obj.target/addon/src/connect-async-worker.o
CXX(target) Release/obj.target/addon/src/addon.o
SOLINK_MODULE(target) Release/addon.node
clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9
> plv8x@0.7.0 prepublish /Users/gburek/code/plv8x
> env PATH="./node_modules/.bin:$PATH" lsc -cj package.ls &&
env PATH="./node_modules/.bin:$PATH" lsc -bc -o lib src
plv8x@0.7.0 /Users/gburek/code/plv8x
...
gburek@gburek-ltm2:~/code/plv8x
> find . -name plv8x
./bin/plv8x</pre></div>
<p></details></p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">gburek@gburek-ltm2:~/code/plv8x
> ./bin/plv8x -l
plv8x: 392.33 kB</pre></div>
<p>YES IT WORKS. Who would have guessed this npm module was only usable when
executed directly in its source tree and installed both locally and globally?</p>
<p>Let’s see what is happening on the db (Heroku per line logging preamble
omitted):</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">gburek@gburek-ltm2:~/code/pg-twitter
> heroku logs -t
[15-1] LOG: statement:
[15-2] SET client_min_messages TO WARNING;
[15-3] DO $PLV8X_EOF$ BEGIN
[15-4]
[15-5] DROP FUNCTION IF EXISTS plv8x.json_eval (code text,data plv8x.json) CASCADE;
[15-6] EXCEPTION WHEN OTHERS THEN END; $PLV8X_EOF$;
[15-7]
[15-8] CREATE FUNCTION plv8x.json_eval (code text,data plv8x.json) RETURNS plv8x.json AS $PLV8X__BODY__$
[15-9] if (typeof plv8x == 'undefined') plv8.execute('select plv8x.boot()', []);;
[15-10] return JSON.stringify((eval(function (code, data){
[15-11] return eval(plv8x.xpressionToBody(code)).apply(data);
[15-12] }))(code,JSON.parse(data)));
[15-13] $PLV8X__BODY__$ LANGUAGE plv8 IMMUTABLE STRICT;
[16-1] NOTICE: drop cascades to operator <|(text,plv8x.json)
[16-2] CONTEXT: SQL statement "DROP FUNCTION IF EXISTS plv8x.json_eval (code text,data plv8x.json) CASCADE"
[16-3] PL/pgSQL function inline_code_block line 3 at SQL statement
[17-1] LOG: statement:
[17-2] SET client_min_messages TO WARNING;
[17-3] DO $PLV8X_EOF$ BEGIN
[17-4]
[17-5] DROP FUNCTION IF EXISTS plv8x.json_eval_ls (code text) CASCADE;
[17-6] EXCEPTION WHEN OTHERS THEN END; $PLV8X_EOF$;
[17-7]
[17-8] CREATE FUNCTION plv8x.json_eval_ls (code text) RETURNS plv8x.json AS $PLV8X__BODY__$
[17-9] if (typeof plv8x == 'undefined') plv8.execute('select plv8x.boot()', []);;
[17-10] return JSON.stringify((eval(function (code){
[17-11] return eval(plv8x.xpressionToBody("~>" + code)).apply(this);
[17-12] }))(code));
[17-13] $PLV8X__BODY__$ LANGUAGE plv8 IMMUTABLE STRICT;
[18-1] NOTICE: drop cascades to operator ~>(NONE,text)
[18-2] CONTEXT: SQL statement "DROP FUNCTION IF EXISTS plv8x.json_eval_ls (code text) CASCADE"
[18-3] PL/pgSQL function inline_code_block line 3 at SQL statement
[19-1] LOG: statement:
[19-2] SET client_min_messages TO WARNING;
[19-3] DO $PLV8X_EOF$ BEGIN
[19-4]
[19-5] DROP FUNCTION IF EXISTS plv8x.json_eval_ls (data plv8x.json,code text) CASCADE;
[19-6] EXCEPTION WHEN OTHERS THEN END; $PLV8X_EOF$;
[19-7]
[19-8] CREATE FUNCTION plv8x.json_eval_ls (data plv8x.json,code text) RETURNS plv8x.json AS $PLV8X__BODY__$
[19-9] if (typeof plv8x == 'undefined') plv8.execute('select plv8x.boot()', []);;
[19-10] return JSON.stringify((eval(function (data, code){
[19-11] return eval(plv8x.xpressionToBody("~>" + code)).apply(data);
[19-12] }))(JSON.parse(data),code));
[19-13] $PLV8X__BODY__$ LANGUAGE plv8 IMMUTABLE STRICT;
[20-1] NOTICE: drop cascades to operator ~>(plv8x.json,text)
[20-2] CONTEXT: SQL statement "DROP FUNCTION IF EXISTS plv8x.json_eval_ls (data plv8x.json,code text) CASCADE"
[20-3] PL/pgSQL function inline_code_block line 3 at SQL statement
[21-1] LOG: statement:
[21-2] SET client_min_messages TO WARNING;
[21-3] DO $PLV8X_EOF$ BEGIN
[21-4]
[21-5] DROP FUNCTION IF EXISTS plv8x.json_eval_ls (code text,data plv8x.json) CASCADE;
[21-6] EXCEPTION WHEN OTHERS THEN END; $PLV8X_EOF$;
[21-7]
[21-8] CREATE FUNCTION plv8x.json_eval_ls (code text,data plv8x.json) RETURNS plv8x.json AS $PLV8X__BODY__$
[21-9] if (typeof plv8x == 'undefined') plv8.execute('select plv8x.boot()', []);;
[21-10] return JSON.stringify((eval(function (code, data){
[21-11] return eval(plv8x.xpressionToBody("~>" + code)).apply(data);
[21-12] }))(code,JSON.parse(data)));
[21-13] $PLV8X__BODY__$ LANGUAGE plv8 IMMUTABLE STRICT;
[22-1] NOTICE: drop cascades to operator <~(text,plv8x.json)
[22-2] CONTEXT: SQL statement "DROP FUNCTION IF EXISTS plv8x.json_eval_ls (code text,data plv8x.json) CASCADE"
[22-3] PL/pgSQL function inline_code_block line 3 at SQL statement
[23-1] LOG: statement: DROP OPERATOR IF EXISTS |> (NONE, text); CREATE OPERATOR |> (
[23-2] RIGHTARG = text,
[23-3] PROCEDURE = plv8x.json_eval
[23-4] );
[23-5] DROP OPERATOR IF EXISTS |> (plv8x.json, text); CREATE OPERATOR |> (
[23-6] LEFTARG = plv8x.json,
[23-7] RIGHTARG = text,
[23-8] COMMUTATOR = <|,
[23-15] PROCEDURE = plv8x.json_eval
[23-10] );
[23-9] PROCEDURE = plv8x.json_eval
[23-16] );
[23-14] COMMUTATOR = |>,
[23-17]
[23-13] RIGHTARG = plv8x.json,
[23-18] DROP OPERATOR IF EXISTS ~> (NONE, text); CREATE OPERATOR ~> (
[23-11] DROP OPERATOR IF EXISTS <| (text, plv8x.json); CREATE OPERATOR <| (
[23-19] RIGHTARG = text,
[23-12] LEFTARG = text,
[23-20] PROCEDURE = plv8x.json_eval_ls
[23-21] );
[23-22] DROP OPERATOR IF EXISTS ~> (plv8x.json, text); CREATE OPERATOR ~> (
[23-23] LEFTARG = plv8x.json,
[23-24] RIGHTARG = text,
[23-25] COMMUTATOR = <~,
[23-27] );
[23-26] PROCEDURE = plv8x.json_eval_ls
[23-28] DROP OPERATOR IF EXISTS <~ (text, plv8x.json); CREATE OPERATOR <~ (
[23-29] LEFTARG = text,
[23-30] RIGHTARG = plv8x.json,
[23-31] COMMUTATOR = ~>,
[23-32] PROCEDURE = plv8x.json_eval_ls
[23-33] );
[24-1] NOTICE: operator |> does not exist, skipping
[25-1] NOTICE: operator |> does not exist, skipping
[26-1] NOTICE: operator ~> does not exist, skipping
[27-1] NOTICE: operator ~> does not exist, skipping</pre></div>
<p>This is not great. It seems that <code>plv8x</code> wants to create custom operators that
are similar to ones in Livescript (<code>|></code> pipeline operator) and CoffeeScript
(<code>-></code> thin arrows which are translated as <code>~></code>).</p>
<p><del>Custom operators are superuser only and run the risk of crashing the
postmaster, so many Postgres providers do not support them.</del></p>
<p><del>However, it seems that they are not critical to using vanilla js and node, so
we may continue.</del></p>
<h3 id="update-only-custom-default-operators-seem-to-not-work-after-altering-the-search-path-of-the-db-these-appear-to-be-created-properly"><em>UPDATE:</em> Only custom <em>default</em> operators seem to not work. After altering the search_path of the db, these appear to be created properly.</h3>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">> select * from pg_operator where oprcode::text ilike 'plv8%';
oprname | oprnamespace | oprowner | oprkind | oprcanmerge | oprcanhash | oprleft | oprright | oprresult | oprcom | oprnegate | oprcode | oprrest | oprjoin
---------+--------------+----------+---------+-------------+------------+---------+----------+-----------+--------+-----------+--------------------+---------+---------
|> | 17017 | 16384 | l | f | f | 0 | 25 | 17027 | 0 | 0 | plv8x.json_eval | - | -
|> | 17017 | 16384 | b | f | f | 17027 | 25 | 17027 | 17316 | 0 | plv8x.json_eval | - | -
<| | 17017 | 16384 | b | f | f | 25 | 17027 | 17027 | 17317 | 0 | plv8x.json_eval | - | -
~> | 17017 | 16384 | l | f | f | 0 | 25 | 17027 | 0 | 0 | plv8x.json_eval_ls | - | -
~> | 17017 | 16384 | b | f | f | 17027 | 25 | 17027 | 17320 | 0 | plv8x.json_eval_ls | - | -
<~ | 17017 | 16384 | b | f | f | 25 | 17027 | 17027 | 17321 | 0 | plv8x.json_eval_ls | - | -
(6 rows)</pre></div>
<h2 id="how-does-this-work">How does this work?</h2>
<p>Let’s take a look at the db:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">> \dn
List of schemas
Name | Owner
--------+----------------
plv8x | uah8s1lfn60k9k
public | uah8s1lfn60k9k
(2 rows)
> \d plv8x.
Table "plv8x.code"
Column | Type | Modifiers
----------+-----------------------------+-----------
name | text | not null
code | text |
load_seq | integer |
updated | timestamp without time zone |
Indexes:
"code_pkey" PRIMARY KEY, btree (name)
Index "plv8x.code_pkey"
Column | Type | Definition
--------+------+------------
name | text | name
primary key, btree, for table "plv8x.code"
> \d plv8x.code
Table "plv8x.code"
Column | Type | Modifiers
----------+-----------------------------+-----------
name | text | not null
code | text |
load_seq | integer |
updated | timestamp without time zone |
Indexes:
"code_pkey" PRIMARY KEY, btree (name)</pre></div>
<p>Wow, ok so node modules are rows in this table?</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">> select name, substring(code from 1 for 1300), load_seq, updated from plv8x.code;
-[ RECORD 1 ]----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
name | plv8x
substring | !function(e){if("object"==typeof exports&&"undefined"!=typeof module)module.exports=e();else if("function"==typeof define&&define.amd)define([],e);else{var f;"undefined"!=typeof window?f=window:"undefined"!=typeof global?f=global:"undefined"!=typeof self&&(f=self),f.plv8x=e()}}(function(){var define,module,exports;return (function e(t,n,r){function s(o,u){if(!n[o]){if(!t[o]){var a=typeof require=="function"&&require;if(!u&&a)return a(o,!0);if(i)return i(o,!0);var f=new Error("Cannot find module '"+o+"'");throw f.code="MODULE_NOT_FOUND",f}var l=n[o]={exports:{}};t[o][0].call(l.exports,function(e){var n=t[o][1][e];return s(n?n:e)},l,l.exports,e,t,n,r)}return n[o].exports}var i=typeof require=="function"&&require;for(var o=0;o<r.length;o++)s(r[o]);return s})({1:[function(require,module,exports){+
| // Generated by LiveScript 1.2.0 +
| var ref$, _mk_func, compileCoffeescript, compileLivescript, xpressionToBody, plv8xSql, operatorsSql, _eval, _apply, _require, _mk_json_eval, _mk_json_eval_ls, _boot; +
| ref$ = require('..'), _mk_func = ref$._mk_func, compileCoffeescript = ref$.compileCoffeescript, compileLivescript = ref$.compileLivescript, xpressionToBody = ref$.xpressionToBody; +
| ref$ = require('./sql'), plv8xSql = ref$.plv8xSql, operatorsSql = ref$.operatorsSql; +
| module.exports = function(drop,
load_seq |
updated | 2016-10-08 16:27:59</pre></div>
<p>Not even minified. Cool.</p>
<h2 id="install-a-node-package-into-the-db">Install a node package into the db</h2>
<p>Let’s install something.</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">gburek@gburek-ltm2:~/code/plv8x
> ./bin/plv8x -i twitter-node-client
gburek@gburek-ltm2:~/code/plv8x
> ./bin/plv8x -l
plv8x: 392.33 kB
twitter-node-client: 685.9 kB
> select name, octet_length(code), load_seq, updated from plv8x.code;
name | octet_length | load_seq | updated
---------------------+--------------+----------+---------------------
plv8x | 392331 | | 2016-10-08 16:27:59
twitter-node-client | 685900 | | 2016-10-11 17:20:21
(2 rows)</pre></div>
<p>Cool. What about running something simple?</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">gburek@gburek-ltm2:~/code/plv8x
> ./bin/plv8x -e 'require("qs").parse("foo=bar&baz=1")'
WARNING: failed to load module buffer:
WARNING: failed to load module qs: Error: no window object present
WARNING: Error: no window object present
at eval (eval at <anonymous> (boot:27:23), <anonymous>:16168:15)
at Object../lib/request (eval at <anonymous> (boot:27:23), <anonymous>:16200:3)
at s (eval at <anonymous> (boot:27:23), <anony
/Users/gburek/code/plv8x/lib/index.js:29
throw err;
^
Error: ERROR: TypeError: Cannot call method 'parse' of undefined
DETAIL: undefined() LINE 0: ((function(){return require("qs").parse("foo=bar&baz=1")}))()
at Client._readError (/Users/gburek/code/plv8x/node_modules/pg-native/index.js:80:13)
at Client._read (/Users/gburek/code/plv8x/node_modules/pg-native/index.js:121:19)
at emitNone (events.js:86:13)
at PQ.emit (events.js:185:7)</pre></div>
<p>This isn’t good. What about defining a function?</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">gburek@gburek-ltm2:~/code/plv8x
> ./bin/plv8x -f 'plv8x.json parse_qs(text)=qs:parse'
ok plv8x.json parse_qs(text)
> SELECT parse_qs('foo=bar&baz=1') AS qs;
WARNING: 01000: failed to load module buffer:
LOCATION: plv8_Elog, plv8_func.cc:327
WARNING: 01000: failed to load module qs: Error: no window object present
LOCATION: plv8_Elog, plv8_func.cc:327
WARNING: 01000: Error: no window object present
at eval (eval at <anonymous> (boot:27:23), <anonymous>:16168:15)
at Object../lib/request (eval at <anonymous> (boot:27:23), <anonymous>:16200:3)
at s (eval at <anonymous> (boot:27:23), <anony
LOCATION: plv8_Elog, plv8_func.cc:327
ERROR: XX000: TypeError: Cannot read property 'parse' of undefined
DETAIL: parse_qs() LINE 4: return plv8x.require('qs').parse.apply(this, arguments);
LOCATION: rethrow, plv8.cc:1649</pre></div>
<p>It looks like the lack of custom operators is preventing this code from
running.</p>
<h1 id="failure">Failure?</h1>
<h3 id="update-it-seems-that-the-search-path-of-the-db-was-not-wide-enough-and-including-all-possible-schemas-allows-operator-creation-and-for-modules-to-be-loaded-and-run">UPDATE: It seems that the search path of the db was not wide enough, and including all possible schemas, allows operator creation and for modules to be loaded and run:</h3>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">> alter database df5f7ilg16vje set search_path to "$user", public, plv8, plv8x;
ALTER DATABASE
Time: 79.494 ms
> SELECT |>'(require("moment")()).format()';
?column?
-----------------------------
"2016-10-12T20:30:26+00:00"
(1 row)
Time: 76.703 ms</pre></div>
<p>Part 2 will continue down this path to running node/npm in Postgres and will
show how the above was found.</p>
<p><del>After all this exploration, I think using plv8x with a Heroku Postgres db is
not possible. The use of custom operators seems to extend beyond the ability to
use LiveScript and Coffeescript and prevents loading vanilla modules.</del></p>
<p><del>I am not too discouraged, however, as
<a href="https://github.com/langateam/node-plv8">node-plv8</a> and
<a href="https://github.com/mgutz/plv8-bedrock">plv8-bedrock</a> seem like viable
alternatives. I’ll try those next!</del></p>Lessons from Automatic Incident Resolution for a Million Databases - SREcon16 EUhttps://www.gregburek.com/2016/10/06/lessons-from-automatic-incident-resolution-for-a-million-databases-srecon16-eu/Thu, 06 Oct 2016 00:00:00 +0000https://www.gregburek.com/2016/10/06/lessons-from-automatic-incident-resolution-for-a-million-databases-srecon16-eu/<p>In July, I traveled to Dublin, Ireland to give a short talk about automtic
incident resolution as we do it at Heroku Data. The original video may be found
on the <a href="https://www.usenix.org/conference/srecon16europe/program/presentation/burek">SREcon16 EU site</a>.</p>
<p>The slides, including working gifs and a video of the talk are embedded below.</p>
<div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;">
<iframe src="//www.youtube.com/embed/hrNsVSQdXxU" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" allowfullscreen title="YouTube Video"></iframe>
</div>Installing bcc to evaluate BPF and Postgreshttps://www.gregburek.com/2016/10/04/installing-bcc-to-evaluate-bpf-and-postgres/Tue, 04 Oct 2016 00:00:00 +0000https://www.gregburek.com/2016/10/04/installing-bcc-to-evaluate-bpf-and-postgres/<p><a href="https://twitter.com/t_crayford">@t_crayford</a> sent me Brendan Gregg’s latest
missive about performance tracing, this time for <a href="http://www.brendangregg.com/blog/2016-10-04/linux-bcc-mysqld-qslower.html">Linux MySQL Slow Query Tracing with bcc/BPF</a>.</p>
<p><a href="https://github.com/iovisor/bcc">bcc</a> stands for ‘BPF Compiler Collection’ and
<a href="https://en.wikipedia.org/wiki/Berkeley_Packet_Filter">BPF</a> stands for
‘Berkeley Packet Filter’. From the bcc
<a href="https://github.com/iovisor/bcc/blob/60393ea5dd966d33ff24929f6981df09473cbb1b/README.md">README</a>:</p>
<blockquote>
<p>BCC is a toolkit for creating efficient kernel tracing and manipulation
programs, and includes several useful tools and examples. It makes use of
extended BPF (Berkeley Packet Filters), formally known as eBPF, a new feature
that was first added to Linux 3.15. Much of what BCC uses requires Linux 4.1
and above.</p>
<p>eBPF was <a href="https://lkml.org/lkml/2015/4/14/232">described</a> by Ingo Molnár as:</p>
<blockquote>
<p>One of the more interesting features in this cycle is the ability to attach
eBPF programs (user-defined, sandboxed bytecode executed by the kernel) to
kprobes. This allows user-defined instrumentation on a live kernel image
that can never crash, hang or interfere with the kernel negatively.</p>
</blockquote>
<p>BCC makes BPF programs easier to write, with kernel instrumentation in C (and
includes a C wrapper around LLVM), and front-ends in Python and lua. It is
suited for many tasks, including performance analysis and network traffic
control.</p>
</blockquote>
<p>As I work for Heroku Postgres, I wanted to investigate something similar for
Postgres, running on our infrastructure. First thing to check was if it was
even possible on our systems, using <a href="https://github.com/iovisor/bcc/blob/60393ea5dd966d33ff24929f6981df09473cbb1b/INSTALL.md">bcc’s INSTALL
instructions</a>.</p>
<p>New Postgres databases get Ubuntu Trusty instances with
<code>linux-generic-lts-xenial</code> kernels of the 4.4 series:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">=> select version();
version
-------------------------------------------------------------------------------------------------
PostgreSQL 9.5.4 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2, 64-bit
(1 row)</pre></div><div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">~$ uname -a
Linux ip-10-0-10-230 4.4.0-38-generic #57~14.04.1-Ubuntu SMP Tue Sep 6 17:20:43 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux</pre></div>
<p>This seems to satisfy the <code>Linux kernel version 4.1 or newer</code> requirement.</p>
<p>Next thing to check is if the kernel has been compiled properly:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">~$ cat /boot/config-4.4.0-38-generic | grep BPF
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_HAVE_BPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_TEST_BPF=m</pre></div>
<p>This looks ok! Next up is to install the repo and tools:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><span style="color:#960050;background-color:#1e0010">~$</span> <span style="color:#a6e22e">sudo</span> <span style="color:#a6e22e">apt</span><span style="color:#f92672">-</span><span style="color:#a6e22e">key</span> <span style="color:#a6e22e">adv</span> <span style="color:#f92672">--</span><span style="color:#a6e22e">keyserver</span> <span style="color:#a6e22e">keyserver</span>.<span style="color:#a6e22e">ubuntu</span>.<span style="color:#a6e22e">com</span> <span style="color:#f92672">--</span><span style="color:#a6e22e">recv</span><span style="color:#f92672">-</span><span style="color:#a6e22e">keys</span> <span style="color:#a6e22e">D4284CDD</span>
<span style="color:#f92672">...</span>
<span style="color:#a6e22e">gpg</span>: <span style="color:#a6e22e">requesting</span> <span style="color:#a6e22e">key</span> <span style="color:#a6e22e">D4284CDD</span> <span style="color:#a6e22e">from</span> <span style="color:#a6e22e">hkp</span> <span style="color:#a6e22e">server</span> <span style="color:#a6e22e">keyserver</span>.<span style="color:#a6e22e">ubuntu</span>.<span style="color:#a6e22e">com</span>
<span style="color:#a6e22e">gpg</span>: <span style="color:#a6e22e">key</span> <span style="color:#a6e22e">D4284CDD</span>: <span style="color:#a6e22e">public</span> <span style="color:#a6e22e">key</span> <span style="color:#e6db74">"Brenden Blanco <bblanco@plumgrid.com>"</span> <span style="color:#a6e22e">imported</span>
<span style="color:#a6e22e">gpg</span>: <span style="color:#a6e22e">Total</span> <span style="color:#a6e22e">number</span> <span style="color:#a6e22e">processed</span>: <span style="color:#ae81ff">1</span>
<span style="color:#a6e22e">gpg</span>: <span style="color:#a6e22e">imported</span>: <span style="color:#ae81ff">1</span> (<span style="color:#a6e22e">RSA</span>: <span style="color:#ae81ff">1</span>)
<span style="color:#960050;background-color:#1e0010">~$</span> <span style="color:#a6e22e">echo</span> <span style="color:#e6db74">"deb https://repo.iovisor.org/apt trusty main"</span> | <span style="color:#a6e22e">sudo</span> <span style="color:#a6e22e">tee</span> <span style="color:#f92672">/</span><span style="color:#a6e22e">etc</span><span style="color:#f92672">/</span><span style="color:#a6e22e">apt</span><span style="color:#f92672">/</span><span style="color:#a6e22e">sources</span>.<span style="color:#a6e22e">list</span>.<span style="color:#a6e22e">d</span><span style="color:#f92672">/</span><span style="color:#a6e22e">iovisor</span>.<span style="color:#a6e22e">list</span>
<span style="color:#a6e22e">deb</span> <span style="color:#a6e22e">https</span>:<span style="color:#75715e">//repo.iovisor.org/apt trusty main
</span><span style="color:#75715e"></span><span style="color:#960050;background-color:#1e0010">~$</span> <span style="color:#a6e22e">sudo</span> <span style="color:#a6e22e">apt</span><span style="color:#f92672">-</span><span style="color:#a6e22e">get</span> <span style="color:#a6e22e">update</span>
<span style="color:#f92672">...</span>
<span style="color:#a6e22e">Fetched</span> <span style="color:#ae81ff">4</span>,<span style="color:#ae81ff">322</span> <span style="color:#a6e22e">kB</span> <span style="color:#a6e22e">in</span> <span style="color:#ae81ff">3</span><span style="color:#a6e22e">s</span> (<span style="color:#ae81ff">1</span>,<span style="color:#ae81ff">269</span> <span style="color:#a6e22e">kB</span><span style="color:#f92672">/</span><span style="color:#a6e22e">s</span>)
<span style="color:#a6e22e">Reading</span> <span style="color:#f92672">package</span> <span style="color:#a6e22e">lists</span><span style="color:#f92672">...</span> <span style="color:#a6e22e">Done</span>
<span style="color:#960050;background-color:#1e0010">~$</span> <span style="color:#a6e22e">sudo</span> <span style="color:#a6e22e">apt</span><span style="color:#f92672">-</span><span style="color:#a6e22e">get</span> <span style="color:#a6e22e">install</span> <span style="color:#a6e22e">binutils</span> <span style="color:#a6e22e">bcc</span> <span style="color:#a6e22e">bcc</span><span style="color:#f92672">-</span><span style="color:#a6e22e">tools</span> <span style="color:#a6e22e">libbcc</span><span style="color:#f92672">-</span><span style="color:#a6e22e">examples</span> <span style="color:#a6e22e">python</span><span style="color:#f92672">-</span><span style="color:#a6e22e">bcc</span>
<span style="color:#a6e22e">Reading</span> <span style="color:#f92672">package</span> <span style="color:#a6e22e">lists</span><span style="color:#f92672">...</span> <span style="color:#a6e22e">Done</span>
<span style="color:#a6e22e">Building</span> <span style="color:#a6e22e">dependency</span> <span style="color:#a6e22e">tree</span>
<span style="color:#a6e22e">Reading</span> <span style="color:#a6e22e">state</span> <span style="color:#a6e22e">information</span><span style="color:#f92672">...</span> <span style="color:#a6e22e">Done</span>
<span style="color:#a6e22e">binutils</span> <span style="color:#a6e22e">is</span> <span style="color:#a6e22e">already</span> <span style="color:#a6e22e">the</span> <span style="color:#a6e22e">newest</span> <span style="color:#a6e22e">version</span>.
<span style="color:#a6e22e">binutils</span> <span style="color:#a6e22e">set</span> <span style="color:#a6e22e">to</span> <span style="color:#a6e22e">manually</span> <span style="color:#a6e22e">installed</span>.
<span style="color:#a6e22e">The</span> <span style="color:#a6e22e">following</span> <span style="color:#a6e22e">extra</span> <span style="color:#a6e22e">packages</span> <span style="color:#a6e22e">will</span> <span style="color:#a6e22e">be</span> <span style="color:#a6e22e">installed</span>:
<span style="color:#a6e22e">bin86</span> <span style="color:#a6e22e">elks</span><span style="color:#f92672">-</span><span style="color:#a6e22e">libc</span> <span style="color:#a6e22e">libbcc</span>
<span style="color:#a6e22e">The</span> <span style="color:#a6e22e">following</span> <span style="color:#a6e22e">NEW</span> <span style="color:#a6e22e">packages</span> <span style="color:#a6e22e">will</span> <span style="color:#a6e22e">be</span> <span style="color:#a6e22e">installed</span>:
<span style="color:#a6e22e">bcc</span> <span style="color:#a6e22e">bcc</span><span style="color:#f92672">-</span><span style="color:#a6e22e">tools</span> <span style="color:#a6e22e">bin86</span> <span style="color:#a6e22e">elks</span><span style="color:#f92672">-</span><span style="color:#a6e22e">libc</span> <span style="color:#a6e22e">libbcc</span> <span style="color:#a6e22e">libbcc</span><span style="color:#f92672">-</span><span style="color:#a6e22e">examples</span> <span style="color:#a6e22e">python</span><span style="color:#f92672">-</span><span style="color:#a6e22e">bcc</span>
<span style="color:#ae81ff">0</span> <span style="color:#a6e22e">upgraded</span>, <span style="color:#ae81ff">7</span> <span style="color:#a6e22e">newly</span> <span style="color:#a6e22e">installed</span>, <span style="color:#ae81ff">0</span> <span style="color:#a6e22e">to</span> <span style="color:#a6e22e">remove</span> <span style="color:#a6e22e">and</span> <span style="color:#ae81ff">30</span> <span style="color:#a6e22e">not</span> <span style="color:#a6e22e">upgraded</span>.
<span style="color:#a6e22e">Need</span> <span style="color:#a6e22e">to</span> <span style="color:#a6e22e">get</span> <span style="color:#ae81ff">10.4</span> <span style="color:#a6e22e">MB</span> <span style="color:#a6e22e">of</span> <span style="color:#a6e22e">archives</span>.
<span style="color:#a6e22e">After</span> <span style="color:#a6e22e">this</span> <span style="color:#a6e22e">operation</span>, <span style="color:#ae81ff">36.6</span> <span style="color:#a6e22e">MB</span> <span style="color:#a6e22e">of</span> <span style="color:#a6e22e">additional</span> <span style="color:#a6e22e">disk</span> <span style="color:#a6e22e">space</span> <span style="color:#a6e22e">will</span> <span style="color:#a6e22e">be</span> <span style="color:#a6e22e">used</span>.
<span style="color:#a6e22e">Do</span> <span style="color:#a6e22e">you</span> <span style="color:#a6e22e">want</span> <span style="color:#a6e22e">to</span> <span style="color:#66d9ef">continue</span><span style="color:#960050;background-color:#1e0010">?</span> [<span style="color:#a6e22e">Y</span><span style="color:#f92672">/</span><span style="color:#a6e22e">n</span>] <span style="color:#a6e22e">y</span>
<span style="color:#f92672">...</span>
<span style="color:#960050;background-color:#1e0010">~$</span></pre></div>
<p>Now to test this out:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"># /usr/share/bcc/tools/tplist -l /usr/lib/postgresql/9.5/bin/postgres
Traceback (most recent call last):
File "/usr/share/bcc/tools/tplist", line 16, in <module>
from bcc import USDTReader
ImportError: cannot import name USDTReader</pre></div>
<p>Welp. Looking at the source of <a href="https://github.com/iovisor/bcc/blob/6e60fbc8a672d8f29cab688ddc0df6d43a96c300/tools/tplist.py">tplist on current master</a>,
the <a href="https://github.com/iovisor/bcc/commit/69e361ac66fbf3baadb1f7cf21762df61ad7a5a9#diff-8189c35f15538919a795b3f18ad0db66L16">most recent commit</a>
removes <code>USDTReader</code>. Time to try the nightly builds:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">~$ echo "deb [trusted=yes] https://repo.iovisor.org/apt/trusty trusty-nightly main" | sudo tee /etc/apt/sources.list.d/iovisor.list
~$ sudo apt-get update
~$ sudo apt-get install bcc-tools
~$ sudo /usr/share/bcc/tools/tplist -l /usr/lib/postgresql/9.5/bin/postgres
'USDT' object has no attribute 'enumerate_probes'</pre></div>
<p>Welp. <code>enumerate_probes</code> was added in another part of the <a href="https://github.com/iovisor/bcc/commit/69e361ac66fbf3baadb1f7cf21762df61ad7a5a9#diff-4cf0bde404ce4b67b2961b61419fa23fR58">above patch</a>, so I think other things
need to be updated, as well.</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><span style="color:#960050;background-color:#1e0010">~$</span> <span style="color:#a6e22e">sudo</span> <span style="color:#a6e22e">apt</span><span style="color:#f92672">-</span><span style="color:#a6e22e">get</span> <span style="color:#a6e22e">install</span> <span style="color:#a6e22e">binutils</span> <span style="color:#a6e22e">bcc</span> <span style="color:#a6e22e">bcc</span><span style="color:#f92672">-</span><span style="color:#a6e22e">tools</span> <span style="color:#a6e22e">libbcc</span><span style="color:#f92672">-</span><span style="color:#a6e22e">examples</span> <span style="color:#a6e22e">python</span><span style="color:#f92672">-</span><span style="color:#a6e22e">bcc</span>
<span style="color:#a6e22e">Reading</span> <span style="color:#f92672">package</span> <span style="color:#a6e22e">lists</span><span style="color:#f92672">...</span> <span style="color:#a6e22e">Done</span>
<span style="color:#a6e22e">Building</span> <span style="color:#a6e22e">dependency</span> <span style="color:#a6e22e">tree</span>
<span style="color:#a6e22e">Reading</span> <span style="color:#a6e22e">state</span> <span style="color:#a6e22e">information</span><span style="color:#f92672">...</span> <span style="color:#a6e22e">Done</span>
<span style="color:#a6e22e">bcc</span> <span style="color:#a6e22e">is</span> <span style="color:#a6e22e">already</span> <span style="color:#a6e22e">the</span> <span style="color:#a6e22e">newest</span> <span style="color:#a6e22e">version</span>.
<span style="color:#a6e22e">binutils</span> <span style="color:#a6e22e">is</span> <span style="color:#a6e22e">already</span> <span style="color:#a6e22e">the</span> <span style="color:#a6e22e">newest</span> <span style="color:#a6e22e">version</span>.
<span style="color:#a6e22e">bcc</span><span style="color:#f92672">-</span><span style="color:#a6e22e">tools</span> <span style="color:#a6e22e">is</span> <span style="color:#a6e22e">already</span> <span style="color:#a6e22e">the</span> <span style="color:#a6e22e">newest</span> <span style="color:#a6e22e">version</span>.
<span style="color:#a6e22e">The</span> <span style="color:#a6e22e">following</span> <span style="color:#a6e22e">packages</span> <span style="color:#a6e22e">will</span> <span style="color:#a6e22e">be</span> <span style="color:#a6e22e">upgraded</span>:
<span style="color:#a6e22e">libbcc</span><span style="color:#f92672">-</span><span style="color:#a6e22e">examples</span> <span style="color:#a6e22e">python</span><span style="color:#f92672">-</span><span style="color:#a6e22e">bcc</span>
<span style="color:#ae81ff">2</span> <span style="color:#a6e22e">upgraded</span>, <span style="color:#ae81ff">0</span> <span style="color:#a6e22e">newly</span> <span style="color:#a6e22e">installed</span>, <span style="color:#ae81ff">0</span> <span style="color:#a6e22e">to</span> <span style="color:#a6e22e">remove</span> <span style="color:#a6e22e">and</span> <span style="color:#ae81ff">31</span> <span style="color:#a6e22e">not</span> <span style="color:#a6e22e">upgraded</span>.
<span style="color:#a6e22e">Need</span> <span style="color:#a6e22e">to</span> <span style="color:#a6e22e">get</span> <span style="color:#ae81ff">302</span> <span style="color:#a6e22e">kB</span> <span style="color:#a6e22e">of</span> <span style="color:#a6e22e">archives</span>.
<span style="color:#a6e22e">After</span> <span style="color:#a6e22e">this</span> <span style="color:#a6e22e">operation</span>, <span style="color:#ae81ff">6</span>,<span style="color:#ae81ff">144</span> <span style="color:#a6e22e">B</span> <span style="color:#a6e22e">of</span> <span style="color:#a6e22e">additional</span> <span style="color:#a6e22e">disk</span> <span style="color:#a6e22e">space</span> <span style="color:#a6e22e">will</span> <span style="color:#a6e22e">be</span> <span style="color:#a6e22e">used</span>.
<span style="color:#a6e22e">Do</span> <span style="color:#a6e22e">you</span> <span style="color:#a6e22e">want</span> <span style="color:#a6e22e">to</span> <span style="color:#66d9ef">continue</span><span style="color:#960050;background-color:#1e0010">?</span> [<span style="color:#a6e22e">Y</span><span style="color:#f92672">/</span><span style="color:#a6e22e">n</span>] <span style="color:#a6e22e">y</span>
<span style="color:#a6e22e">Get</span>:<span style="color:#ae81ff">1</span> <span style="color:#a6e22e">https</span>:<span style="color:#75715e">//repo.iovisor.org/apt/trusty/ trusty-nightly/main libbcc-examples amd64 0.2.0-22.git.12a09dc [267 kB]
</span><span style="color:#75715e"></span><span style="color:#a6e22e">Get</span>:<span style="color:#ae81ff">2</span> <span style="color:#a6e22e">https</span>:<span style="color:#75715e">//repo.iovisor.org/apt/trusty/ trusty-nightly/main python-bcc all 0.2.0-22.git.12a09dc [34.3 kB]
</span><span style="color:#75715e"></span><span style="color:#a6e22e">Fetched</span> <span style="color:#ae81ff">302</span> <span style="color:#a6e22e">kB</span> <span style="color:#a6e22e">in</span> <span style="color:#ae81ff">0</span><span style="color:#a6e22e">s</span> (<span style="color:#ae81ff">319</span> <span style="color:#a6e22e">kB</span><span style="color:#f92672">/</span><span style="color:#a6e22e">s</span>)
(<span style="color:#a6e22e">Reading</span> <span style="color:#a6e22e">database</span> <span style="color:#f92672">...</span> <span style="color:#ae81ff">91427</span> <span style="color:#a6e22e">files</span> <span style="color:#a6e22e">and</span> <span style="color:#a6e22e">directories</span> <span style="color:#a6e22e">currently</span> <span style="color:#a6e22e">installed</span>.)
<span style="color:#a6e22e">Preparing</span> <span style="color:#a6e22e">to</span> <span style="color:#a6e22e">unpack</span> <span style="color:#f92672">.../</span><span style="color:#a6e22e">libbcc</span><span style="color:#f92672">-</span><span style="color:#a6e22e">examples_0</span><span style="color:#ae81ff">.2.0</span><span style="color:#f92672">-</span><span style="color:#ae81ff">22.</span><span style="color:#a6e22e">git</span><span style="color:#ae81ff">.12</span><span style="color:#a6e22e">a09dc_amd64</span>.<span style="color:#a6e22e">deb</span> <span style="color:#f92672">...</span>
<span style="color:#a6e22e">Unpacking</span> <span style="color:#a6e22e">libbcc</span><span style="color:#f92672">-</span><span style="color:#a6e22e">examples</span> (<span style="color:#ae81ff">0.2.0</span><span style="color:#f92672">-</span><span style="color:#ae81ff">22.</span><span style="color:#a6e22e">git</span><span style="color:#ae81ff">.12</span><span style="color:#a6e22e">a09dc</span>) <span style="color:#a6e22e">over</span> (<span style="color:#ae81ff">0.2.0</span><span style="color:#f92672">-</span><span style="color:#ae81ff">1</span>) <span style="color:#f92672">...</span>
<span style="color:#a6e22e">Preparing</span> <span style="color:#a6e22e">to</span> <span style="color:#a6e22e">unpack</span> <span style="color:#f92672">.../</span><span style="color:#a6e22e">python</span><span style="color:#f92672">-</span><span style="color:#a6e22e">bcc_0</span><span style="color:#ae81ff">.2.0</span><span style="color:#f92672">-</span><span style="color:#ae81ff">22.</span><span style="color:#a6e22e">git</span><span style="color:#ae81ff">.12</span><span style="color:#a6e22e">a09dc_all</span>.<span style="color:#a6e22e">deb</span> <span style="color:#f92672">...</span>
<span style="color:#a6e22e">Unpacking</span> <span style="color:#a6e22e">python</span><span style="color:#f92672">-</span><span style="color:#a6e22e">bcc</span> (<span style="color:#ae81ff">0.2.0</span><span style="color:#f92672">-</span><span style="color:#ae81ff">22.</span><span style="color:#a6e22e">git</span><span style="color:#ae81ff">.12</span><span style="color:#a6e22e">a09dc</span>) <span style="color:#a6e22e">over</span> (<span style="color:#ae81ff">0.2.0</span><span style="color:#f92672">-</span><span style="color:#ae81ff">1</span>) <span style="color:#f92672">...</span>
<span style="color:#a6e22e">Setting</span> <span style="color:#a6e22e">up</span> <span style="color:#a6e22e">libbcc</span><span style="color:#f92672">-</span><span style="color:#a6e22e">examples</span> (<span style="color:#ae81ff">0.2.0</span><span style="color:#f92672">-</span><span style="color:#ae81ff">22.</span><span style="color:#a6e22e">git</span><span style="color:#ae81ff">.12</span><span style="color:#a6e22e">a09dc</span>) <span style="color:#f92672">...</span>
<span style="color:#a6e22e">Setting</span> <span style="color:#a6e22e">up</span> <span style="color:#a6e22e">python</span><span style="color:#f92672">-</span><span style="color:#a6e22e">bcc</span> (<span style="color:#ae81ff">0.2.0</span><span style="color:#f92672">-</span><span style="color:#ae81ff">22.</span><span style="color:#a6e22e">git</span><span style="color:#ae81ff">.12</span><span style="color:#a6e22e">a09dc</span>) <span style="color:#f92672">...</span>
<span style="color:#960050;background-color:#1e0010">~$</span> <span style="color:#a6e22e">less</span> <span style="color:#f92672">/</span><span style="color:#a6e22e">usr</span><span style="color:#f92672">/</span><span style="color:#a6e22e">lib</span><span style="color:#f92672">/</span><span style="color:#a6e22e">python2</span><span style="color:#ae81ff">.7</span><span style="color:#f92672">/</span><span style="color:#a6e22e">dist</span><span style="color:#f92672">-</span><span style="color:#a6e22e">packages</span><span style="color:#f92672">/</span><span style="color:#a6e22e">bcc</span><span style="color:#f92672">/</span><span style="color:#a6e22e">usdt</span>.<span style="color:#a6e22e">py</span>
<span style="color:#960050;background-color:#1e0010">~$</span> <span style="color:#a6e22e">sudo</span> <span style="color:#f92672">/</span><span style="color:#a6e22e">usr</span><span style="color:#f92672">/</span><span style="color:#a6e22e">share</span><span style="color:#f92672">/</span><span style="color:#a6e22e">bcc</span><span style="color:#f92672">/</span><span style="color:#a6e22e">tools</span><span style="color:#f92672">/</span><span style="color:#a6e22e">tplist</span> <span style="color:#f92672">-</span><span style="color:#a6e22e">l</span> <span style="color:#f92672">/</span><span style="color:#a6e22e">usr</span><span style="color:#f92672">/</span><span style="color:#a6e22e">lib</span><span style="color:#f92672">/</span><span style="color:#a6e22e">postgresql</span><span style="color:#f92672">/</span><span style="color:#ae81ff">9.5</span><span style="color:#f92672">/</span><span style="color:#a6e22e">bin</span><span style="color:#f92672">/</span><span style="color:#a6e22e">postgres</span>
<span style="color:#a6e22e">Traceback</span> (<span style="color:#a6e22e">most</span> <span style="color:#a6e22e">recent</span> <span style="color:#a6e22e">call</span> <span style="color:#a6e22e">last</span>):
<span style="color:#a6e22e">File</span> <span style="color:#e6db74">"/usr/share/bcc/tools/tplist"</span>, <span style="color:#a6e22e">line</span> <span style="color:#ae81ff">16</span>, <span style="color:#a6e22e">in</span> <<span style="color:#a6e22e">module</span>>
<span style="color:#a6e22e">from</span> <span style="color:#a6e22e">bcc</span> <span style="color:#f92672">import</span> <span style="color:#a6e22e">USDT</span>
<span style="color:#a6e22e">File</span> <span style="color:#e6db74">"/usr/lib/python2.7/dist-packages/bcc/__init__.py"</span>, <span style="color:#a6e22e">line</span> <span style="color:#ae81ff">28</span>, <span style="color:#a6e22e">in</span> <<span style="color:#a6e22e">module</span>>
<span style="color:#a6e22e">from</span> .<span style="color:#a6e22e">libbcc</span> <span style="color:#f92672">import</span> <span style="color:#a6e22e">lib</span>, <span style="color:#a6e22e">_CB_TYPE</span>, <span style="color:#a6e22e">bcc_symbol</span>
<span style="color:#a6e22e">File</span> <span style="color:#e6db74">"/usr/lib/python2.7/dist-packages/bcc/libbcc.py"</span>, <span style="color:#a6e22e">line</span> <span style="color:#ae81ff">160</span>, <span style="color:#a6e22e">in</span> <<span style="color:#a6e22e">module</span>>
<span style="color:#a6e22e">lib</span>.<span style="color:#a6e22e">bcc_usdt_get_probe_argctype</span>.<span style="color:#a6e22e">restype</span> = <span style="color:#a6e22e">ct</span>.<span style="color:#a6e22e">c_char_p</span>
<span style="color:#a6e22e">File</span> <span style="color:#e6db74">"/usr/lib/python2.7/ctypes/__init__.py"</span>, <span style="color:#a6e22e">line</span> <span style="color:#ae81ff">378</span>, <span style="color:#a6e22e">in</span> <span style="color:#a6e22e">__getattr__</span>
<span style="color:#66d9ef">func</span> = <span style="color:#a6e22e">self</span>.<span style="color:#a6e22e">__getitem__</span>(<span style="color:#a6e22e">name</span>)
<span style="color:#a6e22e">File</span> <span style="color:#e6db74">"/usr/lib/python2.7/ctypes/__init__.py"</span>, <span style="color:#a6e22e">line</span> <span style="color:#ae81ff">383</span>, <span style="color:#a6e22e">in</span> <span style="color:#a6e22e">__getitem__</span>
<span style="color:#66d9ef">func</span> = <span style="color:#a6e22e">self</span>.<span style="color:#a6e22e">_FuncPtr</span>((<span style="color:#a6e22e">name_or_ordinal</span>, <span style="color:#a6e22e">self</span>))
<span style="color:#a6e22e">AttributeError</span>: <span style="color:#f92672">/</span><span style="color:#a6e22e">usr</span><span style="color:#f92672">/</span><span style="color:#a6e22e">lib</span><span style="color:#f92672">/</span><span style="color:#a6e22e">x86_64</span><span style="color:#f92672">-</span><span style="color:#a6e22e">linux</span><span style="color:#f92672">-</span><span style="color:#a6e22e">gnu</span><span style="color:#f92672">/</span><span style="color:#a6e22e">libbcc</span>.<span style="color:#a6e22e">so</span><span style="color:#ae81ff">.0</span>: <span style="color:#a6e22e">undefined</span> <span style="color:#a6e22e">symbol</span>: <span style="color:#a6e22e">bcc_usdt_get_probe_argctype</span></pre></div>
<p>One more error. This time in <code>libbcc</code>. Seems like another package to pull from nightly.</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><span style="color:#960050;background-color:#1e0010">~$</span> <span style="color:#a6e22e">sudo</span> <span style="color:#a6e22e">apt</span><span style="color:#f92672">-</span><span style="color:#a6e22e">get</span> <span style="color:#a6e22e">install</span> <span style="color:#a6e22e">libbcc</span>
<span style="color:#a6e22e">Reading</span> <span style="color:#f92672">package</span> <span style="color:#a6e22e">lists</span><span style="color:#f92672">...</span> <span style="color:#a6e22e">Done</span>
<span style="color:#a6e22e">Building</span> <span style="color:#a6e22e">dependency</span> <span style="color:#a6e22e">tree</span>
<span style="color:#a6e22e">Reading</span> <span style="color:#a6e22e">state</span> <span style="color:#a6e22e">information</span><span style="color:#f92672">...</span> <span style="color:#a6e22e">Done</span>
<span style="color:#a6e22e">The</span> <span style="color:#a6e22e">following</span> <span style="color:#a6e22e">packages</span> <span style="color:#a6e22e">will</span> <span style="color:#a6e22e">be</span> <span style="color:#a6e22e">upgraded</span>:
<span style="color:#a6e22e">libbcc</span>
<span style="color:#ae81ff">1</span> <span style="color:#a6e22e">upgraded</span>, <span style="color:#ae81ff">0</span> <span style="color:#a6e22e">newly</span> <span style="color:#a6e22e">installed</span>, <span style="color:#ae81ff">0</span> <span style="color:#a6e22e">to</span> <span style="color:#a6e22e">remove</span> <span style="color:#a6e22e">and</span> <span style="color:#ae81ff">30</span> <span style="color:#a6e22e">not</span> <span style="color:#a6e22e">upgraded</span>.
<span style="color:#a6e22e">Need</span> <span style="color:#a6e22e">to</span> <span style="color:#a6e22e">get</span> <span style="color:#ae81ff">9</span>,<span style="color:#ae81ff">505</span> <span style="color:#a6e22e">kB</span> <span style="color:#a6e22e">of</span> <span style="color:#a6e22e">archives</span>.
<span style="color:#a6e22e">After</span> <span style="color:#a6e22e">this</span> <span style="color:#a6e22e">operation</span>, <span style="color:#ae81ff">0</span> <span style="color:#a6e22e">B</span> <span style="color:#a6e22e">of</span> <span style="color:#a6e22e">additional</span> <span style="color:#a6e22e">disk</span> <span style="color:#a6e22e">space</span> <span style="color:#a6e22e">will</span> <span style="color:#a6e22e">be</span> <span style="color:#a6e22e">used</span>.
<span style="color:#a6e22e">Get</span>:<span style="color:#ae81ff">1</span> <span style="color:#a6e22e">https</span>:<span style="color:#75715e">//repo.iovisor.org/apt/trusty/ trusty-nightly/main libbcc amd64 0.2.0-22.git.12a09dc [9,505 kB]
</span><span style="color:#75715e"></span><span style="color:#a6e22e">Fetched</span> <span style="color:#ae81ff">9</span>,<span style="color:#ae81ff">505</span> <span style="color:#a6e22e">kB</span> <span style="color:#a6e22e">in</span> <span style="color:#ae81ff">2</span><span style="color:#a6e22e">s</span> (<span style="color:#ae81ff">3</span>,<span style="color:#ae81ff">276</span> <span style="color:#a6e22e">kB</span><span style="color:#f92672">/</span><span style="color:#a6e22e">s</span>)
(<span style="color:#a6e22e">Reading</span> <span style="color:#a6e22e">database</span> <span style="color:#f92672">...</span> <span style="color:#ae81ff">91427</span> <span style="color:#a6e22e">files</span> <span style="color:#a6e22e">and</span> <span style="color:#a6e22e">directories</span> <span style="color:#a6e22e">currently</span> <span style="color:#a6e22e">installed</span>.)
<span style="color:#a6e22e">Preparing</span> <span style="color:#a6e22e">to</span> <span style="color:#a6e22e">unpack</span> <span style="color:#f92672">.../</span><span style="color:#a6e22e">libbcc_0</span><span style="color:#ae81ff">.2.0</span><span style="color:#f92672">-</span><span style="color:#ae81ff">22.</span><span style="color:#a6e22e">git</span><span style="color:#ae81ff">.12</span><span style="color:#a6e22e">a09dc_amd64</span>.<span style="color:#a6e22e">deb</span> <span style="color:#f92672">...</span>
<span style="color:#a6e22e">Unpacking</span> <span style="color:#a6e22e">libbcc</span> (<span style="color:#ae81ff">0.2.0</span><span style="color:#f92672">-</span><span style="color:#ae81ff">22.</span><span style="color:#a6e22e">git</span><span style="color:#ae81ff">.12</span><span style="color:#a6e22e">a09dc</span>) <span style="color:#a6e22e">over</span> (<span style="color:#ae81ff">0.2.0</span><span style="color:#f92672">-</span><span style="color:#ae81ff">1</span>) <span style="color:#f92672">...</span>
<span style="color:#a6e22e">Setting</span> <span style="color:#a6e22e">up</span> <span style="color:#a6e22e">libbcc</span> (<span style="color:#ae81ff">0.2.0</span><span style="color:#f92672">-</span><span style="color:#ae81ff">22.</span><span style="color:#a6e22e">git</span><span style="color:#ae81ff">.12</span><span style="color:#a6e22e">a09dc</span>) <span style="color:#f92672">...</span>
<span style="color:#a6e22e">Processing</span> <span style="color:#a6e22e">triggers</span> <span style="color:#66d9ef">for</span> <span style="color:#a6e22e">libc</span><span style="color:#f92672">-</span><span style="color:#a6e22e">bin</span> (<span style="color:#ae81ff">2.19</span><span style="color:#f92672">-</span><span style="color:#ae81ff">0</span><span style="color:#a6e22e">ubuntu6</span><span style="color:#ae81ff">.9</span>) <span style="color:#f92672">...</span>
<span style="color:#960050;background-color:#1e0010">~$</span> <span style="color:#a6e22e">sudo</span> <span style="color:#f92672">/</span><span style="color:#a6e22e">usr</span><span style="color:#f92672">/</span><span style="color:#a6e22e">share</span><span style="color:#f92672">/</span><span style="color:#a6e22e">bcc</span><span style="color:#f92672">/</span><span style="color:#a6e22e">tools</span><span style="color:#f92672">/</span><span style="color:#a6e22e">tplist</span> <span style="color:#f92672">-</span><span style="color:#a6e22e">l</span> <span style="color:#f92672">/</span><span style="color:#a6e22e">usr</span><span style="color:#f92672">/</span><span style="color:#a6e22e">lib</span><span style="color:#f92672">/</span><span style="color:#a6e22e">postgresql</span><span style="color:#f92672">/</span><span style="color:#ae81ff">9.5</span><span style="color:#f92672">/</span><span style="color:#a6e22e">bin</span><span style="color:#f92672">/</span><span style="color:#a6e22e">postgres</span>
<span style="color:#960050;background-color:#1e0010">~$</span></pre></div>
<p>OK! No errors. This is good, as it answers my questions as to if this postgres
package was compiled with the <code>--enable-dtrace</code>. I can further confirm with
<code>readelf -n</code></p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">~$ readelf -n /usr/lib/postgresql/9.5/bin/postgres
Displaying notes found at file offset 0x00000254 with length 0x00000020:
Owner Data size Description
GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag)
OS: Linux, ABI: 2.6.24
Displaying notes found at file offset 0x00000274 with length 0x00000024:
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: 6990037682e6668adc87ae7a6b82e4640959cf52</pre></div>
<p>There are no USDT or bpf traces found here, so next step is to recompile
postgres with <code>--enable-dtrace</code> and see what probes are available to use with
BPF (spoiler: <a href="https://www.postgresql.org/docs/current/static/dynamic-trace.html">there are a lot of them</a>).</p>Stream a remote file to S3https://www.gregburek.com/2015/06/20/stream-a-remote-file-to-s3/Sat, 20 Jun 2015 00:00:00 +0000https://www.gregburek.com/2015/06/20/stream-a-remote-file-to-s3/<p>While attempting to work with my gif collection, I was experimenting with how
to capture gifs from the internet and place them into a S3 bucket for later
use. I found that it was possible to stream a remote file directly to S3.</p>
<p>Using the <code>aws-sdk</code> <a href="https://github.com/aws/aws-sdk-ruby">gem version 2</a>, and
the <code>open-uri</code> module of the <a href="http://ruby-doc.org/stdlib-2.2.2/libdoc/open-uri/rdoc/OpenURI.html">ruby standard
library</a>,
one can link the two IO streams together fairly easily:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby">require <span style="color:#e6db74">'aws-sdk'</span>
require <span style="color:#e6db74">'open-uri'</span>
require <span style="color:#e6db74">'sequel'</span>
require <span style="color:#e6db74">'digest/md5'</span>
<span style="color:#66d9ef">class</span> <span style="color:#a6e22e">RemoteFile</span>
<span style="color:#66d9ef">def</span> <span style="color:#a6e22e">initialize</span>(url)
@url <span style="color:#f92672">=</span> url
<span style="color:#66d9ef">end</span>
<span style="color:#66d9ef">def</span> <span style="color:#a6e22e">s3</span>
@s3 <span style="color:#f92672">||=</span> <span style="color:#66d9ef">Aws</span><span style="color:#f92672">::</span><span style="color:#66d9ef">S3</span><span style="color:#f92672">::</span><span style="color:#66d9ef">Resource</span><span style="color:#f92672">.</span>new(<span style="color:#e6db74">region</span>: <span style="color:#66d9ef">ENV</span><span style="color:#f92672">[</span><span style="color:#e6db74">'AWS_REGION'</span><span style="color:#f92672">]</span>)
<span style="color:#66d9ef">end</span>
<span style="color:#66d9ef">def</span> <span style="color:#a6e22e">md5_hash</span>
<span style="color:#66d9ef">Digest</span><span style="color:#f92672">::</span><span style="color:#66d9ef">MD5</span><span style="color:#f92672">.</span>hexdigest(@url)
<span style="color:#66d9ef">end</span>
<span style="color:#66d9ef">def</span> <span style="color:#a6e22e">obj</span>(bucket_name)
s3<span style="color:#f92672">.</span>bucket(bucket_name)<span style="color:#f92672">.</span>object(md5_hash)
<span style="color:#66d9ef">end</span>
<span style="color:#66d9ef">def</span> <span style="color:#a6e22e">url</span>
<span style="color:#66d9ef">URI</span><span style="color:#f92672">.</span>parse(<span style="color:#66d9ef">URI</span><span style="color:#f92672">.</span>escape(@url))
<span style="color:#66d9ef">end</span>
<span style="color:#66d9ef">def</span> <span style="color:#a6e22e">upload_to_s3</span>(<span style="color:#e6db74">bucket_name</span>:)
open(url) <span style="color:#66d9ef">do</span> <span style="color:#f92672">|</span>file<span style="color:#f92672">|</span>
obj(bucket_name)<span style="color:#f92672">.</span>put(<span style="color:#e6db74">body</span>: file)
<span style="color:#66d9ef">end</span>
<span style="color:#66d9ef">end</span>
<span style="color:#66d9ef">end</span>
<span style="color:#66d9ef">RemoteFile</span><span style="color:#f92672">.</span>new(<span style="color:#e6db74">'https://i.imgur.com/DO3Hr4A.gif'</span>)
<span style="color:#f92672">.</span>upload_to_s3(<span style="color:#e6db74">bucket_name</span>: <span style="color:#e6db74">'mah_gifs'</span>)</code></pre></div>
<p>This code snippet assumes you have <code>ENV['AWS_ACCESS_KEY_ID']</code> and
<code>ENV['AWS_SECRET_ACCESS_KEY']</code> set.</p>DNS, eglibc and resolv-replace on Herokuhttps://www.gregburek.com/2015/02/22/dns-eglibc-and-resolv-replace-on-heroku/Sun, 22 Feb 2015 00:00:00 +0000https://www.gregburek.com/2015/02/22/dns-eglibc-and-resolv-replace-on-heroku/<p>2015-03-01: Fixed versions of eglibc are available for Ubuntu Precise and
Trusty. Time to update.</p>
<p>I work on the team that runs <a href="https://www.heroku.com/postgres">Heroku Postgres</a>. As we have continued to grow, I
have been tracking an intermitent error with <a href="https://devcenter.heroku.com/articles/rollbar">Rollbar</a> that occurs about once
every 50,000 HTTP requests. As we are doing many hundreds of thousands of API
calls a minute to various services, this error can pop up fairly frequently and
in very inconvenient places. The most common traceback seems to indicate a
failure to resolve DNS:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#75715e">#<SocketError: getaddrinfo: Name or service not known></span>
<span style="color:#e6db74">/app/</span>vendor<span style="color:#e6db74">/ruby-2.1.5/</span>lib<span style="color:#e6db74">/ruby/</span><span style="color:#ae81ff">2</span><span style="color:#f92672">.</span><span style="color:#ae81ff">1</span><span style="color:#f92672">.</span><span style="color:#ae81ff">0</span><span style="color:#f92672">/</span>net<span style="color:#e6db74">/http.rb:879:in 'initialize'
</span><span style="color:#e6db74">/</span>app<span style="color:#e6db74">/vendor/</span>ruby<span style="color:#f92672">-</span><span style="color:#ae81ff">2</span><span style="color:#f92672">.</span><span style="color:#ae81ff">1</span><span style="color:#f92672">.</span><span style="color:#ae81ff">5</span><span style="color:#f92672">/</span>lib<span style="color:#e6db74">/ruby/</span><span style="color:#ae81ff">2</span><span style="color:#f92672">.</span><span style="color:#ae81ff">1</span><span style="color:#f92672">.</span><span style="color:#ae81ff">0</span><span style="color:#f92672">/</span>net<span style="color:#e6db74">/http.rb:879:in 'open'
</span><span style="color:#e6db74">/</span>app<span style="color:#e6db74">/vendor/</span>ruby<span style="color:#f92672">-</span><span style="color:#ae81ff">2</span><span style="color:#f92672">.</span><span style="color:#ae81ff">1</span><span style="color:#f92672">.</span><span style="color:#ae81ff">5</span><span style="color:#f92672">/</span>lib<span style="color:#e6db74">/ruby/</span><span style="color:#ae81ff">2</span><span style="color:#f92672">.</span><span style="color:#ae81ff">1</span><span style="color:#f92672">.</span><span style="color:#ae81ff">0</span><span style="color:#f92672">/</span>net<span style="color:#e6db74">/http.rb:879:in 'block in connect'
</span><span style="color:#e6db74">...</span></code></pre></div>
<p><a href="http://lmgtfy.com/?q=SocketError%3A+getaddrinfo%3A+Name+or+service+not+known+heroku">Google</a> led me to a <a href="http://www.subelsky.com/2014/05/fixing-socketerror-getaddrinfo-name-or.html">pertinent blog post</a> that recommended using ruby’s
<a href="http://apidock.com/ruby/Resolv">Resolv</a> library for all DNS requests via a script called <a href="https://github.com/ruby/ruby/blob/trunk/lib/resolv-replace.rb">resolv-replace</a>.
Adding a single line to our initializers, <code>require resolv-replace</code>, caused errors
while submitting Logplex messages to immediately drop:</p>
<p><img src="https://www.gregburek.com/2015-02-22-dns-eglibc-and-resolv-replace-on-heroku/logplex_errors.png" alt="Logplex Errors" /></p>
<p>As did errors from trying to interact with our monitoring service, Observatory:</p>
<p><img src="https://www.gregburek.com/2015-02-22-dns-eglibc-and-resolv-replace-on-heroku/observatory_errors.png" alt="Observatory Errors" /></p>
<p>In an internal thread, <a href="https://twitter.com/freeformz">Ed Muller</a> pointed out a <a href="https://github.com/golang/go/issues/6336#issuecomment-66085142">golang work around</a>
of a <a href="https://sourceware.org/bugzilla/show_bug.cgi?id=15946">bug in glibc</a> which is very likely to be a factor in this error:</p>
<blockquote>
<p>Under high load, getaddrinfo() starts sending DNS queries to random
file descriptors, e.g. some unrelated socket connected to a remote service.</p>
</blockquote>
<p>As Heroku is a shared platform with multitenant runtime instances, it is
possible for a random runtime to experience high load and the <a href="https://devcenter.heroku.com/articles/cedar-ubuntu-packages">cedar-14 glibc
binaries</a> are known to be impacted by this bug. Version 2.20 of glibc has a
fix and as of <a href="https://bazaar.launchpad.net/~ubuntu-branches/ubuntu/trusty/eglibc/trusty-security/revision/346">2.19-0ubuntu6.6</a> and <a href="https://bazaar.launchpad.net/~ubuntu-branches/ubuntu/precise/eglibc/precise-security/revision/316">2.15-0ubuntu10.11</a> this fix was
backported to Ubuntu Precise and Trusty. <s>However, Ubuntu Precise currently
ships <a href="http://packages.ubuntu.com/precise-updates/libc6">2.15-0ubuntu10.10</a> and Trusty provides <a href="http://packages.ubuntu.com/trusty-updates/libc6">2.19-0ubuntu6.5</a>, so this
bug may continue to be a problem for some time to come.</s></p>
<p>My immediate recommendation is to use language native DNS resolution like
<code>resolv-replace</code> whenever possible, on Heroku or other systems. However, if you
require ipv6 or run into problems with <a href="https://github.com/mperham/sidekiq/issues/1258#issuecomment-27389456">third party gems attempting to resolve
<code>nil</code> addresses</a>, and are stuck with the system DNS, upgrade yourself!
<s>please indicate that this bug affects you on the <a href="https://bugs.launchpad.net/eglibc/+bug/1421393">Launchpad bug report
requesting backporting</a> to supported versions of Ubuntu.</s></p>
<p>Thanks to Ed Muller, Michael Hale, Keiko Oda, Steve Conklin, Terence Lee and
Richard Schneeman for help in figuring this out.</p>Require HTTPS to your Heroku apphttps://www.gregburek.com/2014/10/26/require-https-to-your-heroku-app/Sun, 26 Oct 2014 00:00:00 +0000https://www.gregburek.com/2014/10/26/require-https-to-your-heroku-app/<p>Configuring your Heroku app so that it will redirect insecure HTTP traffic to an
HTTPS endpoint can be finicky and is language/framework specific. I was able to
figure out a general and language independent method thanks to the <a href="https://github.com/ryandotsmith/nginx-buildpack">nginx
buildpack</a>. By using nginx,
you can redirect some or all http traffic to your app to the https verison of
your site.</p>
<p>By adding:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">if ($http_x_forwarded_proto != 'https') {
rewrite ^ https://$host$request_uri? permanent;
}</pre></div>
<p>to the <code>location</code> section of your app’s nginx config file template, any access
to that location will be met with a <code>301 Moved Permanently</code> redirect to the
<code>https</code> version of that site and path.</p>
<p>EDIT: @jacobian <a href="https://twitter.com/jacobian/status/526538110201368576">pointed
out</a> on twitter that
using <a href="http://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security">HTTP Strict Transport Security
(HSTS)</a> headers
will make modern clients prefer HTTPS, even for the <code>/insecure</code> path that lacks
the redirect snippet.</p>
<p>As all apps are accessible at <code>https://<app-name>.herokuapp.com/</code> by using
Heroku’s <code>herokuapp.com</code> SSL cert, this provides a free and easy way to secure
your apps. Custom domain names require custom SSL certs, which are available
from traditional SSL vendors or from Heroku addon <a href="https://www.expeditedssl.com/">Expedited
SSL</a></p>
<p>A sample app can be found at
<a href="https://github.com/gregburek/heroku-force-ssl-sample">https://github.com/gregburek/heroku-force-ssl-sample</a>
and deployed to your Heroku account here:
<a href="https://heroku.com/deploy?template=https://github.com/gregburek/heroku-force-ssl-sample"><img src="https://www.herokucdn.com/deploy/button.png" alt="Deploy" /></a></p>Moving site to Middlemanhttps://www.gregburek.com/2014/07/06/moving-site-to-middleman/Sun, 06 Jul 2014 00:00:00 +0000https://www.gregburek.com/2014/07/06/moving-site-to-middleman/<p>This is me attempting to resurrect my blog after several years of silence. This
has also given me a chance to redesign things and adopt another static site
generator.</p>
<p>Octopress served me well, but I felt like it was difficult to grasp the
fundamentals and hard to pick up after a while away. I went looking for new site
framework and found <a href="http://middlemanapp.com/">Middleman</a>. I liked that it was
written in Ruby and I really liked the tutorials I found for it.</p>
<p>In particular, Julie Pagano’s fantastic
<a href="http://juliepagano.com/blog/2013/11/10/site-redesign-using-middleman/">tutorial</a>
was invaluable in getting me past a bunch of unexpected things.</p>
<p>The plugins I am using are:</p>
<ul>
<li><a href="https://github.com/middleman/middleman-blog">middleman-blog</a> for support of
the article format</li>
<li><a href="https://github.com/neo/middleman-gh-pages">middleman-gh-pages</a> for an easy
GitHub pages integrated workflow</li>
<li><a href="https://github.com/middleman/middleman-blog">middleman-syntax</a> for syntax
highlighting of code snippets</li>
</ul>
<p>Starting from a blank Gemfile, unfortunately, seemed to not give properly
rendered code syntax blocks.</p>
<p>An unmerged <a href="https://github.com/middleman/middleman-syntax/pull/42">PR</a> appears
to address the problem, but I found that using Julie Pagano’s
<a href="https://github.com/juliepagano/juliepagano.com/blob/master/Gemfile.lock">Gemfile.lock</a>
also worked very well. I am very grateful that it was available.</p>
<p>The rest of the work was about adapting my strange icon color scheme into
something that doesn’t repulse and offend. Mixed results, I would say.</p>
<p>Now that this is set up, I have high hopes to use this new system to write more
about technology and my experiences with it.</p>
<p>The code for this site now resides on <a href="https://github.com/gregburek/gregburek.com">GitHub</a></p>Debugging a chef cookbook in vagrant with shefhttps://www.gregburek.com/2012/08/21/debugging-a-chef-cookbook-in-vagrant-with-shef/Tue, 21 Aug 2012 00:00:00 +0000https://www.gregburek.com/2012/08/21/debugging-a-chef-cookbook-in-vagrant-with-shef/<p>Sometimes <a href="http://vagrantup.com" title="Vagrant - Virtualized development for
the masses">Vagrant</a>’s provisioning error messages can be a little cryptic.</p>
<p>When troubleshooting a failing chef-solo run, tweaking a <code>run_list</code> or
debugging a new recipe, I’ve found it very handy to log into the partially
provisioned VM with <code>vagrant ssh</code> and then run: <a href="http://vagrantup.com" title="Vagrant - Virtualized development for the masses">Vagrant</a> is a great tool for
developing and testing new chef cookbooks. After bringing up a new vm, and
editing a cookbook,<code>vagrant provision</code> runs the chef-solo provisioner and tests
things out. However, when things fail, vagrant’s provisioning error messages
can be a little cryptic. <a href="http://wiki.opscode.com/display/chef/Shef" title="Shef
- Chef - Opscode Open Source Wiki">Shef</a> is a good tool for running cookbooks in
isolation, but it needs some help to find all the configuration and attributes
that vagrant provides. If my <code>new_and_broken</code> cookbook is failing on a Ubuntu
12.04 VM, all I need to do is run:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">$ shef -s -c /tmp/vagrant-chef-1/solo.rb -j /tmp/vagrant-chef-1/dna.json
(output snip)
chef > recipe
chef:recipe > include_recipe 'new_and_broken'
(huge output snip)
chef:recipe > run_chef
(where the error happens)</pre></div>
<p>This loads shef in solo mode, with vagrant generated configuration and JSON
attributes files, enters into recipe mode, loads my new and broken cookbook and
runs it. The resulting error messages are usually more helpful than <code>vagrant
provision</code> and I can get back to work.</p>Using veewee 0.2 with Vagrant omnibushttps://www.gregburek.com/2012/08/12/using-veewee-0.2-with-vagrant-omnibus/Sun, 12 Aug 2012 00:00:00 +0000https://www.gregburek.com/2012/08/12/using-veewee-0.2-with-vagrant-omnibus/<p>I’m a huge fan of <a href="http://vagrantup.com" title="Vagrant - Virtualized
development for the masses">Vagrant</a>’s recent omnibus style installer. It makes it so
much easier to recommend to others as they can be up and running with Vagrant
extremely quickly, instead of wondering why their distro packaged ruby
installation isn’t working. However, because Vagrant is using an embedded ruby
installation, other gems which add to vagrant are unable to find it.</p>
<p>One such tool is <a href="http://github.com/jedi4ever/veewee/" title="Veewee on
Github">veewee</a>, which makes it dead simple to automatically build VMs from
kickstarter files and basic scripts. Version 0.2 also adds a great subcommand
to Vagrant called ‘basebox’ which lets you use veewee to build baseboxes that
Vagrant may then use to launch new VMs. But with Vagrant being run from its own
embedded ruby environment, veewee is not able to find the vagrant gem to add
to, making it a little harder to use for ruby newbies.</p>
<p>There is a way around this, though. If you were to run:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">sudo /opt/vagrant/embedded/bin/gem install veewee --no-ri --no-rdoc</code></pre></div>
<p>veewee is installed to the vagrant embedded environment and <code>vagrant basebox</code>
is available and functions as expected. I’ve done this on my OSX 10.8 box that
uses rbenv to manage ruby versions as well on a Ubuntu 12.04 box that uses rvm.</p>
<p>Let me know if this does or doesn’t work for you.</p>Rate Limiting Function Calls in Python with a Decoratorhttps://www.gregburek.com/2011/12/05/rate-limiting-function-calls-in-python-with-a-decorator/Mon, 05 Dec 2011 00:00:00 +0000https://www.gregburek.com/2011/12/05/rate-limiting-function-calls-in-python-with-a-decorator/<p>source: <a href="http://stackoverflow.com/a/667706/586172">Stack Overflow “What’s a good rate limiting algorithm?”</a></p>
<p>I making this into a post because I have found it so handy. Some web
APIs have rate limits on requests per minute or you may want to play nice
and not overwhelm the service. In Python, you can use this decorator to
rate limit a function that may handle the API access:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#f92672">import</span> time
<span style="color:#66d9ef">def</span> <span style="color:#a6e22e">RateLimited</span>(maxPerSecond):
minInterval <span style="color:#f92672">=</span> <span style="color:#ae81ff">1.0</span> <span style="color:#f92672">/</span> float(maxPerSecond)
<span style="color:#66d9ef">def</span> <span style="color:#a6e22e">decorate</span>(func):
lastTimeCalled <span style="color:#f92672">=</span> [<span style="color:#ae81ff">0.0</span>]
<span style="color:#66d9ef">def</span> <span style="color:#a6e22e">rateLimitedFunction</span>(<span style="color:#f92672">*</span>args,<span style="color:#f92672">**</span>kargs):
elapsed <span style="color:#f92672">=</span> time<span style="color:#f92672">.</span>clock() <span style="color:#f92672">-</span> lastTimeCalled[<span style="color:#ae81ff">0</span>]
leftToWait <span style="color:#f92672">=</span> minInterval <span style="color:#f92672">-</span> elapsed
<span style="color:#66d9ef">if</span> leftToWait<span style="color:#f92672">></span><span style="color:#ae81ff">0</span>:
time<span style="color:#f92672">.</span>sleep(leftToWait)
ret <span style="color:#f92672">=</span> func(<span style="color:#f92672">*</span>args,<span style="color:#f92672">**</span>kargs)
lastTimeCalled[<span style="color:#ae81ff">0</span>] <span style="color:#f92672">=</span> time<span style="color:#f92672">.</span>clock()
<span style="color:#66d9ef">return</span> ret
<span style="color:#66d9ef">return</span> rateLimitedFunction
<span style="color:#66d9ef">return</span> decorate
<span style="color:#a6e22e">@RateLimited</span>(<span style="color:#ae81ff">2</span>) <span style="color:#75715e"># 2 per second at most</span>
<span style="color:#66d9ef">def</span> <span style="color:#a6e22e">PrintNumber</span>(num):
<span style="color:#66d9ef">print</span> num
<span style="color:#66d9ef">if</span> __name__ <span style="color:#f92672">==</span> <span style="color:#e6db74">"__main__"</span>:
<span style="color:#66d9ef">print</span> <span style="color:#e6db74">"This should print 1,2,3... at about 2 per second."</span>
<span style="color:#66d9ef">for</span> i <span style="color:#f92672">in</span> range(<span style="color:#ae81ff">1</span>,<span style="color:#ae81ff">100</span>):
PrintNumber(i)</code></pre></div>
<p>This answer is simpler than setting up a queue system and is blocking,
which can be good for sequential jobs.</p>Exporting from Aperture 3.2 to Flickr with location information metadatahttps://www.gregburek.com/2011/11/11/exporting-from-aperture-3.2-to-flickr-with-location-information-metadata/Fri, 11 Nov 2011 00:00:00 +0000https://www.gregburek.com/2011/11/11/exporting-from-aperture-3.2-to-flickr-with-location-information-metadata/<p>Over the month of October, I took an incredible trip to Singapore and Japan.
That trip deserves its own post, but until then,
<a href="http://www.flickr.com/photos/gregburek/sets/72157628032336264/" title="Singapore 2011 on Flickr">these</a>
<a href="http://www.flickr.com/photos/gregburek/sets/72157627963313507/" title="Japan
2011 on Flickr">photos</a> will have to suffice.</p>
<p>However, I ran into a little snag while editing and publishing everything. The
problem was that each photo’s GPS/location data wasn’t making the jump to
Flickr from Aperture. The solution was these two check boxes that needed to be
selected in order for the location info to be correctly exported: one in
Advanced Options and another in Export Preferences.</p>
<p><img src="https://www.gregburek.com/2011-11-11-export-aperture-to-flickr-with-GPS/advanced.jpg" alt="Aperture Advanced Options" /></p>
<p><img src="https://www.gregburek.com/2011-11-11-export-aperture-to-flickr-with-GPS/export.jpg" alt="Aperture Export Preferences" /></p>
<p>So in the end, if you make sure “Include location information for published
photos” and “Include location info in exported photos” are both selected,
everything works beautifully.</p>
<p>Aperture Places view</p>
<p><img src="https://www.gregburek.com/2011-11-11-export-aperture-to-flickr-with-GPS/aperture.png" alt="Aperture Places" /></p>
<p>Flickr Map view</p>
<p><img src="https://www.gregburek.com/2011-11-11-export-aperture-to-flickr-with-GPS/flickr.png" alt="Flickr Map" /></p>Lid Open, Display Offhttps://www.gregburek.com/2011/11/09/lid-open-display-off/Wed, 09 Nov 2011 00:00:00 +0000https://www.gregburek.com/2011/11/09/lid-open-display-off/<p>It has been more than a year since my <a href="https://www.gregburek.com/2010/10/21/macbook-lcd-trick">last
post</a> and that
computer has been retired. However, I still find it useful to run my new
MacBook Air attached to an external monitor and with its internal display
disabled. A few days ago, I found a better and more permanent solution than a
flimsy magnet:</p>
<ol>
<li>Attach the closed MacBook to an external display</li>
<li>Wake the MacBook with an external input device, like usb or bluetooth
keyboard.</li>
<li>Open the lid of the MacBook and the internal display will remain off</li>
</ol>
<p>That’s it. No magnets required.</p>
<p>It takes a little bit more for this to work under OS X 10.7 Lion,
though. Entering the following command in terminal seems to do the
trick:</p>
<p>``` shell
sudo nvram boot-args=“iog=0x0”</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">Undoing this is as simple as typing the following, also in terminal:
``` shell
sudo nvram -d boot-args </pre></div>
<p>You can also zap the PRAM (press Cmd+Opt+p+r at power up) to restore it
to the new Lion behaviour.</p>
<p>This works great and I’ve been using it to improve my laptop’s ventilation and
wireless reception.</p>
<p>I first saw this documented here: <a href="http://hints.macworld.com/article.php?story=20110901113922148">Mac OS X Hints - 10.7: Disable internal
laptop display when external display is
attached</a></p>Trick a Macbook into thinking it is closedhttps://www.gregburek.com/2010/10/21/trick-a-macbook-into-thinking-it-is-closed/Thu, 21 Oct 2010 00:00:00 +0000https://www.gregburek.com/2010/10/21/trick-a-macbook-into-thinking-it-is-closed/<p><em>UPDATED Nov 2011</em>: I found a better way to do this that is posted
<a href="https://www.gregburek.com/2011/11/09/Lid-open-display-off">here</a></p>
<p>Up until now, I have run my laptop at work with the lid closed and everything
on an external monitor on my desk. However, I was starting to get really
concerned about the heat (60C idle and 95C under load) and how the little fan
was struggling to keep up.</p>
<p>So, I started looking around for tips about how to leave the lid open, but
trick my MacBook into thinking the lid was closed and leave the LCD off. I
found a few <a href="http://discussions.apple.com/message.jspa?messageID=9390796" title="Using MacBook Open, with External Monitor, MB LCD Off?">good</a>
<a href="http://forums.macrumors.com/archive/index.php//t-625443.html" title="Trick a
MacBook's Lid">tips</a>, but I still had to hunt with a magnet for that magical switch
point. Well, I found it and the picture is below. Things look pretty frosty so
far, let’s hope this extends the life of my 3 year old laptop a bit more.</p>
<p><a href="http://www.flickr.com/photos/gregburek/5103718702/" title="Macbook with lid trick magnet attached by greg.burek, on Flickr"><img src="https://farm5.static.flickr.com/4111/5103718702_6c0ef9e619.jpg" alt="MacBook with lid-closed magnet
applied" /></a></p>iTunes library at home accessible over SSH tunneled AFPhttps://www.gregburek.com/2010/06/20/itunes-library-at-home-accessible-over-ssh-tunneled-afp/Sun, 20 Jun 2010 00:00:00 +0000https://www.gregburek.com/2010/06/20/itunes-library-at-home-accessible-over-ssh-tunneled-afp/<p>Today I freed 150GB of space from my Macbook by moving my music to my Mac Mini
that sits below my TV, at home, behind two firewalls.</p>
<p>First, I <a href="http://tinyapps.org/docs/ssh_osx_and_sshfs.txt" title="Hardening SSH and Mounting Remote Filesystem in OS X Finder via SSHFS">hardened SSH</a> on the mini and setup my laptop to log in without a
password, but I didn’t go through the SSHFS stuff, as I like AFP better (for
the moment). Now, I can access my Mini by running:</p>
<p>``` shell
LAPTOP$ ssh -f -N -p PORT USER@OUTSIDEIP -L 1202:localhost:5900 -L 1203:localhost:548</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">Where PORT is the random port I choose when [hardening SSH][1], USER is the
username on my mini and OUTSIDEIP is the external IP of my cable modem. Then,
I can exit from Terminal on my laptop, press command-K and run:
``` shell
afp://localhost:1203</pre></div>
<p>and get the list of shared folders. Or I can run:</p>
<p>``` shell
vnc://localhost:1202</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4">and get Screen Sharing with the Mini behind all those firewalls.
Finally, I [moved my iTunes Library to an external harddrive] [2] and hooked up
that beast to the Mini. By SSHing into my Mini and then running the AFP
command, I can now access my very large iTunes library from anywhere with an
internet connection.
If I would like to end the SSH tunnel, I run:
``` shell
ps auxww | grep -i ssh</pre></div>
<p>After finding the ID of the process I do:</p>
<p><code>shell
kill -9 SSH_PID
</code>`</p>
<p>with the SSH ID.</p>
<p>I will use some applescript to make this “connect, mount, launch iTunes” dance
a little bit more simple, but I think this is progress, as it has breathed new
life into my 2.5 year-old laptop.</p>Automatically upload new files to Google Documents from OSXhttps://www.gregburek.com/2010/03/12/automatically-upload-new-files-to-google-documents-from-osx/Fri, 12 Mar 2010 00:00:00 +0000https://www.gregburek.com/2010/03/12/automatically-upload-new-files-to-google-documents-from-osx/<p>##The Problem</p>
<p>I get many, many, many presentations, spreadsheets and PDFs sent
to me over email. I usually View them through Google Docs or Download them and
let the files launch in their proper programs. There are some times, however,
when I wish I could view or edit these files on my iPhone or on another
computer, usually located in our <a href="http://www.nanotech.ucsb.edu/">cleanroom</a>.
<a href="http://www.dropbox.com/">Dropbox</a> is an incredible service for this, with an
<a href="https://www.dropbox.com/iphoneapp">iPhone app</a> and multi-platform support, but
those fab computers are locked down hard and I can’t edit things in the iPhone
app.</p>
<p>##The Caveat</p>
<p>Since I use Gmail for everything, I could just view the file
through the web interface on my iPhone or another computer and import things to
Google Documents with a click.</p>
<p>I recommend this way. I really do. Mostly because my solution is a folder
action applescript with your gmail password in plain text. This is stupid.
Very stupid. Beyond moronic. I would like to get Keychain Access from the
applescript for a better password handling, but a few hours of fiddling yielded
nothing and further thought showed it to be even more insecure to have
scriptable readouts of passwords globally enabled.</p>
<p>So, for the record, this is my hacked together solution.</p>
<p>##The Solution</p>
<ol>
<li><p>Download <a href="http://code.google.com/p/google-docs-upload/">google-docs-upload</a> and put it
somewhere. I used /Users/USERNAME/Library/Scripts/GoogleDocs/</p></li>
<li><p>Open Script Editor.app and it should pop up with an Untitled and blank
script window. Copy this into the window:</p></li>
</ol>
<p><code>applescript
on adding folder items to this_folder after receiving added_items
repeat with aFile in added_items
do shell script "java -jar /SOMEWHERE/google-docs-upload-1.3.2.jar
aFile -rf Downloads --skip-all -u YOURUSERNAME -p
YOURPASSWORD >> /SOMEWHERE/GDocs-upload-log.txt"
end repeat
end adding folder items to
</code>`</p>
<ol>
<li><p>Replace SOMEWHERE with the path to google-docs-upload, USERNAME with your
OSX username, YOURUSERNAME with your Google username and YOURPASSWORD with your
google password. This script will run when a file is placed in the folder.
Each file is passed to google-docs-upload along with your username and password
and it decides whether it can upload the file. Any output goes to that log
file tacked on the end.</p></li>
<li><p>Save this script under ./Library/Scripts/Folder\ Action<br />
Scripts/UploadGDocs.scpt</p></li>
<li><p>Right click on your Downloads folder and hover over “More” until it expands.
Go and click on “Configure Folder Actions…”</p></li>
<li><p>Make sure the “Enable Folder Actions” check box is ticked, then click on the
left “+” sign, select the Downloads folder and click the Open button. Then,
select UploadGDocs.scpt from the next menu to drop down and click Attach.
Close the Script Editor and the Folder Actions Setup.</p></li>
</ol>
<p>That should do it. Any new file placed in your Downloads folder will be kicked
automatically to Google Docs.</p>
<p>PS: For some reason, whenever I tweak this script, my Downloads folder becomes
Read-Only for my user. I fix this by right clicking on my Downloads folder,
clicking on Get Info, clicking the lock icon in the new window, inputting my
password and then setting the drop-down menu next to my name to Read&Write.
If you know what is causing this, let me know.</p>